SA617 - Cleaning ASHE strings and preparing SOC knowledgebase and SOC DIRECT LOOKUP

[peter-spencer-ons]

✨ Summary

Prepare datasets for knowledgebase and Direct Lookup for SOC, using data from ASHE.

Changes include adapting prompts and LLM methods that are not meant to be used in the main branch, but exclusively in SA617, without merging.

📜 Changes Introduced

• Adapt prompts and methods within src to be usable for the task.
• Add prompt and method to correct misspelled words from ASHE dataset
• Prepare scripts for:
  • correcting misspellings
  • creating knowledgebase
  • assigning SOC codes with LLM use
  • selecting unambiguous codes for Direct Lookup

✅ Checklist

Please confirm you've completed these checks before requesting a review.

• Code is formatted using Black
• Imports are sorted using isort
• Code passes linting with Ruff, Pylint, and Mypy
• Security checks pass using Bandit
• API and Unit tests are written and pass using pytest
• Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
• DocStrings follow Google-style and are added as per Pylint recommendations
• Documentation has been updated if needed

🔍 How to Test

select a subset of ASHE dataset
create .env file (please reach out for details)
run:

• ashe_clean_2026_04.py
• assign_soc_code_2026_03.py
• soc_kb_2026_04.ipynb
• create_soc_lookup_2026_04.ipynb

[ivyONS]

what makes it even more interesting is comparison with 52570

[ivyONS]

achieved the desired output - one time process, no need to merge