SA-752 SOC prompt engineering: Top candidate

[ivyONS]

✨ Summary

Adds a new top-1 SOC selection flow for shortlist-based classification, with a strict prompt, typed response model, robust parsing fallback, and targeted tests. This is better aligned with clerical coding practices on SOC specifically (that is assigning one code always instead of higher groups used for SIC).

📜 Changes Introduced

• Added a new method: top_one_soc_code in the LLM classification flow.
• Added a new response model: TopOneResponse with fields for selected SOC code, title, likelihood score, and reasoning.
• Added a dedicated top-one prompt that enforces:
  - exactly one shortlist code selection
  - shortlist-only output
  - calibrated likelihood scoring using fixed values (0.2, 0.4, 0.6, 0.8, 0.9)
  - explicit reasoning and separation from next-best alternatives
• Added fallback behaviour in top_one_soc_code when chain/parsing fails: returns first shortlist item with low confidence and error-context reasoning
• Updated SOC candidate formatting labels to clearer wording:
  - Example job titles
  - Description
  - Example job tasks
• Expanded tests to cover:
  - prompt constraints for top-one selection
  - typed return behavior of top_one_soc_code
  - updated candidate-text expectations
• Updated poetry.lock (regenerated with Poetry 2.4.1; transitive dependency refresh).

✅ Checklist

• Code is formatted using ruff format
• Imports are sorted using ruff
• Code passes linting with ruff and mypy
• Security checks pass using Bandit
• API and Unit tests are written and pass using pytest
• Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
• DocStrings follow Google-style and are added as per ruff recommendations
• Documentation has been updated if needed

🔍 How to Test

Ran unit tests.
Invoke the llm interaction from survey-assist-evel (paired PR).