Panel pipeline: full plumbing for per-year snapshots (closes #345)

[vahid-ahmadi]

Summary

Lands all six steps of the panel-data plan in #345. No change to the default behaviour of the existing single-year pipeline — the new code is plumbing and new helpers that consumers opt into. No data sourcing and no modification of audited upload paths.

Step 1 — panel ID contract

• policyengine_uk_data/utils/panel_ids.py: PANEL_ID_COLUMNS, get_panel_ids, assert_panel_id_consistency, classify_panel_ids + PanelIDTransition.
• README.md section documenting that household_id, benunit_id, person_id are the panel keys.

Step 2 — create_yearly_snapshots helper

• policyengine_uk_data/datasets/yearly_snapshots.py with create_yearly_snapshots(base, years, output_dir, *, filename_template=...).
• Uprates per year, asserts panel IDs, saves enhanced_frs_<year>.h5.

Step 3 — demographic ageing

• policyengine_uk_data/utils/demographic_ageing.py with age_dataset(base, years, *, seed, mortality_rates, fertility_rates).
• Per-year loop: mortality (row removal) → fertility (fresh person_ids attached to mother's benunit + household) → age increment.
• Deterministic via seed. Ships _PLACEHOLDER rates — explicitly not ONS data; real rates are follow-up work.

Step 4 — year-aware calibration targets

• Two real bug fixes: local_authorities/loss.py read household_weight at a hard-coded 2025; constituencies/loss.py used dataset.time_period instead of the time_period argument in two places. Both now honour the argument.
• Extracts year-resolution into resolve_target_value(target, year, *, tolerance=3), names YEAR_FALLBACK_TOLERANCE = 3 as a constant, documents the policy (exact → nearest past year within tolerance → None; no backwards extrapolation; VOA population scaling preserved).
• docs/targets_coverage.md documents year coverage across every target category and surfaces the real remaining gaps (DWP 2026+ forecasts, annual local-area CSV refreshes).
• No new data sourced in this PR — plumbing + documentation only.

Step 5 — cross-year weight smoothness

• utils/calibrate.py gains compute_log_weight_smoothness_penalty (pure helper) plus two keyword-only kwargs on calibrate_local_areas: prior_weights and smoothness_penalty.
• When both are supplied, the training loss is augmented with a log-space L2 penalty pulling the current weights towards the prior year's final weights. Zero-prior entries (households outside an area's country) are excluded from the mean.
• Defaults reproduce pre-step-5 behaviour exactly. Shape mismatch raises a clear ValueError.
• The penalty is computed from the underlying log-space weights, not the dropout-augmented tensor fed into the fit loss, so the regulariser does not double-count the dropout noise.

Step 6 — downstream & consumer changes

• storage/upload_yearly_snapshots.py — parallel uploader to upload_completed_datasets.py. Same private destination; the existing audited path is not modified. Destination constants are not exposed as function arguments — redirecting the upload requires a code edit reviewed under CLAUDE.md's data-protection rules.
• tests/conftest.py — new enhanced_frs_for_year factory fixture that resolves enhanced_frs_<year>.h5 and falls back to the legacy enhanced_frs_2023_24.h5 for 2023. Skips (rather than errors) when the year's file is missing so partial panel builds run cleanly.
• docs/panel_downstream.md — coordination note for the sibling policyengine-uk repo: runtime-uprating skip options, fixture migration pattern, default year set, out-of-scope items.

Together these six pieces give a callable, tested pipeline for going from a single imputed base to a set of uprated + demographically-aged per-year snapshots with year-aware targets, smooth cross-year weights, a safe upload path, and test fixtures that consumers can use without rewrites.

What's not in this PR

Explicit non-goals, all deferred so this change stays reviewable:

• Real ONS life tables and fertility rates — step 3 ships placeholders.
• DWP 2026+ benefit caseloads, annual local-area CSV refreshes, NTS 2025+ — data-sourcing work, policy-dependent for DWP.
• Marriage, separation, leaving-home, migration — step 3 extension.
• Refactoring create_datasets.py:main() to invoke create_yearly_snapshots automatically — kept as a follow-up so that the change-of-default conversation happens in its own PR.
• Changes to policyengine-uk — documented in docs/panel_downstream.md, left to the sibling repo.
• Tuning the smoothness-penalty coefficient on real data — empirical question best answered against full loss matrices.

Test plan

All 80 new tests green locally; formatting and lint clean.

• pytest policyengine_uk_data/tests/test_panel_ids.py — 10/10
• pytest policyengine_uk_data/tests/test_yearly_snapshots.py — 7/7
• pytest policyengine_uk_data/tests/test_demographic_ageing.py — 22/22
• pytest policyengine_uk_data/tests/test_resolve_target_value.py — 12/12
• pytest policyengine_uk_data/tests/test_smoothness_penalty.py — 10/10
• pytest policyengine_uk_data/tests/test_calibrate_smoothness_integration.py — 5/5
• pytest policyengine_uk_data/tests/test_upload_yearly_snapshots.py — 7/7
• pytest policyengine_uk_data/tests/test_conftest_fixtures.py — 7/7

Coverage summary

Eight new test files, one pure-function and integration harness per subsystem. All utility/ageing/calibration/upload tests use small in-memory fixtures or monkeypatched dependencies so they run without any real FRS data or network access.

Notable coverage:

• Upload safety: the private HF repo, HF repo type and GCS bucket are locked by test; the function signature is asserted to only allow years and storage_folder as kwargs (no way to redirect the upload).
• No-partial-upload invariant: a missing file aborts the upload call before any network traffic.
• Fixture factory: resolves both the legacy 2023_24 name and the new per-year name, prefers the new one when both exist, coerces str/int years.
• Smoothness regulariser: gradient masking on zero-prior entries, log-space symmetry, large-penalty integration test that proves pull-towards-prior.

Files touched

policyengine_uk_data/utils/panel_ids.py                        (new + classify_panel_ids)
policyengine_uk_data/utils/demographic_ageing.py               (new)
policyengine_uk_data/utils/calibrate.py                        (+ helper + two kwargs)
policyengine_uk_data/datasets/yearly_snapshots.py              (new)
policyengine_uk_data/targets/build_loss_matrix.py              (extract resolve_target_value)
policyengine_uk_data/datasets/local_areas/constituencies/loss.py    (honour time_period)
policyengine_uk_data/datasets/local_areas/local_authorities/loss.py (honour time_period)
policyengine_uk_data/storage/upload_yearly_snapshots.py        (new)
policyengine_uk_data/tests/conftest.py                         (+ enhanced_frs_for_year)
policyengine_uk_data/tests/test_panel_ids.py                   (new, 10 tests)
policyengine_uk_data/tests/test_yearly_snapshots.py            (new,  7 tests)
policyengine_uk_data/tests/test_demographic_ageing.py          (new, 22 tests)
policyengine_uk_data/tests/test_resolve_target_value.py        (new, 12 tests)
policyengine_uk_data/tests/test_smoothness_penalty.py          (new, 10 tests)
policyengine_uk_data/tests/test_calibrate_smoothness_integration.py (new, 5 tests)
policyengine_uk_data/tests/test_upload_yearly_snapshots.py     (new,  7 tests)
policyengine_uk_data/tests/test_conftest_fixtures.py           (new,  7 tests)
docs/targets_coverage.md                                       (new)
docs/panel_downstream.md                                       (new)
README.md                                                      (new section)
changelog.d/345.md                                             (new)

Tracks and closes: #345.