Swap the microcosm spine into the UK national build: self-policing build, retired June path, hardened parity instruments (#757)

[juaristi22]

Executes the swap increment of #757 (WS-E of #145, epic #665), stacking on merged #747 and #743. Eleven commits, ordered as review units; each independently verified (ruff, ci_test_groups.py --verify, the uk and shared-spec groups — no US file is touched).

Closes #770. Closes #771. Closes #772. Closes #773. Closes #774. Closes #775.

The swap machinery

1. Evidence layer (f76228ba) — every spine stage records a health receipt (sidecar stage_evidence 5→12 stages, duck-typed over the executed roster); the four donor-support stages emit clip receipts where the clip was silent; the seam binds and inherits the spine sidecar. Payload-inert, proven on licensed data.
2. Published-fact translations (d9746b87) — any_child_under binds as the age predicate it means here; the NHS anchor composes from the three delivery columns the stage produces; both declared, receipted, never dropped.
3. The spine build polices itself (c57d4da4) — a scoped battery at the assembled/transferred checkpoints: 13 per-stage health gates on the stages' own receipts, the CGT donor stage gated at stage time on the formerly-dead band bounds (closes the third Port the CGT structure, salary sacrifice, and student loan stages onto the UK spine at FRS 2024-25 #740 sign-off), BRMA enum at assembled. Gate ownership is a refuse-on-gap, refuse-on-undeclared-overlap partition (uk_aggregate_admin the one declared duplicate). The W9 family gate resolves against the plan that builds each family; the seam refuses inputs without a green spine battery report.
4. June path retired (0dfd9566) — driver, national_build.py, hmrc_restoration.py deleted whole (−8,434 lines); UK_CANDIDATE_DATASET_NAME = "microcosm_uk_2024"; the scorer names candidates from the artifact it reads; no incumbent byte feeds any build (the parity reference, input-mass descriptor and contract pins stay as comparison instruments until B4 re-arms against the spine's own line).
5. Parity instruments refuse manufactured verdicts (45865198, Merged UK spine: final age-tail breaks all six stored NHS identities #770–Correct stale 1.56.14 main-residence evidence in the 1.56.16 UK register #775) — structured quantitative register expectations enforced (direction, magnitude, exact count deltas) with a hermetic pin to the packaged reference; strict acceptance pinned to the 0.02 contract band (diagnostic otherwise, defect wins); closed-world identity-bound weighted parity; the E6 identity check recomputes NHS at stage-time age (exact inversion), with a reversion-failing regression. The age-first NHS derivation is Derive NHS allocation (and audit age-conditioned imputations) from the disaggregated age surface #785, a successor method change.
6. Licensed certification of the 25-stage candidate (05748419) — e4–e8 ladder green (the run itself caught the Merged UK spine: final age-tail breaks all six stored NHS identities #770 class a second time in e8's donor-carrier recompute — both sites clamped); all 26 register bounds re-measured on the candidate at declared 1e-4 grain; strict parity signed_parity, 0 unsigned; committed acceptance receipt + roster-drift binder.
7. Reviewed exclusions gain lifetimes (b768788f) — the measure-exclusion register carries approver/adjudication/ISO window (María, 3 months), enforced at apply time with correct-or-renew; owned_land re-signed on a fresh 25-stage stability receipt (53.8% national / 96.7% worst-region seed swing persists; expiry 2026-09-26).
8. Four targets re-bound to the facts they publish (0e38b8ad) — obr.ni → total_national_insurance (was bit-identical to the employee-only component; measured recovery −76.7% → −2.9%); the ONS composition family regains its one-family/one-person restrictions (union previously counted 38.6m households on a 28.5m frame; couple_no_children +63.7% → +34.2% pending the multi-family exclusions; the lone-person categories now exact and honest). The unrelated/lone-parent split stays unfixed by design — it needs a relationship-to-head surface the frame lacks; exclusions pending that column.
   9–10. Review + shakedown hardening (28a701d8, 87fca709, ae5e9805) — @vahid-ahmadi's four findings all confirmed and fixed (dispositions: comment above); the battery's licensed shakedown contributed the un-run-stage evidence defect (independently converged), the boundary engine artifact, the E7 reviewed-absent declaration, and the CGT receipt stage-stamp. Support-clip allowances pinned at the receipted zero baseline with fail-closed on absence; the assembled boundary is a declared stage name, not an index; release-candidate strictness is an explicit flag.
9. The battery-built candidate certified (0c6f4206) — spine-g: first UK spine built end-to-end under the fully armed regime (14/14 gates, no phase blocked). Acceptance receipt re-cut to it with a spine_battery block the binder enforces; three code vintages (spine-d/e/g) pairwise payload-identical — the proof receipts and gates never moved a byte.

Evidence (licensed, data/ukds/acceptance/757-swap/)

Seam live proofs: 403 targets bound (+25 vs the campaign baseline, none dropped), shared-target fit statistically identical to the campaign's best run; the post-rebind run improves loss to 0.1362 and correctly blocks at uk_target_fit on the known-catalogue gaps — that diagnostics file is the working packet for the adjudicated exclusion pass.

Deliberately out of scope (the successor increment on #757, sequenced because each needs the first certified line or an adjudication in flight)

• B4: uk_input_mass_parity re-armed against the spine's own first certified line; uk_qrf_tail_concentration armed from the Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 L3 baselines. Cannot precede the cut that creates the line.
• B5: the release-cut certification producer — the executable home of the 16 declared national preflight/terminal gates (their June runner retired here; no live path lost coverage, since everything that runs today is gated by the spine battery, the seam battery, and the cross-run refusals), the multi-part certification union, the seam refusing --release-candidate, and the build-record shippable literal's removal.
• The uk_target_fit dispositions per the adjudicated partition: ~33 signed exclusions, the would_claim_uc lever run, and the fuel-duties ledger-side retarget question.
• UK_REFERENCE_DATASET_NAME re-semantics (meaningful only once a second certified line exists); the cross-lane markdown sweep (separate PR by ruling).

🤖 Generated with Claude Code

[vahid-ahmadi]

Automated review pass (Claude Code, diff only — no build or test execution). Short review as asked, so nits are omitted.

First, the thing worth stating given policyengine-uk-data#468 landed today: nothing in this PR touches local-area target compilation, area joins, or UC targets, so that positional-join hazard has no surface here. The only join-shaped code added, hmrc_restoration._distributional_mass_shares, maps household weights by household_id and is correctly keyed.

Four findings.

1. tools/build_uk_frs_spine.py:~646 — the assembled phase collects evidence from stages that have not run

In _run_plan_with_spine_sampling, the assembled gate phase calls stage_evidence_provider(), which runs _collect_stage_evidence over all of _STAGE_NAMES — including stages that have not executed at that boundary. checkpoint_metadata() on UKWASWealthStageTransform raises RuntimeError("checkpoint metadata requires a completed stage run.") when last_result is None, so any not-yet-run stage exposing that hook aborts the build at the assembled phase rather than yielding empty evidence.

The last_result-based collector this replaces was null-safe; the new hook path is not, so this is a regression introduced by the refactor rather than a pre-existing gap. Either scope the collection to stages that have run at that boundary, or have the hook return empty evidence instead of raising when there is no completed run.

2. tools/build_uk_frs_spine.py — the gate phase boundary is a hardcoded index

assembled_end = min(11, len(plan.stages)) splits the declared stage plan into the assembled and transferred gate phases by position rather than by stage name. Inserting or reordering any stage in country_stage_plan silently moves the boundary, and the gates then run against the wrong frame with no error — a stage plan change in a different file breaks the gate semantics here, and nothing connects the two.

Worth naming the pattern: this is the same shape as the defect in policyengine-uk-data#468 — a position standing in for a key, correct only while two independently-maintained orderings happen to agree. Splitting on a named stage (or a declared phase marker on the stage itself) makes a plan change either correct or loud.

3. packages/microcosm-build/src/microcosm/build/uk/gates.json — the new support-clip gates cannot fail on their stated subject

The new uk_stage_*_support entries are release_blocking but declare max_clipped_low_rows_by_column: {} and max_clipped_high_rows_by_column: {}. In _support_clip_gate an absent per-column allowance means no comparison is performed, so these gates assert receipt shape only: a stage clipping 100% of its rows passes them.

That makes them the same class as the E7 receipt on #747 — a green result reflecting the absence of a check rather than the presence of agreement — and release_blocking makes the claim stronger than what is measured. Either populate the allowances or drop the blocking status until they are populated.

4. tools/build_uk_frs_spine.py — release_candidate is now inferred from sample fraction

release_candidate=args.sample_fraction == 1.0 makes every full-scale developer build a release candidate, which flips evidence_absent gaps from tolerated to blocking. It was previously an explicit caller flag in build_uk_national_dataset, so this changes dev-build behaviour with no way to opt out — a developer running at full scale for an unrelated reason now gets release-candidate strictness. If the inference is deliberate, an explicit override flag would keep the ergonomics; if not, the caller flag was the clearer contract.

---

2 and 3 are the ones I would want resolved before merge. Both are gates that pass without testing what they name, which is the same failure mode this lane has been closing systematically over the last week — worth catching here rather than after the next receipt is cited as evidence.

[juaristi22]

Review dispositions (all four confirmed; fixed at 28a701d and ae5e980, alongside two further findings from the battery's licensed shakedown runs):

1. Assembled-phase evidence from un-run stages — confirmed, fixed before the review posted. The first licensed battery build hit exactly this at the assembled boundary; fix landed at 28a701d with the executed-prefix scoping (each boundary offers only the stages the plan has run) and a two-sided regression. The un-run-stage refusal stays deliberately intact — asking for evidence of a run that has not happened should raise; the fix is not asking.
2. Positional phase boundary — confirmed, fixed (ae5e980). assembled_end = min(11, …) is now the declared UK_SPINE_ASSEMBLED_FINAL_STAGE = "frs_brma", resolved by name; an armed battery refuses a plan lacking the boundary stage, so a stage-plan change is either correct or loud. The constant's comment names the uk-data#468 position-for-key class.
3. Shape-only support gates — confirmed, fixed at both layers (ae5e980). Allowances populated per column at the receipted spine baseline (zero rows clipped on every non-exempt column across all four stages, measured on the licensed 25-stage candidate), and _support_clip_gate now fails closed on a missing allowance with a pin-or-exempt message — the green-by-absence class cannot re-enter through a future entry. One additional same-class finding while in there: the LCFS exempt columns were disjoint from the declared column set, leaving the exempt-marking assertion unable to fire; they now join it.
4. Inferred release-candidate — confirmed, fixed (ae5e980). Explicit --release-candidate flag, default off; the sample-fraction inference is gone.

The shakedown's transferred phase surfaced two more, both fixed in the same commit: the source-signal gate's reviewed list now carries the E7 reviewed-absent hmrc_spi_incapacity_benefit_income (the battery agrees with the standing #683 adjudication rather than re-litigating it each build), and the CGT spine stage re-stamps the shared imputation summary with its own stage name — the gate was rightly enforcing the distinct-receipts-per-family rule against a receipt claiming the certified family's name.

Battery digests re-cut from the live producer; ruff + partition + uk + shared-spec groups green. A fresh licensed spine build under the final gate regime follows, and the committed acceptance receipt re-cuts to that battery-built candidate.

[juaristi22]

The UK gate landscape

Every check in the spine-build and calibration process at uk-spine-swap-757 @ 0c6f4206: 35 declared gates plus the advisory instruments. The graph shows who runs which gates when — color marks the owning battery, not progress; dashed marks gates declared today but without a runner until B5. The table carries the per-gate detail.

flowchart LR
  subgraph SPINE["Spine build battery · runs during build_uk_frs_spine"]
    direction TB
    subgraph ASM["assembled — after frs_brma"]
      BRMA["brma_enum_domain"]
    end
    subgraph TRF["transferred — after age_tail"]
      CLIP["4× stage support clips<br/>receipts vs pinned zero baseline"]
      SIG["spine_leaves_signal"]
      SPI1["spi_support_channel_mass"]
      SPI2["spi_income_spine_identity"]
      CGT1["cgt_incidence_clone_mass"]
      CGT2["cgt_band_donors_support"]
      CGT3["cgt_gains_spine_summary"]
      REAL["2× realization<br/>salsac + student loans"]
      AGE["age_tail_targets"]
    end
  end

  subgraph JOIN["Cross-run refusals"]
    SIDE["sidecar binds artifact"]
    REP["battery report green"]
  end

  subgraph SEAM["Calibration seam battery · terminal"]
    FIT["target_fit ≤25%"]
    ESS["weight_ess"]
    RATIO["weight_ratio"]
    ZWS["zero_weight_strata"]
    ADMIN1["aggregate_admin<br/>(calibrated frame)"]
    COV["reference_coverage"]
  end

  subgraph NAT["National release-cut battery · runner lands with B5"]
    direction TB
    subgraph PRE["preflight"]
      MAN["coverage manifest current"]
      FAM["family_build_stages"]
      LCP["2× ledger_compile_parity"]
    end
    subgraph TERM["terminal"]
      RIC["release_input_coverage"]
      DEG["degenerate_release_surface"]
      WAUD["weights_audit"]
      NONNEG["nonnegative_columns"]
      SUP["support (reviewed bounds)"]
      ADMIN2["aggregate_admin<br/>(release frame)"]
      EXP["export_surface"]
      TGT["target_surface"]
      TAKE["take_up_signal ±0.05"]
      SLENUM["student_loan_plan enum"]
      IMP["input_mass_parity"]
      QRF["qrf_tail_concentration"]
    end
  end

  subgraph ADV["Advisory instruments"]
    PAR["strict parity + signed register"]
    LADDER["identity ladder e4–e8"]
    PAYLD["payload twins"]
    RCPT["acceptance receipt binder (CI)"]
  end

  SPINE -->|"h5 + sidecar + battery report"| JOIN --> SEAM
  SEAM -->|"calibrated candidate"| NAT
  SPINE -.->|"evidence"| ADV

  classDef spine fill:#dbeafe,stroke:#1d4ed8,color:#111
  classDef seam fill:#dcfce7,stroke:#15803d,color:#111
  classDef nat fill:#fef3c7,stroke:#b45309,color:#111,stroke-dasharray:5 4
  classDef join fill:#f3e8ff,stroke:#7e22ce,color:#111
  classDef adv fill:#e5e7eb,stroke:#6b7280,color:#111
  class BRMA,CLIP,SIG,SPI1,SPI2,CGT1,CGT2,CGT3,REAL,AGE spine
  class FIT,ESS,RATIO,ZWS,ADMIN1,COV seam
  class MAN,FAM,LCP,RIC,DEG,WAUD,NONNEG,SUP,ADMIN2,EXP,TGT,TAKE,SLENUM,IMP,QRF nat
  class SIDE,REP join
  class PAR,LADDER,PAYLD,RCPT adv

Loading

Blue = spine battery (14 gates, runs live; first green build spine-g, 14/14). Green = calibration seam (6 gates, runs live). Amber dashed = national battery: 16 gates declared and partitioned, runner retired with the June driver, executable home lands with B5. Purple = refusals joining the batteries. Grey = advisory. aggregate_admin appears twice by declaration (UK_SHARED_GATE_IDS): same anchors, two frames.

Per-gate detail

Status: ✅ enforcing today · 🟡 declared, runner lands with B5 · 🔶 enforcing with a signed follow-up · ⚪ advisory.

Spine build battery (14 · assembled/transferred)

gate	measures	against	status
uk_brma_enum_domain	BRMA values are legal enum members	live policyengine-uk enum domain	✅ moved to assembled: weight-independent, fires 14 stages earlier
uk_stage_{was_wealth, lcfs_consumption, etb_vat, etb_services}_support	rows clipped to donor support per column (13/18/1/3); LCFS's 3 energy columns must be marked exempt	per-column allowances pinned at the receipted spine baseline (0 clipped); missing allowance fails closed	✅
uk_stage_frs_hmrc_spine_leaves_signal	each SPI leaf has ≥1 source-signal row	FRS 2024-25 tabs; reviewed-absent list carries the E7 incapacity adjudication	🔶 stage fence promotes the column if a future vintage carries signal
uk_stage_spi_support_channel_mass	SPI prior-mass share and household count	declared 0.5 share ± tolerance	✅
uk_stage_hmrc_spi_income_spine_identity	post-draw identity exact; per-column effective-mass floors	coverage manifest effective_mass_requirements	✅
uk_stage_cgt_incidence_clone_mass	clone-flag mass halves balance	declared factor 1.0 ± cap	✅
uk_stage_cgt_band_donors_support	realized donor gains inside their band; none below £12,300	cgt_band_donor_support_bounds.json (HMRC Table 2.1a), at stage time per adjudication	✅ closes #740 sign-off 3
uk_stage_hmrc_cgt_gains_spine_summary	summary present, ≥1 band row, stamped by the spine stage	published HMRC Table 3 surface	✅
uk_stage_salary_sacrifice_realization	realization deviation, headcount vs target, cap binding	contract values in the stage receipt	✅
uk_stage_student_loans_realization	per-plan shortfall and realization; England counts	SLC liable-stocks resource	✅
uk_stage_age_tail_targets	achieved weighted 85+ band mass	ONS band populations; each cell names its ons.population.* register target	✅

Cross-run refusals (seam input loader)

check	measures	against	status
sidecar binding	the input h5 is the artifact its sidecar describes	entity row counts, weight kind, weight total	✅
battery-report refusal	the input's build was gated and green	<spine>.spine_gates.json: present, unblocked, all release-blocking passed	✅ refused the pre-gate spine-e on first contact

Calibration seam battery (6 · terminal)

gate	measures	against	status
uk_target_fit	per-target relative error ≤ 25%	compiled chronicle register (403 refs @2025)	🔶 blocks on the 49 catalogued gaps until the adjudicated exclusion pass lands
uk_weight_ess	effective sample size ≥ floor	#578 doctrine constants	✅
uk_weight_ratio	max/median weight ratio ≤ cap	#578 doctrine constants	✅
uk_zero_weight_strata	no undeclared zero-weight strata	declared strata	✅
uk_aggregate_admin (seam)	weighted admin anchors on the calibrated frame	NEED means + NHS budget (composed from 3 delivery columns, declared translation)	✅ shared id
uk_calibration_reference_coverage	every compiled reference bindable or accounted	register vs materialized measures + windowed exclusion register	🔶 exclusions carry approver + expiry (2026-11-25); outside the window the run refuses

National release-cut battery (16 · declared; runner lands with B5)

gate	measures	against	status
uk_release_input_coverage_manifest_current (pre)	committed coverage manifest matches the engine surface	policyengine-uk variables	🟡
uk_release_family_build_stages (pre)	all 12 required families in the build plan	coverage manifest, re-homed to the spine plan (W9)	🟡
uk_ledger_compile_parity_production_2023 (pre)	compiled targets vs the certified June 2023 surface	frozen fixture + signed differences	🟡
uk_ledger_compile_parity_incumbent_2025 (pre)	compiled targets vs the incumbent registry @2025	frozen Fixture-B + signed differences	🟡
uk_release_input_coverage	release frame carries every required column	coverage manifest	🟡
uk_degenerate_release_surface	no near-constant release column	reviewed exclusions (owned_land exp 2026-09-26; charitable gifts exp 2027-02-20)	🟡🔶
uk_weights_audit	every fit emitted typed weight records	stage FitWeightRecords	🟡
uk_nonnegative_columns	declared non-negative outputs hold	per-stage declarations	🟡
uk_support	terminal values inside reviewed donor bounds	4 committed outward-rounded bounds resources	🟡
uk_aggregate_admin (national)	same anchors on the release frame	same anchor resources	🟡 shared id
uk_export_surface	export surface = declared surface	parity evidence	🟡
uk_target_surface	realized target surface vs declared registry	register at name@period grain (register-sourced; the gate note says so)	🟡
uk_take_up_signal	weighted take-up shares within ±0.05 of contract	take-up contract (HMRC/DWP/Scottish Government statistics; UC 0.55 = U8)	🟡🔶 kept terminal: it detects when the UC lever is needed
uk_student_loan_plan_enum_domain	plan values are legal	engine enum	🟡
uk_input_mass_parity	weighted input-mass totals vs reference	today the incumbent 1.56.16 descriptor; B4 flips to the spine's first certified line	🟡🔶
uk_qrf_tail_concentration	top-k weight share of QRF outputs	spec-armed; B4 re-arms from the #686 L3 baselines	🟡🔶

Advisory instruments (document; never block)

instrument	measures	against	status
verify_uk_spine_parity --strict	candidate surface: counts, shares, weighted totals, with enforced direction, magnitude, and count expectations	frozen eFRS 1.56.16 extraction + the quantitative signed register (26 bounds at 1e-4 grain)	⚪ signed_parity, 0 unsigned on spine-g
identity ladder e4–e8	stored columns re-derive exactly (stage-time age on e6/e8)	the artifact's own declared derivations	⚪ green on spine-g
compare_uk_h5_payload	twin payload identity	pairwise builds (d/e/g: three code vintages, one payload)	⚪
acceptance receipt + binder	committed evidence names the real candidate: sha, roster, ladder, parity, battery report	the driver's executed plan; roster drift fails CI	⚪ CI-enforced

Signed follow-ups (each with an owner)

1. National battery runner — B5's release-cut certification producer; until then shippable stays false by construction.
2. uk_input_mass_parity reference — B4 flips it to the spine's own first certified line; the incumbent descriptor retires.
3. uk_qrf_tail_concentration — B4 re-arms from the L3 baselines with run provenance.
4. uk_target_fit's 49 blocks — 4 binding fixes landed (obr.ni −76.7% → −2.9%); the exclusion pass and UC lever run follow.
5. Windowed registers — measure exclusions exp 2026-11-25; owned_land exp 2026-09-26; charitable gifts exp 2027-02-20. All fail closed on expiry.
6. Reviewed-absent incapacity — declared per E7; the stage fence, not the gate, promotes it when a vintage carries signal.