Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686)

changelog.d/686-uk-identity-ladder-e8-stacking.fixed.md

@@ -0,0 +1 @@
+Fix the UK identity-stability ladder for E8's stacking layers (#686). Building a spine that carries every stage through E8 and running the whole ladder surfaced that the e4, e5 and e6 receipts had gone stale: each was written when the SPI support channel was the only stage stacking rows, and E8 added the capital-gains incidence clone and the CGT band donors after them. The failures were in the instrument, not the spine — proved by running the unchanged tool against two artifacts, where the pre-E8 spine passes and the post-E8 spine fails. Three distinct mechanisms were involved, which is why one fix did not cover them: e4 recomputes identity-keyed draws, and a stacked row carries a value copied from its source that was never drawn for its own id; e5's regional property uprating scales to a per-region mean over the owner households in the frame, so stacked rows shift the denominator; and e6's NHS allocation normalizes against an absolute budget, so it needs the stage-time *weights* rather than only the stage-time population, and it was dividing out the SPI channel's reserved share while the clone's `mass_split` went unrestored. Scoping now excludes every stacked layer through a single declared flag list, and the weight restoration reads its factors from the declared operations rather than hardcoding them. Crucially the divisor is driven by which stacking flags the artifact actually carries, not by the committed roster: the first implementation used the roster and divided the clone factor out of a spine built before that stage existed, which the pre-E8 artifact caught immediately. Both vintages now pass. The standing hazard is recorded at the helper: a new mass-redistributing operation kind must be registered there, and the failure mode if it is not is a receipt that silently compares against the wrong grossing scale.

changelog.d/686-uk-incumbent-repin-1-56-16.changed.md

@@ -0,0 +1 @@
+Re-pin the UK enhanced-FRS parity reference from the incumbent data package's 1.56.14 to its 1.56.16 (#686). The previous pin carried uk-data#461: from the 2024-25 FRS release the raw benunit table is no longer ordered by `sernum`, so every benunit-level variable — `benunit_id` itself included — landed on the wrong benefit unit relative to the model's sorted-id entity order. Because the parity screen compares unweighted nonzero shares, it is permutation-blind to that defect: `is_married` reports a byte-identical 0.256587 across both artifacts while every one of its values sits on a different row. Signing whole-spine parity against 1.56.14 would therefore have frozen an upstream defect into the contract. The re-pin moves the source identity (revision `a9e52499…`, sha256 `e433e532…`, 126,553,300 bytes, new self-describing `source.version` field), the `uk_input_mass_parity` reference-registry identity and its `totals_sha256`, the release-input coverage manifest's reference block, and the four gate-battery mirrors in the data shard. The microcosm spine needs no corresponding fix: `frs_spine` has sorted the raw benunit table by `benunit_id` since the original ingest commit (2026-08-14), four days before the upstream fix, so it was never exposed. Column surface and entity counts are unchanged (145 layers; 113,617 / 61,223 / 52,846), and the raw-FRS zip pin in `uk/frs_release.json` is a different artifact and stays at its own immutable revision. Reference-side movement is recorded in `experiments/686-uk-spine-swap-receipts.md`: 39 of 145 unweighted shares move, none by more than ±0.0046, so the ±0.02 parity screen is undisturbed; the licensed weighted register moves on all 128 comparable columns because 1.56.15 changed the Universal Credit caseload targets and the incumbent's calibration re-solves with unseeded dropout, which is the already-signed register-realization class and stays far inside the gross-mass fence.

changelog.d/686-uk-parity-acceptance-band.changed.md

@@ -0,0 +1 @@
+Hold the whole-spine parity instrument's share surface to the #723 acceptance band rather than to the reference's six-decimal grain. The spine re-runs every stochastic stage, so ninety of the file's columns land inside the band on third-decimal drift; requiring a permanent adjudication for each would have filled the register with entries describing noise and blanket-covering the columns they name, which is the failure mode the register exists to prevent. The band governs only which magnitudes must be adjudicated, never what is reported: every difference down to the six-decimal grain still appears in the receipt, now partitioned into `differing` and `within_band` with the in-band maximum carried alongside, and `--share-band 0` restores the exact-grain check. Structural differences are deliberately outside the band's reach — a column appearing or vanishing, and every entity count, still signs exactly, at any band. Alongside this, `--strict` now separates a register entry that matched nothing on a surface this run compared from one whose surface was never examined: the weighted-totals surface stays unexamined until there is a calibrated candidate to compare, so an entry scoped to it is reported as dormant instead of failing the swap-acceptance posture, and becomes an ordinary unused entry again as soon as a run supplies the sidecars.

changelog.d/686-uk-proof-machinery-review.fixed.md

@@ -0,0 +1 @@
+Fix seven defects in the machinery that certifies the UK spine (#686 review). Three could change what a proof means. The signed-differences register and the payload comparator spoke different surface vocabularies — every committed entry is scoped to `nonzero_shares`, `weighted_totals` or `entity_counts`, while `--structure-only` looked entries up as `payload_column`, so the swap-acceptance verdict could only ever fail while reporting all thirteen entries as unused; lookup now bridges the value-bearing surfaces to the payload surface, because both instruments read the same adjudicated fact through different measurements. `expectation` was validated at load and then never consulted, so an entry signing a column's *appearance* also silently signed an arbitrarily large *value* divergence in it — the failure mode the register's own scope note names; every lookup now matches on expectation, and structural expectations never excuse a value difference. The E7 identity receipt returned an empty recomputation when the support-channel layer was absent, so its mismatch loops never ran and it reported green on an artifact where nothing had been checked — it now refuses, names the columns it compared, and treats a certified column missing from the store as a mismatch rather than a narrower comparison. Four more were quieter: the anti-self-comparison fence passed vacuously for a candidate extraction that omitted its source identity, which is now required; a zero reference total produced `float("inf")`, which `allow_nan=False` then refused, turning a real divergence into "no verdict possible", and is now reported as an explicit flag; `--strict` counted an entry matching a within-band column as unused and false-failed on it; and the Scottish water helper's discount fallback rested on an unasserted claim about the vintage's domain, which now raises at build time rather than silently paying sewerage at gross into `council_tax`.

changelog.d/686-uk-scottish-water-charges.fixed.md

@@ -0,0 +1 @@
+Fix the Scottish water and sewerage charge for the FRS 2024-25 vintage (#686, closing the `water_and_sewerage_charges` item on #736). FRS 2024-25 retired `CWATAMT` and `CSEWAMT` ("Wat./Sew. Charge: Final value after discount"): the headers survive but carry no data in any of the 16,288 households, and the FRS replaced them with the derived Scotland-only `CWATAMT1`/`CSEWAMT1` ("Weeklyised gross annual dom. water/sew. charge on bill", "DV created in 2024-25 as variable was removed from the dataset"). Two consequences were live. First, the incumbent's `np.where(scotland, csewamt + cwatamtd, watsewrt).fillna(0)` adds the retired cell *before* filling, so a wholly blank `CSEWAMT` propagates NaN and zeroes the charge for all 1,663 Scottish households that have one; microcosm's per-column fill left it standing, which is the entire +0.1009 nonzero-share divergence flagged on #736 — microcosm correct, incumbent defective, and the gap reproduces on the raw tab to +0.1021 before composition. Second, microcosm's own level was short: `CWATAMTD` is the water charge alone, so it emitted about £185 per Scottish household against roughly £490 for England and Wales on `WATSEWRT`. Both sites now call one shared `scottish_water_and_sewerage_weekly` helper — the spine's `water_and_sewerage_charges` and the `frs_council_tax` netting, so the amount removed from the council tax bill is exactly the amount charged — which adds `CSEWAMT1` discounted at the household's own observed factor `CWATAMTD / CWATAMT1`. That preserves the retired cells' after-discount semantics rather than silently switching to a gross basis; the factor is well defined and within (1/3, 1] for the 1,641 households with a positive gross bill, and the two remaining domains (22 with a recorded `CWATAMTD` but no gross cell and zero sewerage, 21 with no council-tax cells at all) are unaffected by construction and now covered by tests. The nonzero share is unchanged by the level fix — the same households are charged either way — so the share difference against the incumbent stands alone as a signed incumbent-defect divergence. The unit fixtures supplied a non-missing `CSEWAMT` and so never exercised what the tab actually contains; they now carry the retired cells blank.

changelog.d/686-uk-signed-difference-entity-scope.fixed.md

@@ -0,0 +1 @@
+Honour `scope.entities` when the signed-differences register is consulted (#686 re-review). Every committed entry names the entity its columns belong to, but `covers()` matched on surface and column alone — harmless while only the share surface consulted it, because a Frame's column names are globally unique, and load-bearing the moment the payload bridge let a lookup reach a comparator that iterates one table at a time. A household-scoped adjudication could then sign a same-named column on the person or benunit table: a previously-unsigned divergence becoming silently signed, which is the one failure the register exists to prevent. Lookups now take the entity, the payload comparator passes the store key it is iterating, and the bridge carries the entity scope across rather than widening it; a caller that cannot determine an entity passes none and gets no entity filtering, which is safe on the surfaces where the column namespace is already global. Honouring the scope immediately caught a mis-declared one: `student_loan_balance` was scoped to `household` and is a person column, so it is re-scoped here. Two related holes close with it — the loader now refuses an entry on a column surface that names no columns or no entities, so the surface-wide form stays available only to `entity_counts` where the "column" is itself an entity name and a blanket entry cannot absorb anything unrelated.

changelog.d/686-uk-signed-differences-register.added.md

@@ -0,0 +1 @@
+Add the committed UK spine-swap signed-differences register (#686): `uk/spine_swap_signed_differences.json` plus its loader `uk_runtime/signed_differences.py`. The whole-spine comparison has one rule — anything differing between the microcosm-built spine and the frozen enhanced-FRS incumbent that is not signed here is a defect — so each entry names the class of difference, the exact surface (`nonzero_shares`, `entity_counts`, `weighted_totals`, `payload_column`, `root_attr`) and columns it is expected to appear on, disclosure-safe magnitude evidence, the adjudicator, and the date. The loader enforces the three vocabularies, unique kebab-case ids, ISO dates, and precise scoping; a test additionally refuses a column-surface entry that names no columns, since an unscoped entry would absorb unrelated divergences wholesale. The register sits **above** the per-gate reviewed-exclusion registers rather than replacing them: those stay per-gate, per-reference, and expiring, because a suppression must not outlive its reason, whereas a signed difference is a permanent adjudicated fact — so the loader rejects `expires_on` outright and points the author at the exclusion registers instead. It ships seeded with the two Scottish water adjudications, which are scoped separately on purpose: the incumbent's NaN-zeroing signs the share surface, while the successor-cell level change signs weighted totals and deliberately does not sign the share, which it leaves untouched. The E4–E8 method classes recorded in the licensed acceptance receipts, and the columns the #723 screen placed beyond the parity band, are transcribed as each is re-measured against the re-pinned reference and scoped to the divergence actually observed, rather than carried across on their prose classification.

changelog.d/686-uk-spine-defect-batch.fixed.md

@@ -0,0 +1 @@
+Fix two spine defects the first armed calibration campaign surfaced, before the swap rather than after it (#686). **The SPI channel no longer ships NaN.** Twelve person columns are full income concepts the FRS instrument does not measure, and the SPI stage left them NaN on the FRS channel — assessment-era honesty that made the artifact unloadable in practice: PolicyEngine-UK's `validate()` refuses NaN inputs, and any finiteness fence on the calibration path refuses the frame, so the campaign had to zero-fill them outside the build before it could calibrate at all. Zero is the adjudicated stage-time semantics for a concept the instrument never asked about, the auxiliary-crosswalk guard already stops the QRF mistaking the fill for measured data, and a regression test now asserts the stage leaves no NaN in any float column. **The FRS age top-code is disaggregated.** The licensed delivery records no age above 80, so every 80+ person arrives piled at exactly 80: the 85-89 and 90+ population targets are structurally unbindable and the 80-84 band starts at roughly double its target carrying the whole 80+ population — a defect the incumbent shares. The new `age_tail` stage reassigns each piled person a band drawn from a sex-specific inverse CDF over the ONS mid-year populations and a uniform integer age within it, keyed on `person_source_id` so a household and its capital-gains clone twin receive the same age, deterministic under a declared seed with no global RNG. The six band populations ship as a committed resource in which every cell records the calibration target id it must agree with, so the imputation source and the target denominators cannot drift apart. It runs last in the spine plan, downstream of every stage that conditions on age, so imputation conditioning is unchanged and the stage's whole effect is the pile's dispersal; ages are assigned toward the ONS distribution while calibration still owns the totals.

changelog.d/686-uk-spine-swap-signed-differences.changed.md

@@ -0,0 +1 @@
+Sign the twenty-six beyond-band divergences between the microcosm-built UK spine and the re-pinned 1.56.16 incumbent (#686), taking the whole-spine parity verdict from `defect` to `signed_parity` with nothing unsigned. The register grows from two entries to thirteen, and the split between them is the substance: no entry covers both columns where the spine is closer to its donor and columns where the incumbent is, because the direction of the evidence is part of what is being signed. So the LCFS consumption class is three entries rather than one — ten columns where the regime-gated draw lands closer to the donor on incidence, `petrol_spending` and `diesel_spending` where the `has_fuel` gate under-places incidence and the entry says so, and `transport_consumption` alone where the incumbent is marginally closer on share and the spine closer on level. All donor evidence is re-measured through each stage's own committed cleaning function over its own pinned tab, on the survey-weighted basis, which is the convention that reproduces the E6 acceptance receipt's education figure exactly; that measurement also settles the ETB weight-basis question that was blocking three rows, and shows the incumbent's `dfe_education_spending` to be degenerate — fourteen nonzero households in 52,846 against a donor incidence of 0.28 — so those two columns are signed as a defect fix on the incumbent side rather than as a method preference. `student_loan_balance` is scoped alone because no like-for-like donor benchmark exists for it, which the entry states rather than papers over. Entity counts are signed scoped to `person` and `benunit` rather than surface-wide, so that a future divergence in the household count stays a defect.

changelog.d/686-uk-structure-only-verdict.added.md

@@ -0,0 +1 @@
+Add a `--structure-only` verdict mode to `compare_uk_h5_payload.py` (#686). The swap comparison puts a control build and a candidate build side by side and expects them to share a surface exactly while differing in values only where a difference has been adjudicated, which is a different question from the payload-identity the tool was built to answer. The mode re-verdicts the same measurements rather than relaxing them: every structural predicate stays strict — the same store keys and stored kinds, row counts, column lists in order, dtypes, indexes, and root-attribute names — and each differing column or root attribute must name an entry in the committed signed-differences register, so a signature excuses a differing value and never a differing surface. `payload_identical` is still computed and reported in both modes, so a structure-only receipt stays comparable with a full-mode one, and `--signed-differences` is refused outside the mode rather than silently ignored. Exit codes keep their meaning: 0 when the structures match and every value difference is signed, 1 otherwise, 2 when no verdict is possible.

changelog.d/686-uk-whole-spine-parity-instrument.added.md

@@ -0,0 +1 @@
+Add the whole-spine parity instrument (#686). `build_uk_efrs_parity_reference.py` gains a candidate mode — `--candidate-h5` with `--emit-candidate-json` for the parity surface and `--emit-weighted-totals` for the candidate-side licensed register — so both sides of the comparison are measured by the *same* producer, with the same engine, alias handling and rounding. Otherwise the diff would be between two measurement methods rather than two artifacts. A candidate is identified by its own sha256 rather than checked against the incumbent pin, and the mode is structurally unable to write the committed reference: it refuses any destination inside the country package, and `--check` is refused outright since a candidate can never satisfy a check against the incumbent pin. The new `tools/verify_uk_spine_parity.py` then diffs that extraction against the committed reference across three surfaces — the record-count identity exactly, per-column nonzero shares at the reference's own six-decimal grain with the column-set difference in both directions, and optionally the two licensed weighted registers as relative deltas only — and holds every difference to the committed signed-differences register, so anything differing that is not signed is a defect and exits 1. Two fences keep the verdict from being manufactured: the reference side is always the committed instrument, and a candidate whose extraction claims the incumbent's own sha256 is refused rather than compared, since a copied reference would pass by construction. `--strict`, the swap-acceptance posture, additionally fails when a register entry matched nothing, so the register cannot decay into a blanket amnesty as the spine changes. Verified end to end against the E8 spine artifact: 142 columns compared, the record-count identity holding on households while the known donor-composition deltas show on persons and benefit units, and `water_and_sewerage_charges` reproducing at +0.100897 and binding to its signed entry rather than counting as unsigned.