Split private_pension_wealth out of corporate_wealth on the UK WAS stage (#750)

changelog.d/750-pension-wealth-split.changed.md

@@ -0,0 +1 @@
+The UK WAS stage now emits `private_pension_wealth` (WAS private pension wealth less current-employment DB, uk-data#452 M2) as its own household output, drawn first in the second chain segment; `corporate_wealth` keeps the share-like holdings and the stocks-and-shares-ISA fold, row identity old corporate wealth == new corporate wealth + pension wealth holds on the donor, and the `uk` extra now requires policyengine-uk 2.92, whose means tests disregard the new column.

changelog.d/792-band-edges-compiled-register.fixed.md

@@ -0,0 +1 @@
+Banded calibration targets now derive upper edges from the compiled pre-exclusion register, fixing microcosm#792 so excluding a banded target no longer widens its surviving lower neighbour. The materialization seam now refuses a pruned register unless callers provide its compiled band-edge source.

packages/microcosm-build/pyproject.toml

@@ -38,7 +38,7 @@ us = [
 # The UK extra adds the rules engine for local metric generation from a
 # Microcosm UK H5. Target tables remain explicit inputs, and the base package
 # still does not import policyengine-uk at import time.
-uk = ["policyengine-uk>=2.88", "h5py>=3", "tables>=3"]
+uk = ["policyengine-uk>=2.92", "h5py>=3", "tables>=3"]
 
 [project.scripts]
 microcosm-export-us-l0-refit-h5 = "microcosm.build.us_runtime.l0_refit_export:main"

packages/microcosm-build/src/microcosm/build/target_materialization.py

@@ -21,9 +21,11 @@
 # encodings are in use — a numeric lower bound (HMRC SPI income bands) and a
 # published range label (DWP award bands, in monthly units, hence
 # ``band_period_factor``) — and both reduce to one lower edge, because no
-# reference anywhere declares an upper bound. A band's upper edge is
-# therefore its sibling's lower edge within the same contract target, and the
-# top band runs to infinity.
+# reference anywhere declares an upper bound. A band's upper edge is its
+# sibling's lower edge within the same compiled contract target. Edges must
+# never derive from an exclusion-pruned roster, or excluding a band silently
+# widens its lower neighbour; only the compiled register's top band runs to
+# infinity.
 # Entity-count indicators: a value of one per record of the owning entity.
 _COUNT_VALUE_VARIABLES = frozenset(
     {"household_count", "person_count", "benunit_count"}
@@ -126,6 +128,7 @@ def resolve_target_measures(
     period: int | str,
     max_rounds: int = 8,
     contract_targets: Mapping[str, Mapping[str, Any]] | None = None,
+    band_edge_registry: TargetRegistry | None = None,
 ) -> MeasureResolution:
     """Resolve provider-computed inputs until target materialization binds.
 
@@ -159,6 +162,7 @@ def resolve_target_measures(
             registry,
             contract,
             period=period,
+            band_edge_registry=band_edge_registry,
         )
         skipped = tuple(result.skipped)
         round_receipt = {
@@ -414,6 +418,13 @@ def _band_bounds(
             f"groupby_variable {binding['groupby_variable']!r} but its spec "
             "carries no readable band edge"
         )
+    if lower not in band_edges:
+        raise ValueError(
+            f"banded measure {getattr(spec, 'measure', '?')!r} carries band lower edge "
+            f"{lower!r} that is absent from its contract target's band-edge set "
+            f"{list(band_edges)!r}; the band-edge register does not cover this spec's "
+            "band — pass the compiled (pre-exclusion) register this spec was pruned from."
+        )
     upper = math.inf
     for edge in band_edges:
         if edge > lower:
@@ -429,11 +440,15 @@ def materialize_target_bindings(
     *,
     period: int | str,
     providers: Mapping[str, Provider] | None = None,
+    band_edge_registry: TargetRegistry | None = None,
 ) -> TargetMaterializationResult:
     """Prepare measure columns declared by compiled Ledger target specs."""
 
     provider_registry = {**default_provider_registry(), **(providers or {})}
-    band_edges = _band_edges_by_group(registry, contract_targets)
+    band_edges = _band_edges_by_group(
+        registry if band_edge_registry is None else band_edge_registry,
+        contract_targets,
+    )
     skipped: list[MaterializationSkip] = []
     for spec in registry.specs:
         if hasattr(adapter, "has_column") and adapter.has_column(

packages/microcosm-build/src/microcosm/build/uk/efrs_parity_known_gaps.json

@@ -14,6 +14,7 @@
    "attends_private_school_random_draw": "person",
    "brma": "household",
    "bsp_reported": "person",
+   "bus_fare_spending": "household",
    "bus_subsidy_spending": "household",
    "capital_gains": "person",
    "carers_allowance_reported": "person",
@@ -43,6 +44,7 @@
    "employee_pension_contributions": "person",
    "employer_pension_contributions": "person",
    "employment_income": "person",
+   "employment_sector": "person",
    "employment_status": "person",
    "esa_contrib_reported": "person",
    "esa_income_reported": "person",
@@ -120,6 +122,7 @@
    "savings_interest_income": "person",
    "sda_reported": "person",
    "self_employment_income": "person",
+   "sic_industry_division": "person",
    "ssmg_reported": "person",
    "state_pension_reported": "person",
    "statutory_maternity_pay": "person",
@@ -168,6 +171,7 @@
    "attends_private_school_random_draw": 1.0,
    "brma": 0.998307955714,
    "bsp_reported": 0.002269750231,
+   "bus_fare_spending": 0.176972057506,
    "bus_subsidy_spending": 0.181612426148,
    "capital_gains": 0.028136660276,
    "carers_allowance_reported": 0.013745759756,
@@ -197,6 +201,7 @@
    "employee_pension_contributions": 0.213260351368,
    "employer_pension_contributions": 0.273574493593,
    "employment_income": 0.387291454309,
+   "employment_sector": 0.488599258708,
    "employment_status": 0.985687923006,
    "esa_contrib_reported": 0.004342270757,
    "esa_income_reported": 0.008391805365,
@@ -274,6 +279,7 @@
    "savings_interest_income": 0.357580855206,
    "sda_reported": 0.000107122652,
    "self_employment_income": 0.066184105292,
+   "sic_industry_division": 0.713824072671,
    "ssmg_reported": 0.000359030478,
    "state_pension_reported": 0.181567916228,
    "statutory_maternity_pay": 0.002379481691,
@@ -301,15 +307,15 @@
    "would_claim_universal_childcare": 0.429571577769,
    "would_evade_tv_licence_fee": 0.106472398355
   },
-  "effective_signal_columns": 143,
+  "effective_signal_columns": 146,
   "engine": {
    "h5_input_aliases": {
     "capital_gains": "capital_gains_before_response",
     "employee_pension_contributions": "employee_pension_contributions_reported",
     "employment_income": "employment_income_before_lsr"
    },
    "package": "policyengine-uk",
-   "version": "2.89.0"
+   "version": "2.92.0"
   },
   "entity_records": {
    "benunit": 618980,
@@ -335,6 +341,7 @@
    "attends_private_school_random_draw": 1.0,
    "brma": 0.997869,
    "bsp_reported": 0.005877,
+   "bus_fare_spending": 0.204941,
    "bus_subsidy_spending": 0.208829,
    "capital_gains": 0.231216,
    "carers_allowance_reported": 0.011287,
@@ -364,6 +371,7 @@
    "employee_pension_contributions": 0.233878,
    "employer_pension_contributions": 0.295463,
    "employment_income": 0.478282,
+   "employment_sector": 0.458405,
    "employment_status": 0.989284,
    "esa_contrib_reported": 0.005514,
    "esa_income_reported": 0.009507,
@@ -441,6 +449,7 @@
    "savings_interest_income": 0.335736,
    "sda_reported": 0.000104,
    "self_employment_income": 0.056123,
+   "sic_industry_division": 0.733091,
    "ssmg_reported": 0.000415,
    "state_pension_reported": 0.235745,
    "statutory_maternity_pay": 0.002506,
@@ -482,6 +491,7 @@
    "attends_private_school_random_draw": 1.0,
    "brma": 1.0,
    "bsp_reported": 0.005877,
+   "bus_fare_spending": 0.204941,
    "bus_subsidy_spending": 0.208829,
    "capital_gains": 0.231216,
    "carers_allowance_reported": 0.011287,
@@ -511,6 +521,7 @@
    "employee_pension_contributions": 0.233878,
    "employer_pension_contributions": 0.295463,
    "employment_income": 0.478282,
+   "employment_sector": 1.0,
    "employment_status": 1.0,
    "esa_contrib_reported": 0.005514,
    "esa_income_reported": 0.009507,
@@ -588,6 +599,7 @@
    "savings_interest_income": 0.335736,
    "sda_reported": 0.000104,
    "self_employment_income": 0.056123,
+   "sic_industry_division": 0.733091,
    "ssmg_reported": 0.000415,
    "state_pension_reported": 0.235745,
    "statutory_maternity_pay": 0.002506,
@@ -615,8 +627,8 @@
    "would_claim_universal_childcare": 0.560923,
    "would_evade_tv_licence_fee": 0.105666
   },
-  "reference_columns_evaluated": 145,
-  "signal_columns": 145,
+  "reference_columns_evaluated": 148,
+  "signal_columns": 148,
   "source": {
    "filename": "populace_uk_2023.h5",
    "hf_commit": "a75a9a831d6b07aaffbd09713f2a1124f5c0f08f",

packages/microcosm-build/src/microcosm/build/uk/efrs_parity_reference.json

@@ -1,7 +1,7 @@
 {
  "description": "Frozen enhanced-FRS parity reference for the UK release input-coverage contract. Every PolicyEngine-UK loader variable the pinned artifact populates is recorded as its unweighted owning-entity nonzero share. This includes formula-owned persisted overrides because the UK Simulation loader passes every engine-known H5 column to set_input; pipeline scratch columns, structural IDs, and all-zero loader layers are not requirements.",
  "engine": {
-  "engine_known_persisted_variable_count": 866,
+  "engine_known_persisted_variable_count": 881,
   "formula_owned_persisted_overrides_included": [
    "current_education",
    "is_benunit_head",
@@ -22,7 +22,7 @@
    "employee_pension_contributions": "employee_pension_contributions_reported",
    "employment_income": "employment_income_before_lsr"
   },
-  "input_variable_count": 223,
+  "input_variable_count": 227,
   "input_variable_scope": "Every CountryTaxBenefitSystem variable persisted in the H5: Simulation.build_from_multi_year_dataset calls set_input for all engine-known columns, including formula-owned overrides",
   "package": "policyengine-uk",
   "structural_columns_excluded": [
@@ -33,10 +33,8 @@
    "household_id"
   ],
   "unknown_export_columns_excluded": [
-   "bus_fare_spending",
    "clone_index",
    "constituency_code_oa",
-   "employment_sector",
    "esa_health_condition_proxy",
    "esa_support_group_proxy",
    "free_school_breakfasts",
@@ -53,11 +51,10 @@
    "region_code_oa",
    "salary_sacrifice_asked",
    "salary_sacrifice_reported",
-   "sic_industry_division",
    "source_household_id",
    "source_year"
   ],
-  "version": "2.89.0",
+  "version": "2.92.0",
   "zero_share_input_columns_excluded": [
    "disabled_students_allowance_eligible_expenses",
    "incapacity_benefit_reported",
@@ -73,14 +70,14 @@
   },
   "household": {
    "export_columns": 66,
-   "input_columns": 51,
-   "populated_input_columns": 51,
+   "input_columns": 52,
+   "populated_input_columns": 52,
    "records": 52846
   },
   "person": {
    "export_columns": 99,
-   "input_columns": 87,
-   "populated_input_columns": 84,
+   "input_columns": 89,
+   "populated_input_columns": 86,
    "records": 113617
   }
  },
@@ -98,6 +95,7 @@
   "attends_private_school_random_draw": "person",
   "brma": "household",
   "bsp_reported": "person",
+  "bus_fare_spending": "household",
   "bus_subsidy_spending": "household",
   "capital_gains": "person",
   "carers_allowance_reported": "person",
@@ -127,6 +125,7 @@
   "employee_pension_contributions": "person",
   "employer_pension_contributions": "person",
   "employment_income": "person",
+  "employment_sector": "person",
   "employment_status": "person",
   "esa_contrib_reported": "person",
   "esa_income_reported": "person",
@@ -204,6 +203,7 @@
   "savings_interest_income": "person",
   "sda_reported": "person",
   "self_employment_income": "person",
+  "sic_industry_division": "person",
   "ssmg_reported": "person",
   "state_pension_reported": "person",
   "statutory_maternity_pay": "person",
@@ -245,6 +245,7 @@
   "attends_private_school_random_draw": 1.0,
   "brma": 1.0,
   "bsp_reported": 0.004929,
+  "bus_fare_spending": 0.202872,
   "bus_subsidy_spending": 0.316675,
   "capital_gains": 0.23375,
   "carers_allowance_reported": 0.010535,
@@ -274,6 +275,7 @@
   "employee_pension_contributions": 0.273489,
   "employer_pension_contributions": 0.314865,
   "employment_income": 0.492928,
+  "employment_sector": 1.0,
   "employment_status": 1.0,
   "esa_contrib_reported": 0.008053,
   "esa_income_reported": 0.009558,
@@ -351,6 +353,7 @@
   "savings_interest_income": 0.424963,
   "sda_reported": 8.8e-05,
   "self_employment_income": 0.054939,
+  "sic_industry_division": 0.734371,
   "ssmg_reported": 0.000405,
   "state_pension_reported": 0.245694,
   "statutory_maternity_pay": 0.002139,

packages/microcosm-build/src/microcosm/build/uk/gates.json

@@ -70,6 +70,7 @@
           "num_vehicles",
           "other_residential_property_value",
           "owned_land",
+          "private_pension_wealth",
           "property_wealth",
           "savings",
           "stocks_and_shares_isa",
@@ -85,6 +86,7 @@
           "num_vehicles": 0,
           "other_residential_property_value": 0,
           "owned_land": 0,
+          "private_pension_wealth": 0,
           "property_wealth": 0,
           "savings": 0,
           "stocks_and_shares_isa": 0,
@@ -100,6 +102,7 @@
           "num_vehicles": 0,
           "other_residential_property_value": 0,
           "owned_land": 0,
+          "private_pension_wealth": 0,
           "property_wealth": 0,
           "savings": 0,
           "stocks_and_shares_isa": 0,
@@ -526,6 +529,7 @@
           "household.msoa_code",
           "household.num_vehicles",
           "household.oa_code",
+          "household.private_pension_wealth",
           "household.property_purchased",
           "household.rail_usage",
           "household.region_code_oa",
@@ -662,7 +666,7 @@
         "max_top_share": 0.9994670564654868,
         "min_nonzero_records": 104
       },
-      "notes": "Weighted tail-concentration audit of the QRF-imputed output columns derived from the HMRC source manifest. Armed from the #686 L3 baselines per #757 B4, each threshold recording the run that measured it: the 47-column qrf_tail grids of uk_weighted_integrity_baselines_686.json (sha256 8b3f6c4e9c522445bf3e05d3451ee8557f4e14d25524ceac3af8bfe05401a146, licensed acceptance dir 686-spine-swap), measured on spine-a.h5 (sha256 a65f2132736d1dcecb91709a513a0d54a5887635ea03760629cc888d188ee306, 113,649 person rows, design weights) for the #609/#578 threshold adjudication. top_k 100 is the measurement grid anchor [10, 100, 500, 1000]; max_top_share is the exact measured maximum over the checked surface (hmrc_spi_other_social_security_income, 104 carriers), no headroom; min_nonzero_records is the thinnest measured column above the grid anchor (the same column) - the policy domain requires min_nonzero_records > top_k, so the three saturated sub-anchor columns (hmrc_spi_taxable_termination_pay 12, charitable_investment_gifts 22, sda_reported 24 carriers, each top-100 share 1.0) sit below it and go thin visibly in the signed details on every run, never a silent pass. The baselines are design-weight measurements and the terminal gate runs on the calibrated release frame; a calibrated-weight breach names its column and is a finding, not noise. The 12 household_qrf_tail grids (was_wealth surface) are measured in the same baselines file and not yet armed - a household-surface gate entry is a declared follow-up. Reviewed exclusions live in the named register resource of this package."
+      "notes": "Weighted tail-concentration audit of the QRF-imputed output columns derived from the HMRC source manifest. Armed from the #686 L3 baselines per #757 B4, each threshold recording the run that measured it: the 47-column qrf_tail grids of uk_weighted_integrity_baselines_686.json (sha256 8b3f6c4e9c522445bf3e05d3451ee8557f4e14d25524ceac3af8bfe05401a146, licensed acceptance dir 686-spine-swap), measured on spine-a.h5 (sha256 a65f2132736d1dcecb91709a513a0d54a5887635ea03760629cc888d188ee306, 113,649 person rows, design weights) for the #609/#578 threshold adjudication. top_k 100 is the measurement grid anchor [10, 100, 500, 1000]; max_top_share is the exact measured maximum over the checked surface (hmrc_spi_other_social_security_income, 104 carriers), no headroom; min_nonzero_records is the thinnest measured column above the grid anchor (the same column) - the policy domain requires min_nonzero_records > top_k, so the three saturated sub-anchor columns (hmrc_spi_taxable_termination_pay 12, charitable_investment_gifts 22, sda_reported 24 carriers, each top-100 share 1.0) sit below it and go thin visibly in the signed details on every run, never a silent pass. The baselines are design-weight measurements and the terminal gate runs on the calibrated release frame; a calibrated-weight breach names its column and is a finding, not noise. The 12 household_qrf_tail grids (was_wealth surface) are measured in the same baselines file and not yet armed - a household-surface gate entry is a declared follow-up. The #750 split adds a 13th household QRF column (private_pension_wealth) not yet in the measured baselines; the household-surface arming follow-up on microcosm#796 re-measures. Reviewed exclusions live in the named register resource of this package."
     }
   ]
 }