Skip to content

Bind stacked-pool CD-vintage provenance for the release contract #763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MaxGhenis merged 3 commits into from
Aug 25, 2026
Merged

Bind stacked-pool CD-vintage provenance for the release contract #763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5925f80
Start CD vintage provenance lane
MaxGhenis Aug 24, 2026
19854a9
Bind stacked pool CD vintage provenance
MaxGhenis Aug 24, 2026
7207d57
Skip the pandas fixed-format inspector fixture when PyTables is absent
MaxGhenis Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 123 additions & 112 deletions FINAL_REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,119 +1,130 @@
# Final report: PolicyEngine-US 1.819.0 lock bump
# Final report: stacked-pool to release CD-vintage provenance

## Outcome

The branch is ready for the owner to open the PR. `uv.lock` now resolves
`policyengine-us==1.819.0` and `policyengine-core==3.31.0`; the complete
resolver version movement is exactly:

```text
policyengine-core 3.26.11 -> 3.31.0
policyengine-us 1.764.6 -> 1.819.0
```

NumPy remains 2.4.6 and Torch remains 2.12.0. The lock binds the official
PE-US 1.819.0 wheel SHA-256
`525bdf8b238c3eb11cd60c5f4f7a7b0c57bc7eea5c1cf4346c261241b061be45`
(`uv.lock:1366-1421`). The required upgraded
`uv sync --all-packages --extra us` completed for all five workspace packages
from that lock and task-local official PyPI artifacts.

## Compatibility repairs

- WIC's upstream input is now `takes_up_wic_if_eligible`; Microcosm writes
that verified successor while retaining the historical draw salt, and the
six current cross-entity consumers are frozen with direct-receiver and
aggregation ownership
(`packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py:106-109,392-409`;
`packages/microcosm-frame/src/microcosm/frame/adapters/_policyengine_us_source_index.py:1694-1731,2004-2090`;
`packages/microcosm-build/tests/test_us_pool_input_consumers.py:235-359`).
- Reported TANF now writes the verified input successor `receives_tanf`, since
PE-US owns `is_tanf_enrolled` as a formula
(`packages/microcosm-build/src/microcosm/build/us_runtime/cps_carried.py:127-160,266-289,485-500`).
- New CA, CO, and NM premium-assistance take-up leaves remain explicit
`engine_default`/`rate_unsourced` inputs because Microcosm has no reviewed
participation source; no rate was invented
(`packages/microcosm-build/src/microcosm/build/us/spec/take_up.yaml:507-563`).
- Mortgage-interest, Oklahoma pension, SNAP proration/missing-hours, and the
expanded AL/NY/OK/PA `weeks_unemployed` consumers were updated or guarded
against the installed graph
(`packages/microcosm-build/tests/test_us_pool_input_consumers.py:491-575`;
`packages/microcosm-build/tests/test_us_child_support.py:448-489`;
`packages/microcosm-build/tests/test_us_weeks_unemployed.py:695-751`).
- The published US dataset remains certified for the prior engine lock.
Certified loading still fails closed before construction on model/core
mismatch; only the live smoke test recognizes that specific typed mismatch
as an expected skip under this new development lock
(`packages/microcosm-data/src/microcosm/data/loader.py:63-64,425-445`;
`packages/microcosm-data/tests/test_loader.py:326-356,403-419`).

No pool-consumed upstream variable was genuinely removed without a successor,
so the owner-question stop condition did not trigger. The full installed-input
audit and mechanism-by-mechanism code citations are in `_LANE-NOTES.md` under
“verified upstream compatibility repairs.”

## Identities and compatibility note

Repository generators refreshed every affected raw-resource, generated-source,
engine-ABI, remaining-input, field-usage, inventory, seed-protocol, authority,
spec, coverage, and golden identity. Source bytes are part of the attested seed
protocol, so those module edits legitimately move dependent identities even
though NumPy and Torch did not move
(`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:320-367`;
`packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py:114-152`).
The final US spec is
`3189d90dec95c8ea7090e41b5283fa52b1e6855bed4a776dfa02820f2bd11c62`.

`_LANE-NOTES.md`, under “final tool-generated identities,” records all 46 final
values by mechanism. The compatibility commit body records every full old-to-
new digest mapping, including the four lock artifact hashes.

The requested short release-range note is in `_LANE-NOTES.md` under
“compatibility note for 1.764.6 through 1.819.0.” It flags receipt/take-up and
SNAP changes, OBBBA follow-through, major cash/health/housing and tax formula
changes, school-meal child-support treatment, and newly added state programs.
It is intentionally an owner-facing plausibility scan, not an exhaustive
policy audit.
The producer/consumer defect is fixed. The production stacked route now
produces the same authenticated congressional-district contract that release
preflight requires: the canonical crosswalk SHA, target vintage
`119th_congress`, and positive household
`congressional_district_geoid` support. Geography remains outside source
assembly, and the release guard still fails closed on every mismatch
(`packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py:346-406`;
`tools/build_us_fiscal_refresh_release.py:2565-2613,8608-8612`).

## Five-part repair

1. The stacked CLI requires path/SHA pairs for the national PUMA ladder and
canonical 117th-to-119th CD crosswalk. Both declared hashes must equal the
repository pins before the source bytes are authenticated
(`tools/build_us_multispine_pool.py:563-594,897-1004`). The real geography
operator runs immediately after operator-free source assembly and before
gap fill or cloning
(`packages/microcosm-build/src/microcosm/build/us_runtime/operator_boundary.py:346-406`;
`tools/build_us_multispine_pool.py:5290-5334`).

2. The configured namespace, checkpoint identity, persisted stage receipts,
and terminal manifest bind both byte authorities, source and target
vintages, assignment declaration, overlap algorithm, operator order, seed
site/stream/value, and ordered native-household output receipt
(`tools/build_us_multispine_pool.py:814-928,1300-1469,1711-1930,3530-3635,3989-4027,4133-4215`).

3. Nullable H5 publication accepts validated root attributes, writes them in
the same temporary HDFStore as the fixed entity tables, verifies their
exact round trip, and atomically replaces the destination only after all
checks pass. The stacked publisher supplies the crosswalk-SHA and target-
vintage attrs
(`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:1463-1699`;
`tools/build_us_multispine_pool.py:4423-4519`). Schema-9 loading also binds
those physical attrs and the live household geography/clone lineage back to
the manifest
(`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:671-923,1032-1193,1321-1364`).

4. Release preflight now reads root attrs and the household frame from one
HDFStore snapshot through the shared fixed/table-aware reader. Its existing
equality checks for crosswalk SHA, current target vintage, and positive
household district support are unchanged
(`tools/build_us_fiscal_refresh_release.py:2565-2677`).

5. A tiny fixture invokes the real stacked entry point, post-assembly
geography operator, fixed-H5 publisher, and real release assertion. It
proves both root attrs and positive household district support; consumer
tests reject changed attrs, changed native geography, missing lineage, and
divergent clone geography
(`packages/microcosm-build/tests/test_us_multispine_pool_tool.py:2279-2335`;
`packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py:1357-1440`;
`packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py:1061-1184`).

## Assignment design and anti-rot

A one-value deterministic PUMA-to-CD lookup is not defensible because 2020
PUMAs and congressional districts do not nest. The existing national ladder
preserves observed ACS PUMA, assigns missing ASEC PUMA proportional to 2020
PUMA population, then draws CD and county within PUMA proportional to block-
population overlap. Stable row/state/PUMA order and the ledgered
`legacy_puma_ladder` / `geography_legacy` / seed `0` contract make the draw
reproducible for fixed inputs
(`packages/microcosm-build/src/microcosm/build/us_runtime/puma_ladder.py:1-20,293-383,638-698`;
`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:870-887`;
`packages/microcosm-build/src/microcosm/build/us/spec/spine.yaml:421-432`).

The spec and schemas now declare the crosswalk source/vintage authority and
geography assignment, while field-usage and inventory owners cover the new
surface exactly
(`packages/microcosm-build/src/microcosm/build/us/spec/geography.yaml:3-37`;
`packages/microcosm-build/src/microcosm/build/us/spec/sources.yaml:69-86`;
`packages/microcosm-build/src/microcosm/build/us/spec/vintages.yaml:45-53`;
`packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py:387-393,671-692,803-822`;
`packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py:348-380,1649-1712`).

The generated US spec SHA is
`5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e`.
Coverage is exact at 42,120/42,120 fields, 49 claims, and 41/41 inventory
checks. Principal legitimate movements are pointer inventory
`6d7353c6...` to `bc4a948a...`, full checkpoint `b6a47fac...` to
`a128a85f...`, and the new geography identity `f49425ca...`; executable
versions move checkpoint 11 to 12, terminal manifest 8 to 9, and nullable H5
materializer 2 to 3
(`tools/spec_engine_coverage.py:42-45`;
`packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py:348-380`;
`docs/evidence/spec-engine/us-f0-coverage.json:10,775-788,1890,2027-2057,2602`;
`tools/build_us_multispine_pool.py:332-360`;
`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:75-105`).

## Verification

- `microcosm-calibrate`: 203 passed; peak RSS 462,896 KiB.
- `microcosm-data`: 318 passed / 2 skipped; peak RSS 769,408 KiB.
- `microcosm-fit`: 93 passed; peak RSS 872,432 KiB.
- `microcosm-frame`: 295 passed / 36 skipped; peak RSS 6,904,032 KiB.
- `microcosm-build`: exact complete inventory, 6,304 passed / 39 skipped. A
canonical one-process run was green but retained 18,548,960 KiB and was not
accepted under the 15 GiB ceiling. The exact 6,341 collected items were
proven as disjoint 4,161-item and 2,180-item serial fresh-process partitions:
4,127 passed / 36 skipped at 12,596,384 KiB, then 2,177 passed / 3 skipped at
13,363,984 KiB. This follows the repository's documented fresh-process shard
rationale (`.github/workflows/test.yml:24-34`) and changes no test assertion
or model behavior.

Accepted total: **7,213 passed / 77 skipped / 0 failed**, with every accepted
process below 15 GiB RSS.

Final generated checks are green:

- release-input manifest: 163 required, 7 reviewed exclusions, 41 reform
probes;
- target parity: 32 compiled, 52 reviewed exclusions;
- `tools/generate_us_bundle_from_constants.py --check`: US spec SHA above;
- `tools/spec_engine_coverage.py --check`: 42,096/42,096 configuration fields
and 40/40 inventory checks. Both check paths reject stale bytes rather than
using a tolerance
(`tools/generate_us_bundle_from_constants.py:355-367,410-437`;
- `microcosm-calibrate`: 203 passed.
- `microcosm-data`: 318 passed / 2 skipped.
- `microcosm-fit`: 93 passed.
- `microcosm-frame`: 295 passed / 36 skipped.
- `microcosm-build` partition A: 4,155 passed / 36 skipped.
- `microcosm-build` partition B: 2,177 passed / 3 skipped.
- Accepted workspace total: **7,241 passed / 77 skipped / 0 failed**.
- The high-risk real integration and H5 consumer cases are included in those
build partitions
(`packages/microcosm-build/tests/test_us_multispine_pool_tool.py:2279-2335`;
`packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py:1357-1440`).
- Bundle freshness reports the spec SHA above; coverage freshness reports
42,120/42,120 fields and 41/41 inventory checks
(`tools/generate_us_bundle_from_constants.py:355-437`;
`tools/spec_engine_coverage.py:378-398`).

Repository-wide `ruff check .` and `git diff --check` are green. No gate,
threshold, tolerance, or band was tuned. No pool or release build ran, no PR
was opened, and nothing was pushed.

## Commits and handoff

- `514964d4` — start the PolicyEngine-US 1.819.0 bump lane and commit the
standing progress journal.
- The compatibility commit containing this report — lock bump, verified
repairs, generated identities, complete digest mapping, and green receipts.

Next: the owner opens the PR from `bump-policyengine-us`.
- Repository-wide Ruff, smoke-script shell syntax, and `git diff --check` pass.
The source-blind import-graph anti-rot pin now covers the exact 69-module
graph reached by the three new shared validators
(`packages/microcosm-build/tests/test_us_spine_blindness.py:3270-3315`).

No production pool or release build ran, no push occurred, and
`logbook-pending-chain.txt` was not touched.

## Candidate handoff

Stage 1 must add the canonical PUMA-ladder and CD-crosswalk path/SHA pairs
listed exactly at the end of `_LANE-NOTES.md`, run off-chain, and rebuild under
the new checkpoint identity. The old smoke checkpoints and pool publication
are not reusable: configured identity now includes both new input pins and the
geography contract, checkpoint materializer is 12, terminal manifest schema is
9, and nullable H5 materializer is 3
(`tools/build_us_multispine_pool.py:332-360,1300-1555`;
`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:75-105`).
The six pre-existing immutable source artifacts can be reused; the Stage-1
checkpoint namespace and pool H5 must be regenerated with the two additional
authenticated authorities (`tools/build_us_multispine_pool.py:931-1004,1292-1319`).
99 changes: 53 additions & 46 deletions PROGRESS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,56 +1,63 @@
# Progress: PolicyEngine-US 1.819.0 lock bump
# Progress: stacked-pool to release CD-vintage provenance

## State

Complete and ready for the owner to open the PR. The lock resolves
`policyengine-us==1.819.0` and `policyengine-core==3.31.0`; all required
compatibility repairs, generated contracts, attested identities, package test
shards, generator checks, and Ruff are green. The code-cited mechanism audit,
identity inventory, compatibility note, and validation receipts are in
`_LANE-NOTES.md`.
Implementation and validation are complete. The stacked
producer now authenticates and applies household geography after source
assembly, carries its authority through checkpoint and publication identities,
publishes verified CD-vintage H5 attributes, and reaches the unchanged release
guard through the shared fixed/table-aware reader
(`tools/build_us_multispine_pool.py:814-928,1300-1469,1857-1930,5290-5334`;
`packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py:1032-1193,1463-1699`;
`tools/build_us_fiscal_refresh_release.py:2565-2677`).

No pool-consumed variable was removed without a verified successor, so no
owner question is pending. No pool/release build, push, gate change, threshold
change, tolerance change, or band change occurred.
No pool/release build, push, guard weakening, operator-boundary weakening, or
`logbook-pending-chain.txt` access has occurred.

## Done

- Started and committed this standing progress log at `514964d4` after
rebasing the lane start onto current `origin/main`.
- Ran the ordered initial sync attempt, documented the sandbox cache/network
limits, and completed the unchanged-lock and upgraded-lock all-package US
syncs from exact official artifacts in a task-local cache.
- Ran `uv lock --upgrade-package policyengine-us`. The complete version
movement is exactly `policyengine-core 3.26.11 -> 3.31.0` and
`policyengine-us 1.764.6 -> 1.819.0`; NumPy 2.4.6 and Torch 2.12.0 did not
move.
- Verified upstream variable reality in the installed PE-US 1.819.0 package,
adapted Microcosm's input ownership and consumer guards, and retained
fail-closed certified-dataset version checks. Each mechanism is cited in
`_LANE-NOTES.md` under “verified upstream compatibility repairs.”
- Regenerated the release-input, parity, source/take-up, engine-ABI, spec,
seed, coverage, and golden identities with repository tools. The final
46-value inventory is in `_LANE-NOTES.md`; the final commit body carries the
exhaustive old-to-new mapping.
- Added the requested owner-facing PE-US 1.764.6-to-1.819.0 compatibility note
from the installed release changelog, including major SNAP, receipt,
OBBBA-follow-through, cash/health/housing, tax, and new-program changes.
- Passed every package test: 7,213 passed / 77 skipped / 0 failed. Calibrate,
data, fit, and frame ran as individual package shards. The build inventory
was proven as a complete disjoint 4,161-item + 2,180-item partition after
the otherwise-green single process retained more than the binding memory
ceiling; accepted peaks were 12,596,384 and 13,363,984 KiB, both below 15
GiB. This uses the repository's fresh-process shard rationale
(`.github/workflows/test.yml:24-34`) without changing any assertion or model
behavior.
- Reproduced 163 required release inputs / 7 reviewed exclusions / 41 reform
probes; 32 compiled parity targets / 52 reviewed exclusions; US spec SHA-256
`3189d90dec95c8ea7090e41b5283fa52b1e6855bed4a776dfa02820f2bd11c62`;
and 42,096/42,096 configuration fields plus 40/40 inventory checks.
- Passed repository-wide Ruff and `git diff --check`; wrote the final handoff
to `FINAL_REPORT.md`.
- Reconciled the worktree with salvage commit `ca26ea21`: its tracked tree was
already exactly present, so the best salvage was retained and audited rather
than reimplemented.
- Completed the required all-package US environment sync from the exact lock
using the writable offline uv cache after the managed sandbox refused the
default cache.
- Added the two required pinned geography authority pairs, the ledgered seeded
PUMA-overlap assignment, schema-12 checkpoint identity, assignment receipts,
schema-9 terminal manifest validation, and schema-3 H5 materializer binding.
- Added atomic nullable-H5 root-attribute write/verification and authenticated
manifest-to-H5 geography/clone-lineage validation.
- Made release preflight read root attributes and fixed household frames in one
`HDFStore` handle while retaining all existing SHA, target-vintage, and
positive-support checks.
- Added a real tiny stacked-pool publication to release-preflight integration
test plus negative attr, digest, missing-lineage, and divergent-clone tests
(`packages/microcosm-build/tests/test_us_multispine_pool_tool.py:2279-2335`;
`packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py:1357-1440`).
- Regenerated and checked the US anti-rot chain: spec SHA
`5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e`,
42,120/42,120 configuration fields, 49 claims, and 41/41 inventory checks.
- Passed the complete accepted workspace suite: 7,241 passed, 77 skipped, and
0 failed. The two memory-bounded build partitions were 4,155 passed / 36
skipped and 2,177 passed / 3 skipped; calibrate, data, fit, and frame account
for the remaining 909 passed / 38 skipped.
- Passed repository-wide Ruff, bundle freshness, coverage freshness, smoke
script syntax, and `git diff --check`.
- Recorded the exact candidate Stage-1 path/SHA additions and the checkpoint
invalidation verdict in `_LANE-NOTES.md` and `FINAL_REPORT.md`.

## Next

The owner opens the PR. No push or additional build is required from this
lane.
1. Candidate Stage 1 adds both authenticated geography authority pairs and
rebuilds under checkpoint materializer 12.
2. Candidate Stage 2 consumes the new schema-9 manifest/materializer-3 H5 and
passes the unchanged release preflight.
3. Do not reuse pre-fix Stage-1 checkpoints or pool publications; immutable
source artifacts remain reusable.

## Historical prior lane

The PolicyEngine-US 1.819.0 lock-bump journal previously in this file is
historical: that lane merged into `origin/main` at `7b90bb18` on 2026-08-24.
Its final state remains available at commit `05d254aa` and its detailed
receipts remain in the historical section of `_LANE-NOTES.md`.
Loading
Loading