Skip to content

Preserve rare-tail carriers and persist realized regimes in receipts #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MaxGhenis wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Preserve rare-tail carriers and persist realized regimes in receipts #769

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a5c7310
docs: initialize rare signed-tail lane journal
MaxGhenis Aug 21, 2026
5c44b40
Fix rare signed-tail support and regime receipts
MaxGhenis Aug 23, 2026
0fa4802
Record tails 1% no-regression evidence
MaxGhenis Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 75 additions & 105 deletions FINAL_REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,111 +1,81 @@
# Final report: microcosm #462 register alignment
# Final report: battery blocker lane — rare signed tails

## Outcome

Completed the split-PR remediation on `loss-contract-alignment`, based on
`origin/main` at `7b6e10b`. The change is now register alignment only: one
shared critical-target register, one shared congressional-district classifier,
two consumers, builder contract-row gating, and behavioral containment of the
publish contract.

The critical-row loss multiplier was removed entirely per
[microcosm#492](https://github.com/PolicyEngine/microcosm/issues/492). There is no
constant, CLI option, validation, loss overlay, telemetry, diagnostics/scorer
provenance, or historical replay pin left. `_fiscal_target_loss_weights` is
source-identical to `origin/main`, and its output therefore preserves main's
bit-level behavior for the same registry and family multipliers.

## Sol round-1 findings

1. **Table 1.4 selector parity:** removed the builder-only
`accepted_name_prefixes=("irs_soi.",)` constraint. The adapter now has
exactly the shared requirement's substring and suffix selectors. The
outside-prefix reproduction is builder-rejected.
2. **Congressional-district parity:** added exported, stdlib-only
`is_congressional_district_target(name, metadata)` and made the publisher
and builder classifiers thin wrappers. It ORs layout dimension, source-id
token, geography level, geography scope, truthy CD GEOID, and name token.
The builder's exact/semantic, Table 1.4, and zero-support paths now see the
same registry metadata.
3. **Recorded relative-error shape:** a matched row with missing/`None`
`relative_error` now fails with the publish-contract message instead of
silently passing after recomputation. Existing non-numeric and stale-value
checks remain.
4. **Behavioral anti-drift:** the load-bearing test now runs adversarial rows
through both consumers for exact-name, family+role, Table pattern,
missing/non-finite values, and a disallowed incumbent escape at the 0.25
hard stop. A production Ledger compile supplies six separate CD evidence
rows; builder and publisher exclude identical six-name sets and counts.
Field comparisons remain as fast checks, and any added conjunctive prefix
is proven to trip the guard.

The [#490](https://github.com/PolicyEngine/microcosm/issues/490) medical 0.25
adjudication tolerance and its adjacent comment in `us_critical_targets.py`
remain byte-for-byte unchanged, as required.

## Reproduction receipts

The Table 1.4 prefix reproduction now returns:

```text
SOI Table 1.4 national dollar fit failed: other.table_1_4.all.bad_amount@2024: relative_error=1 exceeds 0.25 for SOI Pub 1304 Table 1.4 national dollar rows (soi_table_1_4_national_dollar_rows); target=100.0, final_estimate=200.0.
```

The missing-relative-error reproduction now returns:

```text
SOI Table 1.4 national dollar fit failed: irs_soi.ty2023.table_1_4.all.adversarial_amount@2024: missing recorded relative_error; the publish contract requires a numeric value.
```

The CD reproduction has the owner-mandated exclusion result:

```text
builder_excluded=True
publisher_excluded=True
builder_failures=[]
```

Calling that row "rejected" would contradict the required OR-union exclusion
semantics. The two malformed critical rows are rejected; the CD row is
symmetrically excluded by both consumers.
The lane is complete locally. All 48 frozen red QED checks were classified
from frozen donor support, the one proven generating-code defect was fixed,
and exact realized-regime provenance is now durable. The branch was not pushed
and no pool build or logbook-chain operation was started.

The full adjudicated report is
`experiments/battery_rare_signed_tails/REPORT.md`; the machine-readable
48-check matrix is `experiments/battery_rare_signed_tails/realized_regimes.json`.

## Findings and changes

- All 48 QED checks recompute to gated fits: 35
`zero_inflated_positive` and 13 `three_sign` at check level (35/7 across 42
targets). None is degenerate or single-sign, so regime substitution or
sign-conditional draw logic is not an honest fix. The QRF regime machinery
is at `packages/microcosm-fit/src/microcosm/fit/qrf.py:92-150,950-1003,
1333-1380` (authority/main numbering).
- Transfer ownership is 17 early gap-fill checks and 31 late
producer-complement checks. Five intact sparse donors remain evidence
blockers: ordinals 16, 28, 33, 46, and 75. Their verified sign-carrier
counts are +18, +61, +27, -89, and -48; all exceed the five-carrier QED
support floor (`stacked_spine.py:3027-3031`).
- Ordinals 78/80/82 and the Keogh leg share upstream retirement-support
deletion. The old uniform 5,000-row cap retained 102/2,057 401k carriers,
4/161 403b, 2/61 SEP, and 0/2 Keogh. The fixed helper retains the union of
all four targets' nonzero rows, samples only common zeros, and calibrates
only sampled-zero weights
(`retirement_distributions.py:337-450,486-584`). The cap and seed values did
not move.
- Realized QRF regimes now survive ordinary and banked transfer
(`acs_transfer.py:1348-1418,1813-1823`), target-bank persistence
(`acs_transfer_bank.py:25,47-53,249,357`), and exact early/late/aggregate
receipts (`stacked_spine.py:3754-3772,8215-8360,9260-9268,9315-9331`).
Authentic canonical JSON with sorted regime-map keys resumes correctly;
missing target keys still fail closed.

## Keogh disposition

Keogh is not structurally absent. Native ASEC contains positive values 2,040
and 30,000; the old cap dropped both before transfer fitting, and all 1,736,840
finite frozen bank draws are zero. ACS absence was therefore manufactured on
the transfer donor path and is fixed by preserving source support.

The declared-absence route is rejected. If the signal had been structurally
impossible, it would require an exact recipient-scope absence equation
(`stacked_spine.py:8168-8195`) or a canonical tolerated-absence receipt
(`stacked_spine.py:6491-6528`), plus clone-exact battery structural receipts
(`stacked_spine.py:7186-7205`). Approval belongs to the US pool owner through
the owner-only reviewed route; this lane added no exclusion.

## Verification

The requested suite ran with `UV_NO_SYNC=1` to use the already-synced workspace
environment in the network-restricted sandbox:

```text
uv run --package microcosm-build --extra us --group dev python -m pytest packages/microcosm-data/tests packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py packages/microcosm-build/tests/test_us_state_files_scorer.py -q
264 passed, 3 skipped (267 collected)
```

Additional receipts:

- Complete `test_gates.py`: passed.
- Required multiplier grep: zero Python hits.
- Ruff check: clean on all ten touched Python files.
- Ruff format check: clean on the eight non-exempt touched Python files; the
two historical experiment files were not reformatted, as instructed.
- `git diff --check`: clean.
- The medical adjudication block compares byte-for-byte equal to pre-fix
commit `068854d`.
- Pytest emitted non-failing macOS temporary-directory cleanup warnings; no
test failed.

## Remediation commits

- `5077f95` — start microcosm#462 Sol remediation progress.
- `c48ba37` — remove the microcosm#462 loss multiplier per microcosm#492.
- `afa910a` — fix Sol finding 1 selector parity.
- `89f74f4` — fix Sol finding 2 CD classifier parity.
- `77040fb` — fix Sol finding 3 relative-error shape.
- `bad7145` — fix Sol finding 4 behavioral containment.
- `3c96514` — apply the finding-2 classifier's required Ruff formatting.

Nothing was pushed at the time of this report; the branch was subsequently
pushed and merged as #491 (2026-07-22).

The sandbox rejected writing
`/Users/maxghenis/PolicyEngine/_reviews/sol-491-fix-out.md` with `Operation not
permitted`; the full completion report is therefore committed here and will be
printed to stdout as the requested fallback.
- calibrate: 201 passed.
- data: 275 passed, 1 skipped.
- fit: 93 passed.
- frame: 294 passed, 36 skipped.
- build: 5,973 passed, 39 skipped.
- repository Ruff: all checks passed.
- US bundle generator check: passed, bundle SHA `5b0014c3…9554`.
- frozen regime-evidence generator check: passed.

The existing 1% before artifact projects to 127 failure lines over 93 legs;
47/48 frozen QED reds are visible. The committed baseline projection reproduced
byte-for-byte, and its self-diff is empty. No actual after-build exists for
this branch because the binding headless instruction forbade this lane from
starting one. The host-queue owner must run
`experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh` and feed its
gates file to `diff_1pct_failures.py`. Keogh itself requires a full-scale build
because a 1% ASEC sample almost surely contains neither native carrier.

## Remaining owner work

Forty intact-support shape failures remain blocked on held-out calibration or
mapping evidence, and five sparse-donor failures remain blocked on denser
evidence. REPORT §2 gives the smallest target-scoped generating-mechanism
change for each of the nine classes. None calls for a gate, band, ceiling,
floor, fold, seed, cap-value, or exclusion change.
123 changes: 84 additions & 39 deletions PROGRESS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,93 @@
# Progress
# Rare signed-tail battery lane progress

## State

Microcosm #516 whole-row donor outlier screen is complete on
`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515
interim carve merged as #525). The `puf_tax_detail` donor now drops tax units
whose grouped raw mortgage interest reaches $10M before the #515 carve
(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T
of phantom mortgage-interest mass), with the checkpoint schema bumped to v3
so post-carve pre-screen checkpoints rebuild.
Complete on `battery-rare-signed-tails`, based on `2c7a7218`. The mechanism
fix, frozen-evidence classification, receipt closure, regression coverage,
full per-package test suite, Ruff check, journal, and final reports are in the
solution commit containing this file. No pool build was started by this lane.

No gate, incidence band, QED ceiling, support floor, training-cap value, fold,
seed value, exclusion register, logbook chain, or pending-chain file changed.
The host queue still owns the off-chain 1% after-build. The frozen 1% before
projection and deterministic diff tooling are committed, but an actual after
artifact does not exist for this branch.

Authoritative inputs are the arm-split lane's
`experiments/battery_burndown/ADJUDICATION.md` and `adjudication.json`.
Outputs are:

- `FINAL_REPORT.md` — requested completion output;
- `experiments/battery_rare_signed_tails/REPORT.md` — full code-cited report;
- `experiments/battery_rare_signed_tails/realized_regimes.json` — all 48
check classifications and smallest honest remedies;
- `experiments/battery_rare_signed_tails/baseline_1pct_projection.json` —
frozen 1% before side.

## Done

- Confirmed a clean starting worktree at `aef1c56`.
- Read the repository guidance and established the #515 donor carve as the
screen's required downstream boundary.
- Started source-level audits of every donor-frame consumer, checkpoint
validation, row-count pins, and existing donor-fact summaries.
- Attempted the requested GitNexus impact workflow; the managed filesystem
denied its global registry write. Its local index also exposed a broad
`build/` ignore mismatch, so the completed impact audit uses direct source
call sites and tests.
- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the
structural rationale and pinned-artifact receipts.
- Added a whole-row screen on grouped raw person `home_mortgage_interest`
after tax-unit assembly, before the #515 carve, with retained-index reset.
- Confirmed no downstream consumer pairs donor rows to the original HDF arrays
or carries a stale donor-length vector; values and weights always originate
from the same screened frame.
- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale
checkpoint regression track the live constant while retaining literal-v1
corruptions.
- Added regression coverage for the exact grouped boundary, whole-row removal,
retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant.
- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets
139; microcosm-data 138 with 1 skip. The directly affected tail-bound suite
adds 12 passes. Ruff format/check and `git diff --check` are clean.
- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line
audit, expected 208,611-row real-artifact effect, verification results, count
sweep, and deliberately untouched surfaces.
- Initialized and committed the lane journal before product work
(`a5c7310c`). Inspected all four salvage snapshots (`6597ca87`, `8131e7af`,
`930a3de1`, and newest `bfe794ad`); the newest snapshot supplied the
overlapping working state, while older broad/unrelated material was not
recovered. Exact disposition is in `_LANE-NOTES.md`.
- Completed `uv sync --all-packages --extra us` using the writable lane cache.
- Read the comparator-mechanisms authority in full and recomputed every frozen
fit-boundary regime: 48/48 checks are gated (35
`zero_inflated_positive`, 13 `three_sign`; 42 targets: 35/7), with no
degenerate or single-sign QED target. Verified all 42 banked-draw sign
surfaces against those regimes.
- Classified the 48 checks into nine mechanism classes across fitter regime,
early/late transfer ownership, and donor support. Five checks (16/28/33/46/75)
remain additional-evidence starvation blockers; 40 remain owner-calibration
blockers; three retirement checks (78/80/82) share the fixed support-deletion
mechanism.
- Proved Keogh is not structurally absent: native ASEC has two positive
carriers (`2,040`, `30,000`), the old uniform 5,000-row cap retained 0/2,
and all 1,736,840 finite frozen bank draws are zero. Implemented a
support-preserving carrier-union cap with common-zero weight calibration;
cap and seed values remain unchanged.
- Persisted exact realized QRF regimes through ordinary/banked transfer,
early/late/aggregate receipts, H5 identity, and fail-closed checkpoint
validation. Advanced only the semantic receipt/materializer identities
needed to invalidate regime-free or support-deleting artifacts.
- Fixed canonical-JSON resume validation so sorted regime-map keys do not make
an authentic checkpoint appear corrupt; missing target keys still fail
closed. Added the serialization regression.
- Regenerated and checked all source-attested bundle/coverage identities.
`generate_us_bundle_from_constants.py --check` reports US bundle
`5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554`;
coverage is 41,381/41,381 fields and 40/40 inventory checks.
- Reproduced the frozen baseline projection byte-for-byte: 127 failure lines,
93 legs, and 47/48 red QED checks visible at 1%. A baseline-vs-baseline diff
is exactly empty. Keogh and ordinal 16 are too sparse to be visible at 1%.

## Suite receipts

Final source tree, one pytest process per package shard:

- `microcosm-calibrate`: 201 passed; 2 warnings; 36.25s.
- `microcosm-data`: 275 passed, 1 skipped; 37.50s.
- `microcosm-fit`: 93 passed; 1 warning; 249.72s.
- `microcosm-frame`: 294 passed, 36 skipped; 1 warning; 888.52s.
- `microcosm-build`: 5,973 passed, 39 skipped; 1,941 warnings; 7,858.53s.
- `uv run ruff check .`: all checks passed.
- `tools/generate_us_bundle_from_constants.py --check`: passed.
- `experiments/battery_rare_signed_tails/build_realized_regime_evidence.py
--check`: passed.

The first complete build-shard run exposed one stale derived-payload test pin;
the authoritative generator was byte-identical, the structural schedule hash
was unchanged, the test pin was corrected to the live derived hash, its focused
node passed, and the complete build shard above was rerun green.

## Next

- PR #527 review cycle, then merge. After both #525 and #527: rebuild the
base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a
run that holds per `us_critical_targets.py`.
- Root record-level ETL carve stays open on microcosm#515.
- Host-queue owner: run the committed 1% off-chain command, then compare its
`pool.gates.json` with the frozen baseline using `diff_1pct_failures.py`.
- Owner-scheduled full-scale build: verify the Keogh gate-level flip, which a
1% sample cannot observe.
- Owners of the seven blocked shape families: supply held-out/dense evidence
before the target-scoped remedies in REPORT §2 are implemented.
- Owners of the five intact-but-sparse donors: supply a dense rung or approve
a target-specific sparse-tail model. No exclusion is proposed.
Loading