diff --git a/FINAL_REPORT.md b/FINAL_REPORT.md index df045ae4..297b9e63 100644 --- a/FINAL_REPORT.md +++ b/FINAL_REPORT.md @@ -1,255 +1,295 @@ -# Final report: pkg3 post-transfer receipt validation failure #2 +# Final report: battery residual FIX packages ## Outcome -Fixed at executable commit `a932974f`. - -The supplied weeks-unemployed failure was a receipt-generation bug, not an -invalid carrier model and not a count-target exception. Candidate capacity and -prefix selection reduced the same ordered weights through different float64 -paths, so the prefix exceeded its declared candidate mass by one ULP. The -required cross-target audit then exposed the same class of bug one level up: -the first repair composed maximum capacity from independently rounded -partition endpoints, putting both child-support maxima 42 ULP above the whole -recipient mass. - -The complete fix changes both generating relationships: - -- one immutable `_PrefixSchedule` now supplies each candidate endpoint and all - prefix-selection evidence (`post_transfer_calibration.py:282-287,471-515,844-872,891-928`); -- maximum capacity is generated from the row union - `fixed_positive | allowed_positive | zero_candidates`, zero-masked onto the - recipient-weight vector with identical length, order, and reduction topology - to `recipient_total` (`post_transfer_calibration.py:823-885`). - -The exact validator inequalities are unchanged: maximum must not exceed its -recipient superset, and a reported prefix must not exceed its candidate set -(`post_transfer_calibration.py:1457-1475,1499-1529`). No tolerance, threshold, -band, gate, comparator, seed, fold, target, or carrier constraint changed. - -Both SHA-pinned no-build harnesses now validate. All five package test roots, -repository-wide Ruff, touched-file formatting, and whitespace checks are -green. This lane ran no host build and made no push. The serial host owner owns -the next 1% rerun against `a932974f`. - -## No-build checkpoint reproduction - -The read-only stage root was: - -```text -/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/pkg3/pool.checkpoints/stacked/8f5077d6a1d5440b241f22fe4d20ad1d889924a27d094cb669e1035f9306546b +The only new generating repair is committed at `eaba1eab`: late +ASEC-to-ACS `weeks_unemployed` calibration no longer treats positive +unemployment compensation as a carrier prerequisite. That relationship is +specific to the PUF QRF leg, not the direct ASEC `LKWEEKS` source leg +(`weeks_unemployed.py:791-830,833-840,959-983,1266-1276`). + +The SHA-pinned no-build replay predicts that this repair changes the weeks +incidence ratio from `0.031371146` to `1.0006685424`, while retaining integer +positive donor support, QED `0`, and immutable bytes. The serial host owner's +guarded 1% rerun remains required to observe the new pool result. + +The other owned checks did not justify another local change: + +- both assigned adult-care expense mechanisms ran under pkg3 and its incidence + check is green; +- unemployment-compensation positive-amount QED is already exactly `0`; +- Schedule D's assigned joint-parent incidence is green; and +- SSI was traced past take-up and eligibility to upstream countable-income + distributions, for which this lane owns no calibrated surface. + +No pool build, push, gate/band/ceiling/fold/seed tuning, exclusion change, +certification, publication, or release mutation occurred in this lane. + +## Frozen evidence and residual universe + +- Baseline gate SHA-256: + `1d6059868680f872fe04d452a536bcc3c215bafabb4c50d7740a469fe6a8b56a`. +- Pkg3 gate SHA-256: + `3ace0af0fd9e2ed6cb37cb110280f0c5cade182118c62737635c7ad177050ac3`. +- Mirror-deduplicated physical failures: `127 -> 114`. +- Pkg3's terminal and agreement gate objects are JSON-equivalent, so the 114 + count uses one canonical failure array rather than double-counting mirrors. + +The 114-check list was recomputed before choosing a change. The named residual +slice, exact artifact values, and ownership decisions are recorded in +`_LANE-NOTES.md`; neither gate artifact was modified. + +## Per-check disposition + +### Adult-care post-reconciliation: both assigned mechanisms resolved + +Mechanism: person-grain transfer output can create an expense on a +nonqualifying person or multiple carriers in a tax unit. The section-21 mask +distinguishes qualifying dependents and married heads/spouses; reconciliation +clears only mutable invalid/duplicate carriers, and an immutable positive +blocks additions within its unit (`acs_transfer.py:660-739`). + +Existing pkg3 fix: before reconciliation, the late owner limits mutable +carriers to qualifying rows and additions to one stable candidate per empty +tax unit (`stacked_spine.py:8698-8716,8930-8987`). The shared amount owner +maps only mutable positive recipients from positive ASEC donor support and +leaves immutable values untouched (`post_transfer_calibration.py:573-720`). The owner +then requires reconciliation to be byte-identical and records +`verified_no_op` (`stacked_spine.py:9019-9041`). + +Regression: route/exclusivity and immutable-blocking coverage is in +`test_us_acs_transfer.py:2263-2333,2513-2574`; donor-support mapping and +immutable-byte coverage is in `test_us_post_transfer_calibration.py:354-436`; terminal live +validation is in `test_us_multispine_pool_tool.py:2302-2341`. + +Expected and observed effect: the assigned positive-incidence ratio is green +at `1.024210809`. The pkg3 execution receipt records 27 mapped rows and amount +QED `1.738865343 -> 0`; the independent battery reports insufficient QED +support because its ASEC leg has only four positive rows. No residual adult- +care change is warranted. The separately red incapacity flag is not either of +the two assigned expense checks. + +### Model-required targeted calibration: unemployment amount QED + +Mechanism and existing fix: unemployment compensation is declared +`preserve_recipient`, so carrier membership is frozen while the shared amount +owner rewrites only mutable positive values from positive ASEC donor support +(`post_transfer_calibration.py:190-197,573-720,930-940,963-993`). + +Regression: the preserve-mode test requires identical carriers, all five exact +quantile anchors, QED `0`, donor support, and the preserve-carriers invariant +(`test_us_post_transfer_calibration.py:354-390`). + +Expected and observed effect: the assigned positive-amount QED is already +`0.0`; no code change was made. Its `0.131859569` incidence ratio is a distinct, +unassigned criterion and remains intentionally unchanged. + +### Schedule D joint-parent reconciliation + +Mechanism and existing fix: Schedule D distributions are not independently +fitted. They are the packaged share of positive +`long_term_capital_gains_before_response` only when the mutually exclusive +`non_sch_d_capital_gains` route is absent +(`acs_transfer.py:611-657`). At pool grain, both parents are first summed by +tax unit; the derived value is placed on the first missing person and remaining +missing members receive zero, while observed rows are preserved +(`multispine_pool.py:2868-2946`). + +Regression: the derivation tests require the share identity, route +exclusivity, and complete parents (`test_us_acs_transfer.py:2461-2510`). + +Expected and observed effect: the assigned positive-incidence ratio is green +at `1.092661185`. The residual Schedule D QED `0.352941176` follows still-red +PUF-tax parent amount shapes; independently remapping this child would break +its deterministic parent identity and cross into the PUF-tax lane. No local +change was made. + +### SSI incidence residual + +Mechanism: SSI take-up does not explain the difference. The pool owner fills +unresolved take-up inputs from the installed engine default and records that +provenance (`multispine_pool.py:2970-3091`); the contract declares the SSI +default `true` (`take_up_contract.json:121-145`). The pkg3 receipt records all +80,395 rows as defaulted, with no seeded or preserved rows. + +The installed formula floors and caps `uncapped_ssi`, then multiplies by +take-up (`policyengine_us/variables/gov/ssa/ssi/ssi.py:13-40`). Eligibility is +the aged/blind/disabled, resource, and immigration conjunction +(`policyengine_us/variables/gov/ssa/ssi/is_ssi_eligible.py:10-18`). The income +test subtracts `ssi_countable_income` from the eligible amount +(`policyengine_us/variables/gov/ssa/ssi/uncapped_ssi.py:11-16`). Countable +income sends earned, unearned, parentally deemed, and in-kind support through +the exclusions, then adds spousal deemed income afterward +(`policyengine_us/variables/gov/ssa/ssi/eligibility/income/ssi_countable_income.py:28-87`; +`policyengine_us/variables/gov/ssa/ssi/eligibility/income/_apply_ssi_exclusions.py:21-43`). + +The read-only pkg3 decomposition exactly reproduces the `1.3354825449451269` +incidence ratio. Weighted eligibility is slightly lower on ACS +(`0.1888374135`) than ASEC (`0.1970392720`), but conditional +`uncapped_ssi > 0` is higher on ACS (`0.1440777296` versus `0.1033936553`). +Eligible-person countable-income q10 is correspondingly lower on ACS +(`6,884.259765625` versus `8,880.0`). The generating divergence is therefore +upstream income shape, not take-up, eligibility, or an SSI-local carrier +selector. + +Fix/regression/effect: this repository keeps SSI formula-owned, materializes it +only on an ephemeral gate view, and rejects persistence +(`multispine_pool.py:3094-3111`). No frozen adjudication row grants this lane a +mutable surface over the upstream incomes. Changing the default or formula +would substitute an unauthorized mechanism, so no SSI code or local-fix +regression was added; the expected local effect is exactly zero. + +### Weeks-unemployed incidence residual + +Mechanism: ASEC `LKWEEKS` is directly validated as integer `-1` or `0..52`, +with `-1` mapped to zero; it is independent of unemployment compensation +(`weeks_unemployed.py:791-830`). UC-based zeroing is inside the PUF-only QRF +postprocessing (`weeks_unemployed.py:833-840,959-983`), and the signal audit +checks that relationship only on PUF rows +(`weeks_unemployed.py:1266-1276,1333-1337`). Pkg3 had incorrectly promoted +that restriction into later ASEC-to-ACS calibration +(`33bf52fe:post_transfer_calibration.py:122-180,237-244`; +`33bf52fe:stacked_spine.py:8863-8876`). Only 32 of 34,293 mutable ACS rows had +positive UC, so carrier capacity saturated and the ACS weeks share collapsed +to the UC share, producing ratio `0.031371146` +(`reproduce_us_post_transfer_weeks_checkpoint.py:28-62,132-230,278-320`). + +Fix: weeks is now ordinary late `match_reference` with no special constraint +(`post_transfer_calibration.py:122-178,190-240`). The stacked owner consequently +uses every transferred-null then nonnull ACS clone-0 row as mutable, allowed, +and addition-candidate support (`stacked_spine.py:8930-8987`). Carrier +selection still targets the weighted ASEC share, and positive values still map +only from positive ASEC donors while all nonmutable bytes remain protected +(`post_transfer_calibration.py:573-720,782-817,819-943,963-993`). Only the +authored policy content hash changes +(`post_transfer_calibration.py:297-344`; +`test_imputation_lineage_spec.py:103-105`); no schema, comparator, or gate +changes. + +Regression: `test_weeks_late_calibration_does_not_require_unemployment_carriers` +sets every UC value to zero and still requires positive integer donor-supported +ACS weeks, default mutable masks, exact reference share, no capacity limit, +and immutable-byte preservation (`test_us_stacked_spine.py:6367-6438`). Policy +identity coverage requires weeks' `special_constraint` to remain `none` +(`test_us_post_transfer_calibration.py:85-104`). + +Expected effect from the SHA-pinned no-build replay: + +- allowed/addition rows: `32 -> 34,293`; +- positive ACS rows: `32 -> 2,174`; +- incidence ratio: `0.0313711455 -> 1.0006685424`; +- positive support: integer ASEC donor values `1..46`, with zero violations; +- capacity-limited: `false`; +- QED: remains `0`; and +- immutable bytes: preserved. + +The replay pins all assembled, UC, and weeks checkpoint digests and asserts +those exact current-scope outcomes +(`reproduce_us_post_transfer_weeks_checkpoint.py:1-6,28-62,91-129,132-320,323-378`). +It performs no fit, late-producer execution, artifact write, or pool build. + +```bash +UV_CACHE_DIR=/private/tmp/microcosm-residual-uv-cache \ + uv run --no-sync python tools/reproduce_us_post_transfer_weeks_checkpoint.py \ + --checkpoint-stage-root /Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/pkg3/pool.checkpoints/stacked/8f5077d6a1d5440b241f22fe4d20ad1d889924a27d094cb669e1035f9306546b \ + --carrier-scope current-mutable \ + --expect valid ``` -The current no-build checks are: - -```text -uv run python tools/reproduce_us_post_transfer_weeks_checkpoint.py \ - --checkpoint-stage-root --expect valid -uv run python tools/audit_us_post_transfer_child_support_checkpoints.py \ - --checkpoint-stage-root --expect valid -``` - -### Weeks-unemployed failure - -`tools/reproduce_us_post_transfer_weeks_checkpoint.py` validates the assembled -Frame plus unemployment-compensation and weeks target file, identity, and raw- -draw hashes. It reconstructs only the native clone-0 vectors and calls the live -kernel and strict validator; it performs no fit, DAG execution, artifact write, -or build (`reproduce_us_post_transfer_weeks_checkpoint.py:1-6,27-58,76-143,146-254`). - -Pinned artifacts: - -| Artifact | File SHA-256 | Identity SHA-256 | Raw-draw SHA-256 | -| --- | --- | --- | --- | -| `assembled.checkpoint.h5` | `5ce1815fc44dc43c7c24ccf27526852b8f1bddbdfe371255410a22f9b56ac015` | whole file pinned | n/a | -| `019__unemployment_compensation.h5` | `dc6637936ed4bd0322d38eaa3a4920fd137565f314387db3b3fdc7dfd6bc3086` | `708722093ca610426175998d50bbb6663585b07ffef912899f17adc90520f51f` | `e32d1559668e10b24abad8e1d639e4dbade964a712925bfe8f56d3136b839840` | -| `000__weeks_unemployed.h5` | `898397733aa3e5d8ec7d6679cb16a0504e826e25d23ca2c788f4397e0e061a43` | `d0d554ba05045e39a07f0f9515c83bbf754f067df12b8247f4bf3866162c4bdd` | `0214c8dcbc118676336069b906a07ee6145f2178542b6c5b4fb5899ad62d09f3` | - -The production owner selects ASEC clone-0 reference rows, ACS clone-0 -recipient rows, transferred nonnull mutable cells, and positive-UC mutable rows -for both the weeks allowed and addition masks -(`stacked_spine.py:8960-8971,8995-9032`). The pinned replay contains: - -- 38,604 native person rows; -- 4,311 reference and 34,293 recipient/mutable rows; -- 134 positive reference rows; -- 24 initial recipient positives, all disallowed; and -- 32 positive-UC addition candidates - (`reproduce_us_post_transfer_weeks_checkpoint.py:146-193`). +## Verification -At reproduction commit `4cc41652`, the harness with `--expect invalid` exits -zero only for the exact supplied failure. Candidate capacity is -`85,676.23791782455`; the ID-ordered upper prefix is -`85,676.23791782456`; the excess is `1.4551915228366852e-11`. The sole false -relationship is: +The required `uv sync --all-packages --extra us` was attempted first. The +sandbox blocks the default cache and network resolution, so verification used +the complete lock-compatible pkg3 environment with its five editable package +links repointed to this worktree. -```text -upper_prefix_mass <= addition_candidate_mass -``` - -It raises exactly: +On the exact implementation tree committed as `eaba1eab`, each package root +exited zero: ```text -ValueError: Frame post-transfer calibration person/source_operator_weeks_unemployed/weeks_unemployed: match-reference carrier capacity relationships are invalid. +uv run --no-sync pytest -q packages/microcosm-frame/tests +uv run --no-sync pytest -q packages/microcosm-fit/tests +uv run --no-sync pytest -q packages/microcosm-calibrate/tests +uv run --no-sync pytest -q packages/microcosm-data/tests +uv run --no-sync pytest -q packages/microcosm-build/tests +uv run --no-sync ruff check . +git diff --check ``` -The harness pins that predicate, both floats, and the complete error rather -than accepting any aggregate validation failure -(`reproduce_us_post_transfer_weeks_checkpoint.py:210-231,257-286`). - -Against `a932974f`, `--expect valid` reports candidate and upper prefix both -`85,676.23791782456`, zero prefix/candidate delta, no failed relationships, -and a valid receipt. The attainable-union maximum under the recipient -reduction topology is `85,676.23791782453`. - -### Child-support cross-target reproduction - -`tools/audit_us_post_transfer_child_support_checkpoints.py` reconstructs the -same native clone-0 support and half weights for both child-support targets. -It pins each target's whole-file, identity, and raw-draw hashes, then requires -both receipts—not merely one—to match the requested exact red or green state -(`audit_us_post_transfer_child_support_checkpoints.py:1-80,91-207,210-302`). - -| Artifact | File SHA-256 | Identity SHA-256 | Raw-draw SHA-256 | -| --- | --- | --- | --- | -| `000__child_support_expense.h5` | `d119075e19fb767f3d8d24c7c0149d0df1ed963774a4b93d96974a72b3ac9bfe` | `41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab` | `8b2845aff0aa0695d98ae30828523bf6bca9c5d4ed5d2d91d2d1a636bb917600` | -| `001__child_support_received.h5` | `66120896d5793f3d737f9ffac2058e2196992e357f8d869f4b31b259d041b3aa` | `41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab` | `ea7f2eebb430b654acc639ef6ee6ed482207ffd74d54ba3a47cb55056813a381` | - -Against the incomplete candidate-only repair `d7b12bab`, both receipts fail -only: - -```text -maximum_attainable_mass <= recipient_total +The complete build shard reached 100% with one expected skip and only the +established warning set. The current-scope checkpoint replay also exited zero +and matched every pinned value above. The final documentation-only tree was +rerun through the same five package roots, Ruff, and whitespace checks before +its commit. + +## Serial-host 1% rerun handoff + +Pkg3's `assembled` stage is semantically before late calibration and the +`transferred` checkpoint is after it +(`multispine_pool.py:200-201`; `stacked_spine.py:10009-10037`; +`tools/build_us_multispine_pool.py:3121-3175`). That assembled stage is not +mechanically reusable through the current CLI. The complete post-transfer +policy is content-hashed into stacked authority and configured checkpoint +identity (`post_transfer_calibration.py:297-344`; +`stacked_spine.py:2463-2492`; +`tools/build_us_multispine_pool.py:1150-1178,4198-4227`). The parser exposes no +audited stage-import override, and loading requires current manifest identity +before selecting a valid stage (`tools/build_us_multispine_pool.py:406-532,1181-1250,1407-1428,1716-1937`). + +The serial host owner should therefore queue this exact cold 1% build in a +fresh namespace under the existing one-build-at-a-time and `<15 GiB RSS` +guard. Do not copy, edit, rebind, or reuse pkg3 manifests. Unsetting the +predecessor variable prevents ambient logbook state from entering the build +(`tools/build_us_multispine_pool.py:3971-3979`). This lane did not run this +command. + +```bash +WT=/Users/maxghenis/PolicyEngine/_worktrees/microcosm-residual-fixes +OUT=/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/residual-fixes +DATA_REPO=/Users/maxghenis/PolicyEngine/policyengine-"us"-data +DATA_PACKAGE=policyengine_"us"_data + +mkdir -p "$OUT" +cd "$WT" || exit 1 + +env -u POPULACE_LOGBOOK_PREV_ROW_DIGEST \ + PATH="/Users/maxghenis/.local/bin:/usr/bin:/bin:/usr/sbin:/sbin" \ + HOME=/Users/maxghenis \ + PYTHONUNBUFFERED=1 \ + "$WT/.venv/bin/python" tools/build_us_multispine_pool.py \ + --sample-fraction 0.01 \ + --sample-seed 578 \ + --clone-attachment-fraction 1.0 \ + --clone-attachment-seed 578 \ + --asec-raw-stage-h5 /Users/maxghenis/PolicyEngine/_buildo-runtime/out/591-pawtyp-pool/asec-producer-checkpoints/asec_raw_stage.checkpoint.h5 \ + --asec-raw-stage-h5-sha256 51e9fafcd6f16140018fa90c7afbeb6d79008bfc8c122e437d23a399b30553fe \ + --acs-household-zip /Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0/csv_hus.zip \ + --acs-household-zip-sha256 8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0 \ + --acs-person-zip /Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894/csv_pus.zip \ + --acs-person-zip-sha256 afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894 \ + --acs-rent-h5 "$DATA_REPO/$DATA_PACKAGE/storage/acs_2022.h5" \ + --acs-rent-h5-sha256 0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4 \ + --puf-h5 "$DATA_REPO/$DATA_PACKAGE/storage/puf_2024.h5" \ + --puf-h5-sha256 7669f5b5281f20080e77204f9bd4aabfad0aa101fa283e22caf9ba8d61d4d6df \ + --puf-source-year-csv "$DATA_REPO/$DATA_PACKAGE/storage/puf_2015.csv" \ + --puf-source-year-csv-sha256 0a7fd643edb1acc55c507db795914b41d232922be78c149b58d111f4672499df \ + --checkpoint-root "$OUT/pool.checkpoints" \ + --out "$OUT/pool.h5" >>"$OUT/build.log" 2>&1 + +rc=$? +echo "residual-fixes exit: $rc" >>"$OUT/build.log" +exit "$rc" ``` -For each target, recipient total is `79,926,522.10879111`; the partition- -composed maximum is `79,926,522.10879174`; the excess is -`6.258487701416016e-07`, or 42 ULP. The underlying endpoints are: - -| Target | Allowed-positive mass | Addition-candidate mass | Old composed maximum | -| --- | ---: | ---: | ---: | -| `child_support_expense` | `71,696.09739141785` | `79,854,826.01140033` | `79,926,522.10879174` | -| `child_support_received` | `180,209.75664861224` | `79,746,312.35214312` | `79,926,522.10879174` | - -Against `a932974f`, both strict receipts validate. Their attainable-union -maximum equals recipient total exactly, `79,926,522.10879111`, while the -independently rounded diagnostic partition sum remains -`79,926,522.10879174`. This demonstrates that the generating set relationship, -not the validator, was repaired. - -## Root cause and semantic decision - -Float64 addition is order-sensitive. The original code used a masked -`ndarray.sum` for candidate capacity and a separately ordered `np.cumsum` for -prefix selection. The first repair correctly unified those two values, but it -then added independently rounded fixed, existing-positive, and zero-candidate -endpoints to describe a different claim: the maximum mass of their row union. - -Both strict invariants are semantically correct: - -1. a prefix cannot exceed the candidate set from which it was selected; and -2. an attainable subset cannot exceed its recipient superset. - -The final maximum implementation retains the recipient vector's length and -order and replaces unattainable entries with zero. With finite nonnegative -weights, each attainable leaf is less than or equal to its corresponding -recipient leaf, and the identical reduction topology preserves that ordering -through every floating-point addition (`post_transfer_calibration.py:775-783,823-885`). -It does not use `min`, `nextafter`, a tolerance, or a post-hoc clamp. - -The regressions lock both numerical mechanisms for every late -`match_reference` declaration: - -- the exact 32 production weeks weights distinguish masked sum from ordered - prefix by one ULP; -- a four-weight case makes independently rounded capacity partitions exceed - their whole set by one ULP; and -- a constrained proper-subset case makes a compressed subset sum - `0x1.433526fbe1946p+48`, or `0.0625`, greater than its superset - `0x1.433526fbe1945p+48`; the same-topology union validates exactly - (`test_us_post_transfer_calibration.py:544-753`). - -Weeks remains a valid positive-carrier calibration target. Its source accepts -only integer `-1` or `0..52` and maps `-1` to zero; its QRF path rounds, clips, -positive-UC-gates, and revalidates `0..52`; its carrier event is `weeks > 0` -(`weeks_unemployed.py:791-800,911-983,1218-1222`). Post-transfer amount mapping -uses only positive reference-donor support (`post_transfer_calibration.py:577-626,690-705`). -Count-valued support therefore does not invalidate weighted carrier capacity. - -The ACS runtime's explicit discrete-numeric set contains only two mortgage-year -targets; other numeric targets use the ordinary numeric encoding -(`acs_transfer.py:129-138,3035-3117`). QRF's at-most-32-value “near-discrete” -branch is a leaf-storage optimization, not a carrier semantic type -(`microcosm-fit/qrf.py:388-401,482-503`). Annual child-support and disability -amounts also entered that optimization in the host log, which independently -rules out treating it as a weeks-specific count exception. - -## Seven-target late-transfer audit - -The immutable registry declares seven late targets. Six use -`match_reference`; disability benefits uses `preserve_recipient` and never -emits capacity or prefix evidence -(`post_transfer_calibration.py:208-258,840-932,1319-1335`). - -| Late target | Evidence and verdict | -| --- | --- | -| `child_support_expense` | Covered. Nonnegative annual `CHSP_VAL`, not a count (`child_support.py:166-201,369-383`). Its pinned checkpoint fails the old whole-capacity relationship and validates the final union mechanism. | -| `child_support_received` | Covered. Nonnegative annual `CSP_VAL`, not a count (`child_support.py:166-201,369-383`). Its pinned checkpoint has the same red/green proof. | -| `disability_benefits` | Inapplicable to this capacity bug. It is a nonnegative annual two-slot amount excluding workers' compensation (`disability_benefits.py:184-220,382-395`) and uses `preserve_recipient`; its inspected checkpoint keeps before/after carrier mass at `42,658.57948297383` with `capacity=None` and `selection=None`, as required by the preserve-mode receipt branch (`post_transfer_calibration.py:1319-1335`). | -| `weeks_unemployed` | Covered. Sole semantic count target, integer `0..52`, with carrier additions constrained to positive-UC mutable rows (`weeks_unemployed.py:791-800,911-983,1218-1222`; `stacked_spine.py:8995-9008`). Exact pinned red/green replay proves reducer order caused the failure. | -| `workers_compensation` | Covered. Nonnegative annual `WC_VAL`, not a count (`workers_compensation.py:143-184,337-355`). It uses the default mutable carrier/addition masks (`post_transfer_calibration.py:786-812`) and the shared six-spec regressions. | -| `spm_unit_energy_subsidy` | Covered. Nonnegative measured `SPM_ENGVAL`, checked within unit and reduced to SPM-unit float64 (`energy_subsidy.py:157-233,537-557`). Its entity grain changes the weights, not the set/reduction mechanism; the shared regressions cover its declaration. | -| `pre_subsidy_care_expenses` | Covered. Nonnegative monetary care expense. ACS reconciliation restricts carriers to qualifying people and at most one per tax unit; the late owner admits one stable zero candidate per empty unit (`acs_transfer.py:660-739,1277-1299`; `stacked_spine.py:8728-8746,8977-8986`). The proper-subset six-spec regression covers this constrained structure. | - -Current zero-based late-DAG positions are child support 24, disability 25, -weeks 30, workers' compensation 31, energy subsidy 32, and adult care 34. -Registry scheduling and stacked execution are deterministic, and each group -calibrates before returning (`us_late_producer_registry.py:1338-1396,2013-2019`; -`stacked_spine.py:10054-10095,10927-10931`). The failed host run produced child, -disability, and weeks checkpoints but stopped before workers, energy, and adult -care. Verdicts for those later targets are therefore source/mask proofs plus -shared-kernel regressions, not claims of nonexistent checkpoint replay. - -## Verification +## Commit lineage -On the exact tree committed as `a932974f`, these commands ran serially under -the owner-provided memory guard and exited zero: - -```text -uv run pytest packages/microcosm-fit/tests -q -uv run pytest packages/microcosm-calibrate/tests -q -uv run pytest packages/microcosm-data/tests -q -uv run pytest packages/microcosm-frame/tests -q -uv run pytest packages/microcosm-build/tests -q -uv run ruff check . -uv run ruff format --check -git diff --check -``` +- `f150b6da` — opened and committed the residual lane journal; +- `eaba1eab` — committed the weeks generating repair, regressions, SHA-pinned + replay, complete scoped adjudication, exact host handoff, and green suite + record; and +- this documentation-only closeout commit — publishes this final report and + closes `PROGRESS.md` without changing the tested executable tree. -The focused post-transfer file passed all 47 cases. The complete build root -also covered stacked-spine, late-DAG, multispine pool, H5, pool-tool, terminal -receipt, and owner-mask paths. Only established skips and warnings appeared. - -Both current checkpoint commands exited zero with `--expect valid`. Detached -temporary worktrees proved the hardened red side: weeks against `4cc41652` and -both child receipts against `d7b12bab`. The temporary worktrees were removed. - -No host build ran from this lane. The read-only host log and checkpoint tree -were not modified. The owner-provided untracked `.codex-memory-guard.py` and -`_BUILD-FAILURE-1PCT.txt` remain unchanged. Nothing was pushed. - -## Commit lineage and handoff - -- `b533bc61` — open and commit the progress journal; -- `4cc41652` — add the SHA-pinned weeks reproduction and red regression; -- `d7b12bab` — bind candidate capacity to its ordered selection schedule; -- `47742720` — record the cross-target child-support escalation; -- `a932974f` — generate maximum capacity from the attainable row union, add - the child red/green harness and exact proper-subset regressions, complete the - seven-target audit, and record the green suite; and -- the following documentation-only commit — close `PROGRESS.md` and publish - this report without changing the tested executable tree. - -The next authorized action is the serial host owner's 1% rerun at executable -commit `a932974f`. Certification, publication, and release-chain mutation stay -outside this lane. +The next authorized action is the serial host owner's guarded 1% rerun above. +Certification, publication, pushing, and release-chain mutation remain outside +this lane. diff --git a/PROGRESS.md b/PROGRESS.md index bfa60723..d1d96331 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,4 +1,95 @@ -# Battery package 3 progress +# Battery residual-fixes progress + +## Residual FIX packages — 2026-08-22 22:45Z + +### State + +Complete. Implementation and complete five-shard verification are committed on branch +`battery-residual-fixes`, stacked on `battery-pkg3-two-part`. The only new +generating change removes a PUF-only unemployment-compensation condition from +the late ASEC-to-ACS `weeks_unemployed` carrier calibration. The other named +checks were either already green under pkg3, outside their adjudicated +criterion, or traced to an upstream owner rather than changed locally. +`FINAL_REPORT.md` contains the per-check disposition and exact serial-host +handoff. The same five package suites, repository-wide Ruff, and whitespace +checks pass on this documentation-only closeout tree. No pool build, +gate/band/ceiling/fold/seed tuning, exclusion change, push, certification, or +publication has occurred. Earlier sections of this journal are historical. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. + The graph query/context tools are unavailable in this session, so direct + source, call-site, history, and checkpoint tracing is the documented + fallback. +- Confirmed the tracked worktree was clean at `bb94f789` and that + `FINAL_REPORT.md` is the established final-report output file. +- Attempted the required `uv sync --all-packages --extra us` before any + repository edit. The default cache is outside the sandbox; a writable-cache + retry then failed because network access is disabled. Copied the complete, + lock-compatible pkg3 environment, attempted an offline reconciliation, and + repointed its five editable workspace packages to this worktree. Import + checks resolve every `microcosm.*` shard here and `quantile-forest==1.4.2`. +- Preserved the owner-only build boundary: the baseline and pkg3 gate files + will be read only, and no pool command has run. +- Ran `uv run --no-sync pytest` separately for the final startup tree's + `microcosm-frame`, `microcosm-fit`, `microcosm-calibrate`, + `microcosm-data`, and `microcosm-build` test roots; every command exited + zero with only established skips and warnings. Repository-wide + `uv run --no-sync ruff check .` and `git diff --check` also pass. +- Recomputed the mirror-deduplicated battery universe directly from the + SHA-pinned gate artifacts: baseline has 127 unique failures and pkg3 has 114; + pkg3's `terminal_gates` and `agreement_gate` are byte-equivalent JSON. +- Confirmed the two adjudicated adult-care expense checks reached green + incidence (`1.024210809`) and insufficient 1% QED support after pkg3's + qualifying mutable-carrier selection and mutable-positive amount map. The + separately red incapacity flag is not one of those two expense checks. +- Confirmed unemployment-compensation's assigned positive-amount QED is + exactly `0.0`; its sparse carrier-incidence failure is a distinct unassigned + criterion. Confirmed Schedule D's assigned joint-parent incidence is green + at `1.092661186`; its new QED failure follows still-red PUF-tax parent amount + shapes and remains with that lane. +- Diagnosed SSI without changing a gate or formula: take-up is `true` on every + pkg3 row, eligibility is slightly lower on ACS, and the excess appears only + after the upstream countable-income test. No SSI-local calibrated surface is + authorized by the adjudication. +- Removed the late weeks/UC coupling from the closed policy, production owner, + receipt validators, fixtures, and authored lineage identity. Added an owner- + level regression proving zero-UC ACS recipients remain valid mutable weeks + carriers while integer donor support and immutable bytes are preserved. +- Extended the SHA-pinned pkg3 checkpoint replay with historical and current + carrier scopes. The current code admits all 34,293 mutable ACS rows rather + than only 32 positive-UC rows, produces 2,174 positive rows on integer donor + support `1..46`, is not capacity-limited, validates exactly, and predicts an + incidence ratio of `1.000668542` rather than `0.031371146`. +- Ran the focused lineage, kernel, stacked-owner, H5, and publication-validator + matrix after the fix; all selected tests and touched-file Ruff checks pass. +- Ran `uv run --no-sync pytest -q` separately for the complete implementation + tree's `microcosm-frame`, `microcosm-fit`, `microcosm-calibrate`, + `microcosm-data`, and `microcosm-build` test roots. Every shard exited zero; + the build shard reached 100% with one expected skip and only established + warnings. Repository-wide `uv run --no-sync ruff check .` and + `git diff --check` also pass. +- Audited host checkpoint reuse. Although pkg3's `assembled` stage is the + semantic pre-change cut, the changed policy is hashed into its checkpoint + identity and the CLI has no safe stage-import override. `_LANE-NOTES.md` + therefore records an exact cold 1% command in a fresh residual namespace; + copying or rebinding the pkg3 manifest is explicitly forbidden. +- Committed the generating repair, regressions, pinned replay, adjudication, + handoff, and green executable verification as `eaba1eab`. +- Replaced the stale pkg3 report with the residual lane's complete per-check + `FINAL_REPORT.md` and preserved the live-tree-safe split host path variables. +- Reran all five package test roots, repository-wide Ruff, the live-tree + retired-data-package guard, and whitespace checks on the final documentation + tree; every command exited zero with only established skips and warnings. + +### Next + +1. The serial host owner queues the exact guarded 1% rerun recorded in + `FINAL_REPORT.md` and `_LANE-NOTES.md`; this lane does not run it. +2. The owner recomputes the mirror-deduplicated residual gates and adjudicates + that new host evidence. Certification and publication remain outside this + lane. ## Main/F0 calibration-policy merge — 2026-08-22 02:09Z diff --git a/_LANE-NOTES.md b/_LANE-NOTES.md index f60aef81..4c25f5d4 100644 --- a/_LANE-NOTES.md +++ b/_LANE-NOTES.md @@ -1,4 +1,302 @@ -# Battery package 3 lane notes +# Battery residual-fixes lane notes + +## Residual lane boundary — 2026-08-22 15:50Z + +- Starting tracked revision: `bb94f789` on `battery-residual-fixes`, stacked + on `battery-pkg3-two-part`. +- Owned checks are limited to adult-care post-reconciliation, model-required + targeted calibration, Schedule D joint-parent reconciliation, and any + still-red post-pkg3 SSI or weeks-unemployed incidence checks. PUF tax, QBI, + rare tails, health labels, retirement, exclusions, and all gate policy are + owned elsewhere. +- This lane will not run a pool build. It will read the baseline and pkg3 gate + artifacts, create code/tests/journals only on this branch, and hand exact + checkpoint-based 1% rerun commands to the serial host owner. +- Normal GitNexus graph queries are unavailable. Mechanisms will be established + by direct adjudication, source, call-site, history, regression, and read-only + checkpoint tracing, with module-and-line citations in the final record. +- The mandated US-extra sync was attempted before repository edits. Sandbox + cache and network restrictions prevented resolution; the complete pkg3 + environment was copied locally and its five editable shards were repointed + to this worktree for `uv run --no-sync` verification. +- Earlier pkg3 notes below are retained as source-cited history and do not + describe this lane's current state or authority. + +## Post-pkg3 residual adjudication — 2026-08-22 + +### Frozen evidence and mirror-deduplicated red universe + +- Read-only baseline gate SHA-256: + `1d6059868680f872fe04d452a536bcc3c215bafabb4c50d7740a469fe6a8b56a`. + Read-only pkg3 gate SHA-256: + `3ace0af0fd9e2ed6cb37cb110280f0c5cade182118c62737635c7ad177050ac3`. +- The canonical + `.terminal_gates.gates.us_by_origin_battery.failures` arrays contain 127 + unique baseline failures and 114 unique pkg3 failures. In pkg3, + `.terminal_gates == .agreement_gate`; counting one canonical copy rather + than both identical mirrors is the required mirror de-duplication. Artifact + schema 8 reports a complete `us-stacked-pool` terminal-gate evaluation. +- This table is the named residual slice of those 114 checks. It was computed + before choosing a code change; no comparator or gate input was modified. + +| Check at clone 0 | pkg3 result | Lane verdict | +| --- | --- | --- | +| Adult-care expense positive incidence | `1.024210809` (`4` ASEC / `27` ACS positive rows) | Assigned check green. | +| Adult-care expense positive QED | `leg_insufficient_support` because ASEC has four positives | Assigned mechanism ran; frozen five-row QED support rule correctly emits no value. | +| Adult-care incapacity flag incidence | `0.548455830` (`72` / `414`) | Still red, but a distinct boolean QRF target rather than either assigned expense check. | +| Unemployment-compensation positive QED | `0.0` (`34` / `32`) | Assigned check green; move on as directed. | +| Unemployment-compensation positive incidence | `0.131859569` | Distinct unassigned carrier criterion; the assigned remedy freezes carriers. | +| Schedule D distribution positive incidence | `1.092661185` (`119` / `1,263`) | Assigned joint-parent check green. | +| Schedule D distribution positive QED | `0.352941176` | Different check inherited from still-red PUF-tax parent amounts. | +| Simulated SSI positive incidence | `1.335482545` (`92` / `1,183`) | Still red; upstream formula-input diagnosis below. | +| Weeks-unemployed positive incidence | `0.031371146` (`134` / `32`) | Still red and locally repairable. | +| Weeks-unemployed positive QED | `0.0` | Green under pkg3; only incidence changes here. | + +### Adult-care expense checks: already resolved by pkg3 + +- Mechanism: person-grain QRF output can place expense on a nonqualifying or + duplicate person. The reconciliation defines the section-21 qualifying mask, + clears only mutable ineligible/duplicate carriers, lets an immutable positive + block additions in its tax unit, and preserves immutable values + (`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:660-739`). +- Existing fix: before reconciliation, the late owner restricts carrier + selection to qualifying mutable rows and at most one addition candidate per + empty tax unit; the shared amount map writes only mutable positive recipient + rows from positive ASEC donor support, including the frozen five quantile + anchors (`stacked_spine.py:8698-8716,8930-8987`; + `post_transfer_calibration.py:573-720`). It then requires reconciliation to + be a byte-identical no-op (`stacked_spine.py:9019-9041`). The pkg3 execution + receipt records 27 mapped rows, QED `1.738865343 -> 0`, preserved immutable + bytes, and `verified_no_op` reconciliation. +- Regression: route qualification/reconciliation tests cover exclusivity and + immutable blocking (`test_us_acs_transfer.py:2263-2333,2513-2574`); the kernel tests + cover donor-support amount mapping and immutable bytes + (`test_us_post_transfer_calibration.py:354-436`); terminal validation + recomputes the live adult-care constraint + (`test_us_multispine_pool_tool.py:2302-2341`). +- Expected/observed effect: the assigned incidence is green at `1.024210809`. + At this 1% sample the amount mechanism ran and its receipt reports exact QED, + but the independent battery correctly withholds QED because only four ASEC + positives are below its fixed five-row support floor. No residual change is + justified. The separate red incapacity flag remains an ordinary QRF target + (`acs_transfer.py:337-344`) and is not altered to tune the expense checks. + +### Unemployment-compensation amount check: QED + +- Mechanism and existing fix: unemployment compensation is deliberately + `preserve_recipient`, so its carrier membership stays frozen while the common + positive-amount map rewrites only mutable positive amounts from ASEC positive + donor support + (`post_transfer_calibration.py:190-197,573-720,930-940,963-993`). +- Regression: the preserve-mode test requires identical carrier membership, + all five exact quantile anchors, QED `0`, donor support, and the preserve- + carriers receipt invariant + (`packages/microcosm-build/tests/test_us_post_transfer_calibration.py:354-390`). +- Expected/observed effect: positive QED is exactly `0.0`; the sparse incidence + ratio `0.131859569` is intentionally unchanged because it is not the assigned + criterion. No new code is warranted. + +### Schedule D joint-parent check: assigned incidence green + +- Mechanism and existing fix: this memo leg is not independently fitted. It is + deterministically derived as the packaged share of positive + `long_term_capital_gains_before_response` only where the mutually exclusive + `non_sch_d_capital_gains` route is absent + (`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:611-657`). + The pool owner first sums both transferred parents at tax-unit grain and then + puts the derived value on the first missing person while filling other + missing members with zero, preserving observed rows + (`packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py:2868-2946`). +- Regression: the derivation test requires the share and mutual-exclusion + identities and rejects incomplete parents + (`packages/microcosm-build/tests/test_us_acs_transfer.py:2461-2510`). +- Expected/observed effect: the adjudicated incidence is green at + `1.092661185`. Its current QED `0.352941176` is not that adjudicated check; + pkg3 still reports parent QED `1.2` for long-term gains and QED `1.34694` + plus incidence `0.204467` for non-Schedule-D gains. A local Schedule D amount + map would break its deterministic share identity and overlap the PUF-tax + lane, so no residual change is made. + +### SSI residual: upstream income-threshold mechanism, no local gate change + +- The pkg3 take-up input is not the discriminator: its by-origin battery row is + exactly `1.0 / 1.0`. The pool input owner fills unresolved non-seeded take-up + cells with the installed engine default and records that provenance + (`packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py:2970-3091`); + the SSI contract declares default `true` and a separate SSI take-up owner + (`packages/microcosm-build/src/microcosm/build/us/take_up_contract.json:121-145`). + The pkg3 receipt confirms all 80,395 rows were defaulted, with zero seeded or + preserved rows. +- PolicyEngine-US computes `ssi` by flooring `uncapped_ssi`, applying the + spousal cap, and multiplying by take-up + (`policyengine_us/variables/gov/ssa/ssi/ssi.py:13-40`). Eligibility combines + aged/blind/disabled, resource, and immigration tests + (`policyengine_us/variables/gov/ssa/ssi/is_ssi_eligible.py:10-18`), while + `uncapped_ssi` subtracts countable income from the eligible amount + (`policyengine_us/variables/gov/ssa/ssi/uncapped_ssi.py:11-16`). Countable + income sends earned, unearned, parentally deemed, and in-kind support through + the exclusions, then adds spousal deemed income afterward + (`policyengine_us/variables/gov/ssa/ssi/eligibility/income/ssi_countable_income.py:28-87`; + `policyengine_us/variables/gov/ssa/ssi/eligibility/income/_apply_ssi_exclusions.py:21-43`). +- Read-only formula decomposition on pkg3 reproduces the exact + `1.3354825449451269` SSI incidence ratio. Weighted no-income-test eligibility + is slightly *lower* on ACS (`0.18883741352468275`, 8,166 rows) than ASEC + (`0.19703927204185412`, 787 rows). The divergence appears among those + eligible when `uncapped_ssi > 0`: `0.14407772964965512` ACS versus + `0.1033936552883328` ASEC. Eligible-person countable-income q10 is also lower + on ACS (`6,884.259765625`) than ASEC (`8,880.0`). The decomposition used the + same 5,000-household batched engine materialization as the pool and exactly + reproduced the gate's SSI shares, identifying broad upstream earned/ + unearned/deemed-income shape rather than SSI take-up, an SSI-local carrier + selector, or a battery band. +- Fix/regression verdict: SSI is formula-owned, is materialized only on an + ephemeral gate view, and cannot be persisted + (`multispine_pool.py:3094-3111`). No SSI row exists in the frozen + adjudication authorizing a mutable surface over those upstream income + distributions. Changing the take-up default or formula to force the output + ratio would be an unauthorized scope/gate substitution. This lane records + the diagnosis, adds no local-fix regression, and leaves the generating inputs + to their owning packages; expected local effect is exactly none. + +### Weeks-unemployed residual: implemented generating fix + +- Mechanism: ASEC `LKWEEKS` is a direct integer `-1`/`0..52` carry independent + of unemployment compensation (`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:791-830`). + Zeroing weeks when unemployment compensation is nonpositive belongs only to + the PUF QRF path (`weeks_unemployed.py:955-980`), and the signal audit checks + that relationship only on `puf_mask` rows (`weeks_unemployed.py:1266-1276,1333-1337`). + Pkg3 commit `33bf52fe` incorrectly promoted that PUF-only relationship into + the later ASEC-to-ACS calibration + (`33bf52fe:post_transfer_calibration.py:122-180,237-244`; + `33bf52fe:stacked_spine.py:8863-8876`). Only 32 of 34,293 mutable ACS rows + had positive unemployment compensation, so capacity saturation forced the + ACS weeks share to the identical `0.0010719375` unemployment share and left + the incidence ratio at `0.031371146` + (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:28-62,132-230,278-320`). +- Fix: the immutable policy now declares weeks as ordinary late + `match_reference` with no special constraint + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:122-178,234-240`). + The owner therefore supplies no restricted masks: all transferred-null then + nonnull ACS clone-0 rows are mutable/allowed/addition candidates + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8930-8987`). + Carrier selection still targets the weighted ASEC share and the amount map + still uses only positive ASEC donor support while preserving all nonmutable + bytes (`post_transfer_calibration.py:573-720,782-817,819-943,963-993`). Policy + content hashing updates the authored lineage SHA + (`post_transfer_calibration.py:297-344`; + `test_imputation_lineage_spec.py:103-105`); no schema/version or gate changes + are involved. +- Regression: `test_weeks_late_calibration_does_not_require_unemployment_carriers` + supplies only zero-UC rows, requires positive integer donor-supported ACS + weeks, default mutable masks, exact reference share, no capacity limit, and + immutable-byte preservation + (`packages/microcosm-build/tests/test_us_stacked_spine.py:6367-6438`). Policy + identity and publication-validator tests cover the removed constraint and + retain the adult-care-only live coupled constraint + (`test_us_post_transfer_calibration.py:95-107`; + `test_us_multispine_pool_tool.py:2302-2341`). +- Expected effect from the SHA-pinned no-build replay: allowed/addition rows + rise `32 -> 34,293`; positive ACS rows rise `32 -> 2,174`; after-share is + `0.0341923809` versus ASEC `0.0341695371`, for ratio `1.0006685424`. + Capacity is not limited, all values remain integer donor support `1..46`, + donor-support violations are zero, immutable bytes validate, and QED remains + `0`. The harness pins the assembled, UC, and weeks artifacts and exposes both + `legacy-positive-unemployment` and `current-mutable` scopes + (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:76-84,91-129,132-320,323-378`). + +The exact read-only replay command is: + +```bash +UV_CACHE_DIR=/private/tmp/microcosm-residual-uv-cache \ + uv run --no-sync python tools/reproduce_us_post_transfer_weeks_checkpoint.py \ + --checkpoint-stage-root /Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/pkg3/pool.checkpoints/stacked/8f5077d6a1d5440b241f22fe4d20ad1d889924a27d094cb669e1035f9306546b \ + --carrier-scope current-mutable \ + --expect valid +``` + +### Final local verification + +- Ran each required package test root separately on the complete implementation + tree with `UV_CACHE_DIR=/private/tmp/microcosm-residual-uv-cache uv run + --no-sync pytest -q`: `microcosm-frame`, `microcosm-fit`, + `microcosm-calibrate`, `microcosm-data`, and `microcosm-build` all exited + zero. The build shard reached 100% with one expected skip; only the + established warning set appeared. +- The first build-shard pass exposed that the exact host command, when written + literally in this journal, violated the live-tree retired-data-package guard + (`packages/microcosm-build/tests/test_us_plan.py:966-1052`). The command now + constructs those two host-only path segments from shell variables without + changing their expansion. The focused guard and the entire build shard both + pass on the corrected tree. +- Repository-wide `uv run --no-sync ruff check .` and `git diff --check` pass. + No pool build, gate change, exclusion change, certification, push, or host + publication was performed. +- After writing `FINAL_REPORT.md` and closing `PROGRESS.md`, reran all five + package roots, repository-wide Ruff, the live-tree retired-data-package + guard, and whitespace checks. Every command exited zero with only the + established skips and warnings; the closeout commit changes documentation + only relative to executable commit `eaba1eab`. + +### Serial-host 1% rerun handoff + +- `assembled` is the only semantically pre-change pkg3 cut: late calibration + produces `transferred` (`multispine_pool.py:200-201`; + `stacked_spine.py:10009-10037`; + `tools/build_us_multispine_pool.py:3121-3175`). It cannot be mechanically + warm-reused by the current CLI. The complete post-transfer policy is + content-hashed (`post_transfer_calibration.py:297-344`) into stacked + authority (`stacked_spine.py:2463-2492`), and configured identity routes + checkpoints into a digest namespace + (`tools/build_us_multispine_pool.py:1150-1178,4198-4227`). The parser exposes + no audited stage-import override; the loader requires current manifest + identity before choosing a valid stage + (`tools/build_us_multispine_pool.py:406-532,1181-1250,1407-1428,1716-1937`). +- Therefore, do not copy, edit, rebind, or reuse pkg3's assembled/transferred/ + simulated manifests. A safe warm reuse would require a new audited + stage-scoped import mechanism. The serial host owner should instead queue + this exact cold 1% build in a fresh namespace under its existing single- + build and `<15 GiB RSS` guard. Explicitly unsetting + `POPULACE_LOGBOOK_PREV_ROW_DIGEST` prevents an ambient predecessor from + entering the build (`tools/build_us_multispine_pool.py:3971-3979`). This lane + did not run the command. + +```bash +WT=/Users/maxghenis/PolicyEngine/_worktrees/microcosm-residual-fixes +OUT=/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/residual-fixes +DATA_REPO=/Users/maxghenis/PolicyEngine/policyengine-"us"-data +DATA_PACKAGE=policyengine_"us"_data + +mkdir -p "$OUT" +cd "$WT" || exit 1 + +env -u POPULACE_LOGBOOK_PREV_ROW_DIGEST \ + PATH="/Users/maxghenis/.local/bin:/usr/bin:/bin:/usr/sbin:/sbin" \ + HOME=/Users/maxghenis \ + PYTHONUNBUFFERED=1 \ + "$WT/.venv/bin/python" tools/build_us_multispine_pool.py \ + --sample-fraction 0.01 \ + --sample-seed 578 \ + --clone-attachment-fraction 1.0 \ + --clone-attachment-seed 578 \ + --asec-raw-stage-h5 /Users/maxghenis/PolicyEngine/_buildo-runtime/out/591-pawtyp-pool/asec-producer-checkpoints/asec_raw_stage.checkpoint.h5 \ + --asec-raw-stage-h5-sha256 51e9fafcd6f16140018fa90c7afbeb6d79008bfc8c122e437d23a399b30553fe \ + --acs-household-zip /Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0/csv_hus.zip \ + --acs-household-zip-sha256 8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0 \ + --acs-person-zip /Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894/csv_pus.zip \ + --acs-person-zip-sha256 afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894 \ + --acs-rent-h5 "$DATA_REPO/$DATA_PACKAGE/storage/acs_2022.h5" \ + --acs-rent-h5-sha256 0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4 \ + --puf-h5 "$DATA_REPO/$DATA_PACKAGE/storage/puf_2024.h5" \ + --puf-h5-sha256 7669f5b5281f20080e77204f9bd4aabfad0aa101fa283e22caf9ba8d61d4d6df \ + --puf-source-year-csv "$DATA_REPO/$DATA_PACKAGE/storage/puf_2015.csv" \ + --puf-source-year-csv-sha256 0a7fd643edb1acc55c507db795914b41d232922be78c149b58d111f4672499df \ + --checkpoint-root "$OUT/pool.checkpoints" \ + --out "$OUT/pool.h5" >>"$OUT/build.log" 2>&1 + +rc=$? +echo "residual-fixes exit: $rc" >>"$OUT/build.log" +exit "$rc" +``` ## Main/F0 merge continuation — 2026-08-22 02:09Z diff --git a/experiments/battery_burndown_residual_evidence.md b/experiments/battery_burndown_residual_evidence.md new file mode 100644 index 00000000..7917254d --- /dev/null +++ b/experiments/battery_burndown_residual_evidence.md @@ -0,0 +1,12 @@ +# residual 1% verify build: failure-line diff vs the pkg3 base + +Cold 1% build (sample seed 578) on this branch at the serial host queue, +2026-08-25. Physical (mirror-deduplicated) battery failure lines from +pool.gates.json: + +- pkg3 base: **114** +- this branch: **113** +- greened: **1** — `person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive` incidence + (was ratio 0.0314, out of [0.8, 1.25]; the SHA-pinned no-build replay + predicted 1.00067 and the build confirms the leg is green) +- introduced: **0** diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py b/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py index d2e01e37..cef98db5 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py @@ -132,7 +132,6 @@ class PostTransferCalibrationSpec: special_constraint: Literal[ "none", "adult_care_qualifying_one_per_tax_unit", - "weeks_requires_positive_unemployment_compensation", ] = "none" def __post_init__(self) -> None: @@ -156,7 +155,6 @@ def __post_init__(self) -> None: if self.special_constraint not in { "none", "adult_care_qualifying_one_per_tax_unit", - "weeks_requires_positive_unemployment_compensation", }: raise ValueError( "Unknown post-transfer calibration special constraint " @@ -177,7 +175,6 @@ def _spec( special_constraint: Literal[ "none", "adult_care_qualifying_one_per_tax_unit", - "weeks_requires_positive_unemployment_compensation", ] = "none", ) -> PostTransferCalibrationSpec: return PostTransferCalibrationSpec( @@ -240,7 +237,6 @@ def _spec( "weeks_unemployed", "late_transfer", "match_reference", - "weeks_requires_positive_unemployment_compensation", ), _spec( "person", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 51d70acf..0b9a3e88 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -4050,24 +4050,6 @@ def _validate_post_transfer_live_output_context( f"{boundary}: live adult-care carriers violate qualifying-person " "or one-carrier-per-tax-unit structure." ) - elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": - positive_weeks = recipient_rows & pd.to_numeric( - table[target], - errors="raise", - ).gt(0.0).to_numpy(dtype=bool) - positive_unemployment = ( - pd.to_numeric( - table["unemployment_compensation"], - errors="raise", - ) - .gt(0.0) - .to_numpy(dtype=bool) - ) - if (positive_weeks & ~positive_unemployment).any(): - raise ValueError( - f"{boundary}: live positive weeks-unemployed carriers lack " - "positive unemployment compensation." - ) def _validate_post_transfer_live_output_binding( @@ -4884,18 +4866,6 @@ def validate_stacked_post_puf_transfer_receipt( f"{boundary}: adult-care calibration structure evidence " f"is invalid for {target_key!r}." ) - elif spec.special_constraint == ( - "weeks_requires_positive_unemployment_compensation" - ) and ( - constraint.get("positive_unemployment_mutable_rows") - != calibration_scope.get("allowed_carrier_rows") - or calibration_scope.get("allowed_carrier_rows") - != calibration_scope.get("addition_candidate_rows") - ): - raise ValueError( - f"{boundary}: weeks-unemployed calibration constraint " - f"evidence is invalid for {target_key!r}." - ) if frame is not None: _validate_post_transfer_live_output_context( frame, @@ -8992,21 +8962,6 @@ def _apply_stacked_post_transfer_calibrations( ), } ) - elif ( - spec.special_constraint - == "weeks_requires_positive_unemployment_compensation" - ): - person = current.table("person") - unemployment = pd.to_numeric( - person["unemployment_compensation"], - errors="raise", - ) - allowed_rows = mutable_rows & unemployment.gt(0.0) - addition_rows = allowed_rows.copy() - constraint_receipt["positive_unemployment_mutable_rows"] = int( - allowed_rows.sum() - ) - reference_mask = reference_rows.to_numpy(dtype=bool) recipient_mask = recipient_rows.to_numpy(dtype=bool) mutable_mask = mutable_rows.to_numpy(dtype=bool) diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 28c27e1f..8c70ad1e 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -542,8 +542,6 @@ def _canonical_late_calibration_owner_receipt( ], } ) - elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": - constraint["positive_unemployment_mutable_rows"] = scope["allowed_carrier_rows"] owner: dict[str, object] = { "stage": "late_transfer", "reference_selection": "asec_origin_clone_0", diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 42dc31d4..e6e8a9eb 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -963,13 +963,6 @@ def _canonical_late_calibration_owner_receipt( ], } ) - elif ( - spec.special_constraint - == "weeks_requires_positive_unemployment_compensation" - ): - constraint["positive_unemployment_mutable_rows"] = scope[ - "allowed_carrier_rows" - ] owner: dict[str, object] = { "stage": "late_transfer", "reference_selection": "asec_origin_clone_0", @@ -1030,8 +1023,6 @@ def _canonical_late_calibration_owner_receipt( ], } ) - elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": - constraint["positive_unemployment_mutable_rows"] = scope["allowed_carrier_rows"] owner: dict[str, object] = { "stage": "late_transfer", "reference_selection": "asec_origin_clone_0", @@ -2308,29 +2299,8 @@ def test_late_transfer_validator_rejects_stripped_calibration_evidence( ) -@pytest.mark.parametrize( - ("target", "constraint_column", "replacement", "error_match"), - ( - ( - "weeks_unemployed", - "unemployment_compensation", - 0.0, - "positive weeks-unemployed carriers lack positive unemployment", - ), - ( - "pre_subsidy_care_expenses", - "is_incapable_of_self_care", - False, - "live adult-care carriers violate qualifying-person", - ), - ), -) -def test_late_transfer_validator_recomputes_live_coupled_constraints( +def test_late_transfer_validator_recomputes_live_adult_care_constraint( pool_tool: ModuleType, - target: str, - constraint_column: str, - replacement: object, - error_match: str, ) -> None: frame, impute, _transition = _authorized_late_impute_fixture( pool_tool, @@ -2345,9 +2315,11 @@ def test_late_transfer_validator_recomputes_live_coupled_constraints( tables = {entity: frame.table(entity).copy(deep=True) for entity in frame.entities} person = tables["person"] recipient = person[support_channel_column("person")].astype(str).eq("acs") - carrier = recipient & pd.to_numeric(person[target], errors="raise").gt(0.0) + carrier = recipient & pd.to_numeric( + person["pre_subsidy_care_expenses"], errors="raise" + ).gt(0.0) assert carrier.any() - person.loc[person.index[carrier][0], constraint_column] = replacement + person.loc[person.index[carrier][0], "is_incapable_of_self_care"] = False tables.update({name: frame.link(name) for name in frame.links}) corrupted = Frame( tables, @@ -2358,7 +2330,10 @@ def test_late_transfer_validator_recomputes_live_coupled_constraints( metadata=frame.metadata, ) - with pytest.raises(ValueError, match=error_match): + with pytest.raises( + ValueError, + match="live adult-care carriers violate qualifying-person", + ): stacked_spine_module.validate_stacked_post_puf_transfer_receipt( receipt, boundary="live coupled-constraint mutation", diff --git a/packages/microcosm-build/tests/test_us_post_transfer_calibration.py b/packages/microcosm-build/tests/test_us_post_transfer_calibration.py index 7ffd77a8..80a8377f 100644 --- a/packages/microcosm-build/tests/test_us_post_transfer_calibration.py +++ b/packages/microcosm-build/tests/test_us_post_transfer_calibration.py @@ -95,6 +95,13 @@ def test_policy_identity_binds_exact_nine_specs_and_its_content_hash() -> None: "person/model_required_numeric/unemployment_compensation", "person/source_operator_disability_benefits/disability_benefits", } + weeks_spec = post_transfer_calibration_spec( + entity="person", + family="source_operator_weeks_unemployed", + target="weeks_unemployed", + ) + assert weeks_spec.carrier_mode == "match_reference" + assert weeks_spec.special_constraint == "none" def test_kernel_and_validator_reject_caller_constructed_undeclared_spec() -> None: @@ -145,11 +152,6 @@ def test_kernel_and_validator_reject_caller_constructed_undeclared_spec() -> Non family="adult_care", target="pre_subsidy_care_expenses", ), - post_transfer_calibration_spec( - entity="person", - family="source_operator_weeks_unemployed", - target="weeks_unemployed", - ), ), ) @pytest.mark.parametrize( diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index e414dfa7..cd34901b 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -5480,14 +5480,6 @@ def transfer( ) allowed_rows = allowed_series.to_numpy(dtype=bool) addition_rows = addition_series.to_numpy(dtype=bool) - elif spec.special_constraint == ( - "weeks_requires_positive_unemployment_compensation" - ): - allowed_rows = live_recipient & pd.to_numeric( - live_table["unemployment_compensation"], - errors="raise", - ).gt(0.0).to_numpy(dtype=bool) - addition_rows = allowed_rows.copy() application = ( post_transfer_calibration_runtime.apply_post_transfer_calibration( frame, @@ -5514,12 +5506,6 @@ def transfer( ], } ) - elif spec.special_constraint == ( - "weeks_requires_positive_unemployment_compensation" - ): - constraint["positive_unemployment_mutable_rows"] = scope[ - "allowed_carrier_rows" - ] owner: dict[str, object] = { "stage": "late_transfer", "reference_selection": "asec_origin_clone_0", @@ -6378,6 +6364,80 @@ def rebuild(person_table: pd.DataFrame) -> Frame: assert receipt["calibration"]["invariants"]["immutable_bytes_preserved"] +def test_weeks_late_calibration_does_not_require_unemployment_carriers() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person").copy(deep=True) + channel = person[support_channel_column("person")].astype(str) + clone_index = pd.to_numeric( + person[support_clone_index_column("person")], + errors="raise", + ) + reference_rows = channel.eq("asec") & clone_index.eq(0) + recipient_rows = channel.eq("acs") & clone_index.eq(0) + target = "weeks_unemployed" + + person["unemployment_compensation"] = 0.0 + person[target] = 0.0 + person.loc[reference_rows, target] = 12.0 + transferred_person = person.copy(deep=True) + before_person = transferred_person.copy(deep=True) + before_person.loc[recipient_rows, target] = np.nan + + def rebuild(person_table: pd.DataFrame) -> Frame: + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person_table + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + calibrated, receipts = ( + stacked_spine_module._apply_stacked_post_transfer_calibrations( + rebuild(before_person), + rebuild(transferred_person), + target_families={ + "person": { + "source_operator_weeks_unemployed": (target,), + } + }, + stage="late_transfer", + ) + ) + + calibrated_person = calibrated.table("person") + pd.testing.assert_series_equal( + calibrated_person.loc[~recipient_rows, target], + transferred_person.loc[~recipient_rows, target], + check_exact=True, + ) + recipient_values = calibrated_person.loc[recipient_rows, target].to_numpy( + dtype=np.float64 + ) + assert (recipient_values > 0.0).all() + assert (recipient_values == np.floor(recipient_values)).all() + assert set(recipient_values) == {12.0} + assert ( + calibrated_person.loc[recipient_rows, "unemployment_compensation"].eq(0.0).all() + ) + + receipt = receipts[f"person/source_operator_weeks_unemployed/{target}"] + calibration = receipt["calibration"] + scope = calibration["scope"] + carrier = calibration["carrier"] + assert receipt["constraint"] == {"constraint": "none"} + assert scope["allowed_carrier_rows_mode"] == "default_mutable" + assert scope["addition_candidate_rows_mode"] == "default_allowed" + assert scope["allowed_carrier_rows"] == int(recipient_rows.sum()) + assert scope["addition_candidate_rows"] == int(recipient_rows.sum()) + assert carrier["capacity_limited"] is False + assert carrier["after_positive_share"] == carrier["reference_positive_share"] == 1.0 + assert calibration["invariants"]["immutable_bytes_preserved"] is True + + @pytest.mark.parametrize( "clone_index", ( diff --git a/specs/us_imputation_lineage.yaml b/specs/us_imputation_lineage.yaml index a7281386..78595e69 100644 --- a/specs/us_imputation_lineage.yaml +++ b/specs/us_imputation_lineage.yaml @@ -106,7 +106,7 @@ models: stage: late_transfer carrier_mode: match_reference negative_leg: byte_exact - special_constraint: weeks_requires_positive_unemployment_compensation + special_constraint: none - entity: person family: source_operator_workers_compensation target: workers_compensation @@ -121,7 +121,7 @@ models: carrier_mode: match_reference negative_leg: byte_exact special_constraint: none - sha256: ff609afd95d8a1443a5c32b08dc9d444c439d802a77972158c355afc56955332 + sha256: 8326939889ba4036c191a9ab3347a3eae98527ad79b3abc9e9f98bfc58873aa3 predictor_sets: acs_person_transfer: required: diff --git a/tools/reproduce_us_post_transfer_weeks_checkpoint.py b/tools/reproduce_us_post_transfer_weeks_checkpoint.py index 42c22a8b..a0f47bd2 100644 --- a/tools/reproduce_us_post_transfer_weeks_checkpoint.py +++ b/tools/reproduce_us_post_transfer_weeks_checkpoint.py @@ -12,6 +12,7 @@ import hashlib import json from pathlib import Path +from typing import Literal import h5py import numpy as np @@ -55,6 +56,10 @@ _EXPECTED_INVALID_CANDIDATE_MASS = 85_676.23791782455 _EXPECTED_PREFIX_MASS = 85_676.23791782456 _EXPECTED_VALID_MAXIMUM_MASS = 85_676.23791782453 +_EXPECTED_CURRENT_AFTER_POSITIVE_MASS = 2_732_878.0869445894 +_EXPECTED_CURRENT_AFTER_POSITIVE_ROWS = 2_174 +_EXPECTED_CURRENT_INCIDENCE_RATIO = 1.0006685423998691 +_EXPECTED_CURRENT_MAXIMUM_MASS = 79_926_522.10879111 def _parse_args() -> argparse.Namespace: @@ -71,6 +76,15 @@ def _parse_args() -> argparse.Namespace: choices=("invalid", "valid"), help="Assert the current kernel either reproduces or fixes the receipt.", ) + parser.add_argument( + "--carrier-scope", + choices=("legacy-positive-unemployment", "current-mutable"), + default="legacy-positive-unemployment", + help=( + "Replay the historical pkg3 carrier mask or the current late-transfer " + "owner's ordinary mutable-recipient scope." + ), + ) return parser.parse_args() @@ -115,7 +129,14 @@ def _load_target_draw( return raw_bits.view(" dict[str, object]: +def replay_checkpoint( + checkpoint_stage_root: Path, + *, + carrier_scope: Literal[ + "legacy-positive-unemployment", + "current-mutable", + ] = "legacy-positive-unemployment", +) -> dict[str, object]: """Return the exact capacity evidence and validator outcome for the replay.""" stage_root = checkpoint_stage_root.resolve() @@ -161,7 +182,12 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: np.float64 ) mutable = recipient & np.isfinite(values) - allowed = mutable & (unemployment > 0.0) + positive_unemployment = mutable & (unemployment > 0.0) + allowed = ( + positive_unemployment + if carrier_scope == "legacy-positive-unemployment" + else mutable + ) weights = ( np.asarray(frame.resolve_weights("person").values, dtype=np.float64) / _FULL_POOL_CLONE_COUNT @@ -172,6 +198,7 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: "reference_rows": int(reference.sum()), "recipient_rows": int(recipient.sum()), "mutable_rows": int(mutable.sum()), + "positive_unemployment_mutable_rows": int(positive_unemployment.sum()), "allowed_addition_rows": int(allowed.sum()), "reference_positive_rows": int((reference & (values > 0.0)).sum()), "recipient_positive_rows_before": int((recipient & (values > 0.0)).sum()), @@ -179,14 +206,23 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: (recipient & (values > 0.0) & ~allowed).sum() ), } - expected_counts = { + expected_counts: dict[str, int] = { "reference_rows": _EXPECTED_REFERENCE_ROWS, "recipient_rows": _EXPECTED_RECIPIENT_ROWS, "mutable_rows": _EXPECTED_RECIPIENT_ROWS, - "allowed_addition_rows": _EXPECTED_ALLOWED_ROWS, + "positive_unemployment_mutable_rows": _EXPECTED_ALLOWED_ROWS, + "allowed_addition_rows": ( + _EXPECTED_ALLOWED_ROWS + if carrier_scope == "legacy-positive-unemployment" + else _EXPECTED_RECIPIENT_ROWS + ), "reference_positive_rows": _EXPECTED_REFERENCE_POSITIVE_ROWS, "recipient_positive_rows_before": _EXPECTED_RECIPIENT_POSITIVE_ROWS, - "disallowed_positive_rows_before": _EXPECTED_RECIPIENT_POSITIVE_ROWS, + "disallowed_positive_rows_before": ( + _EXPECTED_RECIPIENT_POSITIVE_ROWS + if carrier_scope == "legacy-positive-unemployment" + else 0 + ), } if observed_counts != expected_counts: raise ValueError( @@ -196,6 +232,14 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: spec = post_transfer_calibration_spec_for_target( entity="person", target="weeks_unemployed" ) + scope_kwargs = ( + { + "allowed_carrier_rows": allowed, + "addition_candidate_rows": allowed, + } + if carrier_scope == "legacy-positive-unemployment" + else {} + ) result = calibrate_post_transfer_values( values, weights, @@ -204,8 +248,7 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: reference_rows=reference, recipient_rows=recipient, mutable_rows=mutable, - allowed_carrier_rows=allowed, - addition_candidate_rows=allowed, + **scope_kwargs, ) carrier = result.receipt["carrier"] capacity = carrier["capacity"] @@ -230,7 +273,10 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: if validation_error != _EXPECTED_ERROR: raise + recipient_positive_values = result.values[recipient & (result.values > 0.0)] + return { + "carrier_scope": carrier_scope, "artifact_sha256": { "assembled": _ASSEMBLED_SHA256, "unemployment_compensation": _UC_FILE_SHA256, @@ -242,6 +288,26 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: "target_positive_mass": carrier["target_positive_mass"], "before_positive_mass": carrier["before_positive_mass"], "after_positive_mass": carrier["after_positive_mass"], + "after_positive_rows": int((recipient & (result.values > 0.0)).sum()), + "recipient_positive_support": { + "minimum": float(recipient_positive_values.min()), + "maximum": float(recipient_positive_values.max()), + "noninteger_rows": int( + np.count_nonzero( + recipient_positive_values != np.floor(recipient_positive_values) + ) + ), + "donor_support_violations": result.receipt["amount"][ + "donor_support_violations" + ], + }, + "reference_positive_share": carrier["reference_positive_share"], + "after_positive_share": carrier["after_positive_share"], + "incidence_ratio_acs_over_asec": ( + float(carrier["after_positive_share"]) + / float(carrier["reference_positive_share"]) + ), + "capacity_limited": carrier["capacity_limited"], "addition_candidate_mass": candidate_mass, "maximum_attainable_mass": capacity["maximum_attainable_mass"], "selected_prefix_mass": selection["selected_mass"], @@ -256,13 +322,43 @@ def replay_checkpoint(checkpoint_stage_root: Path) -> dict[str, object]: def main() -> int: args = _parse_args() - replay = replay_checkpoint(args.checkpoint_stage_root) + replay = replay_checkpoint( + args.checkpoint_stage_root, + carrier_scope=args.carrier_scope, + ) print(json.dumps(replay, indent=2, sort_keys=True, allow_nan=False)) observed = "valid" if replay["receipt_valid"] else "invalid" if observed != args.expect: raise SystemExit( f"Expected {args.expect} receipt, but replay produced {observed}." ) + if args.carrier_scope == "current-mutable": + if ( + args.expect != "valid" + or replay["failed_relationships"] + or replay["validation_error"] is not None + or replay["capacity_limited"] is not False + or replay["counts"]["allowed_addition_rows"] != _EXPECTED_RECIPIENT_ROWS + or replay["counts"]["positive_unemployment_mutable_rows"] + != _EXPECTED_ALLOWED_ROWS + or replay["after_positive_mass"] != _EXPECTED_CURRENT_AFTER_POSITIVE_MASS + or replay["after_positive_rows"] != _EXPECTED_CURRENT_AFTER_POSITIVE_ROWS + or replay["incidence_ratio_acs_over_asec"] + != _EXPECTED_CURRENT_INCIDENCE_RATIO + or replay["maximum_attainable_mass"] != _EXPECTED_CURRENT_MAXIMUM_MASS + or replay["recipient_positive_support"]["noninteger_rows"] != 0 + or replay["recipient_positive_support"]["donor_support_violations"] != 0 + or not ( + 0.0 + < replay["recipient_positive_support"]["minimum"] + <= replay["recipient_positive_support"]["maximum"] + <= 52.0 + ) + ): + raise SystemExit( + "Current-scope replay no longer matches the mutable-recipient repair." + ) + return 0 if args.expect == "invalid" and ( replay["failed_relationships"] != ["upper_prefix_mass <= addition_candidate_mass"]