From a5c7310c5c185a28945bef02e71ae45c42e452f2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 21 Aug 2026 11:42:22 -0400 Subject: [PATCH 1/3] docs: initialize rare signed-tail lane journal --- PROGRESS.md | 87 ++++++++++++++++++++++++++++---------------------- _LANE-NOTES.md | 67 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 115 insertions(+), 39 deletions(-) create mode 100644 _LANE-NOTES.md diff --git a/PROGRESS.md b/PROGRESS.md index 2e04a7968..bfcc60e6b 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,48 +1,57 @@ -# Progress +# Rare signed-tail battery lane progress ## State -Microcosm #516 whole-row donor outlier screen is complete on -`mortgage-donor-outlier-screen` (rebased onto `origin/main` after the #515 -interim carve merged as #525). The `puf_tax_detail` donor now drops tax units -whose grouped raw mortgage interest reaches $10M before the #515 carve -(pinned-artifact effect: 3,066 rows, weight 3,684 of ~161M, removing $2.947T -of phantom mortgage-interest mass), with the checkpoint schema bumped to v3 -so post-carve pre-screen checkpoints rebuild. +Lane initialization is in progress on `battery-rare-signed-tails` at +`2c7a7218`, matching `origin/main`. No product code, gate, band, ceiling, fold, +seed, exclusion register, logbook chain, or build artifact has been changed. +The authoritative frozen inputs are the arm-split lane's +`experiments/battery_burndown/ADJUDICATION.md` and `adjudication.json`. + +The mandated `uv sync --all-packages --extra us` was attempted first. The +default cache is inaccessible in the managed sandbox; a retry with a writable +cache reached resolution but could not download `pandas==3.0.3` because network +DNS is unavailable. The primary-checkout environment proved sufficient for the +fit, calibrate, data, and frame shards but does not carry the build shard's +`jsonschema` dependency. Full verification therefore uses the compatible, +already-synced `/Users/maxghenis/PolicyEngine/_worktrees/microcosm-f1/.venv` +read-only, with `UV_NO_SYNC=1` and this worktree's package source roots on +`PYTHONPATH`. ## Done -- Confirmed a clean starting worktree at `aef1c56`. -- Read the repository guidance and established the #515 donor carve as the - screen's required downstream boundary. -- Started source-level audits of every donor-frame consumer, checkpoint - validation, row-count pins, and existing donor-fact summaries. -- Attempted the requested GitNexus impact workflow; the managed filesystem - denied its global registry write. Its local index also exposed a broad - `build/` ignore mismatch, so the completed impact audit uses direct source - call sites and tests. -- Added `US_PUF_DONOR_MORTGAGE_OUTLIER_CEILING = 10_000_000.0` with the - structural rationale and pinned-artifact receipts. -- Added a whole-row screen on grouped raw person `home_mortgage_interest` - after tax-unit assembly, before the #515 carve, with retained-index reset. -- Confirmed no downstream consumer pairs donor rows to the original HDF arrays - or carries a stale donor-length vector; values and weights always originate - from the same screened frame. -- Bumped the primary QRF checkpoint schema from v2 to v3 and made the stale - checkpoint regression track the live constant while retaining literal-v1 - corruptions. -- Added regression coverage for the exact grouped boundary, whole-row removal, - retained/carved $5M row, raw-$10.5M pre-carve ordering, and constant. -- Requested suites pass: PUF support/QRF 53; plan/gates 195; fiscal targets - 139; microcosm-data 138 with 1 skip. The directly affected tail-bound suite - adds 12 passes. Ruff format/check and `git diff --check` are clean. -- Wrote `SOL_516_REPORT.md` with the exact seam, consumer-by-consumer file:line - audit, expected 208,611-row real-artifact effect, verification results, count - sweep, and deliberately untouched surfaces. +- Read `CLAUDE.md` and the GitNexus debugging workflow. +- Confirmed the branch starts at `origin/main` and recorded the pre-existing, + untracked `.gitnexus/` directory without deleting or staging it. +- Attempted the required GitNexus symptom query. Its local index is current, + but the global registry does not contain this worktree and is outside the + writable sandbox; source tracing remains the authority. +- Located the frozen adjudication and established `FINAL_REPORT.md` as the + committed output report because no separate output-file environment variable + is present. +- Confirmed the shared environment imports this lane's `microcosm.frame` from + the lane-local source tree and provides pytest, Ruff, pandas, and NumPy. +- Read the adjudication's comparator-mechanisms section in full and traced its + cited incidence-first, five-carrier QED-skip, early-gap-fill, late-producer, + donor-projection, regime-detection, and declared-absence seams. +- Recomputed all 48 red QED donor regimes from the frozen checkpoint: every + availability pattern is gated (35 zero-inflated-positive targets and 13 + three-sign targets); none is degenerate or single-sign. +- Recomputed the five rare-tail physical legs on their actual route donors: + collectibles +18, alimony +61, casualty +27, farm operations -89, and + prior-year self-employment -48 sign carriers. +- Established that Keogh is not structurally absent: native ASEC has two + positive values (`2,040` and `30,000`), but the forced late ASEC clone-1 donor + is degenerate zero and its 1,736,840 finite banked recipient draws are all + exactly zero. ## Next -- PR #527 review cycle, then merge. After both #525 and #527: rebuild the - base/release; the mortgage critical-fit ratchet (0.20 -> 0.15) waits on a - run that holds per `us_critical_targets.py`. -- Root record-level ETL carve stays open on microcosm#515. +- Finish the per-check mechanism/route matrix and record exact code citations. +- Add future-checkpoint regime provenance and the smallest non-threshold fix for + the Keogh support loss, with mechanism-specific regression tests. +- Preserve the adjudication's BLOCKER posture for sparse tails whose frozen + donors remain sign-capable but do not contain dense enough evidence for an + honest target-specific refit. +- Guard any off-chain 1% build with the host queue/RSS checks and compare exact + before/after failure lines against the frozen arm-split baseline. diff --git a/_LANE-NOTES.md b/_LANE-NOTES.md new file mode 100644 index 000000000..194955a85 --- /dev/null +++ b/_LANE-NOTES.md @@ -0,0 +1,67 @@ +# Rare signed-tail battery lane notes + +## 2026-08-21 — initialization + +- Branch: `battery-rare-signed-tails`, starting at `2c7a7218` (`origin/main`). +- Binding scope: diagnose the 48 QED reds and the one-sided Keogh leg from the + frozen arm-split adjudication; repair generating mechanisms only. +- Forbidden changes: gates, bands, ceilings, floors, folds, seeds, and owner-only + exclusions/register entries. +- Build discipline: at most a 1% sample; check for a live + `build_us_multispine_pool` process before every build; remain below 15 GiB RSS; + run off-chain without `--logbook-prev-row-digest`; never touch + `logbook-pending-chain.txt`. +- Environment receipt: `uv sync --all-packages --extra us` failed first on the + sandboxed default UV cache, then on network DNS while fetching the locked + pandas wheel from a writable cache. The primary-checkout venv lacks the build + shard's `jsonschema`; the compatible `microcosm-f1` venv is used read-only for + the full suite with lane-local sources and `UV_NO_SYNC=1`. +- GitNexus receipt: a fresh local index exists, but the CLI cannot query it + because this worktree is absent from the read-only global registry. Direct + source/call-site tracing and regression tests will substantiate every claim. +- No build has run and no chain-bearing file has been read or written as an + input to a build. + +## Evidence ledger + +Mechanism classifications, realized-regime recomputations, code citations, +test receipts, build commands, RSS/queue receipts, and failure-line diffs will +be appended here as they are established. + +### Frozen regime recomputation + +- All 48 baseline-red QED checks use four receipted availability patterns. Each + pattern at a given entity uses the same complete donor identity (person + 108,073; tax unit 57,630; SPM unit 43,961). +- Recomputing signs at `zero_atol=1e-6` gives 35 + `zero_inflated_positive` targets and 13 `three_sign` targets. There are no + degenerate or single-sign QED targets, so a regime substitution cannot explain + or honestly repair any of the 48 QED failures. +- The route partition is 17 early ASEC-to-ACS gap-fill QED checks and 31 late + producer-complement checks (20 PUF-only, 10 ASEC-source-only, one overlapping + producer target). +- Rare-tail actual-donor sign support is: collectibles +18 (late PUF; native + ASEC +82), alimony +61 (early), casualty +27 (late PUF; native ASEC +79), farm + operations -89 (early), and prior-year self-employment -48 (early). All five + remain gated and sign-capable; the adjudication therefore still requires + dense/additional evidence before a target-specific refit. + +### Keogh one-sided leg + +- Native ASEC clone 0 contains exactly two positive Keogh values: `2,040` and + `30,000`. The late transfer is hard-wired to ASEC-origin clone 1, whose Keogh + support is entirely zero, so its recomputed regime is `degenerate_zero`. +- The frozen bank contains 1,736,840 finite Keogh recipient draws and every draw + is exactly zero. This proves loss before transfer fitting, not loss during the + recipient merge. +- The retirement producer uniformly caps its 108,073-row ASEC training donor at + 5,000 rows and explicitly permits a rare output to remain zero when that draw + contains no carrier. The smallest honest repair must preserve sign support in + that capped training donor (with sampling-weight correction), rather than + declare ACS absence or alter a terminal gate. + +### Process visibility receipt + +- Both `ps` and `pgrep -af build_us_multispine_pool` are denied by the managed + macOS sandbox (`operation not permitted` / no process list). No build has been + started: the binding pre-build queue check cannot yet be satisfied. From 5c44b40f4dff5be7f66b7c16144514e812652a3e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 23 Aug 2026 04:57:47 +0200 Subject: [PATCH 2/3] Fix rare signed-tail support and regime receipts Preserve the retirement carrier union under the unchanged training cap, persist exact realized QRF regimes through transfer receipts and checkpoints, and add frozen 48-check adjudication evidence.\n\nVerification: calibrate 201 passed; data 275 passed, 1 skipped; fit 93 passed; frame 294 passed, 36 skipped; build 5973 passed, 39 skipped; ruff check passed. --- FINAL_REPORT.md | 180 +- PROGRESS.md | 126 +- _LANE-NOTES.md | 270 +- .../battery-rare-signed-tails.fixed.md | 1 + docs/evidence/spec-engine/us-f0-coverage.json | 50 +- .../battery_rare_signed_tails/README.md | 40 + .../battery_rare_signed_tails/REPORT.md | 323 + .../baseline_1pct_projection.json | 135 + .../build_realized_regime_evidence.py | 554 ++ .../diff_1pct_failures.py | 76 + .../project_baseline_1pct.py | 132 + .../realized_regimes.json | 5876 +++++++++++++++++ .../run_1pct_offchain_build.sh | 85 + .../build/spec_engine/field_usage.py | 8 +- .../build/spec_engine/inventory_coverage.py | 10 +- .../spec_engine/schema/spine.schema.json | 2 +- .../src/microcosm/build/spec_engine/seeds.py | 80 +- .../microcosm/build/us/spec/imputation.yaml | 4 +- .../src/microcosm/build/us/spec/spine.yaml | 2 +- .../build/us_runtime/acs_transfer.py | 71 +- .../build/us_runtime/acs_transfer_bank.py | 53 +- .../src/microcosm/build/us_runtime/h5_io.py | 2 +- .../us_runtime/retirement_distributions.py | 143 +- .../build/us_runtime/stacked_spine.py | 171 +- .../us_runtime/us_late_producer_registry.py | 9 +- .../tests/test_spec_engine_country_bundles.py | 4 +- .../tests/test_spec_engine_coverage_tool.py | 10 +- .../tests/test_spec_engine_field_usage.py | 10 +- .../test_spec_engine_identity_contracts.py | 2 +- .../tests/test_spec_engine_legacy_adapter.py | 2 +- .../tests/test_spec_engine_loader.py | 2 +- .../tests/test_spec_engine_seeds.py | 15 + ...spec_engine_stacked_authority_semantics.py | 4 +- .../tests/test_us_acs_transfer.py | 212 +- .../tests/test_us_exact_k_ladder_e2e.py | 2 +- .../tests/test_us_late_producer_dag.py | 5 +- .../tests/test_us_multispine_pool.py | 3 + .../tests/test_us_multispine_pool_h5_io.py | 45 +- .../tests/test_us_multispine_pool_tool.py | 142 +- .../tests/test_us_retirement_distributions.py | 267 + .../tests/test_us_spec_bundle.py | 2 +- .../tests/test_us_stacked_spine.py | 187 +- tools/build_us_multispine_pool.py | 26 +- tools/spec_engine_coverage.py | 2 +- 44 files changed, 9030 insertions(+), 315 deletions(-) create mode 100644 changelog.d/battery-rare-signed-tails.fixed.md create mode 100644 experiments/battery_rare_signed_tails/README.md create mode 100644 experiments/battery_rare_signed_tails/REPORT.md create mode 100644 experiments/battery_rare_signed_tails/baseline_1pct_projection.json create mode 100644 experiments/battery_rare_signed_tails/build_realized_regime_evidence.py create mode 100644 experiments/battery_rare_signed_tails/diff_1pct_failures.py create mode 100644 experiments/battery_rare_signed_tails/project_baseline_1pct.py create mode 100644 experiments/battery_rare_signed_tails/realized_regimes.json create mode 100755 experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh diff --git a/FINAL_REPORT.md b/FINAL_REPORT.md index 74cdc96d4..912b32e1b 100644 --- a/FINAL_REPORT.md +++ b/FINAL_REPORT.md @@ -1,111 +1,81 @@ -# Final report: microcosm #462 register alignment +# Final report: battery blocker lane — rare signed tails ## Outcome -Completed the split-PR remediation on `loss-contract-alignment`, based on -`origin/main` at `7b6e10b`. The change is now register alignment only: one -shared critical-target register, one shared congressional-district classifier, -two consumers, builder contract-row gating, and behavioral containment of the -publish contract. - -The critical-row loss multiplier was removed entirely per -[microcosm#492](https://github.com/PolicyEngine/microcosm/issues/492). There is no -constant, CLI option, validation, loss overlay, telemetry, diagnostics/scorer -provenance, or historical replay pin left. `_fiscal_target_loss_weights` is -source-identical to `origin/main`, and its output therefore preserves main's -bit-level behavior for the same registry and family multipliers. - -## Sol round-1 findings - -1. **Table 1.4 selector parity:** removed the builder-only - `accepted_name_prefixes=("irs_soi.",)` constraint. The adapter now has - exactly the shared requirement's substring and suffix selectors. The - outside-prefix reproduction is builder-rejected. -2. **Congressional-district parity:** added exported, stdlib-only - `is_congressional_district_target(name, metadata)` and made the publisher - and builder classifiers thin wrappers. It ORs layout dimension, source-id - token, geography level, geography scope, truthy CD GEOID, and name token. - The builder's exact/semantic, Table 1.4, and zero-support paths now see the - same registry metadata. -3. **Recorded relative-error shape:** a matched row with missing/`None` - `relative_error` now fails with the publish-contract message instead of - silently passing after recomputation. Existing non-numeric and stale-value - checks remain. -4. **Behavioral anti-drift:** the load-bearing test now runs adversarial rows - through both consumers for exact-name, family+role, Table pattern, - missing/non-finite values, and a disallowed incumbent escape at the 0.25 - hard stop. A production Ledger compile supplies six separate CD evidence - rows; builder and publisher exclude identical six-name sets and counts. - Field comparisons remain as fast checks, and any added conjunctive prefix - is proven to trip the guard. - -The [#490](https://github.com/PolicyEngine/microcosm/issues/490) medical 0.25 -adjudication tolerance and its adjacent comment in `us_critical_targets.py` -remain byte-for-byte unchanged, as required. - -## Reproduction receipts - -The Table 1.4 prefix reproduction now returns: - -```text -SOI Table 1.4 national dollar fit failed: other.table_1_4.all.bad_amount@2024: relative_error=1 exceeds 0.25 for SOI Pub 1304 Table 1.4 national dollar rows (soi_table_1_4_national_dollar_rows); target=100.0, final_estimate=200.0. -``` - -The missing-relative-error reproduction now returns: - -```text -SOI Table 1.4 national dollar fit failed: irs_soi.ty2023.table_1_4.all.adversarial_amount@2024: missing recorded relative_error; the publish contract requires a numeric value. -``` - -The CD reproduction has the owner-mandated exclusion result: - -```text -builder_excluded=True -publisher_excluded=True -builder_failures=[] -``` - -Calling that row "rejected" would contradict the required OR-union exclusion -semantics. The two malformed critical rows are rejected; the CD row is -symmetrically excluded by both consumers. +The lane is complete locally. All 48 frozen red QED checks were classified +from frozen donor support, the one proven generating-code defect was fixed, +and exact realized-regime provenance is now durable. The branch was not pushed +and no pool build or logbook-chain operation was started. + +The full adjudicated report is +`experiments/battery_rare_signed_tails/REPORT.md`; the machine-readable +48-check matrix is `experiments/battery_rare_signed_tails/realized_regimes.json`. + +## Findings and changes + +- All 48 QED checks recompute to gated fits: 35 + `zero_inflated_positive` and 13 `three_sign` at check level (35/7 across 42 + targets). None is degenerate or single-sign, so regime substitution or + sign-conditional draw logic is not an honest fix. The QRF regime machinery + is at `packages/microcosm-fit/src/microcosm/fit/qrf.py:92-150,950-1003, + 1333-1380` (authority/main numbering). +- Transfer ownership is 17 early gap-fill checks and 31 late + producer-complement checks. Five intact sparse donors remain evidence + blockers: ordinals 16, 28, 33, 46, and 75. Their verified sign-carrier + counts are +18, +61, +27, -89, and -48; all exceed the five-carrier QED + support floor (`stacked_spine.py:3027-3031`). +- Ordinals 78/80/82 and the Keogh leg share upstream retirement-support + deletion. The old uniform 5,000-row cap retained 102/2,057 401k carriers, + 4/161 403b, 2/61 SEP, and 0/2 Keogh. The fixed helper retains the union of + all four targets' nonzero rows, samples only common zeros, and calibrates + only sampled-zero weights + (`retirement_distributions.py:337-450,486-584`). The cap and seed values did + not move. +- Realized QRF regimes now survive ordinary and banked transfer + (`acs_transfer.py:1348-1418,1813-1823`), target-bank persistence + (`acs_transfer_bank.py:25,47-53,249,357`), and exact early/late/aggregate + receipts (`stacked_spine.py:3754-3772,8215-8360,9260-9268,9315-9331`). + Authentic canonical JSON with sorted regime-map keys resumes correctly; + missing target keys still fail closed. + +## Keogh disposition + +Keogh is not structurally absent. Native ASEC contains positive values 2,040 +and 30,000; the old cap dropped both before transfer fitting, and all 1,736,840 +finite frozen bank draws are zero. ACS absence was therefore manufactured on +the transfer donor path and is fixed by preserving source support. + +The declared-absence route is rejected. If the signal had been structurally +impossible, it would require an exact recipient-scope absence equation +(`stacked_spine.py:8168-8195`) or a canonical tolerated-absence receipt +(`stacked_spine.py:6491-6528`), plus clone-exact battery structural receipts +(`stacked_spine.py:7186-7205`). Approval belongs to the US pool owner through +the owner-only reviewed route; this lane added no exclusion. ## Verification -The requested suite ran with `UV_NO_SYNC=1` to use the already-synced workspace -environment in the network-restricted sandbox: - -```text -uv run --package microcosm-build --extra us --group dev python -m pytest packages/microcosm-data/tests packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py packages/microcosm-build/tests/test_us_state_files_scorer.py -q -264 passed, 3 skipped (267 collected) -``` - -Additional receipts: - -- Complete `test_gates.py`: passed. -- Required multiplier grep: zero Python hits. -- Ruff check: clean on all ten touched Python files. -- Ruff format check: clean on the eight non-exempt touched Python files; the - two historical experiment files were not reformatted, as instructed. -- `git diff --check`: clean. -- The medical adjudication block compares byte-for-byte equal to pre-fix - commit `068854d`. -- Pytest emitted non-failing macOS temporary-directory cleanup warnings; no - test failed. - -## Remediation commits - -- `5077f95` — start microcosm#462 Sol remediation progress. -- `c48ba37` — remove the microcosm#462 loss multiplier per microcosm#492. -- `afa910a` — fix Sol finding 1 selector parity. -- `89f74f4` — fix Sol finding 2 CD classifier parity. -- `77040fb` — fix Sol finding 3 relative-error shape. -- `bad7145` — fix Sol finding 4 behavioral containment. -- `3c96514` — apply the finding-2 classifier's required Ruff formatting. - -Nothing was pushed at the time of this report; the branch was subsequently -pushed and merged as #491 (2026-07-22). - -The sandbox rejected writing -`/Users/maxghenis/PolicyEngine/_reviews/sol-491-fix-out.md` with `Operation not -permitted`; the full completion report is therefore committed here and will be -printed to stdout as the requested fallback. +- calibrate: 201 passed. +- data: 275 passed, 1 skipped. +- fit: 93 passed. +- frame: 294 passed, 36 skipped. +- build: 5,973 passed, 39 skipped. +- repository Ruff: all checks passed. +- US bundle generator check: passed, bundle SHA `5b0014c3…9554`. +- frozen regime-evidence generator check: passed. + +The existing 1% before artifact projects to 127 failure lines over 93 legs; +47/48 frozen QED reds are visible. The committed baseline projection reproduced +byte-for-byte, and its self-diff is empty. No actual after-build exists for +this branch because the binding headless instruction forbade this lane from +starting one. The host-queue owner must run +`experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh` and feed its +gates file to `diff_1pct_failures.py`. Keogh itself requires a full-scale build +because a 1% ASEC sample almost surely contains neither native carrier. + +## Remaining owner work + +Forty intact-support shape failures remain blocked on held-out calibration or +mapping evidence, and five sparse-donor failures remain blocked on denser +evidence. REPORT §2 gives the smallest target-scoped generating-mechanism +change for each of the nine classes. None calls for a gate, band, ceiling, +floor, fold, seed, cap-value, or exclusion change. diff --git a/PROGRESS.md b/PROGRESS.md index bfcc60e6b..d3f9dfe36 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -2,56 +2,92 @@ ## State -Lane initialization is in progress on `battery-rare-signed-tails` at -`2c7a7218`, matching `origin/main`. No product code, gate, band, ceiling, fold, -seed, exclusion register, logbook chain, or build artifact has been changed. -The authoritative frozen inputs are the arm-split lane's +Complete on `battery-rare-signed-tails`, based on `2c7a7218`. The mechanism +fix, frozen-evidence classification, receipt closure, regression coverage, +full per-package test suite, Ruff check, journal, and final reports are in the +solution commit containing this file. No pool build was started by this lane. + +No gate, incidence band, QED ceiling, support floor, training-cap value, fold, +seed value, exclusion register, logbook chain, or pending-chain file changed. +The host queue still owns the off-chain 1% after-build. The frozen 1% before +projection and deterministic diff tooling are committed, but an actual after +artifact does not exist for this branch. + +Authoritative inputs are the arm-split lane's `experiments/battery_burndown/ADJUDICATION.md` and `adjudication.json`. +Outputs are: -The mandated `uv sync --all-packages --extra us` was attempted first. The -default cache is inaccessible in the managed sandbox; a retry with a writable -cache reached resolution but could not download `pandas==3.0.3` because network -DNS is unavailable. The primary-checkout environment proved sufficient for the -fit, calibrate, data, and frame shards but does not carry the build shard's -`jsonschema` dependency. Full verification therefore uses the compatible, -already-synced `/Users/maxghenis/PolicyEngine/_worktrees/microcosm-f1/.venv` -read-only, with `UV_NO_SYNC=1` and this worktree's package source roots on -`PYTHONPATH`. +- `FINAL_REPORT.md` — requested completion output; +- `experiments/battery_rare_signed_tails/REPORT.md` — full code-cited report; +- `experiments/battery_rare_signed_tails/realized_regimes.json` — all 48 + check classifications and smallest honest remedies; +- `experiments/battery_rare_signed_tails/baseline_1pct_projection.json` — + frozen 1% before side. ## Done -- Read `CLAUDE.md` and the GitNexus debugging workflow. -- Confirmed the branch starts at `origin/main` and recorded the pre-existing, - untracked `.gitnexus/` directory without deleting or staging it. -- Attempted the required GitNexus symptom query. Its local index is current, - but the global registry does not contain this worktree and is outside the - writable sandbox; source tracing remains the authority. -- Located the frozen adjudication and established `FINAL_REPORT.md` as the - committed output report because no separate output-file environment variable - is present. -- Confirmed the shared environment imports this lane's `microcosm.frame` from - the lane-local source tree and provides pytest, Ruff, pandas, and NumPy. -- Read the adjudication's comparator-mechanisms section in full and traced its - cited incidence-first, five-carrier QED-skip, early-gap-fill, late-producer, - donor-projection, regime-detection, and declared-absence seams. -- Recomputed all 48 red QED donor regimes from the frozen checkpoint: every - availability pattern is gated (35 zero-inflated-positive targets and 13 - three-sign targets); none is degenerate or single-sign. -- Recomputed the five rare-tail physical legs on their actual route donors: - collectibles +18, alimony +61, casualty +27, farm operations -89, and - prior-year self-employment -48 sign carriers. -- Established that Keogh is not structurally absent: native ASEC has two - positive values (`2,040` and `30,000`), but the forced late ASEC clone-1 donor - is degenerate zero and its 1,736,840 finite banked recipient draws are all - exactly zero. +- Initialized and committed the lane journal before product work + (`a5c7310c`). Inspected all four salvage snapshots (`6597ca87`, `8131e7af`, + `930a3de1`, and newest `bfe794ad`); the newest snapshot supplied the + overlapping working state, while older broad/unrelated material was not + recovered. Exact disposition is in `_LANE-NOTES.md`. +- Completed `uv sync --all-packages --extra us` using the writable lane cache. +- Read the comparator-mechanisms authority in full and recomputed every frozen + fit-boundary regime: 48/48 checks are gated (35 + `zero_inflated_positive`, 13 `three_sign`; 42 targets: 35/7), with no + degenerate or single-sign QED target. Verified all 42 banked-draw sign + surfaces against those regimes. +- Classified the 48 checks into nine mechanism classes across fitter regime, + early/late transfer ownership, and donor support. Five checks (16/28/33/46/75) + remain additional-evidence starvation blockers; 40 remain owner-calibration + blockers; three retirement checks (78/80/82) share the fixed support-deletion + mechanism. +- Proved Keogh is not structurally absent: native ASEC has two positive + carriers (`2,040`, `30,000`), the old uniform 5,000-row cap retained 0/2, + and all 1,736,840 finite frozen bank draws are zero. Implemented a + support-preserving carrier-union cap with common-zero weight calibration; + cap and seed values remain unchanged. +- Persisted exact realized QRF regimes through ordinary/banked transfer, + early/late/aggregate receipts, H5 identity, and fail-closed checkpoint + validation. Advanced only the semantic receipt/materializer identities + needed to invalidate regime-free or support-deleting artifacts. +- Fixed canonical-JSON resume validation so sorted regime-map keys do not make + an authentic checkpoint appear corrupt; missing target keys still fail + closed. Added the serialization regression. +- Regenerated and checked all source-attested bundle/coverage identities. + `generate_us_bundle_from_constants.py --check` reports US bundle + `5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554`; + coverage is 41,381/41,381 fields and 40/40 inventory checks. +- Reproduced the frozen baseline projection byte-for-byte: 127 failure lines, + 93 legs, and 47/48 red QED checks visible at 1%. A baseline-vs-baseline diff + is exactly empty. Keogh and ordinal 16 are too sparse to be visible at 1%. + +## Suite receipts + +Final source tree, one pytest process per package shard: + +- `microcosm-calibrate`: 201 passed; 2 warnings; 36.25s. +- `microcosm-data`: 275 passed, 1 skipped; 37.50s. +- `microcosm-fit`: 93 passed; 1 warning; 249.72s. +- `microcosm-frame`: 294 passed, 36 skipped; 1 warning; 888.52s. +- `microcosm-build`: 5,973 passed, 39 skipped; 1,941 warnings; 7,858.53s. +- `uv run ruff check .`: all checks passed. +- `tools/generate_us_bundle_from_constants.py --check`: passed. +- `experiments/battery_rare_signed_tails/build_realized_regime_evidence.py + --check`: passed. + +The first complete build-shard run exposed one stale derived-payload test pin; +the authoritative generator was byte-identical, the structural schedule hash +was unchanged, the test pin was corrected to the live derived hash, its focused +node passed, and the complete build shard above was rerun green. ## Next -- Finish the per-check mechanism/route matrix and record exact code citations. -- Add future-checkpoint regime provenance and the smallest non-threshold fix for - the Keogh support loss, with mechanism-specific regression tests. -- Preserve the adjudication's BLOCKER posture for sparse tails whose frozen - donors remain sign-capable but do not contain dense enough evidence for an - honest target-specific refit. -- Guard any off-chain 1% build with the host queue/RSS checks and compare exact - before/after failure lines against the frozen arm-split baseline. +- Host-queue owner: run the committed 1% off-chain command, then compare its + `pool.gates.json` with the frozen baseline using `diff_1pct_failures.py`. +- Owner-scheduled full-scale build: verify the Keogh gate-level flip, which a + 1% sample cannot observe. +- Owners of the seven blocked shape families: supply held-out/dense evidence + before the target-scoped remedies in REPORT §2 are implemented. +- Owners of the five intact-but-sparse donors: supply a dense rung or approve + a target-specific sparse-tail model. No exclusion is proposed. diff --git a/_LANE-NOTES.md b/_LANE-NOTES.md index 194955a85..7c9c8d3be 100644 --- a/_LANE-NOTES.md +++ b/_LANE-NOTES.md @@ -34,7 +34,7 @@ be appended here as they are established. pattern at a given entity uses the same complete donor identity (person 108,073; tax unit 57,630; SPM unit 43,961). - Recomputing signs at `zero_atol=1e-6` gives 35 - `zero_inflated_positive` targets and 13 `three_sign` targets. There are no + `zero_inflated_positive` checks and 13 `three_sign` checks. There are no degenerate or single-sign QED targets, so a regime substitution cannot explain or honestly repair any of the 48 QED failures. - The route partition is 17 early ASEC-to-ACS gap-fill QED checks and 31 late @@ -65,3 +65,271 @@ be appended here as they are established. - Both `ps` and `pgrep -af build_us_multispine_pool` are denied by the managed macOS sandbox (`operation not permitted` / no process list). No build has been started: the binding pre-build queue check cannot yet be satisfied. + +## 2026-08-22 — salvage disposition + +- Inspected `git show --stat 6597ca87` and the complete diff against HEAD before + altering the salvaged working tree. The salvage combined a sound narrow + retirement fix with a much broader, unverified receipt-closure rewrite and + stale generated hashes; cherry-picking it whole was not suite-safe. +- Recovered and then minimized: + - the carrier-union retirement cap, common-zero weight calibration, and its + mechanism regressions; + - the seed-protocol description of the changed draw consumption; + - exact QRF-regime capture for ordinary and banked ACS transfers; + - bank schema 2, late receipt schema 4, legacy/pool/stacked materializer + invalidation, and the focused receipt regressions. +- Discarded: + - `.gitnexus/` binary/index material (moved recoverably to + `/private/tmp/microcosm-rare-signed-tails-gitnexus-salvage`; the salvage ref + also retains it); + - the descriptor-plan and active-target indexing rewrite because it changed + target-bank execution beyond the proven signed-tail defect; + - the broad target-row/group, routed-wrapper, and sidecar validators and their + tests because they were unrelated to regime persistence and introduced + fixture failures; + - generated bundle/coverage pins and final-report claims, which described the + broad salvage tree rather than this selected implementation. +- Regenerated every retained source-attested hash from the selected tree. No + salvage pin was copied. + +## 2026-08-22 — corrected frozen classification + +- All 48 red QED checks have four availability patterns, not one. The 17 early + person checks use `pattern_00_677f6490`, `pattern_01_5874881e`, + `pattern_02_7c3bceda`, and `pattern_03_04f75638`; the 28 late person checks + use `pattern_00_36785ebf`, `pattern_01_c6777728`, `pattern_02_5e7dd311`, and + `pattern_03_76a0101a`; the three late group-entity checks use + `pattern_00_741356db`, `pattern_01_7c62ad55`, `pattern_02_d9f964c9`, and + `pattern_03_2e5c739d`. +- Every pattern for a target binds the same complete donor identity: person + 108,073 rows/index SHA + `8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2`, + tax unit 57,630 rows/SHA + `da2a8ee2216d1b7ffc77f19d0bf8e74978a8a5d9e46c6ccae7d13e15cb9a2fb0`, + or SPM unit 43,961 rows/SHA + `371654a8deb82ed2dc168c65729647e565b0044e06a6a03ce0d0976d17466865`. +- The correct counting identity is 48 checks, 42 unique targets, 168 unique + target-pattern records, and 192 check-pattern links. At check level the + regime tally is 35 zero-inflated-positive/13 three-sign; at target level it + is 35/seven. No target is degenerate or single-sign. +- Five rows remain additional-evidence starvation blockers: ordinals 16, 28, + 33, 46, and 75. QED ordinals 78, 80, and 82 are separately co-classified as + upstream support starvation caused by the retirement training cap: the old + sample retained 102/2,057, 4/161, and 2/61 native positive carriers. Keogh + retained zero of two. +- `experiments/battery_rare_signed_tails/realized_regimes.json` records every + row, its four authenticated patterns, fit-boundary support, route, realized + regime, terminal carrier/QED values, mechanism class, and smallest honest + owner-scoped remedy. Its generator refuses changed authority hashes or count + identities. +- The retrospective schema-1 bank proves target/pattern identity, predictors, + donor index, weights, and `zero_atol`, but did not persist regimes. Counts are + explicitly labeled fit-boundary reconstructions. Final transferred values are + not a valid substitute for early fits because later producers can mutate + those fields; for example, rental-income fit support is 216/104,057/3,800, + while the final transferred ASEC clone-1 values are + 2,259/101,987/3,827. + +## 2026-08-22 — selected mechanism implementation + +- The retirement training cap now retains every row nonzero for any of its four + targets, samples only the shared all-target-zero stratum under the unchanged + 5,000-row cap and seed, preserves carrier weights, and ratio-calibrates only + the sampled-zero weights. It fails closed when the carrier union leaves no + zero-stratum slot. +- Ordinary and banked ACS transfers now bind every fitted target to a known QRF + regime. Target-bank schema 2 persists that label in each transition and + projects it into operational receipts; schema-1 artifacts rebuild. +- Early direction, late group, and aggregate late receipts expose the same + fitted-pattern regime rows. The canonical aggregate is validated as the exact + concatenation of the 19 group receipts. +- Receipt/materializer identities advance to late receipt 4, legacy checkpoint + 4, pool checkpoint 8, and stacked checkpoint 12. The canonical H5 loader was + updated in lockstep to accept newly emitted legacy materializer 4. +- Source-attested outputs were regenerated from this selected tree: US bundle + SHA `d30642501b9b21cd4510492ebbc931e25f4bfd3576db0f96de4edded1aac8c72`; + F0 field coverage 41,381/41,381; inventory coverage 40/40. +- Focused regressions cover production-call retention of both Keogh carriers, + exact carrier/common-zero weight semantics, invalid/over-cap inputs, regime + label closure, cold/warm bank persistence, schema-1 rebuild, authentic late + aggregation, legacy H5 compatibility, and tamper rejection. Selected Ruff, + H5 loader, field-usage, and coverage checks are green. + +## 2026-08-22 — third-salvage disposition and independent artifact verification + +- Salvage refs two (`8131e7af`) and three (`930a3de1`) are byte-identical + trees; the pre-session working tree (including the untracked + `experiments/battery_rare_signed_tails/` files) matched salvage three + exactly, so recovery required no cherry-picks. Nothing was discarded. +- This environment restored `ps`/`pgrep` and network. The binding + `uv sync --all-packages --extra us` completed in-worktree, replacing the + earlier read-only borrowed-venv arrangement. +- Independent frozen-artifact verification performed this session (not + inherited from the killed attempts): + - All 42 target checkpoints' banked draws are sign-consistent with the + recorded realized regimes at `zero_atol=1e-6`; zero inconsistencies. All + seven `three_sign` targets show both negative and positive decoded draws, + proving both signed classes existed in their fit donors. + - Keogh bank draws: 1,970,973 stored, 1,736,840 finite, zero nonzero, + `max_abs = 0.0` — the fitted transfer regime was degenerate. + - Transferred-frame census: ASEC clone 0 keogh has exactly two nonzero + values `{2,040, 30,000}` over 108,073 rows; ASEC clone 1 and every ACS + clone are all-zero. The ACS absence is manufactured upstream, not + structural. + - The five starvation blockers' donor sign-carrier counts verify exactly: + collectibles +18 (ASEC clone 1) / +82 native, casualty +27 / +79 native, + alimony +61 native, farm operations −89 native, prior-year + self-employment −48 native. +- Citation correction: the pool/stacked/legacy materializer version constants + live in `tools/build_us_multispine_pool.py:231,282,319` (not + `stacked_spine.py`); `h5_io.py:103` mirrors legacy 4 and + `spine.yaml`/`spine.schema.json` pin stacked 12. +- The retirement cap parameter itself is untouched: `sources.yaml:2138` + still declares `max_train_samples: 5000` and `sources.yaml` is not + modified on this branch. Only the sampling design inside the cap changed. + +## 2026-08-22 — final verification session (headless resume) + +- Resumed on the intact working tree (matches salvage three; nothing reset). + `ps` shows no `build_us_multispine_pool` process; the host queue log only + ever waits on disk for its own `pkg1` step. No build was started from this + lane at any point. +- Formatting: `ruff format` was applied to exactly the five files whose + format drift this branch introduced (`acs_transfer.py`, + `acs_transfer_bank.py`, `test_us_acs_transfer.py`, + `test_us_multispine_pool_tool.py`, `test_us_retirement_distributions.py`); + merge-base copies of every other touched file were already unformatted on + `origin/main`, and the CI gate is `uv run ruff check .` (green), so those + were left untouched. +- **Stale first-batch attestation pins found and corrected.** The previous + sessions regenerated source-attested identities in two batches and the + first batch described a tree state that no longer exists. On the final + selected tree the following pins were recomputed through the same code + paths the tests use and corrected: + - BE bundle `spec_sha256` → `91d3a58e669950c7fab0aefeb74687278efca82d413a900e9ab1878b09a7a5a3`, + UK → `54c0b70167b9d36e857a116ca4c80c74af142b6f09d5894ee979de62b9634738` + (`test_spec_engine_country_bundles.py`); the shared `seeds.py` + retirement-site declaration feeds every country bundle. + - Loader golden semantic-hash vector → + `43989f610f9691daf69e9ee74c1aa17f2a8bc3f07b9992489c65fa8b1ad747de` + (`test_spec_engine_loader.py`). + - Inventory `seed_protocol` → + `d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be`, + `seed_map` → + `f9e71d22a4c8608d1e2ca99912f60dd06272e785e0babcca46a452c1a6df9562` + (`inventory_coverage.py` EXPECTED_HASHES). + - US bundle `spec_sha256` is + `5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554` on the + final tree. The earlier journal claim of `d30642501b9b…` described the + intermediate first-batch tree and is historicized here; the committed + report and `test_us_multispine_pool_tool.py` pins were corrected. + - `docs/evidence/spec-engine/us-f0-coverage.json` was regenerated with + `uv run python tools/spec_engine_coverage.py` — 41,381/41,381 + configuration fields, 40/40 inventory checks, exit 0. + - `field_usage.py` pins and the identity-contract tests already matched the + final tree (second-batch regeneration) and were not touched. + - Two further cross-pins of the moved stacked-authority digest surfaced in + the full-shard run and were corrected the same way: + `test_spec_engine_legacy_adapter.py:200` (aggregate `34e86d1e…`) and + `test_spec_engine_stacked_authority_semantics.py:77,93-95` (aggregate + plus its `late_producer_schedule` component `bf95c78e…` → + `870c38d740cf92d6ce7a96cac4c44864b03d87634d59177d701fecf3b2d94436`; all + other components verified unchanged against the live projection). A + repo-wide sweep for every pre-move value then returned zero hits. + - A third same-class find: the exact-k ladder e2e downstream-consumer + fixture fabricated a legacy publication manifest with + `materializer_version: 3` (`test_us_exact_k_ladder_e2e.py:140`), which + the branch's loader now correctly rejects (that rejection is itself a + branch regression test). The fixture now models the current v4 + publication; the file re-ran green (3/3). A repo-wide sweep found no + other stale version literals in test fixtures. + - Suite-receipt composition note: the full build-shard run was launched + before these three expectation fixes; its F marks (collection indices + 847, 945, and 2993-2995) are exactly those tests. All three files were + re-run green in isolation on the final tree (6/6, 4/4, 3/3), and no + product source changed between the runs, so the shard receipt composes: + full run green everywhere else + isolated green for the three fixed + expectation files. +- **Keogh leg identity correction:** the terminal leg is + `person/source_operator_retirement_distributions/keogh_distributions[clone_0]/positive` + (not `taxable_keogh_…`). Full-scale f025 emits + `weighted positive-leg incidence ratio 0 is outside [0.8, 1.25] + (asec=1.43573e-05, acs=0)`, confirming the adjudication: incidence fails + before the five-carrier floor skips only the sign's QED + (branch `stacked_spine.py:12068-12087`, floor at `3027-3031`). +- **Frozen-artifact re-verification (this session's own receipts):** the + Keogh target bank (`late_producer_dag/person/source_operator_retirement_distributions/targets/000__keogh_distributions.h5`) + decodes to 1,970,973 stored / 1,736,840 finite draws with zero nonzero and + `max_abs = 0.0`; the transferred checkpoint's person + `keogh_distributions` column holds exactly two nonzero values + `{2,040, 30,000}` across all 1,970,973 pooled person rows. Both match the + prior sessions' censuses exactly. +- **Baseline 1% projection** (`baseline_1pct_projection.json`, generated by + `project_baseline_1pct.py` against the host-built + `battery-verify/baseline1pct/pool.gates.json`): 127 failure lines over 93 + legs; 47 of the 48 frozen reds are red at 1%; ordinal 16 (collectibles + positive) is the only frozen red without a 1% line; the Keogh leg emits no + 1% line in either direction because a 1% ASEC sample almost surely carries + neither of the two native carriers. Gate-level Keogh proof therefore + requires a full-scale build; the 1% after-build proves the 401k/403b/sep + starvation repairs and overall non-regression instead. + +## 2026-08-22 — newest salvage, completion audit, and final receipts + +- Inspected newest snapshot `bfe794ad` with `git show --stat` and compared it + with HEAD and the three older snapshots. It supersedes `930a3de1`, + `8131e7af`, and `6597ca87` wherever they overlap. The pre-existing tracked + working state matched the newest selected implementation, and its experiment + and changelog files were present untracked, so no cherry-pick or reset was + needed. Retained: the narrow retirement carrier-union fix, exact regime + receipts/materializer invalidation, regressions, and experiment evidence. + Discarded from older overlapping snapshots: `.gitnexus` binary state, + descriptor/active-target rewrites, and unrelated broad validator work. +- The binding sync completed with a writable cache: + `UV_CACHE_DIR=/private/tmp/microcosm-rare-signed-tails-uv-cache uv sync + --all-packages --extra us` → `Checked 100 packages`. +- Re-ran `build_realized_regime_evidence.py --check`; all authority hashes, + 48/42/168/192 identities, pattern IDs, donor lengths, and donor hashes are + current. An independent raw-draw audit of all 42 target checkpoints found 35 + zero-inflated-positive and seven three-sign targets, zero regime/sign + inconsistencies, and both signed classes in every three-sign target. +- Re-censused the frozen retirement mechanism from the selective H5 columns: + native ASEC clone 0 has exactly two Keogh carriers `{2,040, 30,000}`; ASEC + clone 1 and every ACS clone are zero. Replaying the old seeded uniform cap + retained Keogh 0/2, 401k 102/2,057, 403b 4/161, and SEP 2/61. The five intact + sparse route donors remain +18/+61/+27/-89/-48 carriers. +- Found one resume-validation defect in the newest salvage: canonical JSON is + written with sorted mapping keys, but the regime validator compared mapping + iteration order with model-target order. A valid durable checkpoint therefore + rebuilt unnecessarily. `stacked_spine.py` now checks exact key-set equality; + the separately persisted `model_targets` list remains the ordering authority. + `test_realized_regime_receipt_survives_canonical_json_key_sorting` proves a + sorted authentic map resumes and a missing target fails closed. The exact + durable-resume boundary test also passes. +- The first complete build-shard run found one remaining stale source-attested + *test assertion*: `late_producer_schedule_receipt.payload_sha256`. The + structural `schedule_sha256` stayed unchanged. The authoritative US bundle + generator passed byte-for-byte at bundle SHA `5b0014c3…9554`; the derived + payload assertion was updated from `7766f2e…` to live `6af0cc77…`, its + focused node passed, and the entire build shard was rerun green. +- Final mandatory verification on the same source tree: + - calibrate: 201 passed, 2 warnings, 36.25s; + - data: 275 passed, 1 skipped, 37.50s; + - fit: 93 passed, 1 warning, 249.72s; + - frame: 294 passed, 36 skipped, 1 warning, 888.52s; + - build: 5,973 passed, 39 skipped, 1,941 warnings, 7,858.53s; + - repo-wide Ruff: `All checks passed!`; + - US bundle generator `--check`: passed; + - realized-regime evidence generator `--check`: passed. +- Regenerated the 1% baseline projection into `/private/tmp`; it is + byte-identical to the committed projection (127 failure lines, 93 legs, + 47/48 frozen QED reds visible). Baseline-vs-baseline use of the diff tool + reports 127 shared and zero added, removed, or changed lines. +- No branch-specific after artifact exists. The host path named `pkg3` belongs + to the unrelated `microcosm-pkg3-two-part` worktree, and the queue contains no + rare-signed-tail step; using it would fabricate evidence. Current managed + `ps`/`pgrep` calls are sandbox-denied, superseding the earlier historical + process-visibility note. No pool build was started, no chain input was used, + and `logbook-pending-chain.txt` was never touched. The actual 1% after diff + and the full-scale Keogh gate proof remain explicit host-owner actions. diff --git a/changelog.d/battery-rare-signed-tails.fixed.md b/changelog.d/battery-rare-signed-tails.fixed.md new file mode 100644 index 000000000..750646322 --- /dev/null +++ b/changelog.d/battery-rare-signed-tails.fixed.md @@ -0,0 +1 @@ +The US retirement-distribution training cap now retains every rare signed-tail carrier (fixing the manufactured one-sided Keogh leg and the 401k/403b/SEP starvation reds), and ACS-transfer provenance, target-bank schema 2, and late receipts persist each fitted target's realized QRF regime with fail-closed validation. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 7dde306af..8458de947 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -704,8 +704,8 @@ "legacy_sinks": [], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 823, - "pointer_sha256": "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + "pointer_count": 824, + "pointer_sha256": "82eeb3578054e6eab7e714571ba1534a641814f10f57ddea46bbee9f2095ce76", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/resolved/seed_protocol", @@ -740,21 +740,21 @@ "verifier": "vintages" } ], - "configuration_field_count": 41380, - "consumed_field_count": 41380, + "configuration_field_count": 41381, + "consumed_field_count": 41381, "generation0_effect_counts": { "legacy_behavior": 37718, - "no_generation0_effect": 3662 + "no_generation0_effect": 3663 }, "mode_counts": { - "compiler_semantic": 27661, + "compiler_semantic": 27662, "front_end_validation": 346, "identity_only": 128, "legacy_behavior": 13245 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "b63ffc9bf7cb2fb8ac17e314f64c7b661aba14512071c1a697e5f6ff9ebc74c5", - "resolved_binding_field_count": 9219, + "pointer_inventory_sha256": "85fb15ef978600029ba69850659fd190e2919bc8b9dcbbf527f2215eb0c79345", + "resolved_binding_field_count": 9220, "unused_field_count": 0 }, "inventory_coverage": { @@ -1289,7 +1289,7 @@ "expected": "complete execution-row and transition-authority object", "failures": [], "observed": { - "sha256": "503428f6e9d98f19ed3a6ada5bc9883ae44c1b5dc27f09e60b7ac98895a99bc0" + "sha256": "8f4f2ac2ac21d6bea7d3f80a2388489d3d93ba35cbc07cee282c9bc9f0419451" }, "status": "covered" }, @@ -1329,12 +1329,12 @@ ], "expected": { "producer_count": 38, - "sha256": "9206aa072ce360ef87dc7d832351fcae2e0d1da71abcacc5af0d43ef88582dee" + "sha256": "e048fa629487c76c768f26b461d8346fb94e692b99b6bc673f79e342092a0ce4" }, "failures": [], "observed": { "producer_count": 38, - "sha256": "9206aa072ce360ef87dc7d832351fcae2e0d1da71abcacc5af0d43ef88582dee" + "sha256": "e048fa629487c76c768f26b461d8346fb94e692b99b6bc673f79e342092a0ce4" }, "status": "covered" }, @@ -1624,13 +1624,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "protocol_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec" + "map_sha256": "f9e71d22a4c8608d1e2ca99912f60dd06272e785e0babcca46a452c1a6df9562", + "protocol_sha256": "d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be" }, "failures": [], "observed": { - "map_sha256": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "protocol_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec" + "map_sha256": "f9e71d22a4c8608d1e2ca99912f60dd06272e785e0babcca46a452c1a6df9562", + "protocol_sha256": "d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be" }, "status": "covered" }, @@ -1645,7 +1645,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "implementation_sha256": "d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1666,7 +1666,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "implementation_sha256": "d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1704,7 +1704,7 @@ }, "failures": [], "observed": { - "sha256": "99710bf212c3801cfd0434d56d2d1d2e4902a2a958978cdbd91cd9acce81d2fb", + "sha256": "8094ada1cd81ae0b84adb90eefdc256557a080cd003058a02a6a9aaa353e5f7a", "sites": 53 }, "status": "covered" @@ -1806,11 +1806,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "sha256": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "sha256": "34e86d1e5b885e9fef604209bbe28755134b8d6551e38b5b61fb740e0b43e076" }, "failures": [], "observed": { - "sha256": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "sha256": "34e86d1e5b885e9fef604209bbe28755134b8d6551e38b5b61fb740e0b43e076" }, "status": "covered" }, @@ -1852,7 +1852,7 @@ "alpha", "zeta" ], - "sha256": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8" + "sha256": "f4515d69c804c737a9b0b42cc4c6d29db081751a6f9e73958e375fdcd47df2f3" }, "failures": [], "observed": { @@ -1875,7 +1875,7 @@ "alpha", "zeta" ], - "sha256": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8" + "sha256": "f4515d69c804c737a9b0b42cc4c6d29db081751a6f9e73958e375fdcd47df2f3" }, "status": "covered" }, @@ -1939,7 +1939,7 @@ "take_up_contract", "us_qbi_reconciliation_contract" ], - "sha256": "8df3775167f89e79b44cc3606b52b54d488edaee8fc34dd757163dd8b6510fa7" + "sha256": "f325800bb3179582f46cced9254b78106ffc8ddc01d66b3c0a8c0fc204084219" }, "status": "covered" }, @@ -2438,7 +2438,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "6e9dce8f0fd3e3f0101103a14d6a08ac8527b90b82d48fa8bad2c4cc70dbdfde" + "spec_sha256": "5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554" } }, "report_schema_version": 3, @@ -2448,7 +2448,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "6e9dce8f0fd3e3f0101103a14d6a08ac8527b90b82d48fa8bad2c4cc70dbdfde" + "spec_sha256": "5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554" }, "status": "pass" } diff --git a/experiments/battery_rare_signed_tails/README.md b/experiments/battery_rare_signed_tails/README.md new file mode 100644 index 000000000..64c0f1b6d --- /dev/null +++ b/experiments/battery_rare_signed_tails/README.md @@ -0,0 +1,40 @@ +# Rare signed-tail regime evidence + +[`realized_regimes.json`](realized_regimes.json) is the deterministic projection +of the 48 red quantile-envelope checks in the frozen f025 arm-split +adjudication. It records all three requested mechanism dimensions for every +check: + +- reconstructed donor sign support and realized fitter regime; +- early gap-fill versus late producer-complement ownership; and +- donor-support starvation, including the separately identified upstream + retirement-cap deletion. + +The result is 48 checks over 42 unique target checkpoints. Every target has four +recipient availability patterns, for 168 unique target-pattern records and 192 +check-pattern links. All patterns for a target bind the same complete donor +identity. At check level, 35 regimes are `zero_inflated_positive` and 13 are +`three_sign`; none is degenerate or single-sign. + +The schema-1 checkpoint metadata authenticates the target, pattern, predictors, +donor index, weight vector, and zero tolerance, but not the realized regime. +The output therefore labels its sign counts and regimes +`fit_boundary_reconstruction`. Do not infer an early target's support from the +final transferred pool because later producers can mutate that field. The +schema-2 target-bank change on this branch persists the fit result directly for +future builds. + +Rebuild or check the evidence with the frozen artifacts: + +```bash +python experiments/battery_rare_signed_tails/build_realized_regime_evidence.py \ + --adjudication-dir /Users/maxghenis/PolicyEngine/_worktrees/microcosm-arm-split/experiments/battery_burndown \ + --bank-root /Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/populace_us_2024_stacked_pool.checkpoints/stacked/9be8ecdf82356f38998e8b620ee36d9134f554fe89a8eafd8406f438e2b5aad6/acs-transfer/d20ba8f3d33d235485b5da720e7bbe8fc79a72fab7b14df4b9fe2fc72ad756c5 \ + --f025-gates /Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/populace_us_2024_stacked_pool.gates.json \ + --transferred-receipts /Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/populace_us_2024_stacked_pool.checkpoints/stacked/9be8ecdf82356f38998e8b620ee36d9134f554fe89a8eafd8406f438e2b5aad6/transferred.checkpoint.receipts.json \ + --check +``` + +The extractor refuses changed authority hashes, missing or extra QED checks, +changed pattern inventories, donor/support non-closure, or a changed 48/42/168 +counting identity. diff --git a/experiments/battery_rare_signed_tails/REPORT.md b/experiments/battery_rare_signed_tails/REPORT.md new file mode 100644 index 000000000..536e9ebfb --- /dev/null +++ b/experiments/battery_rare_signed_tails/REPORT.md @@ -0,0 +1,323 @@ +# Rare signed-tail battery lane — adjudicated report + +Branch `battery-rare-signed-tails`, cut at `2c7a7218` (`origin/main`). +Authority: the frozen arm-split adjudication +(`/Users/maxghenis/PolicyEngine/_worktrees/microcosm-arm-split/experiments/battery_burndown/ADJUDICATION.md` +and `adjudication.json`) — 48 red QED checks plus the one-sided Keogh leg. +Every line-number citation below is against this branch's tree unless marked +otherwise. No gate, incidence band, QED ceiling, support floor, training-cap +value, fold, seed value, exclusion register, or logbook chain changed on this +branch. The lane started no pool build; the host queue owns all builds. + +## 1. Classification of the 48 red checks + +[`realized_regimes.json`](realized_regimes.json) classifies every check on +the three mechanism dimensions the charter requires, recomputed from the +frozen fit boundary per the adjudication's remediation rule (recompute the +regime from frozen donor support for each availability pattern before any +regime-specific fix; ADJUDICATION.md §Comparator mechanisms, receipt-gap +bullet). The extractor +([`build_realized_regime_evidence.py`](build_realized_regime_evidence.py)) +refuses changed authority hashes, changed pattern inventories, donor/support +non-closure, or a changed 48-check/42-target/168-record counting identity. + +**Fitter regime (recomputed, not assumed).** At `zero_atol = 1e-6` every one +of the 48 checks realized a *gated* regime at its fit boundary: 35 +`zero_inflated_positive` and 13 `three_sign` (42 unique targets: 35/7). None +is `degenerate_zero`, single-sign, or sign-only. All 42 target checkpoints' +banked draws are sign-consistent with these regimes, and all seven +`three_sign` targets show both signed classes in their decoded draws. A +mis-realized regime therefore explains **zero** of the 48 failures, and no +sign-conditional draw change is honest for any of them — the sign machinery +engaged exactly as designed +(`packages/microcosm-fit/src/microcosm/fit/qrf.py:92-150,950-1003,1333-1380`, +main-numbering citations from the adjudication). + +**Transfer partition.** 17 checks are early ASEC→ACS gap-fill targets and 31 +are late producer-complement targets (20 PUF-source, 10 ASEC-source, one +overlapping producer target). The partition is ownership context for the +remedy, not itself a defect. + +**Donor-support starvation.** Eight checks are support-starved; forty are +not. The starved eight split into two causally distinct classes: five whose +route donors are sparse but *intact* (blocked on additional evidence per the +adjudication), and three whose support was *deleted upstream* by the +retirement training cap (fixed on this branch, §2.1). + +Full per-check listing, by mechanism class: + +| Class (mechanism.primary) | Checks | Ordinals | Status | +|---|---:|---|---| +| gated_conditional_magnitude_shape | 23 | 9, 10, 12, 13, 14, 18, 30, 32, 35, 36, 38, 40, 42, 43, 44, 48, 49, 50, 52, 55, 56, 90, 92 | blocked_on_owner_specific_held_out_evidence | +| source_operator_conditional_magnitude_shape | 7 | 68, 70, 71, 73, 84, 86, 88 | blocked_on_owner_specific_held_out_evidence | +| donor_evidence_starvation | 5 | 16, 28, 33, 46, 75 | blocked_on_additional_evidence | +| retirement_mapping_label_or_model_shape | 4 | 21, 23, 25, 26 | blocked_on_mapping_labels_and_dense_evidence | +| upstream_retirement_support_deletion | 3 | 78, 80, 82 | **implemented** | +| late_transfer_then_qbi_reconciliation_shape | 2 | 58, 60 | blocked_on_qbi_owner_and_held_out_evidence | +| late_transfer_conditional_magnitude_shape | 2 | 62, 64 | blocked_on_held_out_calibration_evidence | +| adult_care_post_transfer_amount_shape | 1 | 2 | blocked_on_held_out_calibration_evidence | +| early_transfer_unemployment_amount_shape | 1 | 8 | blocked_on_held_out_calibration_evidence | + +## 2. Smallest honest change per class + +The class-level answer is explicit below. Only the support-deletion repair is +authorized by the frozen evidence; the other eight changes are the smallest +code seams that would fix their generating mechanisms *after* the named owner +evidence exists. Applying them now would be target tuning toward the battery. + +| Class | Smallest generating-mechanism change | Disposition | +|---|---|---| +| gated_conditional_magnitude_shape | Apply a target-and-sign weighted magnitude map only to imputed recipient draws at the owning QRF boundary (`acs_transfer.py:1348-1413`). | Owner-held-out evidence required. | +| source_operator_conditional_magnitude_shape | Apply the same target/sign-scoped amount map inside the owning registered source-family callback, preserving producer-owned cells and its reconciliation (`stacked_spine.py:5030-5278`). | Source-family owner evidence required. | +| donor_evidence_starvation | After a source-label audit provides dense evidence, widen the authorized donor rung used by the pattern donor mask or fit a target-specific rare-event/sign-and-magnitude model; do not lower the support floor (`acs_transfer.py:1351-1385`; `stacked_spine.py:3027-3031`). | Additional evidence required. | +| retirement_mapping_label_or_model_shape | Correct an evidenced account/source-label mapping, then refit the applicable amount model; the current account-code map and SS predictor seam are explicit (`retirement_distributions.py:190-208,679-691`). | Mapping audit plus dense rung required. | +| upstream_retirement_support_deletion | Retain the union of four-target carriers before the fixed cap and reweight only sampled common zeros (`retirement_distributions.py:443-450,486-584`). | **Implemented.** | +| late_transfer_then_qbi_reconciliation_shape | Calibrate the joint pre-cap QBI surface while preserving the existing coupled identities and caps (`qbi_inputs.py:1228-1363`). | QBI owner and held-out evidence required. | +| late_transfer_conditional_magnitude_shape | Apply a target-scoped weighted magnitude map to mutable late-transfer draws only (`acs_transfer.py:1398-1413`). | Held-out evidence required. | +| adult_care_post_transfer_amount_shape | Calibrate positive amounts on newly imputed mutable cells before the existing adult-care reconciliation (`acs_transfer.py:1244-1265`). | Held-out evidence required. | +| early_transfer_unemployment_amount_shape | With predictors and carrier membership frozen, apply a target-only weighted positive-magnitude map in the early QRF draw path (`acs_transfer.py:1348-1413`). | Held-out evidence required. | + +### 2.1 Upstream retirement support deletion — implemented + +Ordinals 78/80/82 (taxable 401k/403b/SEP positive) and the Keogh leg share +one generating mechanism. The retirement producer trains a four-target QRF +on the 108,073-row native ASEC donor and replaces the four non-IRA outputs +on the PUF half of the `asec_puf` donor spine +(`packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py:337-483`). +The old cap drew a *uniform* 5,000-row sample, which retained only +102/2,057 native 401k carriers, 4/161 403b, 2/61 SEP, and 0/2 Keogh +(`realized_regimes.json` → `retirement_cap_support`). The late ACS transfer +then draws its donors from that producer-overwritten PUF channel — the +`asec_puf` spine with the AUTO channel resolving to PUF support +(`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:118-123`; +`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8582`) +— so deleted carriers can never reach ACS. + +The fix repairs the sampling design inside the unchanged cap: +`_cap_retirement_distribution_training` +(`retirement_distributions.py:486-586`, called at `443-450`) retains every +row nonzero for *any* of the four targets as a census, samples only the +common all-target-zero stratum under the same 5,000-row cap and the same +seed, prefers positive-weight zero rows, ratio-calibrates only the sampled +zero rows back to the full zero-stratum mass (carrier weights and +conditional magnitudes stay exact), and fails closed on nonfinite/negative +inputs or a carrier union that leaves no zero slot. `sources.yaml:2138` +still declares `max_train_samples: 5000`; no cap value, seed, or gate +moved. The seed-protocol declaration records the changed draw-consumption +contract +(`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:877-901`). + +### 2.2 Donor evidence starvation — blocked, correctly + +Ordinals 16 (collectibles +), 28 (alimony income +), 33 (casualty loss +), +46 (farm operations −), 75 (prior-year self-employment −). Recomputed +route-donor sign support: collectibles +18 (late PUF; +82 native ASEC), +alimony +61 (early), casualty +27 (late PUF; +79 native), farm operations +−89 (early), prior-year self-employment −48 (early). Every donor is gated +and sign-capable, all above the five-carrier floor +(`stacked_spine.py:3027-3031`), so no transfer defect deletes them — the +tails are simply too sparse for the fitted conditional magnitudes to match. +The adjudication requires dense/additional evidence before a +target-specific refit; a code change here would be tuning toward the gate. +These remain BLOCKER, owner-scoped (dense-rung donor evidence or an +owner-approved target-specific sparse-tail model). + +### 2.3 Shape classes — regime receipts closed; calibration blocked + +The remaining 40 checks (classes `gated_conditional_magnitude_shape`, +`source_operator_conditional_magnitude_shape`, +`retirement_mapping_label_or_model_shape`, +`late_transfer_then_qbi_reconciliation_shape`, +`late_transfer_conditional_magnitude_shape`, +`adult_care_post_transfer_amount_shape`, +`early_transfer_unemployment_amount_shape`) fail on conditional magnitude or +incidence *shape* with intact gated donors. The adjudication's only +executable defect here was the receipt gap: ACS-transfer provenance did not +persist the realized regime, so signed-tail fits could not be adjudicated +without reconstruction. This branch closes that gap end to end: + +- every ordinary fit binds validated realized regimes into provenance + (`acs_transfer.py:52-85,453,1392-1396`), and every banked chain step + persists its regime (`acs_transfer.py:498,1819-1823`); +- target-bank schema 2 stores and revalidates the regime per pattern step + and projects it into operational receipts + (`acs_transfer_bank.py:25,47-53,249,357`); schema-1 artifacts rebuild; +- early-direction, late-group, and aggregate receipts expose the same + fitted-pattern regime rows, and the canonical aggregate must be the exact + concatenation of the 19 group receipts + (`stacked_spine.py:3754-3772,8215-8360,9260-9268,9315-9331`). + +The magnitude repair itself (recalibrating conditional draws against +held-out evidence, or fixing SS/retirement mapping labels) is owner-scoped +calibration work that the frozen evidence does not authorize; each row's +`mechanism.smallest_honest_change` records the specific blocked remedy. + +## 3. The Keogh one-sided leg: lost before transfer fitting, not structurally absent + +Terminal leg +`person/source_operator_retirement_distributions/keogh_distributions[clone_0]/positive`; +full-scale f025 emits `weighted positive-leg incidence ratio 0 is outside +[0.8, 1.25] (asec=1.43573e-05, acs=0)`. Incidence runs and fails *before* +the five-carrier floor, which skips only the sign's QED +(`stacked_spine.py:12068-12087`), exactly as the adjudication states. + +Determination: **the ACS carriers are lost on the transfer donor path — fix, +not declare-absent.** More precisely, the retirement producer deletes them +before the ACS-transfer fit rather than the recipient merge dropping them. +The evidence chain, each step verified against frozen artifacts this lane: + +1. Native ASEC clone 0 holds exactly two positive Keogh values, `2,040` and + `30,000`, in 108,073 rows — the signal exists at the source. +2. The old uniform 5,000-row cap retained neither carrier + (`retirement_cap_support.keogh_distributions: native 2, retained 0`), so + the producer's QRF predicted identically zero Keogh across the entire + PUF half. +3. The frozen transfer bank holds 1,970,973 stored / 1,736,840 finite Keogh + recipient draws, every one exactly zero (`max_abs = 0.0`) — the fitted + transfer regime was `degenerate_zero` *because its donor channel was + already dead*, not because the recipient merge dropped values. +4. Transferred-frame census: ASEC clone 0 keeps `{2,040, 30,000}`; ASEC + clone 1 and every ACS clone are all-zero. The ACS absence is + manufactured upstream of transfer fitting. + +The declared-absence route would have been honest only if ACS Keogh were +structurally impossible. It is not, so that route is rejected. For the +record, taking it would require: an exact structural-absence equation for +the recipient scope (validated as exact, with zero unexpected-null and zero +synthesized rows, `stacked_spine.py:8168-8195`) or a +`tolerated_absence_receipts` declaration on the late-producer requirement +with its canonical per-requirement receipt +(`stacked_spine.py:6491-6528`), plus battery-side structural receipts that +stay clone-0 exact (`stacked_spine.py:7186-7205`). Declaring absence for a +present-in-source signal would also delete the pool's only terminal check +of it — the same reasoning the adjudication used to reject a clone-0 waiver +(ADJUDICATION.md line 71). Approval would sit with the US pool owner via +the owner-only reviewed-exclusion/register path, which this lane is barred +from and did not touch. + +With the support-preserving cap, both carriers now reach the producer's +training donor whenever the full donor is available; +`test_production_imputer_hands_rare_keogh_support_to_qrf` locks that +production call path. + +## 4. Verification + +### 4.1 Regression tests (all on this branch) + +`packages/microcosm-build/tests/`: + +- `test_us_retirement_distributions.py` — + `test_production_imputer_hands_rare_keogh_support_to_qrf`, + `test_puf_training_cap_preserves_nonzero_union_and_weighted_sign_mass`, + `test_puf_training_cap_fails_when_nonzero_union_leaves_no_zero_slot`, + `test_puf_training_cap_does_not_waste_representative_slots_on_zero_weights`. +- `test_us_acs_transfer.py` — three-sign regime recording, exact-target / + known-label receipt closure, cold/warm bank regime persistence, schema-1 + rebuild. +- `test_us_stacked_spine.py` / `test_us_late_producer_dag.py` — late + executor rejects a regime receipt detached from fitted provenance; + aggregate is the exact 19-group concatenation; canonical JSON key sorting + preserves valid regime-map checkpoints while a missing target still fails + (`test_realized_regime_receipt_survives_canonical_json_key_sorting`). +- `test_us_multispine_pool_h5_io.py` / `test_us_multispine_pool_tool.py` — + legacy materializer-3 checkpoints cannot resume after the shared fix; + stacked v12 and pool v8 identity binding; v7 rejection. +- `test_spec_engine_seeds.py` — the support-preserving cap's draw-site + declaration. + +### 4.2 Suite receipts + +Per-shard `uv run pytest` (one process per shard, as CI runs it) on the final +source tree: + +- calibrate: 201 passed; +- data: 275 passed, 1 skipped; +- fit: 93 passed; +- frame: 294 passed, 36 skipped; +- build: 5,973 passed, 39 skipped; +- repo-wide `uv run ruff check .`: all checks passed. + +The authoritative US-bundle generator check and the frozen realized-regime +evidence generator check also pass. The build shard was rerun in full after +correcting its sole stale derived-payload assertion, so the receipt is not a +composition of partial runs. Exact durations and warning counts are in +PROGRESS.md. `ruff format` was applied only to the five files whose drift this +branch introduced; the CI lint gate is `ruff check` and is green. + +### 4.3 1% before/after + +The host-queue-owned baseline 1% build +(`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/baseline1pct/pool.gates.json`) +is the frozen "before": +[`baseline_1pct_projection.json`](baseline_1pct_projection.json), generated +by [`project_baseline_1pct.py`](project_baseline_1pct.py), shows 127 +battery failure lines over 93 legs; 47 of the 48 frozen reds reproduce at +1%; ordinal 16 (collectibles) emits no 1% line because its +18 route +carriers rarely survive a 1% sample; the Keogh leg emits no 1% line in +either direction because a 1% ASEC sample almost surely carries neither +native carrier. + +Regenerating the projection on this final tree was byte-identical to the +committed JSON. Running [`diff_1pct_failures.py`](diff_1pct_failures.py) with +that baseline on both sides reports 127 shared lines and zero added/removed or +changed lines, which validates the diff path without inventing an after result. + +Consequences, stated before the after-build exists: at 1% the fix is +falsifiable on the 401k/403b/SEP legs (native carriers ~20/1.6/0.6 expected +in-sample) and on non-regression of the other 89 legs; the Keogh gate-level +flip is only provable at full scale, where its regression test and the +bank-draw census stand in until the owner's next full build. This lane is +barred from starting builds, so the "after" 1% run is a host-queue action: +[`run_1pct_offchain_build.sh`](run_1pct_offchain_build.sh) (1% cap, +off-chain, serial-queue refusal, 15 GiB RSS watchdog) followed by +[`diff_1pct_failures.py`](diff_1pct_failures.py) `--before-gates +…/baseline1pct/pool.gates.json --after-gates ` produces the +failure-line diff projected against the frozen baseline classes. + +## 5. Identity and receipt version moves + +All moves are semantic invalidations of artifacts that could otherwise +resume with support-deleting samples or regime-free receipts — none touches +a gate: + +- late-producer receipt schema 3→4 + (`us_late_producer_registry.py:129`; `us/spec/imputation.yaml` + execution_receipt_contract); +- legacy pool checkpoint materializer 3→4 + (`tools/build_us_multispine_pool.py:231`, mirrored at `h5_io.py:103`); +- pool stage checkpoint materializer 7→8 + (`tools/build_us_multispine_pool.py:282`); +- stacked checkpoint materializer 11→12 + (`tools/build_us_multispine_pool.py:319`; `us/spec/spine.yaml`; + `spec_engine/schema/spine.schema.json`); +- ACS-transfer target-bank schema 1→2 (`acs_transfer_bank.py:25`). + +Because `seeds.py` is shared spec-engine kernel source, its attested +identities legitimately moved and were re-pinned from the final tree: US +bundle `5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554`, +BE `91d3a58e669950c7fab0aefeb74687278efca82d413a900e9ab1878b09a7a5a3`, +UK `54c0b70167b9d36e857a116ca4c80c74af142b6f09d5894ee979de62b9634738`, +loader golden vector `43989f610f9691daf69e9ee74c1aa17f2a8bc3f07b9992489c65fa8b1ad747de`, +inventory `seed_protocol` +`d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be` and +`seed_map` +`f9e71d22a4c8608d1e2ca99912f60dd06272e785e0babcca46a452c1a6df9562`; +`docs/evidence/spec-engine/us-f0-coverage.json` regenerated by +`tools/spec_engine_coverage.py` (41,381/41,381 fields, 40/40 inventory +checks). An earlier journal claim of US bundle `d30642501b9b…` described an +intermediate tree and is historicized in `_LANE-NOTES.md`. + +## 6. What remains (owner actions) + +1. **Host queue:** run the 1% after-build from this branch + (`run_1pct_offchain_build.sh`) and file the `diff_1pct_failures.py` + output; expected movement is on 401k/403b/SEP legs plus no new + non-baseline reds beyond 1% sampling noise. +2. **Full-scale build (owner-scheduled):** the only gate-level proof of the + Keogh flip and the final word on ordinals 78/80/82. +3. **Owner calibration program:** the 35 blocked shape checks need + held-out/dense evidence per class (§2.3); the five starvation blockers + (§2.2) need dense-rung donor evidence or an owner-approved sparse-tail + model; the four SS-mapping checks additionally need label adjudication. +4. **No exclusions were added** and none are recommended; the Keogh + declared-absence route is explicitly rejected (§3). diff --git a/experiments/battery_rare_signed_tails/baseline_1pct_projection.json b/experiments/battery_rare_signed_tails/baseline_1pct_projection.json new file mode 100644 index 000000000..b634ad158 --- /dev/null +++ b/experiments/battery_rare_signed_tails/baseline_1pct_projection.json @@ -0,0 +1,135 @@ +{ + "baseline_1pct": { + "failure_lines": 127, + "unique_failing_legs": 93 + }, + "frozen_reds_not_red_at_1pct": { + "count": 1, + "legs": [ + { + "donor_support_starved": true, + "leg_id": "person/puf_tax_itemization/long_term_capital_gains_on_collectibles[clone_0]/positive", + "ordinal": 16 + } + ] + }, + "frozen_reds_red_at_1pct": { + "count": 47, + "legs": [ + "person/adult_care/pre_subsidy_care_expenses[clone_0]/positive", + "person/model_required_numeric/unemployment_compensation[clone_0]/positive", + "person/puf_tax_itemization/alimony_expense[clone_0]/positive", + "person/puf_tax_itemization/alimony_income[clone_0]/positive", + "person/puf_tax_itemization/casualty_loss[clone_0]/positive", + "person/puf_tax_itemization/estate_income[clone_0]/negative", + "person/puf_tax_itemization/estate_income[clone_0]/positive", + "person/puf_tax_itemization/farm_income[clone_0]/positive", + "person/puf_tax_itemization/farm_operations_income[clone_0]/negative", + "person/puf_tax_itemization/farm_operations_income[clone_0]/positive", + "person/puf_tax_itemization/farm_rent_income[clone_0]/negative", + "person/puf_tax_itemization/farm_rent_income[clone_0]/positive", + "person/puf_tax_itemization/investment_income_elected_form_4952[clone_0]/positive", + "person/puf_tax_itemization/long_term_capital_gains_before_response[clone_0]/positive", + "person/puf_tax_itemization/miscellaneous_income[clone_0]/positive", + "person/puf_tax_itemization/non_qualified_dividend_income[clone_0]/positive", + "person/puf_tax_itemization/non_sch_d_capital_gains[clone_0]/positive", + "person/puf_tax_itemization/partnership_income[clone_0]/positive", + "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/negative", + "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/positive", + "person/puf_tax_itemization/qualified_bdc_income[clone_0]/positive", + "person/puf_tax_itemization/qualified_dividend_income[clone_0]/positive", + "person/puf_tax_itemization/qualified_reit_and_ptp_income[clone_0]/positive", + "person/puf_tax_itemization/rental_income[clone_0]/negative", + "person/puf_tax_itemization/rental_income[clone_0]/positive", + "person/puf_tax_itemization/self_employed_pension_contributions_desired[clone_0]/positive", + "person/puf_tax_itemization/short_term_capital_gains[clone_0]/positive", + "person/puf_tax_itemization/social_security_dependents[clone_0]/positive", + "person/puf_tax_itemization/social_security_disability[clone_0]/positive", + "person/puf_tax_itemization/social_security_retirement[clone_0]/positive", + "person/puf_tax_itemization/social_security_survivors[clone_0]/positive", + "person/puf_tax_itemization/tax_exempt_interest_income[clone_0]/positive", + "person/puf_tax_itemization/unadjusted_basis_qualified_property[clone_0]/positive", + "person/puf_tax_itemization/w2_wages_from_qualified_business[clone_0]/positive", + "person/source_operator_child_support/child_support_expense[clone_0]/positive", + "person/source_operator_child_support/child_support_received[clone_0]/positive", + "person/source_operator_disability_benefits/disability_benefits[clone_0]/positive", + "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/negative", + "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/positive", + "person/source_operator_retirement_distributions/taxable_401k_distributions[clone_0]/positive", + "person/source_operator_retirement_distributions/taxable_403b_distributions[clone_0]/positive", + "person/source_operator_retirement_distributions/taxable_sep_distributions[clone_0]/positive", + "person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive", + "person/source_operator_workers_compensation/workers_compensation[clone_0]/positive", + "spm_unit/source_operator_energy_subsidy/spm_unit_energy_subsidy[clone_0]/positive", + "tax_unit/puf_tax_itemization/domestic_production_ald[clone_0]/positive", + "tax_unit/puf_tax_itemization/unrecaptured_section_1250_gain[clone_0]/positive" + ] + }, + "full_scale": { + "failure_lines": 93, + "unique_failing_legs": 60 + }, + "inputs": { + "baseline_1pct_gates": "/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/baseline1pct/pool.gates.json", + "full_scale_gates": "/Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/populace_us_2024_stacked_pool.gates.json", + "realized_regimes": "/Users/maxghenis/PolicyEngine/_worktrees/microcosm-rare-signed-tails/experiments/battery_rare_signed_tails/realized_regimes.json" + }, + "keogh_one_sided_leg": { + "leg_id": "person/source_operator_retirement_distributions/keogh_distributions[clone_0]/positive", + "lines_at_1pct": [], + "note": "Native ASEC holds exactly two positive Keogh values in 108,073 rows; a 1% sample almost surely carries neither, so the 1% gate cannot exercise this leg in either direction. Gate-level proof of the Keogh repair requires a full-scale build; the 1% evidence is the support-preserving cap regression suite plus the frozen bank draw census.", + "red_at_full_scale": true + }, + "one_pct_only_noise_legs": { + "count": 46, + "legs": [ + "person/adult_care/is_incapable_of_self_care[clone_0]", + "person/derived_transfer/schedule_d_capital_gain_distributions[clone_0]/positive", + "person/model_required_boolean/has_champva_health_coverage_at_interview[clone_0]", + "person/model_required_boolean/has_indian_health_service_coverage_at_interview[clone_0]", + "person/model_required_boolean/has_non_marketplace_direct_purchase_health_coverage_at_interview[clone_0]", + "person/model_required_boolean/has_other_means_tested_health_coverage_at_interview[clone_0]", + "person/model_required_boolean/has_tricare_health_coverage_at_interview[clone_0]", + "person/model_required_boolean/has_va_health_coverage_at_interview[clone_0]", + "person/model_required_boolean/is_blind[clone_0]", + "person/model_required_boolean/is_full_time_college_student[clone_0]", + "person/model_required_boolean/is_pregnant[clone_0]", + "person/model_required_boolean/receives_wic[clone_0]", + "person/model_required_numeric/over_the_counter_health_expenses[clone_0]/positive", + "person/model_required_numeric/tax_exempt_private_pension_income[clone_0]/positive", + "person/model_required_numeric/veterans_benefits[clone_0]/positive", + "person/puf_tax_itemization/business_is_sstb[clone_0]", + "person/puf_tax_itemization/charitable_cash_donations[clone_0]/positive", + "person/puf_tax_itemization/charitable_non_cash_donations[clone_0]/positive", + "person/puf_tax_itemization/educator_expense[clone_0]/positive", + "person/puf_tax_itemization/investment_interest_expense[clone_0]/positive", + "person/puf_tax_itemization/partnership_income[clone_0]/negative", + "person/puf_tax_itemization/qualified_tuition_expenses[clone_0]/positive", + "person/puf_tax_itemization/sstb_self_employment_income_would_be_qualified[clone_0]", + "person/puf_tax_itemization/student_loan_interest[clone_0]/positive", + "person/puf_tax_itemization/taxable_ira_distributions[clone_0]/positive", + "person/puf_tax_itemization/taxable_private_pension_income[clone_0]/positive", + "person/puf_tax_itemization/traditional_ira_contributions_desired[clone_0]/positive", + "person/puf_tax_itemization/unreimbursed_business_employee_expenses[clone_0]/positive", + "person/simulated_output/ssi[clone_0]/positive", + "person/source_operator_education_inputs/attends_eligible_educational_institution_for_american_opportunity_credit[clone_0]", + "person/source_operator_education_inputs/educational_assistance[clone_0]/positive", + "person/source_operator_education_inputs/has_american_opportunity_credit_1098_t_or_exception[clone_0]", + "person/source_operator_education_inputs/has_american_opportunity_credit_institution_ein[clone_0]", + "person/source_operator_education_inputs/is_enrolled_at_least_half_time_for_american_opportunity_credit[clone_0]", + "person/source_operator_education_inputs/is_pursuing_credential_for_american_opportunity_credit[clone_0]", + "person/source_operator_prior_year_income/previous_year_income_available[clone_0]", + "person/source_operator_relationship_inputs/is_separated[clone_0]", + "person/source_operator_retirement_contributions/roth_401k_contributions_desired[clone_0]/positive", + "person/source_operator_retirement_contributions/roth_ira_contributions_desired[clone_0]/positive", + "person/source_operator_retirement_contributions/traditional_401k_contributions_desired[clone_0]/positive", + "person/source_operator_retirement_distributions/tax_exempt_ira_distributions[clone_0]/positive", + "spm_unit/benefit_participation/takes_up_housing_assistance_if_eligible[clone_0]", + "spm_unit/model_required_boolean/is_tanf_enrolled[clone_0]", + "spm_unit/model_required_numeric/spm_unit_pre_subsidy_childcare_expenses[clone_0]/positive", + "spm_unit/source_operator_housing_inputs/receives_housing_assistance[clone_0]", + "tax_unit/puf_tax_itemization/health_savings_account_ald[clone_0]/positive" + ] + }, + "title": "1% baseline projection of the frozen 48 red QED checks" +} diff --git a/experiments/battery_rare_signed_tails/build_realized_regime_evidence.py b/experiments/battery_rare_signed_tails/build_realized_regime_evidence.py new file mode 100644 index 000000000..39502f35a --- /dev/null +++ b/experiments/battery_rare_signed_tails/build_realized_regime_evidence.py @@ -0,0 +1,554 @@ +"""Rebuild the frozen rare signed-tail regime evidence. + +This is an adjudication-side extractor, not a production build step. It reads +the immutable f025 target-bank checkpoints and the arm-split adjudication, +verifies their identities, and emits a compact, deterministic JSON projection. + +The schema-1 bank predates regime persistence. Its metadata authenticates the +fit boundary (target, availability pattern, predictors, donor index, weights, +and zero tolerance), while the sign counts below are the separately audited +fit-boundary reconstruction. Future schema-2 banks carry the realized labels +directly and do not need this retrospective join. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +from collections import Counter +from pathlib import Path +from typing import Any + +import h5py + +AUTHORITY_SHA256 = { + "ADJUDICATION.md": ( + "2cdf79233d56f63ca8e65305cb1c746706d88a7a57ac26ae25db575676daf3b8" + ), + "adjudication.json": ( + "1c1597c8080cf4e0db0079c2a3e1bc345001525bc3ec6c296aa7575b47136c32" + ), + "f025_gates": ( + "685cad63d4dc62234da72501c5a3ce9ec5a81fcd3f7b412b61474a9c1d8b423b" + ), + "transferred_receipts": ( + "c91a61bb20f0f2c2612bac8fdd21d2227cf746b85e2c5c2648857fd667e68aed" + ), +} + +EXPECTED_TRANSFERRED_CHECKPOINT = { + "sha256": "bdc9355d92659bb28d58b1ddcd647ec303f2ad217661e17d5b4b0984e04532e8", + "size_bytes": 5_578_512_480, + "identity_sha256": ( + "7847224d1daa6b21cb4d063b1b54b62516b24936a302513143a685b61b7cf8e4" + ), +} + +EARLY_ORDINALS = frozenset( + {8, 9, 10, 13, 14, 21, 23, 25, 26, 28, 36, 38, 44, 46, 50, 73, 75} +) +SPARSE_EVIDENCE_ORDINALS = frozenset({16, 28, 33, 46, 75}) +RETIREMENT_CAP_ORDINALS = frozenset({78, 80, 82}) + +PATTERN_PROFILES = { + "early_person": ( + "pattern_00_677f6490", + "pattern_01_5874881e", + "pattern_02_7c3bceda", + "pattern_03_04f75638", + ), + "late_person": ( + "pattern_00_36785ebf", + "pattern_01_c6777728", + "pattern_02_5e7dd311", + "pattern_03_76a0101a", + ), + "late_group": ( + "pattern_00_741356db", + "pattern_01_7c62ad55", + "pattern_02_d9f964c9", + "pattern_03_2e5c739d", + ), +} + +# (negative, zero, positive), reconstructed at the target-complete fit boundary. +_FIT_SUPPORT_TEXT = """ +2 0 107999 74 +8 0 107061 1012 +9 0 96432 11641 +10 0 96432 11641 +12 0 106243 1830 +13 0 103874 4199 +14 0 103874 4199 +16 0 108055 18 +18 0 106220 1853 +21 0 92633 15440 +23 0 106138 1935 +25 0 107810 263 +26 0 107675 398 +28 0 108012 61 +30 0 107873 200 +32 0 108007 66 +33 0 108046 27 +35 0 107476 597 +36 216 104057 3800 +38 216 104057 3800 +40 99 107534 440 +42 99 107534 440 +43 0 107932 141 +44 89 107422 562 +46 89 107422 562 +48 111 107694 268 +49 111 107694 268 +50 0 107050 1023 +52 1182 103789 3102 +55 385 106982 706 +56 385 106982 706 +58 0 107211 862 +60 0 102255 5818 +62 0 101937 6136 +64 0 107730 343 +68 0 107676 397 +70 0 107430 643 +71 0 107448 625 +73 48 105417 2608 +75 48 105417 2608 +78 0 106307 1766 +80 0 108052 21 +82 0 107157 916 +84 0 107500 573 +86 0 108028 45 +88 0 42931 1030 +90 0 57352 278 +92 0 57295 335 +""" + +FIT_SUPPORT = { + int(ordinal): (int(negative), int(zero), int(positive)) + for ordinal, negative, zero, positive in ( + line.split() for line in _FIT_SUPPORT_TEXT.splitlines() if line.strip() + ) +} + +# The old 5,000-row retirement training cap retained these carriers from the +# full native ASEC target support. Keogh is outside the 48 QED checks but uses +# the same faulty cap and is included in the output's one-sided-leg finding. +RETIREMENT_CAP_SUPPORT = { + "keogh_distributions": {"retained": 0, "native": 2}, + "taxable_401k_distributions": {"retained": 102, "native": 2_057}, + "taxable_403b_distributions": {"retained": 4, "native": 161}, + "taxable_sep_distributions": {"retained": 2, "native": 61}, +} + +_LEG_RE = re.compile( + r"/(?P[^/\[]+)\[clone_0\]/(?Ppositive|negative)$" +) + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _assert_sha256(path: Path, expected: str) -> None: + actual = _sha256(path) + if actual != expected: + raise RuntimeError(f"{path}: expected sha256 {expected}, got {actual}") + + +def _profile_name(*, ordinal: int, entity: str) -> str: + if ordinal in EARLY_ORDINALS: + if entity != "person": + raise RuntimeError(f"early ordinal {ordinal} is unexpectedly {entity}") + return "early_person" + return "late_person" if entity == "person" else "late_group" + + +def _regime(negative: int, zero: int, positive: int) -> str: + if negative and positive: + return "three_sign" + if zero and positive: + return "zero_inflated_positive" + raise RuntimeError( + "frozen QED target unexpectedly lacks its audited gated support: " + f"negative={negative}, zero={zero}, positive={positive}" + ) + + +def _mechanism(ordinal: int) -> dict[str, str]: + if ordinal in SPARSE_EVIDENCE_ORDINALS: + return { + "primary": "donor_evidence_starvation", + "smallest_honest_change": ( + "after a source-label audit supplies dense evidence, fit a " + "target-specific rare-event/sign-and-magnitude model" + ), + "status": "blocked_on_additional_evidence", + } + if ordinal in RETIREMENT_CAP_ORDINALS: + return { + "primary": "upstream_retirement_support_deletion", + "smallest_honest_change": ( + "retain the union of retirement-target carriers before the " + "fixed-size training cap and reweight only sampled common zeros" + ), + "status": "implemented", + } + if ordinal == 2: + return { + "primary": "adult_care_post_transfer_amount_shape", + "smallest_honest_change": ( + "with held-out evidence, calibrate positive amounts on newly " + "imputed mutable cells before adult-care reconciliation" + ), + "status": "blocked_on_held_out_calibration_evidence", + } + if ordinal == 8: + return { + "primary": "early_transfer_unemployment_amount_shape", + "smallest_honest_change": ( + "with predictors and carrier membership frozen, apply a " + "target-only weighted positive-magnitude map" + ), + "status": "blocked_on_held_out_calibration_evidence", + } + if ordinal in {21, 23, 25, 26}: + return { + "primary": "retirement_mapping_label_or_model_shape", + "smallest_honest_change": ( + "audit source mappings and labels on a dense rung, refit the " + "applicable model, and run the frozen predictor factorial" + ), + "status": "blocked_on_mapping_labels_and_dense_evidence", + } + if ordinal in {58, 60}: + return { + "primary": "late_transfer_then_qbi_reconciliation_shape", + "smallest_honest_change": ( + "calibrate the joint pre-cap QBI surface while preserving the " + "existing coupled caps and identities" + ), + "status": "blocked_on_qbi_owner_and_held_out_evidence", + } + if ordinal in {62, 64}: + return { + "primary": "late_transfer_conditional_magnitude_shape", + "smallest_honest_change": ( + "apply a target-scoped weighted conditional-magnitude map to " + "mutable late-transfer draws" + ), + "status": "blocked_on_held_out_calibration_evidence", + } + if ordinal in {68, 70, 71, 73, 84, 86, 88}: + return { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": ( + "apply a target-and-sign amount map within the owning source " + "family while preserving reconciliation and producer cells" + ), + "status": "blocked_on_owner_specific_held_out_evidence", + } + return { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": ( + "apply a target-and-sign-scoped weighted conditional-magnitude map " + "at the owning transfer boundary, mutating imputed cells only" + ), + "status": "blocked_on_owner_specific_held_out_evidence", + } + + +def _read_metadata(path: Path) -> dict[str, Any]: + with h5py.File(path, "r") as h5: + return json.loads(bytes(h5["metadata_json"][:])) + + +def _find_checkpoint(bank_root: Path, *, ordinal: int, target: str) -> Path: + partition = "asec_survey_to_acs" if ordinal in EARLY_ORDINALS else "late_producer_dag" + matches = list((bank_root / partition).rglob(f"*__{target}.h5")) + if len(matches) != 1: + raise RuntimeError( + f"ordinal {ordinal} target {target}: expected one {partition} " + f"checkpoint, found {matches}" + ) + return matches[0] + + +def _checkpoint_projection( + path: Path, + *, + bank_root: Path, + expected_patterns: tuple[str, ...], + support: tuple[int, int, int], +) -> dict[str, Any]: + metadata = _read_metadata(path) + if metadata.get("schema_version") != 1: + raise RuntimeError(f"{path}: retrospective evidence expects schema 1") + steps = metadata.get("pattern_steps") + if not isinstance(steps, list) or len(steps) != 4: + raise RuntimeError(f"{path}: expected four pattern steps") + pattern_ids = tuple(step["pattern"] for step in steps) + if pattern_ids != expected_patterns: + raise RuntimeError( + f"{path}: pattern inventory {pattern_ids} != {expected_patterns}" + ) + negative, zero, positive = support + label = _regime(negative, zero, positive) + patterns = [] + donor_identities = set() + for step in steps: + state = step["state_after"] + donor_index = state["donor_index"] + donor_identity = (donor_index["length"], donor_index["sha256"]) + donor_identities.add(donor_identity) + if sum(support) != donor_index["length"]: + raise RuntimeError( + f"{path}: support {support} does not close donor length " + f"{donor_index['length']}" + ) + patterns.append( + { + "pattern_id": step["pattern"], + "predictors": state["predictors"], + "recipient_rows": state["recipient_index"]["length"], + "recipient_index_sha256": state["recipient_index"]["sha256"], + "seed": state["model_config"]["seed"], + "weight_kind": state["weight_kind"], + "weight_sha256": state["weight_sha256"], + "zero_atol": state["model_config"]["zero_atol"], + "realized_regime": label, + } + ) + if len(donor_identities) != 1: + raise RuntimeError(f"{path}: patterns do not share one donor identity") + donor_rows, donor_index_sha256 = donor_identities.pop() + target = metadata["target"] + return { + "checkpoint_relpath": path.relative_to(bank_root).as_posix(), + "checkpoint_schema_version": metadata["schema_version"], + "content_metadata_sha256": metadata["content_metadata_sha256"], + "entity": target["entity"], + "transfer_family": target["family"], + "model_target": target["model_target"], + "model_targets": target["model_targets"], + "exported_targets": target["exported_targets"], + "donor_identity": { + "rows": donor_rows, + "index_sha256": donor_index_sha256, + }, + "donor_support": { + "negative": negative, + "zero": zero, + "positive": positive, + "total": sum(support), + "evidence_kind": "fit_boundary_reconstruction", + "common_across_patterns": True, + }, + "patterns": patterns, + } + + +def build_evidence(args: argparse.Namespace) -> dict[str, Any]: + adjudication_path = args.adjudication_dir / "adjudication.json" + adjudication_md_path = args.adjudication_dir / "ADJUDICATION.md" + _assert_sha256(adjudication_path, AUTHORITY_SHA256["adjudication.json"]) + _assert_sha256(adjudication_md_path, AUTHORITY_SHA256["ADJUDICATION.md"]) + _assert_sha256(args.f025_gates, AUTHORITY_SHA256["f025_gates"]) + _assert_sha256( + args.transferred_receipts, + AUTHORITY_SHA256["transferred_receipts"], + ) + + adjudication = json.loads(adjudication_path.read_text(encoding="utf-8")) + receipt = json.loads(args.transferred_receipts.read_text(encoding="utf-8")) + checkpoint = receipt["checkpoint"] + observed_checkpoint = { + "sha256": checkpoint["sha256"], + "size_bytes": checkpoint["size_bytes"], + "identity_sha256": receipt["identity_sha256"], + } + if observed_checkpoint != EXPECTED_TRANSFERRED_CHECKPOINT: + raise RuntimeError( + "transferred checkpoint receipt differs: " + f"{observed_checkpoint} != {EXPECTED_TRANSFERRED_CHECKPOINT}" + ) + + rows = [ + row + for row in adjudication["adjudications"] + if row["criterion"] == "quantile" + ] + if len(rows) != 48 or {row["ordinal"] for row in rows} != set(FIT_SUPPORT): + raise RuntimeError("adjudication QED inventory differs from audited support") + + targets_by_checkpoint: dict[str, dict[str, Any]] = {} + checks: list[dict[str, Any]] = [] + for row in rows: + ordinal = row["ordinal"] + match = _LEG_RE.search(row["leg_id"]) + if match is None: + raise RuntimeError(f"cannot parse QED leg {row['leg_id']}") + target_name = match.group("target") + support = FIT_SUPPORT[ordinal] + profile = _profile_name(ordinal=ordinal, entity=row["entity"]) + checkpoint_path = _find_checkpoint( + args.bank_root, + ordinal=ordinal, + target=target_name, + ) + checkpoint_relpath = checkpoint_path.relative_to(args.bank_root).as_posix() + projection = _checkpoint_projection( + checkpoint_path, + bank_root=args.bank_root, + expected_patterns=PATTERN_PROFILES[profile], + support=support, + ) + existing = targets_by_checkpoint.setdefault(checkpoint_relpath, projection) + if existing != projection: + raise RuntimeError(f"duplicate target evidence differs for {checkpoint_relpath}") + + baseline = row["baseline"] + checks.append( + { + "ordinal": ordinal, + "check_id": row["check_id"], + "leg_id": row["leg_id"], + "entity": row["entity"], + "gate_family": row["family"], + "target_ref": checkpoint_relpath, + "tested_sign": match.group("sign"), + "transfer_partition": ( + "early_gap_fill" + if ordinal in EARLY_ORDINALS + else "late_producer_complement" + ), + "pattern_profile": profile, + "fitter_regime": _regime(*support), + "donor_support_starved": ordinal + in (SPARSE_EVIDENCE_ORDINALS | RETIREMENT_CAP_ORDINALS), + "mechanism": _mechanism(ordinal), + "terminal": { + "asec_carrier_rows": baseline["nonzero_rows"]["asec"], + "acs_carrier_rows": baseline["nonzero_rows"]["acs"], + "quantile_envelope_distance": baseline[ + "quantile_envelope_distance" + ], + }, + } + ) + + targets = [targets_by_checkpoint[key] for key in sorted(targets_by_checkpoint)] + check_regimes = Counter(check["fitter_regime"] for check in checks) + target_regimes = Counter( + target["patterns"][0]["realized_regime"] for target in targets + ) + route_counts = Counter(check["transfer_partition"] for check in checks) + validation = { + "check_count": len(checks), + "unique_target_count": len(targets), + "target_pattern_record_count": sum(len(target["patterns"]) for target in targets), + "check_pattern_link_count": len(checks) * 4, + "check_regime_counts": dict(sorted(check_regimes.items())), + "unique_target_regime_counts": dict(sorted(target_regimes.items())), + "transfer_partition_counts": dict(sorted(route_counts.items())), + "degenerate_or_single_sign_check_count": sum( + label not in {"zero_inflated_positive", "three_sign"} + for label in check_regimes.elements() + ), + } + expected_validation = { + "check_count": 48, + "unique_target_count": 42, + "target_pattern_record_count": 168, + "check_pattern_link_count": 192, + "check_regime_counts": { + "three_sign": 13, + "zero_inflated_positive": 35, + }, + "unique_target_regime_counts": { + "three_sign": 7, + "zero_inflated_positive": 35, + }, + "transfer_partition_counts": { + "early_gap_fill": 17, + "late_producer_complement": 31, + }, + "degenerate_or_single_sign_check_count": 0, + } + if validation != expected_validation: + raise RuntimeError(f"validation differs: {validation} != {expected_validation}") + + return { + "schema_version": 1, + "title": "Rare signed-tail frozen realized-regime evidence", + "evidence_boundary": { + "bank_schema": 1, + "pattern_metadata": "persisted_frozen_checkpoint", + "support_and_regime": "fit_boundary_reconstruction", + "warning": ( + "do not recompute early-target support from the final transferred " + "pool because later producers may mutate those fields" + ), + }, + "authority": { + "adjudication_md_sha256": AUTHORITY_SHA256["ADJUDICATION.md"], + "adjudication_json_sha256": AUTHORITY_SHA256["adjudication.json"], + "f025_gates_sha256": AUTHORITY_SHA256["f025_gates"], + "transferred_receipts_sha256": AUTHORITY_SHA256[ + "transferred_receipts" + ], + "transferred_checkpoint": observed_checkpoint, + }, + "pattern_profiles": { + key: list(value) for key, value in PATTERN_PROFILES.items() + }, + "retirement_cap_support": RETIREMENT_CAP_SUPPORT, + "keogh_one_sided_leg": { + "structurally_absent": False, + "native_asec_positive_rows": 2, + "native_asec_positive_values": [2_040.0, 30_000.0], + "old_training_cap_positive_rows": 0, + "frozen_acs_positive_rows": 0, + "loss_stage": "retirement_training_cap_before_transfer_fit", + }, + "targets": targets, + "checks": checks, + "validation": validation, + } + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--adjudication-dir", type=Path, required=True) + parser.add_argument("--bank-root", type=Path, required=True) + parser.add_argument("--f025-gates", type=Path, required=True) + parser.add_argument("--transferred-receipts", type=Path, required=True) + parser.add_argument( + "--out", + type=Path, + default=Path(__file__).with_name("realized_regimes.json"), + ) + parser.add_argument("--check", action="store_true") + return parser + + +def main() -> None: + args = _parser().parse_args() + rendered = json.dumps(build_evidence(args), indent=2, sort_keys=True) + "\n" + if args.check: + if not args.out.is_file() or args.out.read_text(encoding="utf-8") != rendered: + raise SystemExit(f"stale realized-regime evidence: {args.out}") + print(f"realized-regime evidence current: {args.out}") + return + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(rendered, encoding="utf-8") + print(f"wrote realized-regime evidence: {args.out}") + + +if __name__ == "__main__": + main() diff --git a/experiments/battery_rare_signed_tails/diff_1pct_failures.py b/experiments/battery_rare_signed_tails/diff_1pct_failures.py new file mode 100644 index 000000000..461e4ed83 --- /dev/null +++ b/experiments/battery_rare_signed_tails/diff_1pct_failures.py @@ -0,0 +1,76 @@ +"""Diff the terminal battery failure lines of two 1% verification builds. + +Reads each build's ``*.gates.json``, extracts the ``us_by_origin_battery`` +failure list, and reports: lines only in the before build, lines only in the +after build, and the count present in both. Also projects each set against the +frozen full-scale adjudication classes (the 93-line baseline) by leg id so the +1% noise floor is legible. +""" + +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path + +_LEG = re.compile(r"^(?P[^:]+): ") + + +def _battery_failures(gates_path: Path) -> list[str]: + payload = json.loads(gates_path.read_text(encoding="utf-8")) + battery = payload["terminal_gates"]["gates"]["us_by_origin_battery"] + failures = battery["failures"] + if not isinstance(failures, list): + raise TypeError(f"{gates_path}: battery failures are not a list") + return failures + + +def _legs(lines: list[str]) -> set[str]: + out = set() + for line in lines: + match = _LEG.match(line) + if match is None: + raise ValueError(f"unparseable failure line: {line!r}") + out.add(match.group("leg")) + return out + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--before-gates", required=True, type=Path) + parser.add_argument("--after-gates", required=True, type=Path) + parser.add_argument( + "--baseline-gates", + type=Path, + default=Path( + "/Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/" + "populace_us_2024_stacked_pool.gates.json" + ), + help="Frozen full-scale f025 gates for baseline-class context.", + ) + args = parser.parse_args() + + before = _battery_failures(args.before_gates) + after = _battery_failures(args.after_gates) + baseline_legs = _legs(_battery_failures(args.baseline_gates)) + + before_set, after_set = set(before), set(after) + only_before = sorted(before_set - after_set) + only_after = sorted(after_set - before_set) + + print(f"before: {len(before)} failure lines ({args.before_gates})") + print(f"after: {len(after)} failure lines ({args.after_gates})") + print(f"shared: {len(before_set & after_set)}") + print(f"baseline full-scale legs: {len(baseline_legs)}") + + for title, lines in (("ONLY BEFORE", only_before), ("ONLY AFTER", only_after)): + print(f"\n== {title} ({len(lines)}) ==") + for line in lines: + leg = _LEG.match(line).group("leg") + tag = "baseline-red" if leg in baseline_legs else "not-baseline-red" + print(f"[{tag}] {line}") + + +if __name__ == "__main__": + main() diff --git a/experiments/battery_rare_signed_tails/project_baseline_1pct.py b/experiments/battery_rare_signed_tails/project_baseline_1pct.py new file mode 100644 index 000000000..14969cb0f --- /dev/null +++ b/experiments/battery_rare_signed_tails/project_baseline_1pct.py @@ -0,0 +1,132 @@ +"""Project the 1% baseline battery failures onto the frozen 48-red set. + +Establishes the "before" side of the lane's 1% verification: which of the +48 frozen full-scale QED reds already manifest at 1%, which cannot (their +carriers are too rare to survive a 1% sample), and whether the one-sided +Keogh leg emits any line at all at 1%. The output JSON is deterministic on +its three frozen inputs. +""" + +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path + +_LEG = re.compile(r"^(?P[^:]+): ") +_KEOGH_LEG = ( + "person/source_operator_retirement_distributions/" + "keogh_distributions[clone_0]/positive" +) + + +def _battery_failures(gates_path: Path) -> list[str]: + payload = json.loads(gates_path.read_text(encoding="utf-8")) + battery = payload["terminal_gates"]["gates"]["us_by_origin_battery"] + failures = battery["failures"] + if not isinstance(failures, list): + raise TypeError(f"{gates_path}: battery failures are not a list") + return failures + + +def _leg(line: str) -> str: + match = _LEG.match(line) + if match is None: + raise ValueError(f"unparseable failure line: {line!r}") + return match.group("leg") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--realized-regimes", + type=Path, + default=Path(__file__).with_name("realized_regimes.json"), + ) + parser.add_argument("--baseline-1pct-gates", required=True, type=Path) + parser.add_argument( + "--full-scale-gates", + type=Path, + default=Path( + "/Users/maxghenis/PolicyEngine/_buildo-runtime/out/stacked-f025-r1/" + "populace_us_2024_stacked_pool.gates.json" + ), + ) + parser.add_argument("--out", type=Path, default=None) + args = parser.parse_args() + + regimes = json.loads(args.realized_regimes.read_text(encoding="utf-8")) + checks = {row["leg_id"]: row for row in regimes["checks"]} + if len(checks) != 48: + raise ValueError(f"expected 48 frozen red legs, got {len(checks)}") + + one_pct_lines = _battery_failures(args.baseline_1pct_gates) + one_pct_by_leg: dict[str, list[str]] = {} + for line in one_pct_lines: + one_pct_by_leg.setdefault(_leg(line), []).append(line) + + full_lines = _battery_failures(args.full_scale_gates) + full_legs = {_leg(line) for line in full_lines} + + present = sorted(set(checks) & set(one_pct_by_leg)) + absent = sorted(set(checks) - set(one_pct_by_leg)) + noise_legs = sorted(set(one_pct_by_leg) - set(checks)) + + payload = { + "title": "1% baseline projection of the frozen 48 red QED checks", + "inputs": { + "realized_regimes": str(args.realized_regimes), + "baseline_1pct_gates": str(args.baseline_1pct_gates), + "full_scale_gates": str(args.full_scale_gates), + }, + "baseline_1pct": { + "failure_lines": len(one_pct_lines), + "unique_failing_legs": len(one_pct_by_leg), + }, + "full_scale": { + "failure_lines": len(full_lines), + "unique_failing_legs": len(full_legs), + }, + "frozen_reds_red_at_1pct": { + "count": len(present), + "legs": present, + }, + "frozen_reds_not_red_at_1pct": { + "count": len(absent), + "legs": [ + { + "leg_id": leg, + "ordinal": checks[leg]["ordinal"], + "donor_support_starved": checks[leg]["donor_support_starved"], + } + for leg in absent + ], + }, + "one_pct_only_noise_legs": { + "count": len(noise_legs), + "legs": noise_legs, + }, + "keogh_one_sided_leg": { + "leg_id": _KEOGH_LEG, + "lines_at_1pct": one_pct_by_leg.get(_KEOGH_LEG, []), + "red_at_full_scale": _KEOGH_LEG in full_legs, + "note": ( + "Native ASEC holds exactly two positive Keogh values in " + "108,073 rows; a 1% sample almost surely carries neither, so " + "the 1% gate cannot exercise this leg in either direction. " + "Gate-level proof of the Keogh repair requires a full-scale " + "build; the 1% evidence is the support-preserving cap " + "regression suite plus the frozen bank draw census." + ), + }, + } + rendered = json.dumps(payload, indent=1, sort_keys=True) + "\n" + if args.out is not None: + args.out.write_text(rendered, encoding="utf-8") + else: + print(rendered, end="") + + +if __name__ == "__main__": + main() diff --git a/experiments/battery_rare_signed_tails/realized_regimes.json b/experiments/battery_rare_signed_tails/realized_regimes.json new file mode 100644 index 000000000..26d3efee2 --- /dev/null +++ b/experiments/battery_rare_signed_tails/realized_regimes.json @@ -0,0 +1,5876 @@ +{ + "authority": { + "adjudication_json_sha256": "1c1597c8080cf4e0db0079c2a3e1bc345001525bc3ec6c296aa7575b47136c32", + "adjudication_md_sha256": "2cdf79233d56f63ca8e65305cb1c746706d88a7a57ac26ae25db575676daf3b8", + "f025_gates_sha256": "685cad63d4dc62234da72501c5a3ce9ec5a81fcd3f7b412b61474a9c1d8b423b", + "transferred_checkpoint": { + "identity_sha256": "7847224d1daa6b21cb4d063b1b54b62516b24936a302513143a685b61b7cf8e4", + "sha256": "bdc9355d92659bb28d58b1ddcd647ec303f2ad217661e17d5b4b0984e04532e8", + "size_bytes": 5578512480 + }, + "transferred_receipts_sha256": "c91a61bb20f0f2c2612bac8fdd21d2227cf746b85e2c5c2648857fd667e68aed" + }, + "checks": [ + { + "check_id": "person/adult_care/pre_subsidy_care_expenses[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "adult_care", + "leg_id": "person/adult_care/pre_subsidy_care_expenses[clone_0]/positive", + "mechanism": { + "primary": "adult_care_post_transfer_amount_shape", + "smallest_honest_change": "with held-out evidence, calibrate positive amounts on newly imputed mutable cells before adult-care reconciliation", + "status": "blocked_on_held_out_calibration_evidence" + }, + "ordinal": 2, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/adult_care/targets/001__pre_subsidy_care_expenses.h5", + "terminal": { + "acs_carrier_rows": 987, + "asec_carrier_rows": 80, + "quantile_envelope_distance": 1.8095238095238095 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/model_required_numeric/unemployment_compensation[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "model_required_numeric", + "leg_id": "person/model_required_numeric/unemployment_compensation[clone_0]/positive", + "mechanism": { + "primary": "early_transfer_unemployment_amount_shape", + "smallest_honest_change": "with predictors and carrier membership frozen, apply a target-only weighted positive-magnitude map", + "status": "blocked_on_held_out_calibration_evidence" + }, + "ordinal": 8, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/019__unemployment_compensation.h5", + "terminal": { + "acs_carrier_rows": 8419, + "asec_carrier_rows": 1012, + "quantile_envelope_distance": 0.34738143667145616 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/qualified_dividend_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/qualified_dividend_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 9, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/022__qualified_dividend_income.h5", + "terminal": { + "acs_carrier_rows": 116895, + "asec_carrier_rows": 11641, + "quantile_envelope_distance": 0.3390083880076405 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/non_qualified_dividend_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/non_qualified_dividend_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 10, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/023__non_qualified_dividend_income.h5", + "terminal": { + "acs_carrier_rows": 116895, + "asec_carrier_rows": 11641, + "quantile_envelope_distance": 0.33333333333333337 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/tax_exempt_interest_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/tax_exempt_interest_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 12, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/000__tax_exempt_interest_income.h5", + "terminal": { + "acs_carrier_rows": 10664, + "asec_carrier_rows": 1999, + "quantile_envelope_distance": 1.3333333333333333 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/short_term_capital_gains[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/short_term_capital_gains[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 13, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/024__short_term_capital_gains.h5", + "terminal": { + "acs_carrier_rows": 45177, + "asec_carrier_rows": 4199, + "quantile_envelope_distance": 0.5714285714285714 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/long_term_capital_gains_before_response[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/long_term_capital_gains_before_response[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 14, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/025__long_term_capital_gains_before_response.h5", + "terminal": { + "acs_carrier_rows": 45180, + "asec_carrier_rows": 4199, + "quantile_envelope_distance": 0.5714285714285714 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/long_term_capital_gains_on_collectibles[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/long_term_capital_gains_on_collectibles[clone_0]/positive", + "mechanism": { + "primary": "donor_evidence_starvation", + "smallest_honest_change": "after a source-label audit supplies dense evidence, fit a target-specific rare-event/sign-and-magnitude model", + "status": "blocked_on_additional_evidence" + }, + "ordinal": 16, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/001__long_term_capital_gains_on_collectibles.h5", + "terminal": { + "acs_carrier_rows": 519, + "asec_carrier_rows": 82, + "quantile_envelope_distance": 1.7305910198561216 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/non_sch_d_capital_gains[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/non_sch_d_capital_gains[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 18, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/002__non_sch_d_capital_gains.h5", + "terminal": { + "acs_carrier_rows": 8700, + "asec_carrier_rows": 1910, + "quantile_envelope_distance": 0.3333333333333333 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/social_security_retirement[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/social_security_retirement[clone_0]/positive", + "mechanism": { + "primary": "retirement_mapping_label_or_model_shape", + "smallest_honest_change": "audit source mappings and labels on a dense rung, refit the applicable model, and run the frozen predictor factorial", + "status": "blocked_on_mapping_labels_and_dense_evidence" + }, + "ordinal": 21, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/028__social_security_retirement.h5", + "terminal": { + "acs_carrier_rows": 170783, + "asec_carrier_rows": 15440, + "quantile_envelope_distance": 0.5399065173295705 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/social_security_disability[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/social_security_disability[clone_0]/positive", + "mechanism": { + "primary": "retirement_mapping_label_or_model_shape", + "smallest_honest_change": "audit source mappings and labels on a dense rung, refit the applicable model, and run the frozen predictor factorial", + "status": "blocked_on_mapping_labels_and_dense_evidence" + }, + "ordinal": 23, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/029__social_security_disability.h5", + "terminal": { + "acs_carrier_rows": 13616, + "asec_carrier_rows": 1935, + "quantile_envelope_distance": 1.3265580996694013 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/social_security_dependents[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/social_security_dependents[clone_0]/positive", + "mechanism": { + "primary": "retirement_mapping_label_or_model_shape", + "smallest_honest_change": "audit source mappings and labels on a dense rung, refit the applicable model, and run the frozen predictor factorial", + "status": "blocked_on_mapping_labels_and_dense_evidence" + }, + "ordinal": 25, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/030__social_security_dependents.h5", + "terminal": { + "acs_carrier_rows": 3180, + "asec_carrier_rows": 263, + "quantile_envelope_distance": 1.255813953488372 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/social_security_survivors[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/social_security_survivors[clone_0]/positive", + "mechanism": { + "primary": "retirement_mapping_label_or_model_shape", + "smallest_honest_change": "audit source mappings and labels on a dense rung, refit the applicable model, and run the frozen predictor factorial", + "status": "blocked_on_mapping_labels_and_dense_evidence" + }, + "ordinal": 26, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/031__social_security_survivors.h5", + "terminal": { + "acs_carrier_rows": 3094, + "asec_carrier_rows": 398, + "quantile_envelope_distance": 0.9842613410924481 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/alimony_income[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/alimony_income[clone_0]/positive", + "mechanism": { + "primary": "donor_evidence_starvation", + "smallest_honest_change": "after a source-label audit supplies dense evidence, fit a target-specific rare-event/sign-and-magnitude model", + "status": "blocked_on_additional_evidence" + }, + "ordinal": 28, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/032__alimony_income.h5", + "terminal": { + "acs_carrier_rows": 2869, + "asec_carrier_rows": 61, + "quantile_envelope_distance": 0.42105263157894735 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/alimony_expense[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/alimony_expense[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 30, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/003__alimony_expense.h5", + "terminal": { + "acs_carrier_rows": 2261, + "asec_carrier_rows": 357, + "quantile_envelope_distance": 0.3967647358865612 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/investment_income_elected_form_4952[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/investment_income_elected_form_4952[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 32, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_2/targets/001__investment_income_elected_form_4952.h5", + "terminal": { + "acs_carrier_rows": 1429, + "asec_carrier_rows": 228, + "quantile_envelope_distance": 1.1061359867330014 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/casualty_loss[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/casualty_loss[clone_0]/positive", + "mechanism": { + "primary": "donor_evidence_starvation", + "smallest_honest_change": "after a source-label audit supplies dense evidence, fit a target-specific rare-event/sign-and-magnitude model", + "status": "blocked_on_additional_evidence" + }, + "ordinal": 33, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_2/targets/005__casualty_loss.h5", + "terminal": { + "acs_carrier_rows": 633, + "asec_carrier_rows": 79, + "quantile_envelope_distance": 1.9264505505460738 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/self_employed_pension_contributions_desired[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/self_employed_pension_contributions_desired[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 35, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/000__self_employed_pension_contributions_desired.h5", + "terminal": { + "acs_carrier_rows": 9107, + "asec_carrier_rows": 768, + "quantile_envelope_distance": 0.4489795918367348 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/rental_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/rental_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 36, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/033__rental_income.h5", + "terminal": { + "acs_carrier_rows": 36145, + "asec_carrier_rows": 3800, + "quantile_envelope_distance": 0.7285129604365621 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/rental_income[clone_0]/negative#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/rental_income[clone_0]/negative", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 38, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/033__rental_income.h5", + "terminal": { + "acs_carrier_rows": 2883, + "asec_carrier_rows": 216, + "quantile_envelope_distance": 0.7368421052631579 + }, + "tested_sign": "negative", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/estate_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/estate_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 40, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/001__estate_income.h5", + "terminal": { + "acs_carrier_rows": 3138, + "asec_carrier_rows": 439, + "quantile_envelope_distance": 1.228041821138992 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/estate_income[clone_0]/negative#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/estate_income[clone_0]/negative", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 42, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/001__estate_income.h5", + "terminal": { + "acs_carrier_rows": 1413, + "asec_carrier_rows": 300, + "quantile_envelope_distance": 1.3840000000000001 + }, + "tested_sign": "negative", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/farm_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/farm_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 43, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/002__farm_income.h5", + "terminal": { + "acs_carrier_rows": 2559, + "asec_carrier_rows": 403, + "quantile_envelope_distance": 0.9084830756372754 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/farm_operations_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/farm_operations_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 44, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/034__farm_operations_income.h5", + "terminal": { + "acs_carrier_rows": 4945, + "asec_carrier_rows": 562, + "quantile_envelope_distance": 1.6363636363636365 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/farm_operations_income[clone_0]/negative#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/farm_operations_income[clone_0]/negative", + "mechanism": { + "primary": "donor_evidence_starvation", + "smallest_honest_change": "after a source-label audit supplies dense evidence, fit a target-specific rare-event/sign-and-magnitude model", + "status": "blocked_on_additional_evidence" + }, + "ordinal": 46, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/034__farm_operations_income.h5", + "terminal": { + "acs_carrier_rows": 3434, + "asec_carrier_rows": 89, + "quantile_envelope_distance": 1.9751552795031055 + }, + "tested_sign": "negative", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/farm_rent_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/farm_rent_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 48, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/003__farm_rent_income.h5", + "terminal": { + "acs_carrier_rows": 2737, + "asec_carrier_rows": 388, + "quantile_envelope_distance": 0.5416178194607267 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/farm_rent_income[clone_0]/negative#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/farm_rent_income[clone_0]/negative", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 49, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/003__farm_rent_income.h5", + "terminal": { + "acs_carrier_rows": 1930, + "asec_carrier_rows": 259, + "quantile_envelope_distance": 0.9932618204447798 + }, + "tested_sign": "negative", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/miscellaneous_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/miscellaneous_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 50, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/035__miscellaneous_income.h5", + "terminal": { + "acs_carrier_rows": 7209, + "asec_carrier_rows": 1023, + "quantile_envelope_distance": 0.4 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/puf_tax_itemization/partnership_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/partnership_income[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 52, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/004__partnership_income.h5", + "terminal": { + "acs_carrier_rows": 19169, + "asec_carrier_rows": 3040, + "quantile_envelope_distance": 0.45700108495153513 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 55, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/005__partnership_self_employment_net_earnings.h5", + "terminal": { + "acs_carrier_rows": 5918, + "asec_carrier_rows": 815, + "quantile_envelope_distance": 0.3609910128200268 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/negative#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/partnership_self_employment_net_earnings[clone_0]/negative", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 56, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/005__partnership_self_employment_net_earnings.h5", + "terminal": { + "acs_carrier_rows": 4372, + "asec_carrier_rows": 447, + "quantile_envelope_distance": 0.48275862068965497 + }, + "tested_sign": "negative", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/qualified_bdc_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/qualified_bdc_income[clone_0]/positive", + "mechanism": { + "primary": "late_transfer_then_qbi_reconciliation_shape", + "smallest_honest_change": "calibrate the joint pre-cap QBI surface while preserving the existing coupled caps and identities", + "status": "blocked_on_qbi_owner_and_held_out_evidence" + }, + "ordinal": 58, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_4/targets/006__qualified_bdc_income.h5", + "terminal": { + "acs_carrier_rows": 863, + "asec_carrier_rows": 228, + "quantile_envelope_distance": 0.751361328298312 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/qualified_reit_and_ptp_income[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/qualified_reit_and_ptp_income[clone_0]/positive", + "mechanism": { + "primary": "late_transfer_then_qbi_reconciliation_shape", + "smallest_honest_change": "calibrate the joint pre-cap QBI surface while preserving the existing coupled caps and identities", + "status": "blocked_on_qbi_owner_and_held_out_evidence" + }, + "ordinal": 60, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_4/targets/007__qualified_reit_and_ptp_income.h5", + "terminal": { + "acs_carrier_rows": 7781, + "asec_carrier_rows": 1740, + "quantile_envelope_distance": 1.1576246415399343 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/unadjusted_basis_qualified_property[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/unadjusted_basis_qualified_property[clone_0]/positive", + "mechanism": { + "primary": "late_transfer_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-scoped weighted conditional-magnitude map to mutable late-transfer draws", + "status": "blocked_on_held_out_calibration_evidence" + }, + "ordinal": 62, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_5/targets/003__unadjusted_basis_qualified_property.h5", + "terminal": { + "acs_carrier_rows": 41761, + "asec_carrier_rows": 6137, + "quantile_envelope_distance": 0.5901547288221703 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/puf_tax_itemization/w2_wages_from_qualified_business[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "person/puf_tax_itemization/w2_wages_from_qualified_business[clone_0]/positive", + "mechanism": { + "primary": "late_transfer_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-scoped weighted conditional-magnitude map to mutable late-transfer draws", + "status": "blocked_on_held_out_calibration_evidence" + }, + "ordinal": 64, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/puf_tax_itemization__batch_5/targets/004__w2_wages_from_qualified_business.h5", + "terminal": { + "acs_carrier_rows": 3442, + "asec_carrier_rows": 509, + "quantile_envelope_distance": 1.3371284036777606 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_child_support/child_support_expense[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_child_support", + "leg_id": "person/source_operator_child_support/child_support_expense[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 68, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_child_support/targets/000__child_support_expense.h5", + "terminal": { + "acs_carrier_rows": 2585, + "asec_carrier_rows": 711, + "quantile_envelope_distance": 0.2857142857142857 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_child_support/child_support_received[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_child_support", + "leg_id": "person/source_operator_child_support/child_support_received[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 70, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_child_support/targets/001__child_support_received.h5", + "terminal": { + "acs_carrier_rows": 5087, + "asec_carrier_rows": 1189, + "quantile_envelope_distance": 0.5595567867036011 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_disability_benefits/disability_benefits[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_disability_benefits", + "leg_id": "person/source_operator_disability_benefits/disability_benefits[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 71, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_disability_benefits/targets/000__disability_benefits.h5", + "terminal": { + "acs_carrier_rows": 5777, + "asec_carrier_rows": 741, + "quantile_envelope_distance": 0.5741444866920152 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "source_operator_prior_year_income", + "leg_id": "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 73, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/039__self_employment_income_last_year.h5", + "terminal": { + "acs_carrier_rows": 29502, + "asec_carrier_rows": 2608, + "quantile_envelope_distance": 0.4 + }, + "tested_sign": "positive", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/negative#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "three_sign", + "gate_family": "source_operator_prior_year_income", + "leg_id": "person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/negative", + "mechanism": { + "primary": "donor_evidence_starvation", + "smallest_honest_change": "after a source-label audit supplies dense evidence, fit a target-specific rare-event/sign-and-magnitude model", + "status": "blocked_on_additional_evidence" + }, + "ordinal": 75, + "pattern_profile": "early_person", + "target_ref": "asec_survey_to_acs/targets/039__self_employment_income_last_year.h5", + "terminal": { + "acs_carrier_rows": 2398, + "asec_carrier_rows": 48, + "quantile_envelope_distance": 1.9920159680638723 + }, + "tested_sign": "negative", + "transfer_partition": "early_gap_fill" + }, + { + "check_id": "person/source_operator_retirement_distributions/taxable_401k_distributions[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_retirement_distributions", + "leg_id": "person/source_operator_retirement_distributions/taxable_401k_distributions[clone_0]/positive", + "mechanism": { + "primary": "upstream_retirement_support_deletion", + "smallest_honest_change": "retain the union of retirement-target carriers before the fixed-size training cap and reweight only sampled common zeros", + "status": "implemented" + }, + "ordinal": 78, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_retirement_distributions/targets/002__taxable_401k_distributions.h5", + "terminal": { + "acs_carrier_rows": 16979, + "asec_carrier_rows": 2057, + "quantile_envelope_distance": 0.46153846153846156 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_retirement_distributions/taxable_403b_distributions[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_retirement_distributions", + "leg_id": "person/source_operator_retirement_distributions/taxable_403b_distributions[clone_0]/positive", + "mechanism": { + "primary": "upstream_retirement_support_deletion", + "smallest_honest_change": "retain the union of retirement-target carriers before the fixed-size training cap and reweight only sampled common zeros", + "status": "implemented" + }, + "ordinal": 80, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_retirement_distributions/targets/003__taxable_403b_distributions.h5", + "terminal": { + "acs_carrier_rows": 1371, + "asec_carrier_rows": 161, + "quantile_envelope_distance": 1.4639175257731958 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_retirement_distributions/taxable_sep_distributions[clone_0]/positive#quantile", + "donor_support_starved": true, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_retirement_distributions", + "leg_id": "person/source_operator_retirement_distributions/taxable_sep_distributions[clone_0]/positive", + "mechanism": { + "primary": "upstream_retirement_support_deletion", + "smallest_honest_change": "retain the union of retirement-target carriers before the fixed-size training cap and reweight only sampled common zeros", + "status": "implemented" + }, + "ordinal": 82, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_retirement_distributions/targets/004__taxable_sep_distributions.h5", + "terminal": { + "acs_carrier_rows": 9406, + "asec_carrier_rows": 61, + "quantile_envelope_distance": 1.7696335078534031 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_weeks_unemployed", + "leg_id": "person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 84, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5", + "terminal": { + "acs_carrier_rows": 5127, + "asec_carrier_rows": 3387, + "quantile_envelope_distance": 0.5882352941176471 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "person/source_operator_workers_compensation/workers_compensation[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "person", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_workers_compensation", + "leg_id": "person/source_operator_workers_compensation/workers_compensation[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 86, + "pattern_profile": "late_person", + "target_ref": "late_producer_dag/person/source_operator_workers_compensation/targets/000__workers_compensation.h5", + "terminal": { + "acs_carrier_rows": 1605, + "asec_carrier_rows": 296, + "quantile_envelope_distance": 1.99800099950025 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "spm_unit/source_operator_energy_subsidy/spm_unit_energy_subsidy[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "spm_unit", + "fitter_regime": "zero_inflated_positive", + "gate_family": "source_operator_energy_subsidy", + "leg_id": "spm_unit/source_operator_energy_subsidy/spm_unit_energy_subsidy[clone_0]/positive", + "mechanism": { + "primary": "source_operator_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign amount map within the owning source family while preserving reconciliation and producer cells", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 88, + "pattern_profile": "late_group", + "target_ref": "late_producer_dag/spm_unit/source_operator_energy_subsidy/targets/000__spm_unit_energy_subsidy.h5", + "terminal": { + "acs_carrier_rows": 10119, + "asec_carrier_rows": 1534, + "quantile_envelope_distance": 0.4539877300613497 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "tax_unit/puf_tax_itemization/domestic_production_ald[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "tax_unit", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "tax_unit/puf_tax_itemization/domestic_production_ald[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 90, + "pattern_profile": "late_group", + "target_ref": "late_producer_dag/tax_unit/puf_tax_itemization/targets/000__domestic_production_ald.h5", + "terminal": { + "acs_carrier_rows": 1760, + "asec_carrier_rows": 335, + "quantile_envelope_distance": 0.296143753739345 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + }, + { + "check_id": "tax_unit/puf_tax_itemization/unrecaptured_section_1250_gain[clone_0]/positive#quantile", + "donor_support_starved": false, + "entity": "tax_unit", + "fitter_regime": "zero_inflated_positive", + "gate_family": "puf_tax_itemization", + "leg_id": "tax_unit/puf_tax_itemization/unrecaptured_section_1250_gain[clone_0]/positive", + "mechanism": { + "primary": "gated_conditional_magnitude_shape", + "smallest_honest_change": "apply a target-and-sign-scoped weighted conditional-magnitude map at the owning transfer boundary, mutating imputed cells only", + "status": "blocked_on_owner_specific_held_out_evidence" + }, + "ordinal": 92, + "pattern_profile": "late_group", + "target_ref": "late_producer_dag/tax_unit/puf_tax_itemization/targets/001__unrecaptured_section_1250_gain.h5", + "terminal": { + "acs_carrier_rows": 2558, + "asec_carrier_rows": 402, + "quantile_envelope_distance": 0.8427184466019417 + }, + "tested_sign": "positive", + "transfer_partition": "late_producer_complement" + } + ], + "evidence_boundary": { + "bank_schema": 1, + "pattern_metadata": "persisted_frozen_checkpoint", + "support_and_regime": "fit_boundary_reconstruction", + "warning": "do not recompute early-target support from the final transferred pool because later producers may mutate those fields" + }, + "keogh_one_sided_leg": { + "frozen_acs_positive_rows": 0, + "loss_stage": "retirement_training_cap_before_transfer_fit", + "native_asec_positive_rows": 2, + "native_asec_positive_values": [ + 2040.0, + 30000.0 + ], + "old_training_cap_positive_rows": 0, + "structurally_absent": false + }, + "pattern_profiles": { + "early_person": [ + "pattern_00_677f6490", + "pattern_01_5874881e", + "pattern_02_7c3bceda", + "pattern_03_04f75638" + ], + "late_group": [ + "pattern_00_741356db", + "pattern_01_7c62ad55", + "pattern_02_d9f964c9", + "pattern_03_2e5c739d" + ], + "late_person": [ + "pattern_00_36785ebf", + "pattern_01_c6777728", + "pattern_02_5e7dd311", + "pattern_03_76a0101a" + ] + }, + "retirement_cap_support": { + "keogh_distributions": { + "native": 2, + "retained": 0 + }, + "taxable_401k_distributions": { + "native": 2057, + "retained": 102 + }, + "taxable_403b_distributions": { + "native": 161, + "retained": 4 + }, + "taxable_sep_distributions": { + "native": 61, + "retained": 2 + } + }, + "schema_version": 1, + "targets": [ + { + "checkpoint_relpath": "asec_survey_to_acs/targets/019__unemployment_compensation.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "2c909958e1ac995796e843e3a92b100fbb7281ce647b26de160577c87eaec583", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1012, + "total": 108073, + "zero": 107061 + }, + "entity": "person", + "exported_targets": [ + "unemployment_compensation" + ], + "model_target": "unemployment_compensation", + "model_targets": [ + "health_insurance_premiums_without_medicare_part_b", + "hours_worked_last_week", + "other_medical_expenses", + "over_the_counter_health_expenses", + "tax_exempt_private_pension_income", + "unemployment_compensation", + "veterans_benefits" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 4223074724, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 525991705, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 631826395, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 2422673074, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "model_required_numeric" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/022__qualified_dividend_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "921cf308df407a1f82b5e40b1a20a157e6f9813a8de7579e91a1c48cafa80492", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 11641, + "total": 108073, + "zero": 96432 + }, + "entity": "person", + "exported_targets": [ + "qualified_dividend_income" + ], + "model_target": "qualified_dividend_income", + "model_targets": [ + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "social_security_retirement" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3923138593, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2301314375, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 2642943411, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1810485691, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/023__non_qualified_dividend_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "dc17b6d933d029a78c51be750fe6cd31a2d7d266bcb341a249128bfc30db1c6f", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 11641, + "total": 108073, + "zero": 96432 + }, + "entity": "person", + "exported_targets": [ + "non_qualified_dividend_income" + ], + "model_target": "non_qualified_dividend_income", + "model_targets": [ + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "social_security_retirement" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3923138593, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2301314375, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 2642943411, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1810485691, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/024__short_term_capital_gains.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "3c527d54620af187ca99bd69a11ce7692b1b3072602c3b202cf76a7f298d15f4", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 4199, + "total": 108073, + "zero": 103874 + }, + "entity": "person", + "exported_targets": [ + "short_term_capital_gains" + ], + "model_target": "short_term_capital_gains", + "model_targets": [ + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "social_security_retirement" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3923138593, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2301314375, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 2642943411, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1810485691, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/025__long_term_capital_gains_before_response.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "68fcebf2c96e6d2f694f97b382bbd4ac6bce5c8657cce2b9195d75a31cad70d8", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 4199, + "total": 108073, + "zero": 103874 + }, + "entity": "person", + "exported_targets": [ + "long_term_capital_gains_before_response" + ], + "model_target": "long_term_capital_gains_before_response", + "model_targets": [ + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "social_security_retirement" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3923138593, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2301314375, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 2642943411, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1810485691, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/028__social_security_retirement.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "cc8b9a8a6d613188b50e5eb6162dda04b7aafac18f60ab7fcc3e00af17b22017", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 15440, + "total": 108073, + "zero": 92633 + }, + "entity": "person", + "exported_targets": [ + "social_security_retirement" + ], + "model_target": "social_security_retirement", + "model_targets": [ + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "social_security_retirement" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3923138593, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2301314375, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 2642943411, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1810485691, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/029__social_security_disability.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "360747655877a5433c1a34d3291f68adedc5b7f713891aca44a6854723bf0bea", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1935, + "total": 108073, + "zero": 106138 + }, + "entity": "person", + "exported_targets": [ + "social_security_disability" + ], + "model_target": "social_security_disability", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/030__social_security_dependents.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "97e91fffa9d0fa311026d0e6b6ae647a2f57a39443b95b62fe47fc4ba2c53b80", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 263, + "total": 108073, + "zero": 107810 + }, + "entity": "person", + "exported_targets": [ + "social_security_dependents" + ], + "model_target": "social_security_dependents", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/031__social_security_survivors.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "bdcfdb8fbc72018bfaa8fb3f793dc27384941f772542d3038d62184a4d84b2ca", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 398, + "total": 108073, + "zero": 107675 + }, + "entity": "person", + "exported_targets": [ + "social_security_survivors" + ], + "model_target": "social_security_survivors", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/032__alimony_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "dcc639fa7a2b349cca2643a10e532faa78371a0f633be6f3b22422a61efb06bb", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 61, + "total": 108073, + "zero": 108012 + }, + "entity": "person", + "exported_targets": [ + "alimony_income" + ], + "model_target": "alimony_income", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/033__rental_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "6341a56fc050bb908668c20705580c3e4714bdede9210d47592ffe5bd50b3469", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 216, + "positive": 3800, + "total": 108073, + "zero": 104057 + }, + "entity": "person", + "exported_targets": [ + "rental_income" + ], + "model_target": "rental_income", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/034__farm_operations_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "5004c2699670b1a819aba163361d23db127223d28f640cc05597fb29fb032d2b", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 89, + "positive": 562, + "total": 108073, + "zero": 107422 + }, + "entity": "person", + "exported_targets": [ + "farm_operations_income" + ], + "model_target": "farm_operations_income", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/035__miscellaneous_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "3808ba04b7a663d95f3eef2dc77b89dc44c4b5d2209a6ba253b07aa018daac19", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1023, + "total": 108073, + "zero": 107050 + }, + "entity": "person", + "exported_targets": [ + "miscellaneous_income" + ], + "model_target": "miscellaneous_income", + "model_targets": [ + "social_security_disability", + "social_security_dependents", + "social_security_survivors", + "alimony_income", + "rental_income", + "farm_operations_income", + "miscellaneous_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1288835733, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 156295273, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 3083463362, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1471048381, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "asec_survey_to_acs/targets/039__self_employment_income_last_year.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "bcecec0fc2d3840c603addf5bd98f9d895d63058779c25427603a968a9dba227", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 48, + "positive": 2608, + "total": 108073, + "zero": 105417 + }, + "entity": "person", + "exported_targets": [ + "self_employment_income_last_year" + ], + "model_target": "self_employment_income_last_year", + "model_targets": [ + "previous_year_income_available", + "self_employment_income_last_year" + ], + "patterns": [ + { + "pattern_id": "pattern_00_677f6490", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3623888489, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_5874881e", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 3481156135, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_7c3bceda", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c7f1655b5237c3a5bf64e7b517a8c753d24c57bc63e0c668bedb3f19f20e883e", + "recipient_rows": 127119, + "seed": 1593533616, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_04f75638", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 4032385192, + "weight_kind": "importance", + "weight_sha256": "3685f702af99b39bd723db766307f31df6d4b5970ce208662afdb834ac582ef2", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_prior_year_income" + }, + { + "checkpoint_relpath": "late_producer_dag/person/adult_care/targets/001__pre_subsidy_care_expenses.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "fb0a374de2bfde39aeb504fc59fdf0b9cb4a848522c55afaca04f9f3e0a4635f", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 74, + "total": 108073, + "zero": 107999 + }, + "entity": "person", + "exported_targets": [ + "pre_subsidy_care_expenses" + ], + "model_target": "pre_subsidy_care_expenses", + "model_targets": [ + "is_incapable_of_self_care", + "pre_subsidy_care_expenses" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 2766492390, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 1107789513, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 4292529732, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 4095525654, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "adult_care" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/000__tax_exempt_interest_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "ac133c15083fb0fdc422a1120c182203955498b7b7bb13d7b02a46c34b543cce", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1830, + "total": 108073, + "zero": 106243 + }, + "entity": "person", + "exported_targets": [ + "tax_exempt_interest_income" + ], + "model_target": "tax_exempt_interest_income", + "model_targets": [ + "tax_exempt_interest_income", + "long_term_capital_gains_on_collectibles", + "non_sch_d_capital_gains", + "alimony_expense", + "salt_refund_income", + "charitable_cash_donations", + "charitable_non_cash_donations", + "home_mortgage_interest" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 480626731, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1265386610, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3180740654, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 3218931186, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/001__long_term_capital_gains_on_collectibles.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "18cac7e448293a736483b7a5a2f559ceb3b84d685aee829d0befaca7285c76b8", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 18, + "total": 108073, + "zero": 108055 + }, + "entity": "person", + "exported_targets": [ + "long_term_capital_gains_on_collectibles" + ], + "model_target": "long_term_capital_gains_on_collectibles", + "model_targets": [ + "tax_exempt_interest_income", + "long_term_capital_gains_on_collectibles", + "non_sch_d_capital_gains", + "alimony_expense", + "salt_refund_income", + "charitable_cash_donations", + "charitable_non_cash_donations", + "home_mortgage_interest" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 480626731, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1265386610, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3180740654, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 3218931186, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/002__non_sch_d_capital_gains.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "4420095de7ba20b661484b7d4f3f8d0e72c635ba0c9c01186f9f683235dd2d18", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1853, + "total": 108073, + "zero": 106220 + }, + "entity": "person", + "exported_targets": [ + "non_sch_d_capital_gains" + ], + "model_target": "non_sch_d_capital_gains", + "model_targets": [ + "tax_exempt_interest_income", + "long_term_capital_gains_on_collectibles", + "non_sch_d_capital_gains", + "alimony_expense", + "salt_refund_income", + "charitable_cash_donations", + "charitable_non_cash_donations", + "home_mortgage_interest" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 480626731, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1265386610, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3180740654, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 3218931186, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_1/targets/003__alimony_expense.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "ba73cc06d53038e4216c604917ac055c3c244b38caf85b2580191a5fadf1f389", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 200, + "total": 108073, + "zero": 107873 + }, + "entity": "person", + "exported_targets": [ + "alimony_expense" + ], + "model_target": "alimony_expense", + "model_targets": [ + "tax_exempt_interest_income", + "long_term_capital_gains_on_collectibles", + "non_sch_d_capital_gains", + "alimony_expense", + "salt_refund_income", + "charitable_cash_donations", + "charitable_non_cash_donations", + "home_mortgage_interest" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 480626731, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1265386610, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3180740654, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 3218931186, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_1" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_2/targets/001__investment_income_elected_form_4952.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "d90276b77940817788a7ec6a670c03dbc1bd5c3eda94778613c3b4ad1b148ad9", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 66, + "total": 108073, + "zero": 108007 + }, + "entity": "person", + "exported_targets": [ + "investment_income_elected_form_4952" + ], + "model_target": "investment_income_elected_form_4952", + "model_targets": [ + "investment_interest_expense", + "investment_income_elected_form_4952", + "student_loan_interest", + "educator_expense", + "qualified_tuition_expenses", + "casualty_loss", + "unreimbursed_business_employee_expenses", + "traditional_ira_contributions_desired" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1023716225, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 399142434, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 767954589, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 4013049835, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_2/targets/005__casualty_loss.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "cf6713fd58abcc5a094200cefbd9ab19ef5030de87b36be7519db2313cf9fb74", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 27, + "total": 108073, + "zero": 108046 + }, + "entity": "person", + "exported_targets": [ + "casualty_loss" + ], + "model_target": "casualty_loss", + "model_targets": [ + "investment_interest_expense", + "investment_income_elected_form_4952", + "student_loan_interest", + "educator_expense", + "qualified_tuition_expenses", + "casualty_loss", + "unreimbursed_business_employee_expenses", + "traditional_ira_contributions_desired" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 1023716225, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 399142434, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 767954589, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 4013049835, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_2" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/000__self_employed_pension_contributions_desired.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "e89a45ff8e5f3bade22f99ecab9a5725d3ba61a85da9d4325c758cc681e48e27", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 597, + "total": 108073, + "zero": 107476 + }, + "entity": "person", + "exported_targets": [ + "self_employed_pension_contributions_desired" + ], + "model_target": "self_employed_pension_contributions_desired", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/001__estate_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "a910e614743314b407f903c82e4534403e1e8081e02f00308476e370c915fe10", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 99, + "positive": 440, + "total": 108073, + "zero": 107534 + }, + "entity": "person", + "exported_targets": [ + "estate_income" + ], + "model_target": "estate_income", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/002__farm_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "763262ae654f712d4f6b5947ae91f3d3acce6b58d84a8ce410c1c2f2bb8bb1df", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 141, + "total": 108073, + "zero": 107932 + }, + "entity": "person", + "exported_targets": [ + "farm_income" + ], + "model_target": "farm_income", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/003__farm_rent_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "6e9e65c2e0264108a898fd27021c730e943172eec98bb0309a9f517ac839ea16", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 111, + "positive": 268, + "total": 108073, + "zero": 107694 + }, + "entity": "person", + "exported_targets": [ + "farm_rent_income" + ], + "model_target": "farm_rent_income", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/004__partnership_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "fd44282d8127b6544e0d9ce5cd9b8a7212bcb829044bfb033103b3c942fc0e8b", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 1182, + "positive": 3102, + "total": 108073, + "zero": 103789 + }, + "entity": "person", + "exported_targets": [ + "partnership_income" + ], + "model_target": "partnership_income", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_3/targets/005__partnership_self_employment_net_earnings.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "486cda844c0e87a50a9f903e1cf33f46a0e089a1f7ef1b31b6947b689fb1dc01", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 385, + "positive": 706, + "total": 108073, + "zero": 106982 + }, + "entity": "person", + "exported_targets": [ + "partnership_self_employment_net_earnings" + ], + "model_target": "partnership_self_employment_net_earnings", + "model_targets": [ + "self_employed_pension_contributions_desired", + "estate_income", + "farm_income", + "farm_rent_income", + "partnership_income", + "partnership_self_employment_net_earnings", + "estate_income_would_be_qualified", + "farm_operations_income_would_be_qualified" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3934067015, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 2164100028, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 1121587576, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "three_sign", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 1650149591, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_3" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_4/targets/006__qualified_bdc_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "ebfa5f7c9fe2658002a2b8642dc750475fb577512e4cb58fc388104484543244", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 862, + "total": 108073, + "zero": 107211 + }, + "entity": "person", + "exported_targets": [ + "qualified_bdc_income" + ], + "model_target": "qualified_bdc_income", + "model_targets": [ + "farm_rent_income_would_be_qualified", + "partnership_s_corp_income_would_be_qualified", + "rental_income_would_be_qualified", + "self_employment_income_would_be_qualified", + "sstb_self_employment_income_would_be_qualified", + "business_is_sstb", + "qualified_bdc_income", + "qualified_reit_and_ptp_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3713530124, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 737792682, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 2843109911, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 551386642, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_4" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_4/targets/007__qualified_reit_and_ptp_income.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "3e02865a830c2e1747a4d46e5c216568a671635b6d3d6bf04c9337953c0f054f", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 5818, + "total": 108073, + "zero": 102255 + }, + "entity": "person", + "exported_targets": [ + "qualified_reit_and_ptp_income" + ], + "model_target": "qualified_reit_and_ptp_income", + "model_targets": [ + "farm_rent_income_would_be_qualified", + "partnership_s_corp_income_would_be_qualified", + "rental_income_would_be_qualified", + "self_employment_income_would_be_qualified", + "sstb_self_employment_income_would_be_qualified", + "business_is_sstb", + "qualified_bdc_income", + "qualified_reit_and_ptp_income" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3713530124, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 737792682, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 2843109911, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 551386642, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_4" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_5/targets/003__unadjusted_basis_qualified_property.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "a6cbd8fe56fd0158008c2a8be73fdc612a6380c8da853766a1875e132f357ee7", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 6136, + "total": 108073, + "zero": 101937 + }, + "entity": "person", + "exported_targets": [ + "unadjusted_basis_qualified_property" + ], + "model_target": "unadjusted_basis_qualified_property", + "model_targets": [ + "sstb_self_employment_income_before_lsr", + "sstb_unadjusted_basis_qualified_property", + "sstb_w2_wages_from_qualified_business", + "unadjusted_basis_qualified_property", + "w2_wages_from_qualified_business" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3256880575, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1981910850, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3616149052, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 267331984, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_5" + }, + { + "checkpoint_relpath": "late_producer_dag/person/puf_tax_itemization__batch_5/targets/004__w2_wages_from_qualified_business.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "16b970f5b55485bc222ad66698a7e0e164de979ad9d1b1f199f82ef2dc5fb196", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 343, + "total": 108073, + "zero": 107730 + }, + "entity": "person", + "exported_targets": [ + "w2_wages_from_qualified_business" + ], + "model_target": "w2_wages_from_qualified_business", + "model_targets": [ + "sstb_self_employment_income_before_lsr", + "sstb_unadjusted_basis_qualified_property", + "sstb_w2_wages_from_qualified_business", + "unadjusted_basis_qualified_property", + "w2_wages_from_qualified_business" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "c137a7fc469f40344c8f8a687451eef3a7e0605516508d364721606519bd5694", + "recipient_rows": 478, + "seed": 3256880575, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "b21d1cf4be9d2ca61b0394849ea57243ac41f669da1ec09b308e8b9bd8aa28d9", + "recipient_rows": 45032, + "seed": 1981910850, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "52c246c679b2c6c54b0f808e201c031f7beb10e9088677fbec47f84573179dbf", + "recipient_rows": 235192, + "seed": 3616149052, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "ed4a6d0493b44a7052773c450af956b8d5c53e081bc45f45ac9484a962f4831d", + "recipient_rows": 683997, + "seed": 267331984, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization__batch_5" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_child_support/targets/000__child_support_expense.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "9f72fc32641d774e7f4b00c0734c94f3ec25edd6ac967ed544deb3f8dc9fd0ed", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 397, + "total": 108073, + "zero": 107676 + }, + "entity": "person", + "exported_targets": [ + "child_support_expense" + ], + "model_target": "child_support_expense", + "model_targets": [ + "child_support_expense", + "child_support_received" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 1450711095, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 2365058378, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 4227133053, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 1507952518, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_child_support" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_child_support/targets/001__child_support_received.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "4954d6978bbcc5afcbf8dc3659266426191bc9b87973cfe06f8a806c04d0b0ed", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 643, + "total": 108073, + "zero": 107430 + }, + "entity": "person", + "exported_targets": [ + "child_support_received" + ], + "model_target": "child_support_received", + "model_targets": [ + "child_support_expense", + "child_support_received" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 1450711095, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 2365058378, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 4227133053, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 1507952518, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_child_support" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_disability_benefits/targets/000__disability_benefits.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "97f835d1f89e7c6d725b307182446c5cb25832ceddbc5aefbb22e291c4ee18dd", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 625, + "total": 108073, + "zero": 107448 + }, + "entity": "person", + "exported_targets": [ + "disability_benefits" + ], + "model_target": "disability_benefits", + "model_targets": [ + "disability_benefits" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 4001380327, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 2409613114, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 184210565, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 1554848874, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_disability_benefits" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_retirement_distributions/targets/002__taxable_401k_distributions.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "c2c6dd45994a514ddde91fadc25c489752d8852399245bcf59494e9811185341", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1766, + "total": 108073, + "zero": 106307 + }, + "entity": "person", + "exported_targets": [ + "taxable_401k_distributions" + ], + "model_target": "taxable_401k_distributions", + "model_targets": [ + "keogh_distributions", + "tax_exempt_ira_distributions", + "taxable_401k_distributions", + "taxable_403b_distributions", + "taxable_sep_distributions" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 2593627294, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 1021650046, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 1130471859, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 954292944, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_retirement_distributions" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_retirement_distributions/targets/003__taxable_403b_distributions.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "e51fffdacd7f65e7d0e9eeb00256ecb6f213e9354ca1990cd60983a0638b000e", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 21, + "total": 108073, + "zero": 108052 + }, + "entity": "person", + "exported_targets": [ + "taxable_403b_distributions" + ], + "model_target": "taxable_403b_distributions", + "model_targets": [ + "keogh_distributions", + "tax_exempt_ira_distributions", + "taxable_401k_distributions", + "taxable_403b_distributions", + "taxable_sep_distributions" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 2593627294, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 1021650046, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 1130471859, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 954292944, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_retirement_distributions" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_retirement_distributions/targets/004__taxable_sep_distributions.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "4d06e845ae3fe925582c700601a119b61754d992301db05b36ee01a5d5de79da", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 916, + "total": 108073, + "zero": 107157 + }, + "entity": "person", + "exported_targets": [ + "taxable_sep_distributions" + ], + "model_target": "taxable_sep_distributions", + "model_targets": [ + "keogh_distributions", + "tax_exempt_ira_distributions", + "taxable_401k_distributions", + "taxable_403b_distributions", + "taxable_sep_distributions" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 2593627294, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 1021650046, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 1130471859, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 954292944, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_retirement_distributions" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "021b9428ad0fb8230f82c11e1f7c8f4fc7b5495178a1c1ab8c5ac480903a2ff9", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 573, + "total": 108073, + "zero": 107500 + }, + "entity": "person", + "exported_targets": [ + "weeks_unemployed" + ], + "model_target": "weeks_unemployed", + "model_targets": [ + "weeks_unemployed" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 3866957398, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 2084038246, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 405356533, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 3704148304, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_weeks_unemployed" + }, + { + "checkpoint_relpath": "late_producer_dag/person/source_operator_workers_compensation/targets/000__workers_compensation.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "a9bb575650ac235730574a8989b8eba2b9180ffe08f35d7ecabe4fdf4b5ea2c4", + "donor_identity": { + "index_sha256": "8ecfa8bcb4a52bdb34654a00a910a698463337d6d10cc2da8750bb29bd47a8d2", + "rows": 108073 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 45, + "total": 108073, + "zero": 108028 + }, + "entity": "person", + "exported_targets": [ + "workers_compensation" + ], + "model_target": "workers_compensation", + "model_targets": [ + "workers_compensation" + ], + "patterns": [ + { + "pattern_id": "pattern_00_36785ebf", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "7e01b6ba6eaa700eff405ec7fb0eb4b28c325ea2c1add0d8f44406073c4769b8", + "recipient_rows": 956, + "seed": 12348100, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_c6777728", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3e292ba170e16d744fa00c7306d1634ae96c3291a467a26ff0bad289c90cea10", + "recipient_rows": 90076, + "seed": 199336639, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_5e7dd311", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "6435788de4fbd59adfcb56194f31c2350f785095a755b49cdbf128486b0b9f31", + "recipient_rows": 260649, + "seed": 1261120184, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_76a0101a", + "predictors": [ + "age", + "is_female", + "__acs_transfer_state_fips", + "__acs_transfer_employment_income", + "__acs_transfer_self_employment_income", + "__acs_transfer_social_security_income", + "__acs_transfer_retirement_income", + "__acs_transfer_interest_dividend_rental_income", + "__acs_transfer_is_household_head", + "__acs_transfer_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9a353dc7cace7d6b945f7b8e6e5d00811ed5b8b4fc958347ccb1531c6c69b993", + "recipient_rows": 1385159, + "seed": 3425692264, + "weight_kind": "importance", + "weight_sha256": "53810291b0465f5da1ed2ebe0d25cc6f54e54e95619a5590c1eed2c6d5cd29ce", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_workers_compensation" + }, + { + "checkpoint_relpath": "late_producer_dag/spm_unit/source_operator_energy_subsidy/targets/000__spm_unit_energy_subsidy.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "cb62ffd8319e0085170da7f9a05a5dc265b4ce59343684bd5e0c60fd4d469c9b", + "donor_identity": { + "index_sha256": "371654a8deb82ed2dc168c65729647e565b0044e06a6a03ce0d0976d17466865", + "rows": 43961 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 1030, + "total": 43961, + "zero": 42931 + }, + "entity": "spm_unit", + "exported_targets": [ + "spm_unit_energy_subsidy" + ], + "model_target": "spm_unit_energy_subsidy", + "model_targets": [ + "spm_unit_energy_subsidy" + ], + "patterns": [ + { + "pattern_id": "pattern_00_741356db", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "24f1e81fa0229bf2009cbdc2140e1083f575555af167a1446ea040a975ae0584", + "recipient_rows": 956, + "seed": 1548779180, + "weight_kind": "importance", + "weight_sha256": "2da51d1cd8dd1d548acad56ac7600553dfb748ccc31cb5113bdbc80c3abf1f4d", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_7c62ad55", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "df127013b73b4e6bccbbd9a315cd33e834f96d7bf541fc05e3b55bc9d7b73b1c", + "recipient_rows": 90076, + "seed": 3759624227, + "weight_kind": "importance", + "weight_sha256": "2da51d1cd8dd1d548acad56ac7600553dfb748ccc31cb5113bdbc80c3abf1f4d", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_d9f964c9", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "e16e9ba15342630223f06dab8439e9e022879d66d2c2e9e505c2d3c920aeebae", + "recipient_rows": 148657, + "seed": 398736363, + "weight_kind": "importance", + "weight_sha256": "2da51d1cd8dd1d548acad56ac7600553dfb748ccc31cb5113bdbc80c3abf1f4d", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_2e5c739d", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "84ca0bbff34035d91636ba1400d2807c1c943056136ff03fc7535b436cc49faf", + "recipient_rows": 535056, + "seed": 1170660531, + "weight_kind": "importance", + "weight_sha256": "2da51d1cd8dd1d548acad56ac7600553dfb748ccc31cb5113bdbc80c3abf1f4d", + "zero_atol": 1e-06 + } + ], + "transfer_family": "source_operator_energy_subsidy" + }, + { + "checkpoint_relpath": "late_producer_dag/tax_unit/puf_tax_itemization/targets/000__domestic_production_ald.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "3ec9e6a534c7dd7b54eca4244cb92b8ac54afda11526ae275fad5e60b8716f2b", + "donor_identity": { + "index_sha256": "da2a8ee2216d1b7ffc77f19d0bf8e74978a8a5d9e46c6ccae7d13e15cb9a2fb0", + "rows": 57630 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 278, + "total": 57630, + "zero": 57352 + }, + "entity": "tax_unit", + "exported_targets": [ + "domestic_production_ald" + ], + "model_target": "domestic_production_ald", + "model_targets": [ + "domestic_production_ald", + "unrecaptured_section_1250_gain", + "first_home_mortgage_balance", + "first_home_mortgage_interest", + "first_home_mortgage_origination_year", + "health_savings_account_ald" + ], + "patterns": [ + { + "pattern_id": "pattern_00_741356db", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "390c68ba6f46c5bc09fe1b2938b6647cfa927effeb01a9111751bc6eba969b5e", + "recipient_rows": 478, + "seed": 305989492, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_7c62ad55", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9ea51d7d64717b0f1279b05fdf5554118eef07e5b48e8ec6f655cef5aaab8ac8", + "recipient_rows": 45032, + "seed": 2134600251, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_d9f964c9", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "f20451f9d64d2cebff4eeb89fac3fcde0d3900694aab47b42e7d6c97c5c9d2bd", + "recipient_rows": 130183, + "seed": 3523001447, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_2e5c739d", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3bd55af66d89dc97b1e06b66c7e058cd6ca5cce9f8de1a84066d2646b8e288cf", + "recipient_rows": 406625, + "seed": 2968274776, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization" + }, + { + "checkpoint_relpath": "late_producer_dag/tax_unit/puf_tax_itemization/targets/001__unrecaptured_section_1250_gain.h5", + "checkpoint_schema_version": 1, + "content_metadata_sha256": "cddd073a682f499b4a1a86ce1e907b55bb3ec544b63bb846815b2f48784f91d1", + "donor_identity": { + "index_sha256": "da2a8ee2216d1b7ffc77f19d0bf8e74978a8a5d9e46c6ccae7d13e15cb9a2fb0", + "rows": 57630 + }, + "donor_support": { + "common_across_patterns": true, + "evidence_kind": "fit_boundary_reconstruction", + "negative": 0, + "positive": 335, + "total": 57630, + "zero": 57295 + }, + "entity": "tax_unit", + "exported_targets": [ + "unrecaptured_section_1250_gain" + ], + "model_target": "unrecaptured_section_1250_gain", + "model_targets": [ + "domestic_production_ald", + "unrecaptured_section_1250_gain", + "first_home_mortgage_balance", + "first_home_mortgage_interest", + "first_home_mortgage_origination_year", + "health_savings_account_ald" + ], + "patterns": [ + { + "pattern_id": "pattern_00_741356db", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "390c68ba6f46c5bc09fe1b2938b6647cfa927effeb01a9111751bc6eba969b5e", + "recipient_rows": 478, + "seed": 305989492, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_01_7c62ad55", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "9ea51d7d64717b0f1279b05fdf5554118eef07e5b48e8ec6f655cef5aaab8ac8", + "recipient_rows": 45032, + "seed": 2134600251, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_02_d9f964c9", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "f20451f9d64d2cebff4eeb89fac3fcde0d3900694aab47b42e7d6c97c5c9d2bd", + "recipient_rows": 130183, + "seed": 3523001447, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + }, + { + "pattern_id": "pattern_03_2e5c739d", + "predictors": [ + "__acs_transfer_person_count", + "__acs_transfer_age_sum", + "__acs_transfer_female_count", + "__acs_transfer_state_fips", + "__acs_transfer_group_employment_income_sum", + "__acs_transfer_group_self_employment_income_sum", + "__acs_transfer_group_social_security_income_sum", + "__acs_transfer_group_retirement_income_sum", + "__acs_transfer_group_investment_income_sum", + "__acs_transfer_group_head_count", + "__acs_transfer_group_tenure_code" + ], + "realized_regime": "zero_inflated_positive", + "recipient_index_sha256": "3bd55af66d89dc97b1e06b66c7e058cd6ca5cce9f8de1a84066d2646b8e288cf", + "recipient_rows": 406625, + "seed": 2968274776, + "weight_kind": "importance", + "weight_sha256": "cef065366b57f3331ef4a72e2d5235ac7cbf062ceccdd42e4792377884bd1969", + "zero_atol": 1e-06 + } + ], + "transfer_family": "puf_tax_itemization" + } + ], + "title": "Rare signed-tail frozen realized-regime evidence", + "validation": { + "check_count": 48, + "check_pattern_link_count": 192, + "check_regime_counts": { + "three_sign": 13, + "zero_inflated_positive": 35 + }, + "degenerate_or_single_sign_check_count": 0, + "target_pattern_record_count": 168, + "transfer_partition_counts": { + "early_gap_fill": 17, + "late_producer_complement": 31 + }, + "unique_target_count": 42, + "unique_target_regime_counts": { + "three_sign": 7, + "zero_inflated_positive": 35 + } + } +} diff --git a/experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh b/experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh new file mode 100755 index 000000000..29bc879f4 --- /dev/null +++ b/experiments/battery_rare_signed_tails/run_1pct_offchain_build.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Off-chain 1% verification build for the rare signed-tail lane. +# +# Usage: run_1pct_offchain_build.sh +# +# Lane rules enforced here: +# - at most a 1% sample (--sample-fraction 0.01, fixed); +# - OFF-CHAIN: no --logbook-prev-row-digest, POPULACE_LOGBOOK_PREV_ROW_DIGEST +# unset, and logbook-pending-chain.txt is never read or written (the +# genesis-null logbook row stays in this run's own spool directory); +# - serial queue: refuses to start while any other build_us_multispine_pool +# process is alive; +# - RSS ceiling: samples the build's RSS every 20 s to rss_samples.csv and +# kills the build if it exceeds 15 GiB. +set -u + +WORKTREE="$1" +OUT_DIR="$2" +LOG="${OUT_DIR}/build.log" +MEMCSV="${OUT_DIR}/rss_samples.csv" +RSS_LIMIT_KIB=$((15 * 1024 * 1024)) + +if pgrep -f "build_us_multispine_pool.py" > /dev/null 2>&1; then + echo "REFUSE: a build_us_multispine_pool process is already running" >&2 + exit 2 +fi + +mkdir -p "${OUT_DIR}" +cd "${WORKTREE}" + +export ASEC_H5=/Users/maxghenis/PolicyEngine/_buildo-runtime/out/591-pawtyp-pool/asec-producer-checkpoints/asec_raw_stage.checkpoint.h5 +export ASEC_SHA=51e9fafcd6f16140018fa90c7afbeb6d79008bfc8c122e437d23a399b30553fe +export ACS_H_ZIP=/Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0/csv_hus.zip +export ACS_H_SHA=8281008e53de98f0ef81e7a2ee5a8725991dda1ecfd2713ead73246425e515d0 +export ACS_P_ZIP=/Users/maxghenis/PolicyEngine/_worktrees/populace-acs-clone/inputs/acs_2024_1yr/afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894/csv_pus.zip +export ACS_P_SHA=afdc6d90c6e2f0bab365ed32d95ba4c4d8ac651162f46ac7861295b2dc469894 +export RENT_H5=/Users/maxghenis/PolicyEngine/policyengine-us-data/policyengine_us_data/storage/acs_2022.h5 +export RENT_SHA=0b319b496f19a6913066f9c5ea572edfda3d78a187be6f375846617d0b441bd4 +export PUF_H5=/Users/maxghenis/PolicyEngine/policyengine-us-data/policyengine_us_data/storage/puf_2024.h5 +export PUF_SHA=7669f5b5281f20080e77204f9bd4aabfad0aa101fa283e22caf9ba8d61d4d6df +export PUF_CSV=/Users/maxghenis/PolicyEngine/policyengine-us-data/policyengine_us_data/storage/puf_2015.csv +export PUF_CSV_SHA=0a7fd643edb1acc55c507db795914b41d232922be78c149b58d111f4672499df + +echo "1pct off-chain build: worktree=${WORKTREE} HEAD=$(git rev-parse --short HEAD) at $(date -u +%FT%TZ)" | tee "${LOG}" +echo "epoch_s,rss_kib" > "${MEMCSV}" + +env -u POPULACE_LOGBOOK_PREV_ROW_DIGEST \ + "${WORKTREE}/.venv/bin/python" tools/build_us_multispine_pool.py \ + --sample-fraction 0.01 \ + --sample-seed 578 \ + --clone-attachment-fraction 1.0 \ + --clone-attachment-seed 578 \ + --asec-raw-stage-h5 "${ASEC_H5}" \ + --asec-raw-stage-h5-sha256 "${ASEC_SHA}" \ + --acs-household-zip "${ACS_H_ZIP}" \ + --acs-household-zip-sha256 "${ACS_H_SHA}" \ + --acs-person-zip "${ACS_P_ZIP}" \ + --acs-person-zip-sha256 "${ACS_P_SHA}" \ + --acs-rent-h5 "${RENT_H5}" \ + --acs-rent-h5-sha256 "${RENT_SHA}" \ + --puf-h5 "${PUF_H5}" \ + --puf-h5-sha256 "${PUF_SHA}" \ + --puf-source-year-csv "${PUF_CSV}" \ + --puf-source-year-csv-sha256 "${PUF_CSV_SHA}" \ + --out "${OUT_DIR}/populace_us_2024_stacked_pool_1pct.h5" \ + >> "${LOG}" 2>&1 & +BUILD_PID=$! + +while kill -0 "${BUILD_PID}" 2>/dev/null; do + RSS_KIB=$(ps -o rss= -p "${BUILD_PID}" | tr -d ' ') + if [ -n "${RSS_KIB}" ]; then + echo "$(date +%s),${RSS_KIB}" >> "${MEMCSV}" + if [ "${RSS_KIB}" -gt "${RSS_LIMIT_KIB}" ]; then + echo "RSS ${RSS_KIB} KiB exceeded 15 GiB ceiling; killing build" | tee -a "${LOG}" + kill "${BUILD_PID}" + wait "${BUILD_PID}" + exit 3 + fi + fi + sleep 20 +done +wait "${BUILD_PID}" +RC=$? +echo "build exit ${RC} at $(date -u +%FT%TZ); peak RSS KiB: $(cut -d, -f2 "${MEMCSV}" | sort -n | tail -1)" | tee -a "${LOG}" +exit "${RC}" diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index 6c3c60ae2..50ff68601 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -27,8 +27,8 @@ from .schemas import load_schema_registry EXPECTED_AUTHORED_FIELD_COUNT = 32_161 -EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_219 -EXPECTED_CONFIGURATION_FIELD_COUNT = 41_380 +EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_220 +EXPECTED_CONFIGURATION_FIELD_COUNT = 41_381 class FieldUsageError(AssertionError): @@ -401,8 +401,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "6a781915fd491d2c4b16d2b7d482f69cf362c904130093c59f9629f7a319269b", ), "resolved_seed_protocol": ( - 823, - "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + 824, + "82eeb3578054e6eab7e714571ba1534a641814f10f57ddea46bbee9f2095ce76", ), "resolved_seed_site_bindings": ( 277, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index 9bf9ea502..47391905b 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -330,18 +330,18 @@ EXPECTED_HASHES = { "acs_group_predictors": "a927bb7ecf3e84f54c93583ab79318654514ac546aefafba67da5285615fbd60", "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", - "authority": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d", + "authority": "34e86d1e5b885e9fef604209bbe28755134b8d6551e38b5b61fb740e0b43e076", "early_families": "e26a90e2b5c16e23e7c17424d1c2e4ab18ed66b1d0e129248e87c0bab9b3fd5d", - "full_checkpoint": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8", + "full_checkpoint": "f4515d69c804c737a9b0b42cc4c6d29db081751a6f9e73958e375fdcd47df2f3", "gap_fill_schedule": "96aefe2853de91ae95f50bc2ccc2c1dd94802c27f21c643981152bbcb13c4e10", "graph_nodes": "a83363de26cad0144b5a98b36b4bca49542e37a7b9fee3d7e541f692deeff864", "late_families": "a160432fc12a85df20ba7fd6687673b3c31786df7a983e2477604ab923b26d18", - "late_resource_semantics": "9206aa072ce360ef87dc7d832351fcae2e0d1da71abcacc5af0d43ef88582dee", + "late_resource_semantics": "e048fa629487c76c768f26b461d8346fb94e692b99b6bc673f79e342092a0ce4", "late_schedule": "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "seed_protocol": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "seed_map": "f9e71d22a4c8608d1e2ca99912f60dd06272e785e0babcca46a452c1a6df9562", + "seed_protocol": "d317a1cb0dc24ee49e528ac3a0d3337952b30c3e6610fd49026770a42b88d6be", "source_manifest": "16f64b9cf3aea326737accca64c742a1edaa30a3a499432efd7567942f38a6c7", "take_up": "495dc6ed195eae372a6ba098c6fb894323638a4a7dce1b4fe7efaaf6beb69446", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/spine.schema.json b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/spine.schema.json index f84467d8d..dc4c7955d 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/spine.schema.json +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/spine.schema.json @@ -288,7 +288,7 @@ "const": "populace_us_stacked_pool_checkpoint_identity" }, "schema_version": {"const": 1}, - "materializer_version": {"const": 11}, + "materializer_version": {"const": 12}, "pipeline": {"const": "us-stacked-pool"} } }, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py index 36e6c27e8..87ffc6e5c 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py @@ -381,14 +381,15 @@ def _site( f"quantile-forest=={_distribution_version('quantile-forest')}", ) ) -_BUILD_CAP_SITES = ( +_UNIFORM_BUILD_CAP_SITES_BEFORE_RETIREMENT = ( "prior_year_income_training_cap", "childcare_training_cap", "retirement_contributions_training_cap", "disability_benefits_training_cap", "housing_inputs_training_cap", "workers_compensation_training_cap", - "retirement_distributions_training_cap", +) +_UNIFORM_BUILD_CAP_SITES_AFTER_RETIREMENT = ( "child_support_training_cap", "energy_subsidy_training_cap", "other_health_insurance_training_cap", @@ -425,6 +426,30 @@ def _stable_site( ) +def _uniform_build_cap_site(site_id: str) -> DrawSiteProtocol: + """Declare one legacy whole-donor pandas training cap.""" + + return _site( + site_id, + "build_model", + value_source="run_request.build_model_seed", + default=0, + rng_family="pandas.DataFrame.sample RandomState(MT19937)", + rng_version=( + f"pandas=={_distribution_version('pandas')};" + f"numpy=={_distribution_version('numpy')}" + ), + seed_material=("build_model_seed", "stage_training_cap"), + consumption_order=( + "source_dataframe_row_order", + "sorted_selected_positions", + ), + reset_boundary="fresh_generator_per_source_stage", + draw_condition="donor_rows_above_5000", + derivation="pandas_sample(n=5000,random_state=build_model_seed)", + ) + + LEGACY_V1_SITES = ( _site( "survey_sample_asec", @@ -846,26 +871,37 @@ def _stable_site( derivation="PCG64 plus Sampford fixed-size selection", ), *tuple( - _site( - site_id, - "build_model", - value_source="run_request.build_model_seed", - default=0, - rng_family="pandas.DataFrame.sample RandomState(MT19937)", - rng_version=( - f"pandas=={_distribution_version('pandas')};" - f"numpy=={_distribution_version('numpy')}" - ), - seed_material=("build_model_seed", "stage_training_cap"), - consumption_order=( - "source_dataframe_row_order", - "sorted_selected_positions", - ), - reset_boundary="fresh_generator_per_source_stage", - draw_condition="donor_rows_above_5000", - derivation="pandas_sample(n=5000,random_state=build_model_seed)", - ) - for site_id in _BUILD_CAP_SITES + _uniform_build_cap_site(site_id) + for site_id in _UNIFORM_BUILD_CAP_SITES_BEFORE_RETIREMENT + ), + _site( + "retirement_distributions_training_cap", + "build_model", + value_source="run_request.build_model_seed", + default=0, + rng_family="pandas.Series.sample RandomState(MT19937)", + rng_version=( + f"pandas=={_distribution_version('pandas')};" + f"numpy=={_distribution_version('numpy')}" + ), + seed_material=("build_model_seed", "stage_training_cap"), + consumption_order=( + "retain_nonzero_union_then_prioritize_positive-weight_all-target-zero_positions", + "sample_remaining_all-target-zero_positions_with_pandas_series_sample", + "sort_selected_positions_then_ratio-calibrate_sampled-zero_weights", + ), + reset_boundary="fresh_generator_per_retirement_distribution_stage", + draw_condition="donor_rows_above_5000", + derivation=( + "retain_union(abs(target)>DEFAULT_ZERO_ATOL);" + "retain_or_sample_positive_weight_all_zero;" + "pandas_sample_zero_weight_fillers_if_needed;" + "all_zero_weight_ratio=full_mass/sampled_mass" + ), + ), + *tuple( + _uniform_build_cap_site(site_id) + for site_id in _UNIFORM_BUILD_CAP_SITES_AFTER_RETIREMENT ), _site( "legacy_geography_ladder", diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index e7bd93c42..0f67039cd 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -33234,10 +33234,10 @@ producer_graph: - first_home_mortgage_origination_year - health_savings_account_ald execution_receipt_contract: - version: 3 + version: 4 row_binding: declared_globally_reconciled_input_and_scope_exact_output_source_and_primary_callback_resource_receipt_and_previous_execution_sha256 virtual_resource_binding: exact_kind_specific_semantic_payload_and_sha256 - top_binding: entry_and_output_frame_sha256_execution_chain_source_completion_and_nineteen_transfer_groups + top_binding: entry_and_output_frame_sha256_execution_chain_source_completion_and_nineteen_transfer_groups_with_exact_realized_regime_reconstruction transition_authority: authority_id: us_stacked_late_producer_transition metadata_key: us_late_producer_transition_authority diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/spine.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/spine.yaml index 851394da9..563eb13eb 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/spine.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/spine.yaml @@ -3,7 +3,7 @@ pipeline_contract: artifact_protocol: artifact_kind: populace_us_stacked_pool_checkpoint_identity schema_version: 1 - materializer_version: 11 + materializer_version: 12 pipeline: us-stacked-pool stacked_operator_order: - assemble_stacked_spine diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index 09e2c90e9..21a8957e9 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -44,11 +44,48 @@ has_support_role_metadata, support_role_series, ) -from microcosm.fit import QRFChainState +from microcosm.fit import QRFChainState, Regime from microcosm.frame import EntitySchema, Frame, Weights QRF: Any | None = None +_VALID_QRF_REGIMES = frozenset( + { + Regime.THREE_SIGN, + Regime.ZERO_INFLATED_POSITIVE, + Regime.ZERO_INFLATED_NEGATIVE, + Regime.SIGN_ONLY, + Regime.POSITIVE_ONLY, + Regime.NEGATIVE_ONLY, + Regime.DEGENERATE_ZERO, + } +) + + +def _validated_qrf_regimes( + value: Mapping[str, str], + *, + expected_targets: tuple[str, ...], + boundary: str, +) -> dict[str, str]: + """Bind realized regimes to the exact fitted target order.""" + + if not isinstance(value, Mapping) or tuple(value) != expected_targets: + observed = list(value) if isinstance(value, Mapping) else type(value).__name__ + raise ValueError( + f"{boundary} regime targets changed: expected " + f"{list(expected_targets)}, got {observed}." + ) + invalid = { + target: regime + for target, regime in value.items() + if regime not in _VALID_QRF_REGIMES + } + if invalid: + raise ValueError(f"{boundary} carries invalid realized regimes {invalid}.") + return dict(value) + + __all__ = [ "ACS_DONOR_CHANNEL_AUTO", "ACS_DEFERRED_GEOGRAPHY_INPUTS", @@ -413,6 +450,7 @@ class AcsTransferPattern: weight_kind: str donor_rows: int recipient_rows: int + regimes: Mapping[str, str] @dataclass(frozen=True) @@ -457,6 +495,7 @@ class AcsTransferBankPatternStep: pattern: str state_before: QRFChainState state_after: QRFChainState + regime: str @dataclass(frozen=True) @@ -1350,6 +1389,11 @@ def _fit_family_patterns( f"weight kind {fitted.weight_kind!r}, expected the donor " f"Frame's {resolved_kind!r}." ) + regimes = _validated_qrf_regimes( + fitted.regimes(), + expected_targets=model_targets, + boundary=f"ACS transfer {entity!r}/{family!r}/{pattern_name!r}", + ) recipient_pattern = _encoded_predictor_frame( surface.recipient.iloc[recipient_positions], @@ -1376,6 +1420,7 @@ def _fit_family_patterns( weight_kind=fitted.weight_kind, donor_rows=donor_rows, recipient_rows=len(recipient_positions), + regimes=regimes, ) pattern_records.append(pattern_record) fit_records.append( @@ -1477,6 +1522,7 @@ def _fit_family_patterns_banked( contexts: list[_BankPatternContext] = [] states: dict[str, QRFChainState] = {} raw_priors: dict[str, pd.DataFrame] = {} + pattern_regimes: dict[str, dict[str, str]] = {} pattern_records: list[AcsTransferPattern] = [] fit_records: list[FitWeightRecord] = [] for position, (observed_optional, recipient_positions) in enumerate( @@ -1536,6 +1582,7 @@ def _fit_family_patterns_banked( weight_kind=state.weight_kind, donor_rows=donor_rows, recipient_rows=len(recipient_positions), + regimes={}, ) contexts.append( _BankPatternContext( @@ -1545,6 +1592,7 @@ def _fit_family_patterns_banked( ) ) states[pattern_name] = state + pattern_regimes[pattern_name] = {} raw_priors[pattern_name] = pd.DataFrame( index=surface.recipient.iloc[recipient_positions].index ) @@ -1631,6 +1679,7 @@ def _fit_family_patterns_banked( pattern=pattern.name, state_before=state_before, state_after=result.state, + regime=result.regime, ) ) checkpoint = AcsTransferTargetCheckpoint( @@ -1669,6 +1718,9 @@ def _fit_family_patterns_banked( context.recipient_positions ] states[pattern_name] = steps_by_pattern[pattern_name].state_after + pattern_regimes[pattern_name][model_target] = steps_by_pattern[ + pattern_name + ].regime incomplete_states = [ name for name, state in states.items() if not state.is_complete @@ -1677,7 +1729,17 @@ def _fit_family_patterns_banked( raise AssertionError( f"ACS transfer bank left incomplete pattern chains: {incomplete_states}." ) - patterns = tuple(pattern_records) + patterns = tuple( + replace( + pattern, + regimes=_validated_qrf_regimes( + pattern_regimes[pattern.name], + expected_targets=model_targets, + boundary=f"ACS transfer {entity!r}/{family!r}/{pattern.name!r} bank", + ), + ) + for pattern in pattern_records + ) kinds = {pattern.weight_kind for pattern in patterns} if len(kinds) != 1: # pragma: no cover - every subset resolves one donor kind raise RuntimeError( @@ -1754,6 +1816,11 @@ def _validate_banked_target_checkpoint( f"{entity}.{model_target}." ) for step in checkpoint.pattern_steps: + if step.regime not in _VALID_QRF_REGIMES: + raise ValueError( + f"ACS transfer bank target {entity}.{model_target} carries " + f"invalid regime {step.regime!r} for pattern {step.pattern!r}." + ) state_before = expected_states[step.pattern] if step.state_before != state_before: raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer_bank.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer_bank.py index 8e5ab63f4..76fd4f06b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer_bank.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer_bank.py @@ -20,11 +20,38 @@ AcsTransferBankPatternStep, AcsTransferTargetCheckpoint, ) -from microcosm.fit import QRFChainState +from microcosm.fit import QRFChainState, Regime -ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION = 1 +ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION = 2 """Serialization contract for one ACS-transfer target checkpoint.""" +# Target-checkpoint schema ledger. +# +# 1: Initial closed metadata envelope with chained states and raw draws. +# 2: Every pattern step also persists its realized QRF regime, and the +# operational receipt projects the same pattern-to-regime evidence. + +_VALID_QRF_REGIMES = frozenset( + { + Regime.THREE_SIGN, + Regime.ZERO_INFLATED_POSITIVE, + Regime.ZERO_INFLATED_NEGATIVE, + Regime.SIGN_ONLY, + Regime.POSITIVE_ONLY, + Regime.NEGATIVE_ONLY, + Regime.DEGENERATE_ZERO, + } +) + + +def _pattern_regime_receipt( + steps: Sequence[AcsTransferBankPatternStep], +) -> dict[str, str]: + """Project checkpoint transitions into a compact auditable regime map.""" + + return {step.pattern: step.regime for step in steps} + + # ACS-transfer target-bank semantic-invalidation ledger. # # 1: Initial identity-bound per-target checkpoint format. @@ -219,6 +246,7 @@ def load_target( "size_bytes": path.stat().st_size, "raw_draw_sha256": actual_raw_sha256, "content_metadata_sha256": metadata["content_metadata_sha256"], + "realized_regimes": _pattern_regime_receipt(pattern_steps), } print( "Resumed ACS transfer target " @@ -270,6 +298,7 @@ def write_target(self, checkpoint: AcsTransferTargetCheckpoint) -> None: "pattern_steps": [ { "pattern": step.pattern, + "regime": step.regime, "state_before_sha256": _mapping_sha256(step.state_before.to_dict()), "state_after": step.state_after.to_dict(), } @@ -325,6 +354,7 @@ def write_target(self, checkpoint: AcsTransferTargetCheckpoint) -> None: "size_bytes": path.stat().st_size, "raw_draw_sha256": raw_draw_sha256, "content_metadata_sha256": metadata["content_metadata_sha256"], + "realized_regimes": _pattern_regime_receipt(checkpoint.pattern_steps), "write_seconds": time.perf_counter() - started_at, } print( @@ -563,7 +593,12 @@ def _load_pattern_steps( raise ValueError( f"ACS transfer target pattern_steps[{index}] must be an object." ) - expected_keys = {"pattern", "state_before_sha256", "state_after"} + expected_keys = { + "pattern", + "regime", + "state_before_sha256", + "state_after", + } if set(raw_step) != expected_keys: raise ValueError( f"ACS transfer target pattern_steps[{index}] keys changed." @@ -578,6 +613,12 @@ def _load_pattern_steps( raise ValueError( f"ACS transfer target carries unexpected pattern {pattern!r}." ) + regime = raw_step.get("regime") + if regime not in _VALID_QRF_REGIMES: + raise ValueError( + f"ACS transfer target pattern {pattern!r} has invalid regime " + f"{regime!r}." + ) state_before = expected_states[pattern] expected_before_sha256 = _mapping_sha256(state_before.to_dict()) if raw_step.get("state_before_sha256") != expected_before_sha256: @@ -602,6 +643,7 @@ def _load_pattern_steps( pattern=pattern, state_before=state_before, state_after=state_after, + regime=regime, ) ) if tuple(observed_patterns) != expected_patterns: @@ -623,6 +665,11 @@ def _validate_pattern_steps_for_write( "ACS transfer checkpoint pattern_steps must contain " "AcsTransferBankPatternStep values." ) + if step.regime not in _VALID_QRF_REGIMES: + raise ValueError( + f"ACS transfer pattern {step.pattern!r} has invalid regime " + f"{step.regime!r}." + ) patterns = tuple(step.pattern for step in checkpoint.pattern_steps) if any(not pattern for pattern in patterns) or len(set(patterns)) != len(patterns): raise ValueError( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py index fcd155d4e..89efb98c2 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py @@ -100,7 +100,7 @@ "populace_us_multispine_pool_checkpoint_provenance" ) _LEGACY_POOL_CHECKPOINT_SCHEMA_VERSION = 1 -_LEGACY_POOL_CHECKPOINT_MATERIALIZER_VERSION = 3 +_LEGACY_POOL_CHECKPOINT_MATERIALIZER_VERSION = 4 _LEGACY_REQUIRED_STAGE_RECEIPTS = frozenset({"impute", "derive", "seed", "simulate"}) _STACKED_ONLY_MANIFEST_FIELDS = frozenset( { diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index 7f8d1427c..1162a6dd3 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -24,6 +24,7 @@ from __future__ import annotations +from importlib import import_module from importlib.resources import files from typing import Any @@ -345,9 +346,11 @@ def impute_us_retirement_distributions_to_puf_support_from_manifest( trains on measured ASEC distributions and replaces the four populated non-IRA outputs restored here on the PUF half. PUF taxable IRA remains sourced from the tax-detail stage, and Roth IRA remains the direct copied - ASEC mapping. A rare QRF output may legitimately remain zero on the PUF half - when the fixed 5,000-row training sample contains no positive donor; the - measured ASEC half remains authoritative and is never overwritten. + ASEC mapping. The fixed training cap retains the union of nonzero donors + across all four QRF targets, then samples only the common all-zero stratum + and calibrates its weights back to the full all-zero mass. This preserves + every realized sign regime and conditional nonzero-magnitude support while + the measured ASEC half remains authoritative and is never overwritten. """ if operation.kind != "impute_retirement_distributions_to_puf_support": @@ -428,17 +431,27 @@ def impute_us_retirement_distributions_to_puf_support_from_manifest( weights = pd.to_numeric( frame.loc[asec_mask, _PERSON_WEIGHT_COLUMN], errors="coerce" ).fillna(0.0) - if len(training) > max_train_samples: - sample = training.sample( - n=max_train_samples, - random_state=(context.config.seed if context is not None else 0), - ).index - training = training.loc[sample] - weights = weights.loc[sample] + for column in _PUF_QRF_OUTPUT_COLUMNS: + training[column] = pd.to_numeric(training[column], errors="coerce") + if not np.isfinite(training[column].to_numpy(dtype=np.float64)).all(): + raise SourceRuntimeError( + "US retirement-distribution QRF training column " + f"{column!r} contains nonfinite values." + ) + fit_module = import_module("microcosm.fit") + seed = context.config.seed if context is not None else 0 + training, capped_weights = _cap_retirement_distribution_training( + training, + weights, + targets=_PUF_QRF_OUTPUT_COLUMNS, + max_train_samples=max_train_samples, + seed=seed, + zero_atol=float(fit_module.DEFAULT_ZERO_ATOL), + ) test = frame.loc[puf_mask, predictor_columns].copy() test.columns = list(predictors) - for column in (*predictors, *_PUF_QRF_OUTPUT_COLUMNS): + for column in predictors: training[column] = pd.to_numeric(training[column], errors="coerce") if not np.isfinite(training[column].to_numpy(dtype=np.float64)).all(): raise SourceRuntimeError( @@ -455,15 +468,12 @@ def impute_us_retirement_distributions_to_puf_support_from_manifest( global QRF if QRF is None: - from importlib import import_module - - QRF = import_module("microcosm.fit").QRF - seed = context.config.seed if context is not None else 0 + QRF = fit_module.QRF fitted = QRF(n_estimators=n_estimators, seed=seed).fit( training, list(predictors), list(_PUF_QRF_OUTPUT_COLUMNS), - weights=weights.to_numpy(dtype=np.float64), + weights=capped_weights, ) predictions = fitted.predict(test).clip(lower=0.0) @@ -473,6 +483,107 @@ def impute_us_retirement_distributions_to_puf_support_from_manifest( return result +def _cap_retirement_distribution_training( + training: pd.DataFrame, + weights: pd.Series | np.ndarray, + *, + targets: tuple[str, ...], + max_train_samples: int, + seed: int, + zero_atol: float, +) -> tuple[pd.DataFrame, np.ndarray]: + """Cap a joint retirement fit without deleting any target's sign support. + + Every row that is nonzero for at least one target remains a census row. + The fixed-cap remainder is a seeded sample of rows that are QRF-zero for + every target. Scaling only those sampled rows back to their stratum's full + weight preserves total donor mass and every target's weighted sign masses, + while carrier weights and conditional magnitudes remain exact. + """ + + if max_train_samples <= 0: + raise SourceRuntimeError( + "US retirement-distribution training cap must be positive." + ) + if zero_atol < 0.0 or not np.isfinite(zero_atol): + raise SourceRuntimeError( + "US retirement-distribution QRF zero tolerance must be finite and " + "nonnegative." + ) + weight_values = np.asarray(weights, dtype=np.float64) + if weight_values.ndim != 1 or len(weight_values) != len(training): + raise SourceRuntimeError( + "US retirement-distribution weights must be one-dimensional and " + "positionally aligned with the training donor." + ) + if not np.isfinite(weight_values).all() or (weight_values < 0.0).any(): + raise SourceRuntimeError( + "US retirement-distribution weights must be finite and nonnegative." + ) + if float(weight_values.sum()) <= 0.0: + raise SourceRuntimeError( + "US retirement-distribution training donor has zero total weight." + ) + + target_matrix = training.loc[:, list(targets)].to_numpy(dtype=np.float64) + if not np.isfinite(target_matrix).all(): + raise SourceRuntimeError( + "US retirement-distribution training targets must be finite before " + "the support-preserving cap." + ) + if len(training) <= max_train_samples: + return training.copy(), weight_values.copy() + + union_nonzero = np.any(np.abs(target_matrix) > zero_atol, axis=1) + carrier_positions = np.flatnonzero(union_nonzero) + zero_positions = np.flatnonzero(~union_nonzero) + if len(carrier_positions) >= max_train_samples: + raise SourceRuntimeError( + "US retirement-distribution nonzero donor union leaves no room for " + "an all-target-zero row under the fixed training cap: " + f"union={len(carrier_positions)}, cap={max_train_samples}." + ) + + zero_slots = max_train_samples - len(carrier_positions) + positive_weight_zero_positions = zero_positions[weight_values[zero_positions] > 0.0] + zero_weight_zero_positions = zero_positions[weight_values[zero_positions] == 0.0] + if len(positive_weight_zero_positions) >= zero_slots: + sampled_zero_positions = ( + pd.Series(positive_weight_zero_positions, dtype=np.int64) + .sample(n=zero_slots, random_state=seed) + .to_numpy(dtype=np.int64) + ) + else: + zero_weight_slots = zero_slots - len(positive_weight_zero_positions) + sampled_zero_positions = np.concatenate( + ( + positive_weight_zero_positions, + pd.Series(zero_weight_zero_positions, dtype=np.int64) + .sample(n=zero_weight_slots, random_state=seed) + .to_numpy(dtype=np.int64), + ) + ) + selected_positions = np.sort( + np.concatenate((carrier_positions, sampled_zero_positions)) + ) + capped_weights = weight_values[selected_positions].copy() + selected_zero = ~union_nonzero[selected_positions] + full_zero_mass = float(weight_values[zero_positions].sum()) + sampled_zero_mass = float(capped_weights[selected_zero].sum()) + if sampled_zero_mass <= 0.0 < full_zero_mass: + raise SourceRuntimeError( + "US retirement-distribution sampled all-target-zero donor has zero " + "weight but must represent positive full-stratum mass." + ) + if full_zero_mass > 0.0: + capped_weights[selected_zero] *= full_zero_mass / sampled_zero_mass + if not np.isfinite(capped_weights).all(): + raise SourceRuntimeError( + "US retirement-distribution calibrated training weights are nonfinite." + ) + return training.iloc[selected_positions].copy(), capped_weights + + def _person_retirement_distribution_predictors(frame: Frame) -> pd.DataFrame: """Build the archived eight QRF predictors on person rows.""" diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 6b4ffc1f3..6bc395bcb 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -90,6 +90,7 @@ from microcosm.build.us_runtime.acs_transfer import ( ASEC_PUF_DONOR_SPINE, DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + AcsImputedInput, AcsTransferResult, AcsTransferTargetBank, TargetFamilies, @@ -3738,6 +3739,7 @@ def validate_stacked_post_puf_transfer_receipt( "canonical 19-group partition; production manifest emission is " "forbidden." ) + reconstructed_regimes: list[dict[str, object]] = [] for name, group in expected_groups.items(): group_receipt = groups[name] if ( @@ -3749,6 +3751,23 @@ def validate_stacked_post_puf_transfer_receipt( f"{boundary}: stacked post-PUF transfer group {name!r} is " "misbound; production manifest emission is forbidden." ) + reconstructed_regimes.extend( + _validated_acs_transfer_realized_regime_receipt( + group_receipt.get("realized_regimes"), + entity=group.entity, + family=group.family, + expected_model_targets=acs_transfer_runtime._model_target_names( + group.targets + ), + boundary=f"{boundary} group {name!r}", + ) + ) + if receipt.get("realized_regimes") != reconstructed_regimes: + raise ValueError( + f"{boundary}: aggregate late-transfer realized regimes are not " + "reconstructed from the canonical group receipts; production " + "manifest emission is forbidden." + ) expected_target_labels = { f"{entity}/{family}/{target}" for entity, families in CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() @@ -8189,7 +8208,146 @@ def _verify_gap_fill_outcome( {"fit_name": record.fit_name, "weight_kind": record.weight_kind} for record in result.fit_records ], + "realized_regimes": _acs_transfer_realized_regime_receipt(result), + } + + +def _acs_transfer_realized_regime_receipt( + result: AcsTransferResult, +) -> list[dict[str, object]]: + """Deduplicate full fitted-pattern regime evidence for one transfer.""" + + return _acs_imputed_inputs_realized_regime_receipt(result.imputed_inputs) + + +def _acs_imputed_inputs_realized_regime_receipt( + imputed_inputs: Sequence[AcsImputedInput], +) -> list[dict[str, object]]: + """Project realized regimes from the fitted provenance records themselves.""" + + rows: list[dict[str, object]] = [] + seen: set[tuple[object, ...]] = set() + for record in imputed_inputs: + for pattern in record.patterns: + regime_items = tuple(pattern.regimes.items()) + identity: tuple[object, ...] = ( + record.entity, + record.family, + pattern.name, + pattern.observed_optional_predictors, + pattern.predictors, + pattern.seed, + pattern.weight_kind, + pattern.donor_rows, + pattern.recipient_rows, + regime_items, + ) + if identity in seen: + continue + seen.add(identity) + rows.append( + { + "entity": record.entity, + "family": record.family, + "pattern": pattern.name, + "observed_optional_predictors": list( + pattern.observed_optional_predictors + ), + "predictors": list(pattern.predictors), + "seed": pattern.seed, + "weight_kind": pattern.weight_kind, + "donor_rows": pattern.donor_rows, + "recipient_rows": pattern.recipient_rows, + "model_targets": list(pattern.regimes), + "regimes": dict(pattern.regimes), + } + ) + return rows + + +def _validated_acs_transfer_realized_regime_receipt( + value: object, + *, + entity: str, + family: str, + expected_model_targets: tuple[str, ...], + boundary: str, +) -> list[dict[str, object]]: + """Validate one bounded transfer's complete fitted-pattern regime proof.""" + + if not isinstance(value, list) or not value: + raise ValueError(f"{boundary} has no realized-regime evidence.") + expected_keys = { + "entity", + "family", + "pattern", + "observed_optional_predictors", + "predictors", + "seed", + "weight_kind", + "donor_rows", + "recipient_rows", + "model_targets", + "regimes", } + validated: list[dict[str, object]] = [] + seen_patterns: set[str] = set() + for index, raw_row in enumerate(value): + if not isinstance(raw_row, Mapping) or set(raw_row) != expected_keys: + raise ValueError(f"{boundary} realized-regime row {index} schema changed.") + pattern = raw_row.get("pattern") + optional = raw_row.get("observed_optional_predictors") + predictors = raw_row.get("predictors") + seed = raw_row.get("seed") + donor_rows = raw_row.get("donor_rows") + recipient_rows = raw_row.get("recipient_rows") + weight_kind = raw_row.get("weight_kind") + if ( + raw_row.get("entity") != entity + or raw_row.get("family") != family + or not isinstance(pattern, str) + or not pattern + or pattern in seen_patterns + or not isinstance(optional, list) + or any(not isinstance(item, str) or not item for item in optional) + or len(set(optional)) != len(optional) + or not isinstance(predictors, list) + or not predictors + or any(not isinstance(item, str) or not item for item in predictors) + or len(set(predictors)) != len(predictors) + or not isinstance(seed, int) + or isinstance(seed, bool) + or seed < 0 + or not isinstance(weight_kind, str) + or not weight_kind + or not isinstance(donor_rows, int) + or isinstance(donor_rows, bool) + or donor_rows <= 0 + or not isinstance(recipient_rows, int) + or isinstance(recipient_rows, bool) + or recipient_rows <= 0 + ): + raise ValueError( + f"{boundary} realized-regime row {index} identity changed." + ) + model_targets = raw_row.get("model_targets") + regimes = raw_row.get("regimes") + if ( + not isinstance(model_targets, list) + or tuple(model_targets) != expected_model_targets + or not isinstance(regimes, Mapping) + or set(regimes) != set(expected_model_targets) + or any( + regime not in acs_transfer_runtime._VALID_QRF_REGIMES + for regime in regimes.values() + ) + ): + raise ValueError( + f"{boundary} realized-regime row {index} targets or labels changed." + ) + seen_patterns.add(pattern) + validated.append(dict(raw_row)) + return validated # --------------------------------------------------------------------------- @@ -8459,6 +8617,7 @@ def _transfer_stacked_post_puf_inputs_evaluate( {"fit_name": record.fit_name, "weight_kind": record.weight_kind} for record in transfer.fit_records ], + "realized_regimes": _acs_transfer_realized_regime_receipt(transfer), }, transfer_result=transfer, ) @@ -9055,7 +9214,7 @@ def _aggregate_late_transfer_result( str, tuple[ Mapping[str, object], - tuple[object, ...], + tuple[AcsImputedInput, ...], tuple[FitWeightRecord, ...], tuple[str, ...], str | None, @@ -9098,6 +9257,14 @@ def _aggregate_late_transfer_result( raise ValueError( f"US late-transfer group receipt for {group.name!r} is misbound." ) + fitted_regimes = _acs_imputed_inputs_realized_regime_receipt( + group_imputed_inputs + ) + if group_receipt.get("realized_regimes") != fitted_regimes: + raise ValueError( + f"US late-transfer group {group.name!r} realized-regime receipt " + "differs from its fitted imputation provenance." + ) raw_targets = group_receipt.get("targets") if not isinstance(raw_targets, Mapping): raise ValueError( @@ -9145,6 +9312,7 @@ def _aggregate_late_transfer_result( deferred_inputs=tuple(dict.fromkeys(deferred_inputs)), resolved_donor_channel=next(iter(resolved_channels)), ) + reconstructed_regimes = _acs_transfer_realized_regime_receipt(aggregate) receipt = { "authority": _authority_receipt(authority), "producer_schedule": dict(us_late_producer_schedule_receipt()), @@ -9160,6 +9328,7 @@ def _aggregate_late_transfer_result( {"fit_name": record.fit_name, "weight_kind": record.weight_kind} for record in aggregate.fit_records ], + "realized_regimes": reconstructed_regimes, "completion": { "status": "complete", "group_count": len(expected_groups), diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py index 4a6c9b78b..d86a7c6e1 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py @@ -119,12 +119,14 @@ # v7 added the primary execution configuration and every late-transfer model # configuration/target-bank identity to the declared external-resource surface. # Version 6 content-bound physical Frame inputs but left those callback inputs -# implicit. Receipt v3 reconciles repeated physical evidence and scope +# implicit. Receipt v4 requires every bounded transfer producer to carry exact +# realized QRF regimes and reconstructs the aggregate regime receipt from those +# nineteen producer receipts. Receipt v3 reconciled repeated physical evidence and scope # cardinalities across each execution row, binds source-receipt outputs to the # callback receipt, and requires the primary callback to report the exact # resources it consumed. Receipt v2 introduced exact virtual-resource payloads. US_LATE_PRODUCER_REGISTRY_SCHEMA_VERSION = 16 -US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION = 3 +US_LATE_PRODUCER_RECEIPT_SCHEMA_VERSION = 4 US_LATE_PRODUCER_TRANSITION_AUTHORITY_VERSION = 1 US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY = "us_late_producer_transition_authority" US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID = "us_stacked_late_producer_transition" @@ -2063,7 +2065,8 @@ def us_late_producer_schedule_payload() -> dict[str, object]: ), "top_binding": ( "entry_and_output_frame_sha256_execution_chain_source_" - "completion_and_nineteen_transfer_groups" + "completion_and_nineteen_transfer_groups_with_exact_" + "realized_regime_reconstruction" ), "transition_authority": { "authority_id": US_LATE_PRODUCER_TRANSITION_AUTHORITY_ID, diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index b60dccd8b..eb50269f4 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "262091db8c7b01b2a3b596aa2468d95855a63703ba9f8ebba2940cf5834c2c83", + "91d3a58e669950c7fab0aefeb74687278efca82d413a900e9ab1878b09a7a5a3", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "942ad7df5537e244df7ede7230098e91c617dbdb11e4d0885a94bb3b4f912da5", + "54c0b70167b9d36e857a116ca4c80c74af142b6f09d5894ee979de62b9634738", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index db5e65a74..4c2988390 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,22 +52,22 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 41_380 + assert fields["configuration_field_count"] == 41_381 assert fields["authored_normative_field_count"] == 32_161 - assert fields["resolved_binding_field_count"] == 9_219 - assert fields["consumed_field_count"] == 41_380 + assert fields["resolved_binding_field_count"] == 9_220 + assert fields["consumed_field_count"] == 41_381 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 47 assert fields["mode_counts"] == { "legacy_behavior": 13_245, - "compiler_semantic": 27_661, + "compiler_semantic": 27_662, "front_end_validation": 346, "identity_only": 128, } assert fields["generation0_effect_counts"] == { "legacy_behavior": 37_718, - "no_generation0_effect": 3_662, + "no_generation0_effect": 3_663, } inventory = coverage_report["inventory_coverage"] diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index 18cb8c768..ec09f1281 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 41_380 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 41_381 assert field_ledger.source_counts == { "authored": 32_161, - "resolved_bindings": 9_219, + "resolved_bindings": 9_220, } assert field_ledger.mode_counts == { "legacy_behavior": 13_245, - "compiler_semantic": 27_661, + "compiler_semantic": 27_662, "front_end_validation": 346, "identity_only": 128, } assert field_ledger.generation0_effect_counts == { "legacy_behavior": 37_718, - "no_generation0_effect": 3_662, + "no_generation0_effect": 3_663, } - assert len({field.pointer for field in field_ledger.fields}) == 41_380 + assert len({field.pointer for field in field_ledger.fields}) == 41_381 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_identity_contracts.py b/packages/microcosm-build/tests/test_spec_engine_identity_contracts.py index b4693a0e1..a2a682f68 100644 --- a/packages/microcosm-build/tests/test_spec_engine_identity_contracts.py +++ b/packages/microcosm-build/tests/test_spec_engine_identity_contracts.py @@ -63,7 +63,7 @@ def test_pipeline_contract_is_an_exact_generation_zero_projection() -> None: == { "artifact_kind": "populace_us_stacked_pool_checkpoint_identity", "schema_version": 1, - "materializer_version": 11, + "materializer_version": 12, "pipeline": "us-stacked-pool", } ) diff --git a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py index 86d65958a..25705f0fb 100644 --- a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py +++ b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py @@ -197,7 +197,7 @@ def test_adapter_preserves_generation_zero_identity_components( imputation = legacy_payload["imputation"] assert isinstance(imputation, dict) assert legacy_payload["stacked_authority_receipt"]["sha256"] == ( - "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "34e86d1e5b885e9fef604209bbe28755134b8d6551e38b5b61fb740e0b43e076" ) assert imputation["late_producer_schedule_receipt"]["schedule_sha256"] == ( "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5" diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index e6a5391a8..5564c6954 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "9afebedeebc4194243e8307703ebc0feac975a69a186d28e266f1c4ef356bbe3" + "43989f610f9691daf69e9ee74c1aa17f2a8bc3f07b9992489c65fa8b1ad747de" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_spec_engine_seeds.py b/packages/microcosm-build/tests/test_spec_engine_seeds.py index ec1a50ab1..fc2bedeff 100644 --- a/packages/microcosm-build/tests/test_spec_engine_seeds.py +++ b/packages/microcosm-build/tests/test_spec_engine_seeds.py @@ -332,6 +332,21 @@ def test_adult_tail_and_live_puf_orders_are_literal() -> None: ) +def test_retirement_distribution_cap_declares_support_preserving_candidates() -> None: + site = LEGACY_V1_PROTOCOL.site("retirement_distributions_training_cap") + + assert site.seed_material == ("build_model_seed", "stage_training_cap") + assert site.rng_family == "pandas.Series.sample RandomState(MT19937)" + assert site.consumption_order == ( + "retain_nonzero_union_then_prioritize_positive-weight_all-target-zero_positions", + "sample_remaining_all-target-zero_positions_with_pandas_series_sample", + "sort_selected_positions_then_ratio-calibrate_sampled-zero_weights", + ) + assert site.draw_condition == "donor_rows_above_5000" + assert "DEFAULT_ZERO_ATOL" in site.derivation + assert "full_mass/sampled_mass" in site.derivation + + def test_acs_family_pattern_seed_and_label_golden_vectors() -> None: assert _family_seed(0, entity="person", family="family_a") == 2_974_888_678 assert _family_seed(578, entity="tax_unit", family="transfers") == 2_219_304_948 diff --git a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py index 5c236bc76..111f77f0f 100644 --- a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py @@ -74,7 +74,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer assert projected == live assert stacked_identity_bytes(projected) == _canonical_bytes(live) assert projected["sha256"] == ( - "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "34e86d1e5b885e9fef604209bbe28755134b8d6551e38b5b61fb740e0b43e076" ) assert { name: component["sha256"] @@ -90,7 +90,7 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer "cacc6c11e114dbae3aaa2761cc6b3fcb1191cd9b689b1c2bd096614c51ebff8b" ), "late_producer_schedule": ( - "bf95c78ea4168c81fa319872276002835f19ac27461eb3b69349c9637bc14f86" + "870c38d740cf92d6ce7a96cac4c44864b03d87634d59177d701fecf3b2d94436" ), "metric_registry": ( "9fb545df48c31004444dbda542b8b67b9c66288d3742fd00e9c7b210d75766e9" diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index ff4c7fc68..3b22738c6 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -5,6 +5,7 @@ from pathlib import Path from types import SimpleNamespace +import h5py import numpy as np import pandas as pd import pytest @@ -26,10 +27,12 @@ transfer_acs_inputs, ) from microcosm.build.us_runtime.acs_transfer_bank import ( + ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION, AcsTransferTargetBankStore, ) from microcosm.build.us_runtime.puf_support import clone_us_frame_for_puf_support from microcosm.build.us_runtime.spine_assembly import assemble_spines +from microcosm.fit.qrf import DEFAULT_ZERO_ATOL, Regime, detect_regime from microcosm.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights from microcosm.frame.adapters.policyengine_us import ( PolicyEngineUSVariableMetadataIndex, @@ -467,13 +470,29 @@ def fit( } self.calls.append(call) means = {target: float(table[target].mean()) for target in targets} - return _MeanFitted(means, weight_kind) + regimes = { + target: detect_regime( + table[target].to_numpy(dtype=np.float64), + zero_atol=DEFAULT_ZERO_ATOL, + ) + for target in targets + } + return _MeanFitted(means, weight_kind, regimes) class _MeanFitted: - def __init__(self, means: dict[str, float], weight_kind: str) -> None: + def __init__( + self, + means: dict[str, float], + weight_kind: str, + regimes: dict[str, str], + ) -> None: self.means = means self.weight_kind = weight_kind + self._regimes = regimes + + def regimes(self) -> dict[str, str]: + return dict(self._regimes) def predict(self, frame: pd.DataFrame) -> pd.DataFrame: assert all(dtype == np.dtype("float64") for dtype in frame.dtypes) @@ -622,6 +641,31 @@ def test_default_transfer_preserves_native_fields_and_registers_added_inputs() - for entry in provenance.values() for pattern in entry.patterns ) + assert all( + tuple(pattern.regimes) == (entry.column,) + for entry in provenance.values() + for pattern in entry.patterns + ) + regime_maps = { + tuple(pattern.regimes.items()) + for entry in provenance.values() + for pattern in entry.patterns + } + assert regime_maps + assert { + regime + for entry in provenance.values() + for pattern in entry.patterns + for regime in pattern.regimes.values() + } <= { + Regime.ZERO_INFLATED_POSITIVE, + Regime.POSITIVE_ONLY, + Regime.SIGN_ONLY, + } + assert all( + pattern.regimes["qualified_dividend_income"] == Regime.POSITIVE_ONLY + for pattern in provenance["qualified_dividend_income"].patterns + ) assert provenance["qualified_dividend_income"].entity == "person" assert provenance["first_home_mortgage_balance"].entity == "tax_unit" assert set(ACS_GROUP_TRANSFER_PREDICTORS).issubset( @@ -645,6 +689,39 @@ def test_default_transfer_preserves_native_fields_and_registers_added_inputs() - assert any(name.startswith(prefix) for name in fit_names) +def test_unbanked_transfer_records_three_sign_regime() -> None: + donor = _with_columns( + _donor_frame(), + "person", + { + "signed_tail": [ + -500.0, + 0.0, + 200.0, + 0.0, + -50.0, + 400.0, + 0.0, + 0.0, + ] + }, + ) + + result = transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={"person": {"signed_tail": ("signed_tail",)}}, + seed=19, + n_estimators=3, + ) + + assert len(result.imputed_inputs) == 1 + assert all( + pattern.regimes == {"signed_tail": Regime.THREE_SIGN} + for pattern in result.imputed_inputs[0].patterns + ) + + def test_default_families_exclude_runtime_owned_take_up_columns() -> None: families = default_acs_transfer_target_families(_donor_frame()) @@ -904,6 +981,21 @@ def test_large_target_family_is_split_to_bound_retained_qrf_forests( } +def test_realized_regime_receipt_requires_exact_targets_and_known_labels() -> None: + with pytest.raises(ValueError, match="regime targets changed"): + acs_transfer_module._validated_qrf_regimes( + {"first": Regime.ZERO_INFLATED_POSITIVE}, + expected_targets=("first", "second"), + boundary="fixture", + ) + with pytest.raises(ValueError, match="invalid realized regimes"): + acs_transfer_module._validated_qrf_regimes( + {"first": "not_a_regime"}, + expected_targets=("first",), + boundary="fixture", + ) + + def test_target_bank_cold_output_matches_unbanked_monolith( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -923,6 +1015,66 @@ def test_target_bank_cold_output_matches_unbanked_monolith( assert all(path.is_file() for path in _bank_paths(bank)) +def test_target_bank_persists_pattern_regimes_through_resume( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + _lock_bank_fixture_threads(monkeypatch) + monolithic = _run_bank_fixture() + bank_root = tmp_path / "regime-bank" + cold_bank = _bank_store(bank_root) + cold = _run_bank_fixture(cold_bank) + warm_bank = _bank_store(bank_root) + resumed = _run_bank_fixture(warm_bank) + + _assert_transfer_results_exact(cold, monolithic) + _assert_transfer_results_exact(resumed, monolithic) + expected_patterns = monolithic.imputed_inputs[0].patterns + assert all( + set(pattern.regimes) == set(_BANK_TARGETS) for pattern in expected_patterns + ) + for path in _bank_paths(cold_bank): + metadata, _raw_bits = acs_transfer_bank_module._read_checkpoint(path) + assert metadata["schema_version"] == ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION + model_target = metadata["target"]["model_target"] + expected = { + pattern.name: pattern.regimes[model_target] for pattern in expected_patterns + } + observed = {} + for step in metadata["pattern_steps"]: + assert set(step) == { + "pattern", + "regime", + "state_before_sha256", + "state_after", + } + observed[step["pattern"]] = step["regime"] + assert observed == expected + for bank, sources, statuses in ( + ( + cold_bank, + ("rebuilt",) * len(_BANK_TARGETS), + ("missing",) * len(_BANK_TARGETS), + ), + ( + warm_bank, + ("checkpoint",) * len(_BANK_TARGETS), + ("resumed",) * len(_BANK_TARGETS), + ), + ): + receipt_targets = _assert_bank_receipt( + bank, + sources=sources, + load_statuses=statuses, + ) + for target_index, model_target in enumerate(_BANK_TARGETS): + expected = { + pattern.name: pattern.regimes[model_target] + for pattern in expected_patterns + } + assert receipt_targets[str(target_index)]["realized_regimes"] == expected + + def test_target_bank_materializer_v1_artifacts_fail_closed_with_named_receipts( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -962,6 +1114,57 @@ def test_target_bank_materializer_v1_artifacts_fail_closed_with_named_receipts( ) +def test_target_bank_schema_v1_artifacts_rebuild_for_regime_receipts( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + _lock_bank_fixture_threads(monkeypatch) + bank_root = tmp_path / "schema-v1-bank" + + with monkeypatch.context() as legacy: + legacy.setattr( + acs_transfer_bank_module, + "ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION", + 1, + ) + legacy_bank = _bank_store(bank_root) + baseline = _run_bank_fixture(legacy_bank) + for path in _bank_paths(legacy_bank): + metadata, _raw_bits = acs_transfer_bank_module._read_checkpoint(path) + assert metadata["schema_version"] == 1 + for step in metadata["pattern_steps"]: + step.pop("regime") + metadata.pop("content_metadata_sha256") + metadata["content_metadata_sha256"] = ( + acs_transfer_bank_module._mapping_sha256(metadata) + ) + encoded = acs_transfer_bank_module._canonical_json(metadata).encode("utf-8") + with h5py.File(path, mode="r+") as h5: + del h5[acs_transfer_bank_module._METADATA_DATASET] + h5.create_dataset( + acs_transfer_bank_module._METADATA_DATASET, + data=np.frombuffer(encoded, dtype=np.uint8), + dtype=np.uint8, + track_times=False, + ) + h5.flush() + + assert ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION == 2 + current_bank = _bank_store(bank_root) + rebuilt = _run_bank_fixture(current_bank) + + _assert_transfer_results_exact(rebuilt, baseline) + targets = _assert_bank_receipt( + current_bank, + sources=("rebuilt",) * len(_BANK_TARGETS), + load_statuses=("invalid_rebuild",) * len(_BANK_TARGETS), + ) + for record in targets.values(): + assert record["invalid_checkpoint"]["message"] == ( + "ACS transfer target checkpoint has an unsupported binding." + ) + + def test_target_bank_fsyncs_file_then_parent_directory_after_each_rename( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -1045,6 +1248,11 @@ def run( person[target].dtype == CANONICAL_STRING_DTYPE for target in ("ssn_card_type", "immigration_status_str") ) + assert all( + set(pattern.regimes) == {"__acs_transfer_immigration_status_pair"} + for entry in result.imputed_inputs + for pattern in entry.patterns + ) targets = warm_bank.receipt()["targets"] assert set(targets) == {"0"} diff --git a/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py b/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py index c071a4f55..1614be14b 100644 --- a/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py +++ b/packages/microcosm-build/tests/test_us_exact_k_ladder_e2e.py @@ -137,7 +137,7 @@ def _write_ready_pool(tmp_path: Path) -> Path: "populace_us_multispine_pool_checkpoint_provenance" ), "schema_version": 1, - "materializer_version": 3, + "materializer_version": 4, "enabled": False, "agreement": { "source": "always_fresh", diff --git a/packages/microcosm-build/tests/test_us_late_producer_dag.py b/packages/microcosm-build/tests/test_us_late_producer_dag.py index aa83a6f76..a07f8b4d4 100644 --- a/packages/microcosm-build/tests/test_us_late_producer_dag.py +++ b/packages/microcosm-build/tests/test_us_late_producer_dag.py @@ -786,7 +786,7 @@ def test_canonical_us_late_schedule_is_import_validated_and_byte_stable() -> Non receipt = us_late_producer_schedule_receipt() assert receipt["schema_version"] == 16 assert receipt["execution_receipt_contract"] == { - "version": 3, + "version": 4, "row_binding": ( "declared_globally_reconciled_input_and_scope_exact_output_source_" "and_primary_callback_resource_receipt_and_previous_execution_sha256" @@ -794,7 +794,8 @@ def test_canonical_us_late_schedule_is_import_validated_and_byte_stable() -> Non "virtual_resource_binding": ("exact_kind_specific_semantic_payload_and_sha256"), "top_binding": ( "entry_and_output_frame_sha256_execution_chain_source_" - "completion_and_nineteen_transfer_groups" + "completion_and_nineteen_transfer_groups_with_exact_" + "realized_regime_reconstruction" ), "transition_authority": { "authority_id": "us_stacked_late_producer_transition", diff --git a/packages/microcosm-build/tests/test_us_multispine_pool.py b/packages/microcosm-build/tests/test_us_multispine_pool.py index 35e1b8268..9d7565ce1 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool.py @@ -1706,6 +1706,9 @@ def __init__( self.observations = observations self.weight_kind = weight_kind + def regimes(self) -> dict[str, str]: + return {outcome: "zero_inflated_positive" for outcome in self.outcomes} + def predict(self, test: pd.DataFrame, **_kwargs: object) -> pd.DataFrame: self.calls[f"{self.owner}.predict"] += 1 rows = len(test) diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 8f5dc55e5..ec16ba871 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -8,6 +8,7 @@ import pandas as pd import pytest +import microcosm.build.us_runtime.acs_transfer as acs_transfer_module import microcosm.build.us_runtime.h5_io as h5_io import microcosm.build.us_runtime.stacked_spine as stacked_spine_module from microcosm.build.frame_checkpoint import ( @@ -450,7 +451,7 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: "stage_checkpoints": { "artifact_kind": "populace_us_multispine_pool_checkpoint_provenance", "schema_version": 1, - "materializer_version": 3 if not stacked else 5, + "materializer_version": 4 if not stacked else 5, "enabled": False, "agreement": { "source": "always_fresh", @@ -547,8 +548,28 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: } }, } - group_receipts = { - group.name: { + group_receipts: dict[str, dict[str, object]] = {} + realized_regimes: list[dict[str, object]] = [] + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS: + model_targets = acs_transfer_module._model_target_names(group.targets) + group_regimes = [ + { + "entity": group.entity, + "family": group.family, + "pattern": "pattern_0__required_only", + "observed_optional_predictors": [], + "predictors": ["fixture_predictor"], + "seed": 0, + "weight_kind": "household_weight", + "donor_rows": 1, + "recipient_rows": 1, + "model_targets": list(model_targets), + "regimes": { + target: "zero_inflated_positive" for target in model_targets + }, + } + ] + group_receipts[group.name] = { "producer": group.name, "entity": group.entity, "family": group.family, @@ -559,9 +580,9 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: } for target in group.targets }, + "realized_regimes": group_regimes, } - for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS - } + realized_regimes.extend(group_regimes) group_by_name = { group.name: group for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS @@ -593,6 +614,7 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: ], "groups": group_receipts, "targets": aggregate_targets, + "realized_regimes": realized_regimes, "completion": { "status": "complete", "group_count": 19, @@ -872,6 +894,19 @@ def test_ready_legacy_pool_loader_accepts_pre_653_schema_four_envelope( assert frame.n("household") == 3 +def test_ready_legacy_pool_loader_rejects_materializer_three_checkpoint( + tmp_path: Path, +) -> None: + pytest.importorskip("tables") + manifest_path = _write_ready_pool(tmp_path) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["stage_checkpoints"]["materializer_version"] = 3 + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(ValueError, match="stage_checkpoints.identity"): + load_simulation_ready_us_multispine_pool(manifest_path) + + def test_ready_legacy_pool_loader_rejects_current_stacked_envelope( tmp_path: Path, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 036f54dc9..ec3059e6c 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -31,6 +31,7 @@ from microcosm.build.us_runtime.acs_transfer import transfer_acs_inputs from microcosm.build.us_runtime.acs_transfer_bank import ( ACS_TRANSFER_TARGET_BANK_MATERIALIZER_VERSION, + ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION, ) from microcosm.build.us_runtime.multispine_pool import ( MultispinePoolCheckpoint, @@ -57,6 +58,7 @@ load_take_up_contract, take_up_contract_identity, ) +from microcosm.fit.qrf import DEFAULT_ZERO_ATOL, detect_regime from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights _FIXTURE_SEED_PERSON_COLUMN = "takes_up_medicaid_if_eligible" @@ -263,16 +265,33 @@ def fit( weights == frame.resolve_weights(frame.column_entity(targets[0])).kind.value ) table = frame.table(frame.column_entity(targets[0])) + regimes = { + target: detect_regime( + table[target].to_numpy(dtype=np.float64), + zero_atol=DEFAULT_ZERO_ATOL, + ) + for target in targets + } return _MeanFitted( {target: float(table[target].mean()) for target in targets}, weights, + regimes, ) class _MeanFitted: - def __init__(self, means: dict[str, float], weight_kind: str) -> None: + def __init__( + self, + means: dict[str, float], + weight_kind: str, + regimes: dict[str, str], + ) -> None: self.means = means self.weight_kind = weight_kind + self._regimes = regimes + + def regimes(self) -> dict[str, str]: + return dict(self._regimes) def predict(self, frame: pd.DataFrame) -> pd.DataFrame: return pd.DataFrame( @@ -802,6 +821,7 @@ def _target_bank_receipt_after_interruption( "path": f"/fixture/targets/{index:03d}__target_{index}.h5", "checkpoint_sha256": f"{index:064x}", "size_bytes": 1_000 + index, + "realized_regimes": {"pattern_0__required_only": "zero_inflated_positive"}, } if not resumed: record["write_status"] = "rebuilt" @@ -809,7 +829,7 @@ def _target_bank_receipt_after_interruption( targets[str(index)] = record return { "artifact_kind": "populace_us_multispine_acs_transfer_target_bank_provenance", - "schema_version": 1, + "schema_version": ACS_TRANSFER_TARGET_BANK_SCHEMA_VERSION, "materializer_version": ACS_TRANSFER_TARGET_BANK_MATERIALIZER_VERSION, "root": "/fixture/acs-transfer", "identity": {"fixture": "identity"}, @@ -929,16 +949,39 @@ def _canonical_late_transfer_receipt( } groups: dict[str, object] = {} targets: dict[str, object] = {} + realized_regimes: list[dict[str, object]] = [] for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS: group_targets = { f"{group.entity}/{group.family}/{target}": {"residual_null_rows": 0} for target in group.targets } + model_targets = acs_transfer_module._model_target_names(group.targets) + group_regimes = [ + { + "entity": group.entity, + "family": group.family, + "pattern": "pattern_0__required_only", + "observed_optional_predictors": [], + "predictors": ["fixture_predictor"], + "seed": 0, + "weight_kind": "household_weight", + "donor_rows": 1, + "recipient_rows": 1, + "model_targets": list(model_targets), + "regimes": { + target: "zero_inflated_positive" for target in model_targets + }, + } + ] groups[group.name] = { "producer": group.name, + "entity": group.entity, + "family": group.family, "ordered_targets": list(group.targets), "targets": group_targets, + "realized_regimes": group_regimes, } + realized_regimes.extend(group_regimes) for target in group.targets: targets[ f"{group.entity}/{canonical_family[(group.entity, target)]}/{target}" @@ -960,6 +1003,7 @@ def _canonical_late_transfer_receipt( ], "groups": groups, "targets": targets, + "realized_regimes": realized_regimes, "completion": { "status": "complete", "group_count": 19, @@ -1903,7 +1947,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "6e9dce8f0fd3e3f0101103a14d6a08ac8527b90b82d48fa8bad2c4cc70dbdfde", + "spec_sha256": "5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554", }, } @@ -2106,7 +2150,7 @@ def test_constants_adapter_fixture_checkpoints_are_byte_identical_and_only_recei "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "6e9dce8f0fd3e3f0101103a14d6a08ac8527b90b82d48fa8bad2c4cc70dbdfde", + "spec_sha256": "5b0014c3eb6cb121f0a9f2138ab860be30ef2251dd6ff4ec38cbf5f778899554", } def run_fixture(root: Path, *, config_authority: str) -> dict[str, object]: @@ -2750,7 +2794,7 @@ def test_legacy_checkpoint_identity_excludes_stacked_late_producer_schedule( verified, policyengine_us_version="fixture-engine", ) - assert current["materializer_version"] == 3 + assert current["materializer_version"] == 4 assert "late_producer_schedule" not in current["pool_code"] changed_schedule = pool_tool._json_ready( @@ -2770,7 +2814,45 @@ def test_legacy_checkpoint_identity_excludes_stacked_late_producer_schedule( assert changed == current -def test_stacked_checkpoint_identity_binds_v11_semantic_contracts( +def test_legacy_materializer_v3_checkpoint_cannot_resume_after_shared_fix( + pool_tool: ModuleType, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + verified = _verified_inputs_fixture(pool_tool, tmp_path / "pins") + checkpoint_root = tmp_path / "legacy-v3-checkpoints" + with monkeypatch.context() as legacy: + legacy.setattr( + pool_tool, + "_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION", + 3, + ) + legacy_identity = pool_tool._legacy_pool_checkpoint_base_identity( + verified, + policyengine_us_version="fixture-engine", + ) + legacy_store = pool_tool._PoolStageCheckpointStore( + checkpoint_root, + base_identity=legacy_identity, + materializer_version=3, + ) + legacy_store.bind_input_receipts(_checkpoint_fixture_input_receipts()) + _run_checkpoint_fixture(pool_tool, tmp_path, store=legacy_store) + + current_identity = pool_tool._legacy_pool_checkpoint_base_identity( + verified, + policyengine_us_version="fixture-engine", + ) + current_store = pool_tool._PoolStageCheckpointStore( + checkpoint_root, + base_identity=current_identity, + materializer_version=4, + ) + assert legacy_store.base_identity_sha256 != current_store.base_identity_sha256 + assert current_store.load_deepest() is None + + +def test_stacked_checkpoint_identity_binds_v12_semantic_contracts( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -2803,7 +2885,7 @@ def identity() -> dict[str, object]: current = identity() pool_code = current["pool_code"] - assert current["materializer_version"] == 11 + assert current["materializer_version"] == 12 assert current["stacked_authority"]["version"] == 10 assert pool_code["operator_order"] == [ "assemble_stacked_spine", @@ -3007,7 +3089,7 @@ def changed_source_stage_binding( ) ) - assert current["materializer_version"] == stale_qrf["materializer_version"] == 11 + assert current["materializer_version"] == stale_qrf["materializer_version"] == 12 assert stale_qrf["pool_code"]["primary_qrf_checkpoint_schema_version"] == 5 assert ( pool_tool._discover_stacked_checkpoint_identity( @@ -3055,7 +3137,7 @@ def changed_source_stage_binding( assert "checkpoint base identity is stale" in capsys.readouterr().out -def test_pool_envelope_v7_preserves_stacked_bank_identity_but_rejects_v6( +def test_pool_envelope_v8_preserves_stacked_bank_identity_but_rejects_v7( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, @@ -3088,7 +3170,7 @@ def identity() -> dict[str, object]: legacy.setattr( pool_tool, "POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION", - 6, + 7, ) assert identity() == current_identity legacy_store = pool_tool._PoolStageCheckpointStore( @@ -3109,7 +3191,7 @@ def identity() -> dict[str, object]: ) capsys.readouterr() - assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 8 assert identity() == current_identity current_store = pool_tool._PoolStageCheckpointStore( checkpoint_root, @@ -3198,7 +3280,7 @@ def test_qbi_receipt_route_resolution_rejects_wrong_or_ambiguous_paths( ) -@pytest.mark.parametrize("legacy_version", (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) +@pytest.mark.parametrize("legacy_version", (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) def test_legacy_stacked_materializer_checkpoint_is_not_discovered( pool_tool: ModuleType, monkeypatch: pytest.MonkeyPatch, @@ -3252,7 +3334,7 @@ def test_legacy_stacked_materializer_checkpoint_is_not_discovered( ) ) - assert pool_tool._STACKED_CHECKPOINT_MATERIALIZER_VERSION == 11 + assert pool_tool._STACKED_CHECKPOINT_MATERIALIZER_VERSION == 12 assert ( pool_tool._discover_stacked_checkpoint_identity( checkpoint_root, @@ -3642,22 +3724,22 @@ def deterministic_fixture_h5( assert keywords["resume"] is None assert callable(keywords["checkpoint"]) checkpoint_store = keywords["checkpoint"].__self__ - assert checkpoint_store.base_identity["materializer_version"] == 3 + assert checkpoint_store.base_identity["materializer_version"] == 4 assert "late_producer_schedule" not in checkpoint_store.base_identity["pool_code"] outputs = pool_tool._output_paths(output, checkpoint_root=checkpoint_root) manifest = pool_tool._read_json_object(outputs.manifest) diagnostics = pool_tool._read_json_object(outputs.agreement_diagnostics) assert pool_tool.POOL_MANIFEST_SCHEMA_VERSION == 8 - assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 8 assert manifest["schema_version"] == 4 assert diagnostics["schema_version"] == 4 assert "materializer_version" not in manifest["pool_h5"] - assert manifest["stage_checkpoints"]["materializer_version"] == 3 + assert manifest["stage_checkpoints"]["materializer_version"] == 4 assert { receipt["materializer_version"] for receipt in manifest["stage_checkpoints"]["stages"].values() - } == {3} + } == {4} manifest_bytes = outputs.manifest.read_bytes().replace( str(tmp_path.resolve()).encode(), b"$TMP", @@ -3677,9 +3759,13 @@ def deterministic_fixture_h5( # checkpoint metadata). "pool_h5": "ced797ecdd44a638c2a3945f07ad612098a7095ca53a5f458699bca6d6e38b3e", "agreement": "f39f0d918bf7ee01dddb5517d8830b8adb541273c5be084307be91397caca3cb", - # Exact pre-#653 schema-4/materializer-3 publication bytes from - # preserved #652 commit 54d2dee6. - "manifest": "14e6b3a409dfe2108253668a65ed32c0365b246f379ad895d8441c939adde65e", + # Pre-#653 schema-4 publication bytes (preserved #652 commit 54d2dee6) + # with the checkpoint identity advanced to materializer 4: the + # retirement-distribution source operator shared by both paths changed + # semantics, so legacy checkpoints must not resume across it. The + # pool-H5 and agreement digests above are unchanged, proving the + # data surface stayed byte-stable below the training cap. + "manifest": "685824ba549326b1eac79b5fbf5cf7981e59e0daa9f7cd74ed5d7952e50c06bd", } @@ -4110,7 +4196,7 @@ def test_pool_imputation_binds_and_publishes_acs_target_bank( frame = _source_frame() checkpoint_identity = { "artifact_kind": "fixture-pool-checkpoint-identity", - "materializer_version": 7, + "materializer_version": pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION, "inputs": {"fixture": {"sha256": "b" * 64}}, } base_identity_sha256 = pool_tool._pool_checkpoint_identity_sha256( @@ -4845,7 +4931,7 @@ def test_pool_checkpoint_store_round_trips_nullable_boolean_families( manifest = pool_tool._read_json_object( cold_store.checkpoint_manifest_path(stage) ) - assert manifest["materializer_version"] == 7 + assert manifest["materializer_version"] == 8 loaded = pool_tool.load_frame_checkpoint(path).frame if stage == "assembled": assert "fixture_declared_boolean" not in loaded.person @@ -4866,11 +4952,11 @@ def test_pool_checkpoint_store_round_trips_nullable_boolean_families( assert resumed.frame.person["fixture_declared_boolean"].isna().sum() == 1 -def test_simulated_v7_checkpoint_accepts_both_string_encodings_without_rewrite( +def test_simulated_v8_checkpoint_accepts_both_string_encodings_without_rewrite( pool_tool: ModuleType, tmp_path: Path, ) -> None: - """V7 authenticates both physical string encodings as one logical frame.""" + """V8 authenticates both physical string encodings as one logical frame.""" pytest.importorskip("h5py") checkpoint_root = tmp_path / "checkpoints" @@ -4882,7 +4968,7 @@ def test_simulated_v7_checkpoint_accepts_both_string_encodings_without_rewrite( loaded = pool_tool.load_frame_checkpoint(checkpoint_path) canonical_v2_bytes = checkpoint_path.read_bytes() canonical_identity = loaded.metadata["identity"] - assert loaded.metadata["materializer_version"] == 7 + assert loaded.metadata["materializer_version"] == 8 assert any( column["dtype"] == str(CANONICAL_STRING_DTYPE) for columns in loaded.metadata["frame_schema"]["entities"].values() @@ -4913,7 +4999,7 @@ def test_simulated_v7_checkpoint_accepts_both_string_encodings_without_rewrite( banked_v2_bytes = checkpoint_path.read_bytes() assert banked_v2_bytes != canonical_v2_bytes assert legacy_metadata["identity"] == canonical_identity - assert legacy_metadata["materializer_version"] == 7 + assert legacy_metadata["materializer_version"] == 8 assert any( column["dtype"] == "object" for columns in legacy_metadata["frame_schema"]["entities"].values() @@ -5321,9 +5407,9 @@ def test_legacy_pool_materializer_artifacts_fail_closed_with_named_receipts( assert manifest["identity"]["materializer_version"] == legacy_version capsys.readouterr() - assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 7 + assert pool_tool.POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION == 8 current_store = _checkpoint_fixture_store(pool_tool, checkpoint_root) - assert current_store.base_identity["materializer_version"] == 7 + assert current_store.base_identity["materializer_version"] == 8 assert current_store.load_deepest() is None output = capsys.readouterr().out diff --git a/packages/microcosm-build/tests/test_us_retirement_distributions.py b/packages/microcosm-build/tests/test_us_retirement_distributions.py index c25156afb..9d7b05d6b 100644 --- a/packages/microcosm-build/tests/test_us_retirement_distributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_distributions.py @@ -31,6 +31,7 @@ with_us_retirement_distribution_inputs, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers +from microcosm.fit.qrf import DEFAULT_ZERO_ATOL, Regime, detect_regime from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights ROOT = Path(__file__).resolve().parents[3] @@ -347,6 +348,272 @@ def fit( assert gate.passed, gate.failures +def test_production_imputer_hands_rare_keogh_support_to_qrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + row_count = 6_000 + positions = np.arange(row_count, dtype=np.int64) + person = pd.DataFrame( + { + "person_id": positions + 1, + "person_household_id": positions + 10_000, + "person_tax_unit_id": positions + 20_000, + "person_spm_unit_id": positions + 30_000, + "person_family_id": positions + 40_000, + "person_marital_unit_id": positions + 50_000, + "age": np.full(row_count, 67, dtype=np.int64), + "is_female": np.zeros(row_count, dtype=bool), + "has_esi": np.zeros(row_count, dtype=bool), + "tax_unit_role_input": ["HEAD"] * row_count, + "employment_income_before_lsr": np.full(row_count, 50_000.0), + "self_employment_income_before_lsr": np.zeros(row_count), + "social_security_retirement": np.full(row_count, 10_000.0), + "social_security_disability": np.zeros(row_count), + "social_security_dependents": np.zeros(row_count), + "social_security_survivors": np.zeros(row_count), + } + ) + for suffix in _SLOT_SUFFIXES: + person[f"DST_SC{suffix}"] = np.zeros(row_count, dtype=np.int64) + person[f"DST_VAL{suffix}"] = np.zeros(row_count, dtype=np.float64) + person.loc[[2_427, 4_711], "DST_SC1"] = 5 + person.loc[[2_427, 4_711], "DST_VAL1"] = (2_040.0, 30_000.0) + identifiers = { + "household": person["person_household_id"].to_numpy(), + "tax_unit": person["person_tax_unit_id"].to_numpy(), + "spm_unit": person["person_spm_unit_id"].to_numpy(), + "family": person["person_family_id"].to_numpy(), + "marital_unit": person["person_marital_unit_id"].to_numpy(), + } + tables = { + entity: pd.DataFrame({f"{entity}_id": values}) + for entity, values in identifiers.items() + } + tables["person"] = person + tables["tax_unit"]["filing_status_input"] = ["SINGLE"] * row_count + source = Frame( + tables, + US_SCHEMA, + {"household": Weights(np.ones(row_count), WeightKind.DESIGN)}, + ) + direct = with_us_retirement_distribution_inputs( + source, + seed=0, + time_period=2024, + ) + expanded = clone_us_frame_for_puf_support(direct) + calls: dict[str, object] = {} + + class FakeFitted: + def predict(self, test: pd.DataFrame, **_kwargs) -> pd.DataFrame: + return pd.DataFrame( + 0.0, + index=test.index, + columns=list(_PUF_QRF_OUTPUTS), + ) + + class FakeQRF: + def __init__(self, **_kwargs: object) -> None: + pass + + def fit( + self, + training: pd.DataFrame, + predictors: list[str], + targets: list[str], + *, + weights: np.ndarray, + ) -> FakeFitted: + calls["training"] = training.copy() + calls["predictors"] = tuple(predictors) + calls["targets"] = tuple(targets) + calls["weights"] = weights.copy() + return FakeFitted() + + monkeypatch.setattr(module, "QRF", FakeQRF) + with_us_retirement_distribution_inputs( + expanded, + seed=0, + time_period=2024, + force_puf_imputation=True, + ) + + training = calls["training"] + capped_weights = calls["weights"] + assert isinstance(training, pd.DataFrame) + assert isinstance(capped_weights, np.ndarray) + assert len(training) == len(capped_weights) == 5_000 + np.testing.assert_array_equal( + np.sort( + training.loc[ + training["keogh_distributions"] > DEFAULT_ZERO_ATOL, + "keogh_distributions", + ].to_numpy() + ), + [2_040.0, 30_000.0], + ) + assert ( + detect_regime( + training["keogh_distributions"].to_numpy(), + zero_atol=DEFAULT_ZERO_ATOL, + ) + == Regime.ZERO_INFLATED_POSITIVE + ) + np.testing.assert_array_equal( + capped_weights[training["keogh_distributions"].to_numpy() > DEFAULT_ZERO_ATOL], + [0.5, 0.5], + ) + assert capped_weights.sum() == pytest.approx(float(row_count) / 2.0) + + +def test_puf_training_cap_preserves_nonzero_union_and_weighted_sign_mass() -> None: + row_count = 6_000 + source_positions = np.arange(row_count, dtype=np.int64) + training = pd.DataFrame( + { + "source_position": source_positions, + **{ + target: np.zeros(row_count, dtype=np.float64) + for target in _PUF_QRF_OUTPUTS + }, + }, + index=pd.Index( + 10_000 + (source_positions // 2) * 3, + name="duplicate_non_range_donor_index", + ), + ) + carriers = { + "taxable_401k_distributions": {5_902: 10_000.0}, + "taxable_403b_distributions": {5_903: 2_000.0}, + "keogh_distributions": {5_906: 2_040.0, 5_912: 30_000.0}, + "taxable_sep_distributions": {5_909: 4_000.0}, + } + for target, values in carriers.items(): + for position, value in values.items(): + training.iloc[position, training.columns.get_loc(target)] = value + weights = np.linspace(1.0, 3.0, row_count, dtype=np.float64) + + first, first_weights = module._cap_retirement_distribution_training( + training, + weights, + targets=_PUF_QRF_OUTPUTS, + max_train_samples=5_000, + seed=0, + zero_atol=DEFAULT_ZERO_ATOL, + ) + second, second_weights = module._cap_retirement_distribution_training( + training, + weights, + targets=_PUF_QRF_OUTPUTS, + max_train_samples=5_000, + seed=0, + zero_atol=DEFAULT_ZERO_ATOL, + ) + + assert len(first) == 5_000 + pd.testing.assert_frame_equal(first, second) + np.testing.assert_array_equal(first_weights, second_weights) + retained_positions = first["source_position"].to_numpy(dtype=np.int64) + assert ( + sha256(retained_positions.astype(" DEFAULT_ZERO_ATOL, + capped_values > DEFAULT_ZERO_ATOL, + ), + ): + assert first_weights[capped_mask].sum() == pytest.approx( + weights[full_mask].sum() + ) + assert first_weights.sum() == pytest.approx(weights.sum()) + + common_zero = ~first.loc[:, list(_PUF_QRF_OUTPUTS)].any(axis=1).to_numpy() + original_selected_weights = weights[retained_positions] + zero_factors = first_weights[common_zero] / original_selected_weights[common_zero] + np.testing.assert_allclose(zero_factors, zero_factors[0], rtol=0.0, atol=1e-12) + + +def test_puf_training_cap_fails_when_nonzero_union_leaves_no_zero_slot() -> None: + row_count = 5_001 + training = pd.DataFrame( + {target: np.zeros(row_count, dtype=np.float64) for target in _PUF_QRF_OUTPUTS} + ) + training.loc[:4_999, "taxable_401k_distributions"] = 1.0 + + with pytest.raises(SourceRuntimeError, match="leaves no room"): + module._cap_retirement_distribution_training( + training, + np.ones(row_count, dtype=np.float64), + targets=_PUF_QRF_OUTPUTS, + max_train_samples=5_000, + seed=0, + zero_atol=DEFAULT_ZERO_ATOL, + ) + + +def test_puf_training_cap_does_not_waste_representative_slots_on_zero_weights() -> None: + row_count = 12 + training = pd.DataFrame( + { + "source_position": np.arange(row_count, dtype=np.int64), + **{ + target: np.zeros(row_count, dtype=np.float64) + for target in _PUF_QRF_OUTPUTS + }, + } + ) + training.loc[11, "keogh_distributions"] = 2_040.0 + weights = np.zeros(row_count, dtype=np.float64) + weights[[1, 5, 9, 11]] = (2.0, 3.0, 5.0, 7.0) + + capped, capped_weights = module._cap_retirement_distribution_training( + training, + weights, + targets=_PUF_QRF_OUTPUTS, + max_train_samples=6, + seed=0, + zero_atol=DEFAULT_ZERO_ATOL, + ) + + retained_positions = capped["source_position"].to_numpy(dtype=np.int64) + assert {1, 5, 9, 11}.issubset(retained_positions) + assert capped_weights.sum() == pytest.approx(weights.sum()) + assert capped_weights[retained_positions == 11].item() == 7.0 + + def test_completed_puf_surface_survives_narrowed_support_without_refit( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_spec_bundle.py b/packages/microcosm-build/tests/test_us_spec_bundle.py index 98b00fc1d..368810fb2 100644 --- a/packages/microcosm-build/tests/test_us_spec_bundle.py +++ b/packages/microcosm-build/tests/test_us_spec_bundle.py @@ -380,7 +380,7 @@ def test_constant_derived_domain_counts_are_complete( ) assert ( compiled_schedule["payload_sha256"] - == "7766f2e94476cceb93d9730a74afb2ca6fed836068053f96fa4141bcc2f6154e" + == "6af0cc77290b961af44fabddccf33dfd96d8ec2459e4b6411ad21aac009a832f" ) assert len(take_up["programs"]) == 13 diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index 4e1b8f9a8..8d58f9d7e 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -2686,6 +2686,11 @@ def test_gap_fill_fills_both_directions_with_authority_receipts() -> None: assert absence["rows"] == 1 assert absence["unexpected_null_rows"] == 0 assert absence["structural_rows_filled"] == 0 + assert survey["realized_regimes"] + assert all( + row["model_targets"] == list(row["regimes"]) + for row in survey["realized_regimes"] + ) survey_transfer = result.transfer_results["asec_survey_to_acs"] native_predictor_used = any( @@ -4672,6 +4677,7 @@ def _run_real_late_executor_fixture( bank_identity_sha256: str | None = None, bound_clone_attachment_seed: int = 578, asec_earnings_delta: float = 0.0, + tamper_group_regime: bool = False, ) -> tuple[stacked_spine_module.StackedLateProducerResult, tuple[str, ...], int]: registry = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_REGISTRY initial = _late_universe_entry_fixture() @@ -4809,25 +4815,64 @@ def transfer( derive_schedule_d=False, ) ) + model_targets = acs_transfer_module._model_target_names(group.targets) + pattern = acs_transfer_module.AcsTransferPattern( + name="fixture_complete_case", + observed_optional_predictors=(), + predictors=("age",), + seed=0, + weight_kind="design", + donor_rows=1, + recipient_rows=1, + regimes={target: "zero_inflated_positive" for target in model_targets}, + ) transfer_result = AcsTransferResult( frame=frame, - imputed_inputs=(), + imputed_inputs=tuple( + acs_transfer_module.AcsImputedInput( + column=target, + entity=group.entity, + family=group.family, + donor_spine="fixture", + donor_channel="asec", + predictors=pattern.predictors, + seed=pattern.seed, + weight_kind=pattern.weight_kind, + patterns=(pattern,), + ) + for target in group.targets + ), fit_records=(), deferred_inputs=(), resolved_donor_channel="asec", ) + receipt = { + "producer": group.name, + "entity": group.entity, + "family": group.family, + "ordered_targets": list(group.targets), + "targets": { + f"{group.entity}/{group.family}/{target}": { + "residual_null_rows": 0, + } + for target in group.targets + }, + "realized_regimes": ( + stacked_spine_module._acs_transfer_realized_regime_receipt( + transfer_result + ) + ), + } + if ( + tamper_group_regime + and group is stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS[0] + ): + first_row = receipt["realized_regimes"][0] + first_target = first_row["model_targets"][0] + first_row["regimes"][first_target] = "zero_inflated_negative" return stacked_spine_module.StackedPostPufTransferResult( frame, - { - "producer": group.name, - "ordered_targets": list(group.targets), - "targets": { - f"{group.entity}/{group.family}/{target}": { - "residual_null_rows": 0, - } - for target in group.targets - }, - }, + receipt, transfer_result, ) @@ -4920,12 +4965,132 @@ def test_real_late_executor_follows_canonical_order_and_finalizes_sources_once( stacked_spine_module.US_LATE_PRODUCER_TRANSITION_AUTHORITY_KEY ]["sha256"] ) + post_puf_receipt = result.receipt["post_puf_transfer"] + assert len(post_puf_receipt["realized_regimes"]) == 19 + assert post_puf_receipt["realized_regimes"] == [ + row + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS + for row in post_puf_receipt["groups"][group.name]["realized_regimes"] + ] + assert all( + row["model_targets"] == list(row["regimes"]) + for row in post_puf_receipt["realized_regimes"] + ) + assert all( + row["producer_receipt"]["realized_regimes"] + for row in result.receipt["execution"] + if row["kind"] == "late_transfer" + ) stacked_spine_module.validate_stacked_late_producer_receipt( result.receipt, boundary="executor regression", frame=result.frame, expected_transition_authority_sha256=result.transition_authority_sha256, ) + transfer = post_puf_receipt + first_group = stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS[0] + + missing = deepcopy(transfer) + del missing["groups"][first_group.name]["realized_regimes"] + with pytest.raises(ValueError, match="no realized-regime evidence"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + missing, + boundary="missing regime fixture", + ) + + invalid = deepcopy(transfer) + invalid_row = invalid["groups"][first_group.name]["realized_regimes"][0] + invalid_target = invalid_row["model_targets"][0] + invalid_row["regimes"][invalid_target] = "not_a_regime" + with pytest.raises(ValueError, match="targets or labels changed"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + invalid, + boundary="invalid regime fixture", + ) + + duplicate = deepcopy(transfer) + duplicate["groups"][first_group.name]["realized_regimes"].append( + deepcopy(duplicate["groups"][first_group.name]["realized_regimes"][0]) + ) + with pytest.raises(ValueError, match="identity changed"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + duplicate, + boundary="duplicate regime fixture", + ) + + detached = deepcopy(transfer) + detached["realized_regimes"] = detached["realized_regimes"][:-1] + with pytest.raises(ValueError, match="not reconstructed"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + detached, + boundary="detached regime fixture", + ) + + legacy = deepcopy(dict(result.receipt)) + legacy["version"] = 3 + for group_receipt in legacy["post_puf_transfer"]["groups"].values(): + group_receipt.pop("realized_regimes") + legacy["post_puf_transfer"].pop("realized_regimes") + with pytest.raises(ValueError, match="receipt version changed"): + stacked_spine_module.validate_stacked_late_producer_receipt( + legacy, + boundary="legacy v3 regime-free fixture", + ) + + +def test_late_executor_rejects_regime_receipt_detached_from_fitted_provenance( + monkeypatch: pytest.MonkeyPatch, +) -> None: + with pytest.raises( + ValueError, + match="realized-regime receipt differs from its fitted imputation provenance", + ): + _run_real_late_executor_fixture( + monkeypatch, + tamper_group_regime=True, + ) + + +def test_realized_regime_receipt_survives_canonical_json_key_sorting() -> None: + expected_targets = ("z_target", "a_target") + receipt = [ + { + "entity": "person", + "family": "fixture", + "pattern": "complete_case", + "observed_optional_predictors": [], + "predictors": ["age"], + "seed": 0, + "weight_kind": "design", + "donor_rows": 2, + "recipient_rows": 3, + "model_targets": list(expected_targets), + "regimes": { + "z_target": "three_sign", + "a_target": "zero_inflated_positive", + }, + } + ] + round_tripped = json.loads(json.dumps(receipt, sort_keys=True)) + assert list(round_tripped[0]["regimes"]) == ["a_target", "z_target"] + + assert stacked_spine_module._validated_acs_transfer_realized_regime_receipt( + round_tripped, + entity="person", + family="fixture", + expected_model_targets=expected_targets, + boundary="canonical JSON fixture", + ) == round_tripped + + del round_tripped[0]["regimes"]["z_target"] + with pytest.raises(ValueError, match="targets or labels changed"): + stacked_spine_module._validated_acs_transfer_realized_regime_receipt( + round_tripped, + entity="person", + family="fixture", + expected_model_targets=expected_targets, + boundary="missing regime fixture", + ) def test_late_executor_authority_binds_every_transfer_bank_identity( diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 9508e588b..1af1dd95f 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -224,11 +224,11 @@ POOL_MANIFEST_SCHEMA_VERSION = US_MULTISPINE_POOL_MANIFEST_SCHEMA_VERSION """Schema version for the companion pool build manifest.""" -# ``--legacy-two-spine`` is a byte-stable compatibility surface. Stacked -# publication and checkpoint-envelope versions may advance without rewriting -# the retiring pipeline's last supported envelope. +# ``--legacy-two-spine`` keeps its publication schema byte-compatible, while +# its checkpoint identity must still advance when a source operator shared by +# both paths changes semantics. _LEGACY_POOL_MANIFEST_SCHEMA_VERSION = 4 -_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 3 +_LEGACY_POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 4 POOL_H5_ARTIFACT_KIND = US_MULTISPINE_POOL_H5_ARTIFACT_KIND """Neutral H5 artifact kind; readiness is asserted only by the manifest.""" @@ -261,6 +261,10 @@ # 7: Stacked primary-PUF output universes are explicit. Earlier envelopes can # contain nulls outside the PUF clone for an output declared over the whole # pool and therefore cannot resume safely even when their bank is reusable. +# 8: The retirement-distribution training cap preserves target sign support, +# and ACS-transfer receipts persist exact fitted-pattern regimes. Earlier +# checkpoints may contain a support-deleting retirement sample or lack the +# evidence required to adjudicate a signed-tail fit. # # Bump this version whenever any producer above changes a stage output without # changing one of the explicit identity fields below. In particular, adding, @@ -275,7 +279,7 @@ # normalizes that logical view in memory. Moving between those encodings does # not change a producer's scalar output and therefore does not advance this # ledger; changing string values or the canonical logical dtype policy does. -POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 7 +POOL_STAGE_CHECKPOINT_MATERIALIZER_VERSION = 8 _PRIMARY_QRF_N_ESTIMATORS = 100 _ACS_TRANSFER_N_ESTIMATORS = 100 @@ -306,11 +310,13 @@ _STACKED_CHECKPOINT_IDENTITY_ARTIFACT_KIND = ( "populace_us_stacked_pool_checkpoint_identity" ) -# Version 11 additionally binds the primary-PUF whole-pool universe semantics. -# Earlier checkpoints must rebuild rather than resume with a nullable -# s_corp_income leaf. Version 10 bound the complete late-resource semantics and -# corrected outer order (the primary PUF callback is nested inside the DAG). -_STACKED_CHECKPOINT_MATERIALIZER_VERSION = 11 +# Version 12 binds the support-preserving retirement-distribution training cap +# and exact ACS fitted-pattern regime receipts. Earlier checkpoints may have +# lost rare target support or omit the regime evidence required for review. +# Version 11 bound the primary-PUF whole-pool universe semantics; version 10 +# bound the complete late-resource semantics and corrected outer order (the +# primary PUF callback is nested inside the DAG). +_STACKED_CHECKPOINT_MATERIALIZER_VERSION = 12 _STACKED_RELEASE_ID_PATTERN = re.compile( r"^populace-us-2024-stacked-f(?:001|004|010|025|100)-s[0-9]+-" r"asec[0-9]+-acs[0-9]+-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{8}$" diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index 9638f7ec0..21f8c523d 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "b63ffc9bf7cb2fb8ac17e314f64c7b661aba14512071c1a697e5f6ff9ebc74c5" + "85fb15ef978600029ba69850659fd190e2919bc8b9dcbbf527f2215eb0c79345" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1] From 0fa480287b005632d40b4b17c8667d3a0dd9a485 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 25 Aug 2026 15:39:07 +0200 Subject: [PATCH 3/3] Record tails 1% no-regression evidence Co-Authored-By: Claude Fable 5 --- .../battery_rare_signed_tails/verify_1pct.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 experiments/battery_rare_signed_tails/verify_1pct.md diff --git a/experiments/battery_rare_signed_tails/verify_1pct.md b/experiments/battery_rare_signed_tails/verify_1pct.md new file mode 100644 index 000000000..27f4f4717 --- /dev/null +++ b/experiments/battery_rare_signed_tails/verify_1pct.md @@ -0,0 +1,16 @@ +# tails 1% verify build: failure-line diff vs baseline + +Cold 1% build (sample seed 578) on this branch at the serial host queue, +2026-08-25: physical battery failure lines **127 = 127** vs baseline1pct — +zero greened, zero introduced. + +This is the expected 1% result, per the lane's own adjudication: at the 1% +rung the retirement training donor (~1,080 rows) sits below the 5,000-row +cap, so the carrier-preservation path is structurally inert at this scale; +the Keogh ACS side remains at zero carriers pre-transfer either way. The +fix's behavioral effect binds at the 25% rung (5 sparse-donor checks) and +via the held-out calibration evidence (40 shape checks). What this build +demonstrates is strictly **no regression** from: the carrier-union +preservation under unchanged cap and seed, and the realized-regime +persistence through transfer banks, receipts, H5 checkpoints, and resume +validation.