Launch v4: agent-first surface, shared microsim base, dict reforms, refreshed docs

[MaxGhenis]

Summary

Consolidated v4 launch PR. Collapses the rest of the v4 stack (#291, #292, #293, #294, #295, #297) into a single merge to main after #288 and #290 landed.

What v4 ships

1. Agent-first household calculator (replaces ceremony)

# Before:
from policyengine.tax_benefit_models.us import USHouseholdInput, calculate_household_impact
calculate_household_impact(USHouseholdInput(people=[...], tax_unit={...}, year=2024))

# After:
import policyengine as pe
pe.us.calculate_household(people=[...], tax_unit={...}, year=2026)

• pe.uk.calculate_household / pe.us.calculate_household on the top-level package
• Flat kwargs; HouseholdResult(dict) with __getattr__ dot-access
• Entity-aware input validation (misplaced-variable detection, close-match suggestions, **kwargs catcher for cross-country confusion)
• Reform dicts: {"param.path": value} or {"param.path": {date: value}} — compiled internally

2. Dict reforms on Simulation too (#297, net-new)

The same flat dict is now accepted by population microsim:

Simulation(
    dataset=dataset,
    tax_benefit_model_version=pe.us.model,
    policy={"gov.irs.credits.ctc.amount.base[0].amount": 3_000},
)

No more hand-built Parameter/ParameterValue objects. Path validation and effective-date defaulting use tax_benefit_model_version + dataset.year.

3. Provenance split out of core (+ lazy h5py)

• policyengine.provenance.{manifest,trace} replaces policyengine.core.release_manifest / policyengine.core.trace_tro
• h5py now imports lazily inside WeightReplacementStrategy.apply() — a bare import policyengine no longer pulls h5py

4. Shared MicrosimulationModelVersion base

~300 lines of duplicated __init__ / save / load / variable+parameter loading extracted from PolicyEngineUSLatest / PolicyEngineUKLatest to tax_benefit_models.common.model_version. Country subclasses declare class-level metadata (country_code, package_name, group_entities, entity_variables) and implement four thin hooks. run() stays per-country — subtle divergences in reform application and output post-processing make a shared skeleton actively misleading.

Guarded by byte-level snapshot tests (tests/test_base_extraction_snapshot.py) freezing household outputs for 4 US + 4 UK cases and model-surface aggregates. Zero drift post-refactor.

5. Unified ProgramStatistics

One class replaces split ProgramStatistics / ProgrammeStatistics.

6. Docs refreshed for the agent-first surface

README, core-concepts, economic-impact-analysis, country-models-{uk,us}, regions-and-scoping, dev, and examples all lead with pe.uk.* / pe.us.*. Removed "Legacy filter fields" doc section (dropped in #290). examples/household_impact_example.py rewritten against the v4 API and verified end-to-end.

Breaking changes

• policyengine.tax_benefit_models.{uk,us}.calculate_household_impact / USHouseholdInput / UKHouseholdInput removed (use pe.{uk,us}.calculate_household)
• Simulation.filter_field / Simulation.filter_value removed in [v4] Drop legacy filter_field/filter_value scoping fields #290 (use scoping_strategy=RowFilterStrategy(...))
• policyengine.core.release_manifest / policyengine.core.trace_tro moved to policyengine.provenance.{manifest,trace}
• plotly moved to [plotting] optional extra
• Reform dict is the only supported reform surface for calculate_household; reform JSON sidecars removed

Replacements for v3 stack

Closes / supersedes: #291, #292, #293, #294, #295, #297 (all CI-green; rebased into this single consolidated branch).

Test plan

• 397/397 tests pass locally (Python 3.12)
• Byte-level snapshot test guards household + model-surface outputs across US/UK
• End-to-end microsim with dict reform produces expected -$25.5B revenue impact for CTC-base $3,000 reform on enhanced_cps_2024
• examples/household_impact_example.py runs end-to-end (UK + US)

🤖 Generated with Claude Code

[MaxGhenis]

Review fixes pushed (commit 76ea9a0)

Three reviewer passes (practitioner, code-simplifier, end-to-end verification) ran on the consolidated v4 branch. All pre-launch blockers addressed:

API ergonomics

• Simulation class now has a full __doc__ showing the canonical dict-reform call shape — help(pe.Simulation) was returning Pydantic boilerplate, hiding the headline v4 feature
• RowFilterStrategy.variable_value is now Union[str, int, float] — numeric columns (state_fips, county_fips) were silently unscopable; the docs example for CA used state_code which doesn't exist on the shipped dataset
• pe.__all__ now exports outputs for tab-completion discoverability

Version / branding

• pyproject.toml: 3.6.0 → 4.0.0
• release_manifests/{us,uk}.json: bundle_id + policyengine_version bumped so bundle TRO URLs point at the right git tag

Docs

• README, core-concepts, economic-impact-analysis, country-models-{us,uk}, regions-and-scoping all updated to lead with dict reforms. Lost ~130 lines of deprecated Parameter/ParameterValue ceremony.
• state_code_str → state_code (consistent with us/household.py)
• reform.py docstring documents the [N].amount / [N].threshold indexed-parameter convention so agents don't hit the bracket-head trap

Code simplification (simplifier review)

• except (ValueError, Exception) → except Exception
• compile_reform_to_policy / compile_reform_to_dynamic now share a private helper (was 25-line copy-paste)
• Simulation._compile_dict_reforms loops over (field, compiler) pairs
• test_base_extraction_snapshot.py → test_household_calculator_snapshot.py (matches what it actually pins, not the refactor that motivated it)

Remaining as v4.1 follow-ups (not blocking):

• Lazy-h5py claim is inaccurate — bare import policyengine still pulls h5py via policyengine_core and policyengine_uk.data.dataset_schema. Internal lazy imports work; upstream transitives don't. Would need pe.us.__getattr__ / pe.uk.__getattr__ gating.
• Some tax_benefit_models.common.model_version.py docstrings are multi-paragraph where a line would do — cosmetic only

397 tests pass. ruff + ruff format clean.