Add release-bundle refresh helper + CLI wrapper#309
Merged
Conversation
New policyengine.provenance.bundle module exposing
refresh_release_bundle(country, *, model_version=None, data_version=None).
Given a country and optional new versions, it:
- Fetches fresh wheel metadata (url + sha256) from PyPI JSON API
- Streams the HF dataset file to compute its sha256
- Writes updated data/release_manifests/{country}.json in place
(model_package + data_package + certified_data_artifact +
certification fields, preserves unknown fields untouched)
- Bumps the pyproject.toml [project.optional-dependencies] pin for
the country extra
- Skips PyPI / HF fetches when the respective version hasn't
changed, so data-only and model-only refreshes each hit exactly
the one network endpoint they need
regenerate_trace_tro(country) companion runs the same code path
scripts/generate_trace_tros.py uses, so the refresh flow chains
cleanly to TRO regeneration.
scripts/refresh_release_bundle.py is a thin argparse wrapper:
python scripts/refresh_release_bundle.py \\
--country us --data-version 1.83.4
Tested offline via mocked urlopen. 6 new tests cover: model-only
bump, data-only bump, combined bump, update_pyproject=False,
PyPI "no matching wheel" error, malformed dataset URI error.
All 427 existing tests still pass.
Next step (not in this PR, requires a HF token with read scope for
policyengine/policyengine-us-data): run the actual 1.73.0 -> 1.83.4
bump and regenerate snapshots.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Packages the release-bundle bump process so it's one command instead of six manually-coordinated file edits.
New
policyengine.provenance.bundlemodule exposingrefresh_release_bundle(country, model_version=..., data_version=...). Given optional new versions, it:data/release_manifests/{country}.jsonin place (model_package + data_package + certified_data_artifact + certification fields; preserves unknown fields untouched)pyproject.tomlpin for the country extraCompanion
regenerate_trace_tro(country)chains the TRO-sidecar rebuild, wrapping the existingscripts/generate_trace_tros.pycode path.scripts/refresh_release_bundle.pyis a thin argparse wrapper:Tests
6 new tests in
tests/test_bundle_refresh.py, all offline via mockedurlopen:update_pyproject=Falseshort-circuit427/427 total tests pass (421 existing + 6 new).
Why module vs pure script
The refresh logic is reusable beyond the CLI (e.g., CI automation, per-reform certification in the TRACE TRO flow). Putting the core in
policyengine.provenance.bundlegives us:help(policyengine.provenance)policyengine release refreshonpe.clionce that CLI has a second userWhat this doesn't do
PE_UPDATE_SNAPSHOTS=1 pytest tests/test_household_calculator_snapshot.pystill needs to be a human-reviewed step — the numeric changes deserve attention, not an auto-accept.policyengine/policyengine-us-data, which my local token doesn't have. Follow-up for whoever has the right credentials.Test plan
pytest tests/test_bundle_refresh.pypasses offlinepytest tests/(427/427) passesruff check/ruff format --checkclean🤖 Generated with Claude Code