Add results.json schema validation and source tracking

.github/workflows/push.yaml

@@ -117,7 +117,13 @@ jobs:
         run: pip install yaml-changelog towncrier && make changelog
       - name: Preview changelog update
         run: ".github/get-changelog-diff.sh"
-      - name: Update changelog
+      - name: Install package for TRO regeneration
+        run: pip install -e .
+      - name: Regenerate bundled TRACE TROs
+        env:
+          HUGGING_FACE_TOKEN: ${{ secrets.HUGGING_FACE_TOKEN }}
+        run: python scripts/generate_trace_tros.py
+      - name: Update changelog and TROs
         uses: EndBug/add-and-commit@v9
         with:
           add: "."

changelog.d/trace-tro-hardening.added.md

@@ -0,0 +1,8 @@
+TRACE TRO hardening: bundle TROs now hash the country model wheel (read from
+`PackageVersion.sha256` when present, otherwise fetched from PyPI), use HTTPS
+artifact locations, carry structured `pe:*` certification fields and GitHub
+Actions attestation metadata, and are validated in CI against a shipped JSON
+Schema. Adds a `policyengine trace-tro` CLI, per-simulation TROs through
+`policyengine.results.build_results_trace_tro` / `write_results_with_trace_tro`,
+and restores the `TaxBenefitModelVersion.trace_tro` property and
+`policyengine.core` re-exports that were dropped in #276.

changelog.d/trace-tro-vocabulary-fix.changed.md

@@ -0,0 +1,19 @@
+TRACE TRO emission now conforms to the public TROv 2023/05 vocabulary:
+switched namespace to `https://w3id.org/trace/2023/05/trov#`, flattened
+`trov:hash` nodes to the native `trov:sha256` property, renamed
+`trov:path`→`trov:hasLocation` and the inverse pointer on ArtifactLocation
+to `trov:hasArtifact`, corrected `TrustedResearchSystem`→`TransparentResearchSystem`
+and `TrustedResearchPerformance`→`TransparentResearchPerformance`, and replaced
+the locally-invented `ArrangementBinding` chain with
+`trov:accessedArrangement` as used by the published trov-demos. Every TRO
+now carries `pe:emittedIn` (`"local"` or `"github-actions"`) so a verifier
+can distinguish a CI-emitted TRO from a laptop rebuild. Per-simulation TROs
+accept a `bundle_tro_url` that is recorded as `pe:bundleTroUrl`, letting a
+verifier independently fetch and re-hash the bundle to detect a forged
+reference. The composition fingerprint now joins hashes with `\n` to
+prevent hex-length concatenation collisions. Adds `policyengine
+trace-tro-validate` CLI, removes the broken `--offline` flag, wires
+`scripts/generate_trace_tros.py` into the `Versioning` CI job so bundled
+TROs ship with every release, inlines the real model wheel sha256 on
+`us.json`/`uk.json`, and cleans up the dead `DataReleaseArtifact.https_uri`
+/ `_data_release_manifest_url` helpers.

docs/release-bundles.md

@@ -195,19 +195,139 @@ TRACE sits on top of those manifests as a standards-based export layer.
 
 ### What gets exported
 
-Country `*-data` repos should emit a `trace.tro.jsonld` file for each published data
-release. That TRO should cover:
-
-- the release manifest itself
-- each published artifact hash listed in the release manifest
-- the build-time model provenance recorded in the release manifest
-
-`policyengine.py` should emit a separate certified-bundle TRO. That TRO should cover:
+`policyengine.py` emits a certified-bundle TRO for each supported country. The
+composition pins four artifacts by sha256:
 
 - the bundled country release manifest shipped in `policyengine.py`
 - the country data release manifest resolved for the certified data package version
-- the certified dataset artifact hash
-- the certification basis used to allow runtime reuse
+- the certified dataset artifact
+- the country model wheel published to PyPI (hash read from the bundled manifest
+  when present, otherwise fetched from the PyPI JSON API at emit time)
+
+TROs use the public TROv vocabulary at
+`https://w3id.org/trace/2023/05/trov#`. Every artifact location in the TRO
+is a dereferenceable HTTPS URI or a local path relative to the shipped
+wheel. Certification metadata is carried as structured `pe:*` fields on
+the `trov:TransparentResearchPerformance` node so downstream tooling can
+read `pe:certifiedForModelVersion`, `pe:compatibilityBasis`,
+`pe:builtWithModelVersion`, `pe:dataBuildFingerprint`, and `pe:dataBuildId`
+without parsing prose. Every TRO also carries `pe:emittedIn` set to
+`"github-actions"` or `"local"`; CI-emitted TROs additionally carry
+`pe:ciRunUrl` and `pe:ciGitSha`.
+
+Country `*-data` repos should also emit a matching `trace.tro.jsonld` per
+data release covering the release manifest and every staged artifact hash.
+That is a country-data concern and lives in those repos.
+
+#### Emitting a bundle TRO
+
+From Python:
+
+```python
+from policyengine.core.release_manifest import get_data_release_manifest, get_release_manifest
+from policyengine.core.trace_tro import build_trace_tro_from_release_bundle, serialize_trace_tro
+
+country = get_release_manifest("us")
+tro = build_trace_tro_from_release_bundle(country, get_data_release_manifest("us"))
+Path("us.trace.tro.jsonld").write_bytes(serialize_trace_tro(tro))
+```
+
+From the CLI:
+
+```
+policyengine trace-tro us --out us.trace.tro.jsonld
+```
+
+At release time, `scripts/generate_trace_tros.py` regenerates the bundled
+`data/release_manifests/{country}.trace.tro.jsonld` files, and the
+`Versioning` CI job commits them alongside the changelog so every published
+wheel ships with the matching TRO.
+
+#### Emitting a per-simulation TRO
+
+```python
+from policyengine.results import write_results_with_trace_tro
+
+write_results_with_trace_tro(
+    results,                                # ResultsJson instance
+    "results.json",                         # where to write results
+    bundle_tro=bundle_tro,                  # loaded from the shipped bundle
+    reform_payload={"salt_cap": 0},
+    bundle_tro_url=(
+        "https://raw.githubusercontent.com/PolicyEngine/policyengine.py/"
+        "v3.4.5/src/policyengine/data/release_manifests/us.trace.tro.jsonld"
+    ),
+)
+```
+
+The `bundle_tro_url` is recorded on the performance node as
+`pe:bundleTroUrl`. A verifier can fetch that URL, recompute its sha256,
+and confirm it matches the `bundle_tro` artifact hash in the simulation
+TRO's composition. Without this anchor, the bundle reference is only as
+trustworthy as whoever produced the JSON.
+
+#### Validating a received TRO
+
+Structural validation:
+
+```
+policyengine trace-tro-validate path/to/tro.jsonld
+```
+
+The shipped schema at `policyengine/data/schemas/trace_tro.schema.json`
+checks structural fields, canonical hex-encoded sha256s, the required
+`pe:emittedIn`, and that `trov:hasLocation` uses HTTPS (or the
+well-known local paths `results.json`, `reform.json`,
+`bundle.trace.tro.jsonld`). The same schema is exercised in the test
+suite against generated TROs.
+
+Content validation (the verifier workflow a replication reviewer
+should run):
+
+```python
+import hashlib, json, requests
+from policyengine.core.trace_tro import canonical_json_bytes
+
+sim_tro = json.load(open("results.trace.tro.jsonld"))
+perf = sim_tro["@graph"][0]["trov:hasPerformance"]
+
+# 1. Fetch the bundle TRO from its pinned URL and recompute its hash.
+bundle_bytes = requests.get(perf["pe:bundleTroUrl"]).content
+bundle_hash = hashlib.sha256(canonical_json_bytes(json.loads(bundle_bytes))).hexdigest()
+
+# 2. Compare against the hash recorded in the simulation TRO's composition.
+recorded = next(
+    a["trov:sha256"]
+    for a in sim_tro["@graph"][0]["trov:hasComposition"]["trov:hasArtifact"]
+    if a["@id"].endswith("bundle_tro")
+)
+assert bundle_hash == recorded, "bundle_tro_url content does not match sim TRO"
+
+# 3. Confirm the fingerprint recorded on the performance matches the
+#    fingerprint inside the fetched bundle.
+bundle = json.loads(bundle_bytes)
+bundle_fp = bundle["@graph"][0]["trov:hasComposition"]["trov:hasFingerprint"]["trov:sha256"]
+assert perf["pe:bundleFingerprint"] == bundle_fp
+```
+
+A sim TRO with a swapped `bundle_tro` dict but a truthful
+`pe:bundleTroUrl` will fail step 2; a sim TRO with both swapped will
+fail step 3.
+
+#### Known limitations
+
+- TROs are emitted unsigned. A signed attestation (sigstore or in-toto)
+  is a future addition that will bind TROs to a trusted-system key.
+- The bundle composition does not yet pin a transitive lockfile
+  (`uv.lock`/`poetry.lock`), a Python interpreter version, or an OS. AEA
+  reviewers may demand these; the schema is extensible.
+- The model wheel is hashed by PyPI's published sha256. If a wheel is
+  yanked and re-uploaded under the same version, the hash will change
+  and the TRO becomes invalid — which is the correct behaviour.
+- Country data packages whose data release manifest is private require
+  `HUGGING_FACE_TOKEN` at emit time. The regeneration script skips
+  countries whose data release manifest is unreachable so a partial run
+  does not block other countries.
 
 ### What TRACE does not replace
 

pyproject.toml

@@ -29,6 +29,9 @@ dependencies = [
     "psutil>=5.9.0",
 ]
 
+[project.scripts]
+policyengine = "policyengine.cli:main"
+
 [project.optional-dependencies]
 uk = [
     "policyengine_core>=3.25.0",
@@ -46,6 +49,7 @@ dev = [
     "yaml-changelog>=0.1.7",
     "itables",
     "build",
+    "jsonschema>=4.0.0",
     "pytest-asyncio>=0.26.0",
     "ruff>=0.9.0",
     "policyengine_core>=3.25.0",

scripts/generate_trace_tros.py

@@ -0,0 +1,82 @@
+"""Regenerate bundled TRACE TRO artifacts for every country release manifest.
+
+Writes ``data/release_manifests/{country}.trace.tro.jsonld`` for each
+country whose bundled manifest ships in the wheel. Run this before
+releasing a new ``policyengine.py`` version so the packaged TRO
+matches the pinned bundle. Requires HTTPS access to the data release
+manifest (and ``HUGGING_FACE_TOKEN`` for private country data).
+
+If a country previously had a TRO on disk and the new run cannot
+regenerate it (e.g. a missing secret or an unreachable HF endpoint),
+the script exits non-zero so the release workflow blocks rather than
+silently shipping a stale/missing TRO.
+"""
+
+from __future__ import annotations
+
+import sys
+from importlib.resources import files
+from pathlib import Path
+
+from policyengine.core.release_manifest import (
+    DataReleaseManifestUnavailableError,
+    get_data_release_manifest,
+    get_release_manifest,
+)
+from policyengine.core.trace_tro import (
+    build_trace_tro_from_release_bundle,
+    serialize_trace_tro,
+)
+
+
+def regenerate_all() -> tuple[list[Path], list[tuple[str, Path, str]]]:
+    manifest_root = Path(
+        str(files("policyengine").joinpath("data", "release_manifests"))
+    )
+    written: list[Path] = []
+    regressions: list[tuple[str, Path, str]] = []
+    for manifest_path in sorted(manifest_root.glob("*.json")):
+        country_id = manifest_path.stem
+        tro_path = manifest_path.with_suffix(".trace.tro.jsonld")
+        country_manifest = get_release_manifest(country_id)
+        try:
+            data_release_manifest = get_data_release_manifest(country_id)
+        except DataReleaseManifestUnavailableError as exc:
+            if tro_path.exists():
+                regressions.append((country_id, tro_path, str(exc)))
+            else:
+                print(
+                    f"skipped {country_id}: {exc}",
+                    file=sys.stderr,
+                )
+            continue
+        tro = build_trace_tro_from_release_bundle(
+            country_manifest,
+            data_release_manifest,
+            certification=country_manifest.certification,
+        )
+        tro_path.write_bytes(serialize_trace_tro(tro))
+        written.append(tro_path)
+    return written, regressions
+
+
+def main() -> int:
+    written, regressions = regenerate_all()
+    for path in written:
+        print(f"wrote {path}")
+    for country_id, tro_path, reason in regressions:
+        print(
+            f"error: {country_id} already has {tro_path.name} but regeneration "
+            f"failed: {reason}",
+            file=sys.stderr,
+        )
+    if regressions:
+        return 1
+    if not written:
+        print("no release manifests found", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())