fix(reward): derive what the receipt asserts instead of believing the bundle and the trainer

[abrichr]

An adversarial review found five ways to make the reward receipt assert
something the oracle read did not support. Each one is reproduced below with
its real output before and after. Every fix adds a refusal; none removes one.

The shape is the same in all five: the worker trusted a label or a number its
counterparty supplied. The bundle author chose the oracle tier by naming a
recipe kind. The trainer chose the graded subject after the rollout, and chose
the policy update that decides whether the certificate has expired. The
seeded contract asked about the store's current contents instead of about the
episode. The calibration corpus was a constant, so the bound did not depend on
the contract it was issued for.

1. The oracle tier was a string the bundle author wrote

json_file mapped to channel file (tier 2) and screen_dump to ocr (tier
0), and build_oracle handed both the same JsonDocumentOracle. One
document, two tiers, on the same bytes:

kind=screen_dump  sha256(file)=same  tier=0  certified=False  development_only=True   scalar=1.0
kind=json_file    sha256(file)=same  tier=2  certified=True   development_only=False  scalar=1.0

The channel is now a class attribute of the adapter, and no caller sets it.
Both kinds read through ocr at tier 0, and build_oracle refuses to return
an adapter whose channel differs from the recipe table's, so the table in
models.py and the adapters in oracles.py cannot drift apart quietly.

kind=screen_dump  declares ocr   -> loads, tier 0
kind=json_file    declares file  -> REFUSED: oracle recipe channel ocr does not match the contract channel file
kind=json_file    declares ocr   -> loads, tier 0
sqlite recipe on a JSON file -> REFUSED: ... is not a SQLite database file, so this recipe cannot read through the db channel

Tier 0 is the floor a local JSON document can defend. Whoever writes the file
writes the answer, and nothing in the bytes tells a system-of-record dump from
a screen scrape.

sqlite keeps tier 2 because the worker opens a real database read-only and
runs one SELECT through the engine, and it now checks the file header, so a
screen dump renamed store.db is refused before the worker starts. The seeded
tier-2 bundle moves from mockmed/records.json to a real SQLite store at
mockmed/records.db.

Narrowed rather than enforced: rest and fhir still reach tier 2 by
pointing at any HTTP server that answers with JSON. The worker can verify that
it made a network call. It cannot verify that the endpoint is the customer's
system of record rather than a server the trainer stood up, and a self-signed
synthetic-scope certificate does not attest to it. docs/REWARD_WORKER.md
now says so under "What the tier rests on" and names whoever admits the bundle
as the owner of that check. Adding a hostname or scheme rule would have looked
like enforcement without being any.

2. The graded subject was chosen after the rollout

worker.py:331 read identity = declared_identity or registered_identity, so
the descriptor won over the registration and nothing compared them:

registered by begin_episode: patient-lie-0002
declared by descriptor:      patient-honest-0001
outcome=verified certified=True scalar=1.0
receipt stamps identity: {'patient_id': 'patient-honest-0001'}

The registration is made by the environment before the rollout, when nobody
knows how the episode will end. The descriptor arrives after, when the trainer
does. So the registration decides, and a descriptor that names a different
subject is refused rather than silently overridden, because a trainer that
believes it is grading one subject while the worker grades another has a bug
worth stopping for.

registered patient-lie-0002, descriptor names patient-honest-0001
-> 422 identity_conflict: the environment registered this episode for {patient_id=patient-lie-0002} before the rollout ran, and the descriptor names ...
re-register under another subject -> 409 identity_conflict

Re-registering the same episode under the same subject is still allowed and
only re-reads the baseline. Re-registering an already scored episode is
refused.

3. A rollout that did nothing could earn reward 1

The seeded required effect was a plain record_written, a statement about the
store's current contents. No episode had to run:

required effects: [('record_written', count_new_only=False, requires_baseline=False)]
no episode ran -> outcome=verified scalar=1.0 certified=True

RewardBundle.load now refuses a contract whose required effects contain no
claim about change. count_new_only and exact_new_set are the two kinds the
judge settles against the pre-episode baseline; everything else describes the
store as it stands. The required effects are judged as a conjunction, so one
change claim is enough for verified to mean the episode added a record, and a
field_equals read-back can still ride alongside it.

no write at all           -> wrong_effect scalar=0.0
the row was already there -> wrong_effect scalar=0.0
state-only required effect -> REFUSED: required_effects assert only what the store holds now, so they are satisfied by a record that was already there ...

Narrowed rather than enforced: the task allowed an escape hatch for a
contract that says explicitly it is state-only. I did not add one. The bundle
is written by the same party the rule constrains, and an opt-out that party
writes is defect 1 in a different costume. The claim is narrower instead: this
worker serves contracts whose required effects assert a change, because
verified is a statement about an episode and a state-only contract cannot
make one.

This is also why POST /v1/episodes is new. A baseline is now needed for a
scored episode, begin_episode was in process only, and without a route the
HTTP path could never produce one.

4. Certificate expiry counted a number the counterparty reported

episode.policy_update went from the wire straight to certificate_state
with no comparison to anything seen before:

policy_update=0            certificate_state=current   certified=True  scalar=1.0
policy_update=999          certificate_state=current   certified=True  scalar=1.0
policy_update=1000000000   certificate_state=expired   certified=False scalar=1.0
policy_update=0            certificate_state=current   certified=True  scalar=1.0

The worker now keeps its own high-water mark under
<data-dir>/policy_updates/, beside the episode index, and refuses anything
below it:

policy_update=0            -> current   certified=True
policy_update=999          -> current   certified=True
policy_update=1000000000   -> expired   certified=False
policy_update=0            -> 422 policy_update_regressed: this contract has already scored policy update 1000000000 ...

Stronger than asked, and here is the sixth defect. The task said to key the
mark per contract and per policy checkpoint. Per checkpoint alone, a trainer
resets the counter by calling its checkpoint something else: register a fresh
policy_checkpoint_id, send policy_update: 0, and an expired certificate
reads current again. The mark is therefore per contract, and
test_renaming_the_checkpoint_does_not_reset_expiry drives exactly that. The
ledger still records which checkpoint set the mark, for the error message.

5. The calibration corpus was hard-coded

calibration.py:132 faulted_store always emitted
{"id", "patient_id", "type": "Triage", "status": "saved"}, whatever the
contract asked for:

contract asserts type=Radiology; corpus emits type=Triage only
trials=300 false_accepts=0 epsilon=0.009936  <- the best bound the method can give
FAULT_CLASSES = ('extra_record', 'duplicate_record', 'missing_record', 'wrong_type', 'forbidden_present')
wrong_subject in FAULT_CLASSES: False

The contract matched nothing, so every trial refuted for a reason the planted
fault did not cause, the false-accept count was zero because the checker
rejected everything, and the certificate got the best epsilon 300 trials can
produce. The same 0.009936 a correct contract gets.

corpus_from_effects now reads the records off the contract's own required
and forbidden effects. A field_equals read-back on the same selector merges
into the record a record_written effect already describes, so a paired
contract plants one row rather than two. Each trial gets a real pre-state, so
count_new_only has a baseline to work against.

contract asserts type=Radiology; corpus plants [{'id': 1, 'patient_id': 'p1', 'type': 'Radiology'}]
trials=300 false_accepts=0 epsilon=0.009936
fault classes sampled: ('duplicate_record', 'extra_record', 'forbidden_present', 'missing_record', 'wrong_field', 'wrong_subject')
contradictory required effects -> REFUSED: a clean store built from this contract's own required effects judges wrong_effect, not verified, so the corpus does not exercise the contract ...

Three refusals now stand behind that number. extradup_trials runs a control
trial first and refuses to report a bound unless a clean store built from the
contract's own effects earns VERIFIED, so a zero count that only means "the
checker rejects everything" cannot become a certificate. The corpus digest is
derived from the contract, and RewardBundle.load refuses a certificate or a
certificate policy that names a different one. And a contract no fault class
applies to cannot be calibrated at all.

calibration.json beside the certificate now records the corpus digest and
which fault classes were sampled, so a reader can see what the bound covers
rather than assume.

FAULT_CLASSES: I added the class rather than correcting the docstring.
wrong_subject is now sampled, planting the required record correct in every
field under another subject's identity. RewardOutcomeV1.WRONG_EFFECT calls
itself "a terminal effect that differs from the required one", the judge
already catches that mode, and a bound that never planted it was quiet about
the failure this reward exists to price. The docstring is in openadapt-types
and I did not touch it; it is now accurate as written.

wrong_type is renamed wrong_field, because on a derived corpus it spoils
whichever declared literal the contract has rather than a hardcoded type
column.

What the seeded MockMed run looks like now

The demo got sharper rather than weaker. Because a required effect asserts a
change, an episode has to write something, and the two channels can be pointed
at the same episode:

honest (tier2)             outcome=verified     scalar=1.0  tier=2 certified=True  dev_only=False
banner lie (tier2)         outcome=wrong_effect scalar=0.0  tier=2 certified=True  dev_only=False
banner lie (tier0)         outcome=verified     scalar=1.0  tier=0 certified=False dev_only=True
duplicate (tier2)          outcome=wrong_effect scalar=0.0  tier=2 certified=True  dev_only=False
nothing ran (tier2)        outcome=wrong_effect scalar=0.0  tier=2 certified=True  dev_only=False

The OCR dump agrees with the banner while the database holds nothing. That is
the whole argument for the tier ladder, and the fixture now shows it on one
episode instead of two.

Two published sentences corrected

README.md advertised pip install 'openadapt-flow[reward]' and
openadapt-flow serve-reward --seed-mockmed on the GitHub landing page.
Published 1.34.0 declares no reward extra, ships no openadapt_flow/reward/,
and registers no serve-reward entry point, so a reader got a failed install
and then an unknown command. Both commands stay as what will work when the
release lands, with no version and no date, and a repository-head install is
added for today. docs/REWARD_WORKER.md gets the same treatment, matching the
shape openadapt-ops#212 used for the docs page.

docs/REWARD_WORKER.md line 29 said the control service "issues and revokes
reward certificates". Nothing implements revocation. The sentence now says
expiry is the only thing that ends a certificate.

Tests

tests/test_reward_trust_boundary.py is new: 21 tests, each driving a
reproduction above. tests/test_reward_worker.py is updated so its outcome
tests register an episode and write what the episode would have written,
rather than scoring a store that was already correct. 53 tests pass; ruff,
ruff format and mypy are clean on openadapt_flow.

Opened by an agent session, not the founder.

🤖 Generated with Claude Code