From b72eaa17a5cf6508f4fa540e6fa157133f7e9b9d Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Sat, 22 Aug 2026 12:32:11 -0700 Subject: [PATCH] fix(calibration): a hash check identifies a state, not a trajectory #22 split contract milestones into earnable achievements and unearnable replay-identity hashes. The premise was that a hash pinned from the reference can only be satisfied by reproducing that trajectory. Measurement refutes it. On ALE Breakout both pinned hashes fire after 32 inputs. Over the vocabulary NOOP/FIRE/RIGHT/LEFT that prefix admits 96 single-input substitutions, and 40 of them still reproduce both hashes, at 16 of the 32 turns. All 16 applied at once reproduce them too, so at least 382,205,952 distinct 32-input logs satisfy the pins. FIRE while the ball is in flight is a state no-op, and two logs that differ only there reach a bit-identical emulator state. A hash therefore identifies a STATE, not a trajectory. Any policy that stands in that state earns the milestone, without ever seeing the reference. The real axis is legibility: a reader can see what `score >= 4` demands and cannot see what a hash demands. `checkLegibility` and `contractLegibility` replace `checkRole` and `contractEarnability`, with the same derivation from the check kind and the same walk over `requires`. `MilestoneScore` is `{ verified, total }`, because every milestone is a point: Breakout is 6, not 4. `Attestation.earned`, `EpisodeRecord.earned`, and `earnedMilestones` are removed, since a run earns what it verifies. The second fix in #22 was real and independent, and it stands in corrected form. An opaque milestone no baseline reached still landed in `separating` and carried the separation claim. `separating` now holds legible milestones only and `separates` compares legible counts, because "the reference reached it and no baseline did" states nothing a reader can check. `probeOpaqueCollisions` is the prober #22 lacked. It replaces one input of the reference at a time over the prefix that ends where an opaque check first passes, counts the perturbed logs that still satisfy it, and reports the free turns and a lower bound on the family. It runs inside `calibrateContract` for every opaque milestone. The gate refuses an opaque check the sweep reproduced unless the author accepts the measured weakness by id in `weakChecks`. The eleven packaged contracts are re-reported over the milestone count. The partition by check kind is unchanged; the denominator #22 published was wrong. Replay attestation's input-log hash chain is untouched. --- CHANGELOG.md | 70 +++++--- README.md | 87 +++++---- ale.test.mts | 124 +++++++++---- attestation.ts | 14 +- calibration.test.mts | 310 ++++++++++++++++++++++---------- calibration.ts | 406 +++++++++++++++++++++++++++++++++--------- campaign.test.mts | 7 +- campaign.ts | 5 +- docs/adapters.md | 29 +-- episode-loop.ts | 1 - episode.test.mts | 23 ++- episode.ts | 10 +- pyboy-libbet.test.mts | 51 +++--- schema.ts | 149 +++++++--------- 14 files changed, 855 insertions(+), 431 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4da5232..887307a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,38 +4,66 @@ All notable changes to Playproof are documented here. ## 0.6.0 -### Earnable milestones are separated from replay-identity checks +### A hash check identifies a state, not a trajectory -- **The defect.** `deriveContract` samples frame hashes and save hashes from the reference trajectory and emits them as milestones. On ALE Breakout the derived contract has six milestones, and two of them — `frame-at-first-score` and `save-at-first-score` — pin the exact bytes the reference produced. No independent policy earns them. An authored policy that verified `score-opened`, `score-tier-2`, and `life-lost` read as 3 of 6; a hand-written ball tracker reached the same three. A third of that contract's points were reachable only by a replay of the reference, and every score quoted from it carried that denominator. -- A milestone's role is now derived from its check kind. `save-hash` and `frame-hash` are replay-identity; `state-path`, `save-path`, `frame-path`, and `log-contains` are achievements. Nothing is stored on the milestone, so every existing contract keeps its bytes and its hash, and no author can forget to set it. -- `contractEarnability(contract)` follows `requires` as well. An achievement gated behind a hash is unreachable too, because `MilestoneTracker` admits a milestone only after every prerequisite has passed. The packaged `save-levels` contract is the case in the repo: its `log-contains` milestone is semantic and still unearnable, through its save-hash prerequisite. -- `Attestation` and `EpisodeRecord` keep `verified` unchanged and gain `earned` and `score`. `MilestoneScore` is `{ verified, earned, earnable, total }`, and `formatMilestoneScore` writes it as `3 of 4 earnable (3 of 6 verified, 2 replay-identity)`. A campaign segment report gains `scoreSoFar`, so an analyst reads progress against the earnable denominator mid-run. +- **The correction.** An earlier draft of this release split contract milestones into "earnable" achievements and "replay-identity" hashes, on the premise that a hash pinned from the reference trajectory can only be satisfied by reproducing that trajectory. That premise is false, and it has been measured false. The measurement is below. An earlier draft of this entry published earnable denominators for eleven packaged contracts — Breakout 4 of 6, Libbet 4 of 6, PyBoy Tetris 3 of 5, Gymnasium FrozenLake 2 of 3 — and every one of those denominators was wrong. The denominator is the milestone count. +- **What is true.** A `frame-hash` or `save-hash` check names one exact game state. Many trajectories reach one state, so an independent policy earns a hash milestone by playing, without ever seeing the reference. It is a legitimate achievement check whose requirement happens to be written where nobody can read it. +- The axis is legibility, not earnability. `checkLegibility` and `contractLegibility(contract)` split a contract into **legible** milestones (`state-path`, `save-path`, `frame-path`, `log-contains` — a reader sees what they demand) and **opaque** ones (`save-hash`, `frame-hash`). It follows `requires`, because a legible check gated behind a hash still demands something a reader cannot see. Legibility is derived from the check kind, so every existing contract keeps its bytes and its hash. +- `MilestoneScore` is `{ verified, total }`, and `formatMilestoneScore` writes it as `3 of 6`. Every milestone is a point, hashes included. `Attestation` and `EpisodeRecord` carry `verified` and `score`; the `earned` field and `earnedMilestones` are gone, because a run earns everything it verifies. A campaign segment report carries `scoreSoFar`. - The campaign ledger is unchanged. Its `verified` list and the contract it pins by hash reproduce the score through `scoreMilestones`, so a ledger written by 0.5.0 still loads. +- Replay attestation is untouched. The input-log hash chain is the mechanism that proves a replay reproduced a recorded run, and it is correct. A milestone hash never proved that. -### Calibration refuses a contract with points no policy can score +### Calibration refuses a contract whose points nobody can read -- **A second defect, in the gate itself.** A replay-identity milestone is never earned by a trivial baseline, so it landed in `separating` and the separation test read it as the contract's strongest evidence. Measured on 0.5.0: a contract whose two milestones are hashes over the whole input chain reports `separates: true` with every baseline earning nothing. Unreachable read as hard. -- `CalibrationReport.separating` now holds earnable milestones only, and `separates` compares earnable counts through the new `bestBaselineEarnedCount`. A baseline that beats the reference on real progress can no longer be outvoted by hashes the reference reproduces by construction. -- The report gains `earnable`, `unearnable`, `unearnableReasons`, `unearnableReproduced`, and `referenceScore`. -- `assertContractSeparates(report, { identityChecks })` takes an exact declaration of the identity checks the author accepts. An undeclared one, a stale id, and an identity hash that a trivial baseline reproduced all fail the gate, so a hash milestone added by a later derivation cannot enter a published contract unnoticed. `assertMilestonesEarnable` runs the same check alone, for a target that is not meant to separate. -- Identity checks are not removed and not discouraged. Replay attestation is exactly the claim that one run reproduced another's bytes. The fix is to stop counting them as achievements. +- **A real defect, independent of the model above, and it stands.** An opaque milestone is never earned by a trivial baseline, so it landed in `separating` and the separation test read it as the contract's strongest evidence. Measured on 0.5.0: a contract whose two milestones are hashes over the whole input chain reports `separates: true` with every baseline earning nothing. +- `CalibrationReport.separating` now holds legible milestones only, and `separates` compares legible counts through `bestBaselineLegibleCount`. "The reference reached it and no baseline did" is not evidence when no reader can tell what it was reached for. +- The report gains `legible`, `opaque`, `opacityReasons`, `opaqueReproduced`, `collisions`, and `referenceScore`. +- `assertContractSeparates(report, { opaqueChecks, weakChecks })` takes an exact declaration. An undeclared opaque check, a stale id, and a hash that a trivial baseline reproduced all fail the gate. `assertOpaqueChecksDeclared` runs the same check alone, for a target that is not meant to separate. -### Measured +### The collision sweep, which the earlier draft could not do + +- The earlier draft tested an opaque check against trivial baselines only, found none of them reproduced a hash, and concluded the hash identified one run. That is why the false premise survived: the baseline suite cannot serve a ball, let alone score. +- `probeOpaqueCollisions` is the prober that can. It replaces one input of the reference at a time, over the prefix that ends where the check first passes, and counts the perturbed logs that still satisfy it. It is deterministic, needs no independent policy, and runs inside `calibrateContract` for every opaque milestone. +- Each row carries `firesAfter`, `probedTurns`, `substitutions`, `collisions`, `freeTurns`, `jointCollision`, and `family` — a lower bound on the number of distinct logs that satisfy the check, as the product over probed turns of (1 + surviving alternatives). `collisionTurns` caps the probe for a check that fires very late. +- The gate refuses an opaque check the sweep reproduced, unless the author accepts the measured weakness by id in `weakChecks`. A stale `weakChecks` id fails too. -| Contract | Milestones | Earnable | Reference | Best trivial baseline | -|---|---|---|---|---| -| ALE Breakout, 210 turns, seed 0 | 6 | 4 | 4 of 4 earnable (6 of 6 verified) | 0 earnable | -| Libbet through `pyboy-generic`, 70 turns, seed 0 | 6 | 4 | 3 of 4 earnable (3 of 6 verified) | 3 earnable | -| `save-levels` toy | 2 | 0 | 0 of 0 earnable (2 of 2 verified) | 0 earnable | -| `screen-puzzle` toy | 2 | 0 | 0 of 0 earnable (2 of 2 verified) | 2 verified | +### Measured -- Breakout separates on its earnable milestones and its scores were quoted out of the wrong denominator. Libbet still does not separate, and its four earnable milestones are exactly the ones a constant `a` press already earns. -- `screen-puzzle` renders from one coordinate, so `constant:r` walks to the same square and reproduces both pinned frames. A hash another trajectory reproduces identifies no run, and the gate now says so. +ALE Breakout, ale-py 0.12.1, seed 0, both hashes firing after 32 inputs over `NOOP/FIRE/RIGHT/LEFT`: + +| Measurement | Value | +|---|---| +| single-input substitutions of the 32-turn prefix | 96 | +| substitutions that still reproduce both hashes | 40, at 16 of the 32 turns | +| all 16 free turns substituted at once | still reproduces both | +| distinct 32-input logs that satisfy both hashes | at least 382,205,952 | +| trivial baselines that reproduce either hash | 0 of 8 | + +`FIRE` while the ball is already in flight is a state no-op, so a large family of logs reaches a bit-identical emulator state. An independently written ball tracker reproduced both hashes without seeing the reference; its log diverges at turn 17, where it plays `FIRE` and the reference plays `NOOP`. + +Corrected per-contract figures. The **Milestones** column is the denominator of a score: + +| Contract | Milestones | Legible | Opaque | Reference | Best trivial baseline | +|---|---|---|---|---|---| +| ALE Breakout, 210 turns, seed 0 | 6 | 4 | 2 | 6 of 6 | 0 legible | +| Libbet through `pyboy-generic`, 70 turns, seed 0 | 6 | 4 | 2 | 3 of 6 | 3 legible | +| PyBoy Tetris | 5 | 3 | 2 | — | — | +| stable-retro Airstriker | 5 | 4 | 1 | — | — | +| Gymnasium CartPole | 5 | 4 | 1 | — | — | +| Gymnasium FrozenLake | 3 | 2 | 1 | — | — | +| RetroArch, `n` channels with screen milestones | n + 2 | n + 1 | 1 | — | — | +| `native-2048` | 7 | 7 | 0 | — | — | +| `engine-crawler` toy | 4 | 4 | 0 | — | — | +| `save-levels` toy | 2 | 0 | 2 | 2 of 2 | 2 | +| `screen-puzzle` toy | 2 | 0 | 2 | 2 of 2 | 2 | + +- Breakout separates on its legible milestones. Libbet still does not separate, and its four legible milestones are exactly the ones a constant `a` press already earns. +- Both toys are reproduced in full by a trivial baseline. On `save-levels` the round-robin cycle over `clear`/`grind` replays the reference exactly, and on `screen-puzzle` `constant:r` walks to the same square. A hash a trivial baseline satisfies demands nothing, and the gate says so. Neither toy's substitution sweep finds a collision, because every substitution of those short references breaks the check — the two probers are complementary, and neither replaces the other. ### Replay attestation is unaffected - `verified` keeps its meaning and its contents. The three packaged toy contract hashes are byte-identical across the change, and `calibration.test.mts` pins them, together with the serialized milestone key set, so a contract that gains a field fails the build. -- `ale.test.mts` and `pyboy-libbet.test.mts` verify the same milestone ids on the same runs as before, and now also report the split. +- `ale.test.mts` and `pyboy-libbet.test.mts` verify the same milestone ids on the same runs as before, and now also report the split and the sweep. ## 0.5.0 diff --git a/README.md b/README.md index 6b7f563..aff5312 100644 --- a/README.md +++ b/README.md @@ -270,32 +270,36 @@ Use semantic checks such as `score >= 10` for progression. Exact hashes identify Dependencies between milestones form a declared partial order. A later achievement cannot verify before its prerequisites, even when its raw condition already holds. -### Earnable milestones and replay-identity checks +### Legible checks and opaque ones -A contract holds two kinds of statement, and one score cannot carry both. +A hash check identifies a **state**, not a trajectory. +Many trajectories reach one state, so a hash is an achievement like any other and an independent policy earns it by playing. +What a hash cannot do is tell a reader what it demands. -- An **achievement** is a threshold, a normalized field, or an event. Two different valid trajectories both satisfy it, so an independent policy earns it by playing. -- A **replay-identity check** is a hash over the exact bytes one recorded run produced. It proves that a replay reproduced that run, and no independent policy earns it. +- A **legible** check is a threshold, a normalized field, or an event. A reader sees what the milestone asks for and can judge whether reaching it is progress. +- An **opaque** check is a hash. It names one exact state without saying which, so nobody reading the contract can weigh the point. -The role is derived from the check kind, so no contract changes and no author has to remember to set it. -`save-hash` and `frame-hash` are identity; `state-path`, `save-path`, `frame-path`, and `log-contains` are achievements. -`requires` is followed: an achievement gated behind a hash is unreachable too, because a milestone verifies only after every prerequisite has. +Legibility is derived from the check kind, so no contract changes and no author has to remember to set it. +`save-hash` and `frame-hash` are opaque; `state-path`, `save-path`, `frame-path`, and `log-contains` are legible. +`requires` is followed: a legible check gated behind a hash still demands something a reader cannot see. ```ts -import { contractEarnability, formatMilestoneScore } from '@tangle-network/playproof' +import { contractLegibility, formatMilestoneScore } from '@tangle-network/playproof' -contractEarnability(contract) -// { earnable: ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost'], -// unearnable: ['frame-at-first-score', 'save-at-first-score'], -// reasons: { 'frame-at-first-score': "its frame-hash check pins the reference run's exact bytes", … } } +contractLegibility(contract) +// { legible: ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost'], +// opaque: ['frame-at-first-score', 'save-at-first-score'], +// reasons: { 'frame-at-first-score': 'its frame-hash check states its requirement as a hash, …' } } -formatMilestoneScore(record.score) // '3 of 4 earnable (3 of 6 verified, 2 replay-identity)' +formatMilestoneScore(record.score) // '3 of 6' ``` -`Attestation` and `EpisodeRecord` carry `verified` unchanged, plus `earned` and `score`. -A campaign segment report carries `scoreSoFar`. -Report a run as earned over earnable. -Identity checks stay in the contract and stay in `verified`; they are what replay attestation proves. +Every milestone is a point, hashes included, so the denominator of a score is the contract's milestone count. +`Attestation` and `EpisodeRecord` carry `verified` and `score`; a campaign segment report carries `scoreSoFar`. + +Replay attestation is a different mechanism and is untouched. +The input-log hash chain is what proves a replay reproduced a recorded run. +A milestone hash never proved that, because many logs reach the same state. ## Calibration: does the contract separate? @@ -319,37 +323,50 @@ assertContractSeparates(report) `calibrateContract` replays the reference and a suite of trivial policies through the same attestation path: one constant policy per input word, a word the game cannot interpret, a round-robin cycle over the vocabulary, and a seeded pseudo-random walk over it. Every policy is deterministic in the seed, so a report reproduces from one number. -The report names `separating` (earnable milestones no baseline earned), `trivial` (milestones at least one baseline earned), `earnable` and `unearnable`, and both baseline counts. -`separates` is true only when an **earnable** milestone is out of reach of every baseline **and** the reference earns strictly more earnable milestones than the strongest baseline. +The report names `separating` (legible milestones no baseline earned), `trivial` (milestones at least one baseline earned), `legible` and `opaque`, `collisions`, and both baseline counts. +`separates` is true only when a **legible** milestone is out of reach of every baseline **and** the reference verifies strictly more legible milestones than the strongest baseline. +An opaque milestone cannot carry the claim: "the reference reached it and no baseline did" states nothing a reader can check. `assertContractSeparates` throws otherwise, and the message names every trivial milestone with the baseline that earned it. -### The gate also refuses points no policy can score - -A replay-identity check is never earned by a baseline, so the separation test alone reads it as the contract's strongest evidence. -It is the opposite: a milestone out of reach of every policy measures nothing, and it inflates the denominator of every score quoted from the contract. +### The gate also refuses points nobody can read -Measured on ALE Breakout: the derived contract has six milestones and two of them are hashes of the screen and the save state at the first point. -An authored policy that verified `score-opened`, `score-tier-2`, and `life-lost` read as 3 of 6. -It could never have reached 6. -The honest number is 3 of 4. +Declare every opaque check by id, or state the progression with a legible check: ```ts assertContractSeparates(report, { - identityChecks: ['frame-at-first-score', 'save-at-first-score'], + opaqueChecks: ['frame-at-first-score', 'save-at-first-score'], + weakChecks: ['frame-at-first-score', 'save-at-first-score'], }) ``` -The declaration is an exact set, not a switch: an undeclared hash milestone, a stale id, and an identity hash that a trivial baseline reproduced all fail the gate. +The declaration is an exact set, not a switch: an undeclared hash milestone, a stale id, and a hash that a trivial baseline reproduced all fail the gate. A hash milestone that a later derivation adds therefore cannot enter a published contract unnoticed. -`assertMilestonesEarnable` runs the same check alone, for a demonstration target that is not meant to separate. +`assertOpaqueChecksDeclared` runs the same check alone, for a demonstration target that is not meant to separate. + +### How many logs satisfy a hash? Measure it + +`calibrateContract` also runs `probeOpaqueCollisions`: it replaces one input of the reference at a time, over the prefix that ends where an opaque check first passes, and counts the perturbed logs that still satisfy it. +A hash a large family of logs satisfies is a weak check, and `weakChecks` is where an author accepts the measured number by id. + +Measured on ALE Breakout, whose two hashes fire after 32 inputs over `NOOP/FIRE/RIGHT/LEFT`: + +| Measurement | Value | +|---|---| +| single-input substitutions of the 32-turn prefix | 96 | +| substitutions that still reproduce both hashes | 40, at 16 of the 32 turns | +| all 16 applied at once | still reproduces both | +| distinct 32-input logs that satisfy the hashes | at least 3.82 × 10⁸ | + +`FIRE` while the ball is already in flight is a state no-op, so those logs reach a bit-identical emulator state. +No trivial baseline reproduces either hash, which is exactly why the baseline suite is not enough to judge one. -| Contract | Milestones | Earnable | Reference score | Best trivial baseline | +| Contract | Milestones | Legible | Reference score | Best trivial baseline | |---|---|---|---|---| -| ALE Breakout | 6 | 4 | 4 of 4 earnable | 0 earnable | -| Libbet through `pyboy-generic` | 6 | 4 | 3 of 4 earnable | 3 earnable | +| ALE Breakout | 6 | 4 | 6 of 6 | 0 legible | +| Libbet through `pyboy-generic` | 6 | 4 | 3 of 6 | 3 legible | -Breakout separates and its score was quoted out of the wrong denominator. -Libbet does not separate, and its earnable milestones are exactly the ones a constant button press already earns. +Breakout separates on its legible milestones. +Libbet does not separate, and its legible milestones are exactly the ones a constant button press already earns. ### The measurement that made this exist diff --git a/ale.test.mts b/ale.test.mts index dc167fe..61c8147 100644 --- a/ale.test.mts +++ b/ale.test.mts @@ -8,7 +8,8 @@ * * Battery: contract derivation across three evidence tiers, known-good * attestation, garbage rejection, graded partial credit, calibration with the - * earnable split, cross-process determinism including the save-state hash, + * legible/opaque split and the opaque-collision sweep, cross-process + * determinism including the save-state hash, * checkpoint round-trip, unknown-input no-op, the observation image channel, * and worker teardown. Zero model spend. */ @@ -16,14 +17,40 @@ import { strict as assert } from 'node:assert' import { spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' import { attestRun } from './attestation' -import { assertContractSeparates, assertMilestonesEarnable, calibrateContract } from './calibration' +import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract } from './calibration' import { logFrom, observationOf } from './runtime' import { decodePng, unscale } from './test-png.mts' -import { contractEarnability, formatMilestoneScore, scoreMilestones, validateContract } from './schema' +import { contractLegibility, formatMilestoneScore, scoreMilestones, validateContract } from './schema' import { AleRpc } from './adapters/ale-rpc' import { bundledReference, makeAle, type Ale, type AleState } from './adapters/ale' const GAME = 'breakout' + +/** + * The opaque-collision sweep on the packaged Breakout contract, pinned. + * + * Both hashes fire after 32 inputs. Over the vocabulary NOOP/FIRE/RIGHT/LEFT + * that prefix admits 96 single-input substitutions, and 40 of them still + * reproduce both hashes, at 16 of the 32 turns. Applying one alternative at + * every free turn at once reproduces them too, which is why the family bound + * multiplies out instead of adding. + * + * These numbers are the correction to #22, which concluded from a clean + * trivial-baseline result that a hash could only be reached by replaying the + * reference. A regression here means ale-py, the ROM, or the reference moved. + */ +const ALE_BREAKOUT_COLLISIONS = [ + { + milestone: 'frame-at-first-score', + firesAfter: 32, probedTurns: 32, substitutions: 96, + collisions: 40, freeTurns: 16, jointCollision: true, family: 382205952, + }, + { + milestone: 'save-at-first-score', + firesAfter: 32, probedTurns: 32, substitutions: 96, + collisions: 40, freeTurns: 16, jointCollision: true, family: 382205952, + }, +] const python = process.env.PLAYPROOF_PYTHON ?? 'python3' /** The bundled ROM must be present, not just the package. */ @@ -110,31 +137,32 @@ if (!pythonHasAle()) { assert.deepEqual(partial.verified, ['score-opened', 'frame-at-first-score', 'save-at-first-score', 'score-tier-2']) assert.deepEqual(partial.reasons, ['claimed-not-reproduced:score-tier-4,life-lost']) - // The reference verified all six milestones, and only four of them are - // progress. `frame-at-first-score` and `save-at-first-score` hash the exact - // screen and save state this trajectory produced, so reaching them means - // reproducing this run rather than playing Breakout. - assert.deepEqual(contractEarnability(adapter.contract), { - earnable: ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost'], - unearnable: ['frame-at-first-score', 'save-at-first-score'], + // A hash check identifies a STATE, not a trajectory. Both hashes here name + // the exact screen and save state the reference stood in when it first + // scored, and any policy that stands in that state earns them. They are + // points, so the honest denominator is six. + // + // What they are not is readable: `frame-at-first-score` says nothing a + // contract reader can weigh, while `score-tier-4` says score >= 4. + assert.deepEqual(contractLegibility(adapter.contract), { + legible: ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost'], + opaque: ['frame-at-first-score', 'save-at-first-score'], reasons: { - 'frame-at-first-score': "its frame-hash check pins the reference run's exact bytes", - 'save-at-first-score': "its save-hash check pins the reference run's exact bytes", + 'frame-at-first-score': + 'its frame-hash check states its requirement as a hash, so a reader cannot see what it demands', + 'save-at-first-score': + 'its save-hash check states its requirement as a hash, so a reader cannot see what it demands', }, }) - assert.deepEqual(good.earned, ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost']) - assert.deepEqual(good.score, { verified: 6, earned: 4, earnable: 4, total: 6 }) - assert.equal(formatMilestoneScore(partial.score), '2 of 4 earnable (4 of 6 verified, 2 replay-identity)') + assert.deepEqual(good.score, { verified: 6, total: 6 }) + assert.equal(formatMilestoneScore(partial.score), '4 of 6') - // The measured defect this split exists for. An authored policy and a - // hand-written ball tracker each verified score-opened, score-tier-2 and - // life-lost on this contract, and neither reproduced either hash. Both - // read as three of six. Three of four is the honest statement, and it is - // the whole of what an independent policy can score. + // A run that played Breakout and did not land on the pinned state scores + // three of six. Three of four was the wrong statement: it removed two + // points from the denominator that a policy can, and does, reach. const played = ['score-opened', 'score-tier-2', 'life-lost'] - assert.deepEqual(scoreMilestones(adapter.contract, played), { verified: 3, earned: 3, earnable: 4, total: 6 }) - assert.equal(formatMilestoneScore(scoreMilestones(adapter.contract, played)), - '3 of 4 earnable (3 of 6 verified, 2 replay-identity)') + assert.deepEqual(scoreMilestones(adapter.contract, played), { verified: 3, total: 6 }) + assert.equal(formatMilestoneScore(scoreMilestones(adapter.contract, played)), '3 of 6') // Calibration on the real emulator. Every baseline plays the reference's // 210 turns, so the comparison is length-matched. @@ -146,27 +174,53 @@ if (!pythonHasAle()) { for (const outcome of [calibration.reference, ...calibration.baselines]) { console.log(` ${outcome.id.padEnd(32)} ${String(outcome.verified.length).padStart(2)} ${outcome.verdict} ${outcome.verified.join(',') || '-'}`) } - assert.deepEqual(calibration.earnable, ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost']) - assert.deepEqual(calibration.unearnable, ['frame-at-first-score', 'save-at-first-score']) - assert.deepEqual(calibration.referenceScore, { verified: 6, earned: 4, earnable: 4, total: 6 }) - // No trivial policy reproduced either hash, so both really are identity - // checks on this substrate and not low-entropy channels. - assert.deepEqual(calibration.unearnableReproduced, []) + assert.deepEqual(calibration.legible, ['score-opened', 'score-tier-2', 'score-tier-4', 'life-lost']) + assert.deepEqual(calibration.opaque, ['frame-at-first-score', 'save-at-first-score']) + assert.deepEqual(calibration.referenceScore, { verified: 6, total: 6 }) + // No trivial policy reproduced either hash. That was the whole of the + // evidence #22 had for calling them unearnable, and it proves nothing: the + // baseline suite cannot serve a ball, let alone score. + assert.deepEqual(calibration.opaqueReproduced, []) + + // The substitution sweep is the prober that can. Both hashes fire after 32 + // inputs; the sweep replaces one input of that prefix at a time over + // NOOP/FIRE/RIGHT/LEFT and counts the logs that still satisfy the check. + // FIRE while the ball is already in flight is a state no-op, so a large + // family of logs reaches a bit-identical emulator state. + for (const row of calibration.collisions) { + console.log( + ` ${row.milestone.padEnd(24)} fires after ${row.firesAfter}; ` + + `${row.collisions}/${row.substitutions} substitutions collide at ${row.freeTurns}/${row.probedTurns} turns; ` + + `joint=${row.jointCollision}; family >= ${row.family.toExponential(2)}`, + ) + } + assert.deepEqual(calibration.collisions, ALE_BREAKOUT_COLLISIONS) + // Undeclared, the contract cannot ship, whatever the separation verdict. - assert.throws(() => assertMilestonesEarnable(calibration), /2 of 6 milestone\(s\) no policy can earn/u) - const declared = { identityChecks: ['frame-at-first-score', 'save-at-first-score'] } - assertMilestonesEarnable(calibration, declared) + assert.throws(() => assertOpaqueChecksDeclared(calibration), /states 2 of 6 milestone\(s\) as a hash/u) + // Declaring them opaque is still not enough, because the sweep measured + // the collisions. The author must accept the weakness by id. + assert.throws( + () => assertOpaqueChecksDeclared(calibration, { opaqueChecks: ['frame-at-first-score', 'save-at-first-score'] }), + /2 opaque milestone\(s\) are satisfied by input logs other than the reference/u, + ) + const declared = { + opaqueChecks: ['frame-at-first-score', 'save-at-first-score'], + weakChecks: ['frame-at-first-score', 'save-at-first-score'], + } + assertOpaqueChecksDeclared(calibration, declared) // Whether Breakout separates is a fact about the ROM, not about this - // change: assert only that the earnable comparison is the one being made. + // change: assert only that the legible comparison is the one being made. + const referenceLegible = calibration.reference.verified.filter((id) => calibration.legible.includes(id)).length assert.equal( calibration.separates, - calibration.separating.length > 0 && calibration.referenceScore.earned > calibration.bestBaselineEarnedCount, + calibration.separating.length > 0 && referenceLegible > calibration.bestBaselineLegibleCount, ) if (calibration.separates) assertContractSeparates(calibration, declared) else assert.throws(() => assertContractSeparates(calibration, declared), /does not separate/u) console.log( `ale: calibration — reference ${formatMilestoneScore(calibration.referenceScore)}, ` + - `best trivial baseline ${calibration.bestBaselineEarnedCount} earnable over ${calibration.turns} turns, ` + + `best trivial baseline ${calibration.bestBaselineLegibleCount} legible over ${calibration.turns} turns, ` + `separating=${calibration.separating.join(',') || 'nothing'}, separates=${calibration.separates}`, ) diff --git a/attestation.ts b/attestation.ts index a40e86a..94088cb 100644 --- a/attestation.ts +++ b/attestation.ts @@ -8,23 +8,16 @@ */ import { logFrom } from './runtime' import type { Evidence, Game, Input, InputLog } from './runtime' -import { contractHash, earnedMilestones, scoreMilestones } from './schema' +import { contractHash, scoreMilestones } from './schema' import type { Milestone, MilestoneContract, MilestoneScore, NumericOperator } from './schema' export interface Attestation { gameId: string verdict: 'clean' | 'rejected' reasons: string[] - /** Every milestone the replay reproduced, replay-identity checks included. */ + /** Every milestone the replay reproduced. */ verified: string[] - /** - * The subset of `verified` an independent policy can earn by playing. - * - * A hash check proves a replay reproduced the recorded run. Counting it as - * progress inflates the denominator of every reported score, so the two sets - * are reported apart. `verified` stays the attestation statement. - */ - earned: string[] + /** `verified` over the contract's milestone count. */ score: MilestoneScore checks: { name: string; passed: boolean }[] } @@ -119,7 +112,6 @@ export function attestRun( verdict: reasons.length === 0 ? 'clean' : 'rejected', reasons, verified, - earned: earnedMilestones(contract, verified), score: scoreMilestones(contract, verified), checks: [ { name: 'input-log-chain', passed: chainOk }, diff --git a/calibration.test.mts b/calibration.test.mts index 43f05c8..86eb28c 100644 --- a/calibration.test.mts +++ b/calibration.test.mts @@ -9,26 +9,26 @@ * out-of-process game. * * The second half covers the other way a contract fails to measure: a milestone - * NO policy can earn, because a hash check pins the reference run's exact - * bytes. The trace game makes that case unambiguous, because its hash covers - * the whole input sequence. + * stated as a hash, which a reader cannot judge. The trace game hashes the + * whole input sequence, so exactly one trajectory satisfies it. The ledge walk + * hashes a state many trajectories reach, and only the substitution sweep + * finds that. */ import { strict as assert } from 'node:assert' import { createHash } from 'node:crypto' import { deriveContract } from './authoring' import { assertContractSeparates, - assertMilestonesEarnable, + assertOpaqueChecksDeclared, calibrateContract, trivialBaselines, UNKNOWN_BASELINE_WORD, } from './calibration' -import type { Game } from './runtime' +import { hashString, type Game } from './runtime' import { canonicalContractJson, - contractEarnability, contractHash, - earnedMilestones, + contractLegibility, formatMilestoneScore, scoreMilestones, } from './schema' @@ -286,34 +286,41 @@ try { adapter.dispose() } -// --- earnable milestones vs replay-identity checks --------------------------- +// --- legible checks vs opaque ones ------------------------------------------- +// +// A hash check identifies a STATE, not a trajectory. It is earnable: any policy +// that stands in that state satisfies it, whether or not it ever saw the +// reference. What a hash cannot do is tell a reader what it demands, and that +// is the axis these cases pin. -// (f) an all-achievement contract is unchanged: every milestone is earnable, -// nothing is declared, and the gate passes exactly as it did before. +// (f) an all-legible contract: nothing to declare, and the gate passes exactly +// as it did before opacity existed. { const report = calibrateContract(comboLock, lockContract, { reference: LOCK_REFERENCE, vocabulary: LOCK_VOCABULARY, }) - assert.deepEqual(report.earnable, ['moved', 'lock-opened']) - assert.deepEqual(report.unearnable, []) - assert.deepEqual(report.unearnableReasons, {}) - assert.deepEqual(report.unearnableReproduced, []) - assert.deepEqual(report.referenceScore, { verified: 2, earned: 2, earnable: 2, total: 2 }) - assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2 earnable') - assert.equal(report.bestBaselineEarnedCount, 1) + assert.deepEqual(report.legible, ['moved', 'lock-opened']) + assert.deepEqual(report.opaque, []) + assert.deepEqual(report.opacityReasons, {}) + assert.deepEqual(report.opaqueReproduced, []) + assert.deepEqual(report.collisions, []) + assert.deepEqual(report.referenceScore, { verified: 2, total: 2 }) + assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2') + assert.equal(report.bestBaselineLegibleCount, 1) assert.equal(report.separates, true) assertContractSeparates(report) - assertMilestonesEarnable(report) + assertOpaqueChecksDeclared(report) } -// (g) a contract mixing both kinds reports the right earnable count, and the -// identity check must be declared before the contract ships. +// (g) a contract mixing both kinds. `frame-at-open` hashes the whole input +// chain, so exactly one trajectory reaches the state it names — the sweep +// proves that, rather than assuming it from the check kind. // -// `frame-at-open` is a hash over the whole input chain, so only a replay of the -// reference reproduces it. It is a correct attestation check and it stays in -// the contract; it is not a third point an agent can score. -const lockIdentityContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ +// The hash is still a point an independent policy can score, so it stays in +// the denominator: a run that opens the lock without landing on the pinned +// frame scores 2 of 3, not 2 of 2. +const lockOpaqueContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ { id: 'moved', tier: 'engine-state', @@ -340,20 +347,26 @@ const lockIdentityContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ ]) { - const report = calibrateContract(comboLock, lockIdentityContract, { + const report = calibrateContract(comboLock, lockOpaqueContract, { reference: LOCK_REFERENCE, vocabulary: LOCK_VOCABULARY, }) assert.deepEqual(report.reference.verified, ['moved', 'lock-opened', 'frame-at-open']) - assert.deepEqual(report.earnable, ['moved', 'lock-opened']) - assert.deepEqual(report.unearnable, ['frame-at-open']) - assert.match(report.unearnableReasons['frame-at-open'] ?? '', /frame-hash check pins the reference run's exact bytes/u) - assert.deepEqual(report.unearnableReproduced, []) - // The reference scored three of three, and only two of them were earnable. - assert.deepEqual(report.referenceScore, { verified: 3, earned: 2, earnable: 2, total: 3 }) - assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2 earnable (3 of 3 verified, 1 replay-identity)') - // The identity check is out of reach of every baseline, and that is not - // separation: `separating` names the earnable milestone only. + assert.deepEqual(report.legible, ['moved', 'lock-opened']) + assert.deepEqual(report.opaque, ['frame-at-open']) + assert.match(report.opacityReasons['frame-at-open'] ?? '', /frame-hash check states its requirement as a hash/u) + assert.deepEqual(report.opaqueReproduced, []) + + // The corrected arithmetic. Three milestones, three points, whoever scores + // them. A run that reached the first two scores 2 of 3. + assert.deepEqual(report.referenceScore, { verified: 3, total: 3 }) + assert.equal(formatMilestoneScore(report.referenceScore), '3 of 3') + assert.deepEqual(scoreMilestones(lockOpaqueContract, ['moved', 'lock-opened']), { verified: 2, total: 3 }) + assert.equal(formatMilestoneScore(scoreMilestones(lockOpaqueContract, ['moved', 'lock-opened'])), '2 of 3') + + // The preserved half of the separation fix: an opaque milestone no baseline + // reached is not evidence that the contract separates, because no reader can + // see what it asked for. `separating` names the legible milestone only. assert.deepEqual(report.separating, ['lock-opened']) assert.equal(report.separates, true) @@ -363,40 +376,55 @@ const lockIdentityContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ () => assertContractSeparates(report), (error: unknown) => { const message = (error as Error).message - assert.match(message, /1 of 3 milestone\(s\) no policy can earn/u) - assert.match(message, /2 earnable, 33% unearnable/u) + assert.match(message, /states 1 of 3 milestone\(s\) as a hash/u) + assert.match(message, /2 legible, 33% opaque/u) assert.match(message, /frame-at-open/u) - assert.match(message, /identityChecks: \['frame-at-open'\]/u) + assert.match(message, /opaqueChecks: \['frame-at-open'\]/u) assert.doesNotMatch(message, /does not separate/u) return true }, ) - assert.throws(() => assertMilestonesEarnable(report), /no policy can earn/u) + assert.throws(() => assertOpaqueChecksDeclared(report), /as a hash/u) - // Declared, the same contract passes both gates. - assertContractSeparates(report, { identityChecks: ['frame-at-open'] }) - assertMilestonesEarnable(report, { identityChecks: ['frame-at-open'] }) + // Declared, the same contract passes both gates. Nothing needs declaring as + // weak, because the sweep found no other log that satisfies the hash. + assertContractSeparates(report, { opaqueChecks: ['frame-at-open'] }) + assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-at-open'] }) // A declaration is an exact set. A missing id and a stale id both fail, so a // hash milestone added by a later derivation cannot hide behind it. - assert.throws(() => assertContractSeparates(report, { identityChecks: [] }), /1 undeclared milestone\(s\)/u) + assert.throws(() => assertContractSeparates(report, { opaqueChecks: [] }), /1 undeclared opaque milestone\(s\)/u) assert.throws( - () => assertContractSeparates(report, { identityChecks: ['frame-at-open', 'lock-opened'] }), - /the contract does not pin: lock-opened/u, + () => assertContractSeparates(report, { opaqueChecks: ['frame-at-open', 'lock-opened'] }), + /the contract does not state as a hash: lock-opened/u, ) - // Attestation reports the same split for one run. - const score = scoreMilestones(lockIdentityContract, report.reference.verified) - assert.deepEqual(score, report.referenceScore) - assert.deepEqual(earnedMilestones(lockIdentityContract, report.reference.verified), ['moved', 'lock-opened']) + // A hash over the whole input chain really is reached one way: 40 perturbed + // logs, none of them satisfying it. + assert.deepEqual(report.collisions, [{ + milestone: 'frame-at-open', + firesAfter: 8, + probedTurns: 8, + substitutions: 40, + collisions: 0, + freeTurns: 0, + jointCollision: false, + family: 1, + }]) + // Declaring a check weak when the sweep found no collision is stale too. + assert.throws( + () => assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-at-open'], weakChecks: ['frame-at-open'] }), + /weakChecks names 1 milestone\(s\) the sweep found no colliding log for/u, + ) + + // Attestation reports the same score for one run. + assert.deepEqual(scoreMilestones(lockOpaqueContract, report.reference.verified), report.referenceScore) } -// (h) an all-identity contract reports 0 earnable, and calibration refuses it. -// -// This is the regression. Every baseline earns nothing, so before earnability -// existed both milestones landed in `separating` and the report called the -// contract a benchmark — one on which no policy but a replay of the reference -// can ever score a point. +// (h) an all-opaque contract. This is the regression #22 found and the reason +// its fix survives: every baseline earns nothing, so before the split both +// milestones landed in `separating` and the report called the contract a +// benchmark. Nothing a reader can check separates it from a coin flip. const TRACE_VOCABULARY = ['a', 'b', 'c'] const TRACE_REFERENCE = ['a', 'b', 'b', 'c', 'a'] @@ -441,44 +469,54 @@ const traceContract = deriveContract(traceGame, 0, [...TRACE_REFERENCE], [ vocabulary: TRACE_VOCABULARY, }) assert.deepEqual(report.reference.verified, ['frame-at-three', 'save-at-five']) - assert.deepEqual(report.earnable, []) - assert.deepEqual(report.unearnable, ['frame-at-three', 'save-at-five']) - assert.deepEqual(report.referenceScore, { verified: 2, earned: 0, earnable: 0, total: 2 }) - assert.equal(formatMilestoneScore(report.referenceScore), '0 of 0 earnable (2 of 2 verified, 2 replay-identity)') - - // No baseline earned anything at all, and the contract still does not - // separate. That combination is the whole point: unreachable is not hard. + assert.deepEqual(report.legible, []) + assert.deepEqual(report.opaque, ['frame-at-three', 'save-at-five']) + // Both points are real points; the reference scored both, and so would any + // policy that reached the two states. The denominator is not the problem. + assert.deepEqual(report.referenceScore, { verified: 2, total: 2 }) + assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2') + + // No baseline earned anything, and the contract still does not separate. + // That combination is the whole point: a claim no reader can check is not + // evidence, however few policies satisfy it. assert.deepEqual(report.trivial, []) assert.ok(report.baselines.every((b) => b.verified.length === 0)) assert.deepEqual(report.separating, []) - assert.equal(report.bestBaselineEarnedCount, 0) + assert.equal(report.bestBaselineLegibleCount, 0) assert.equal(report.separates, false) assert.throws( () => assertContractSeparates(report), (error: unknown) => { const message = (error as Error).message - assert.match(message, /2 of 2 milestone\(s\) no policy can earn/u) - assert.match(message, /0 earnable, 100% unearnable/u) - assert.match(message, /the reference itself scored 0 of 0 earnable \(2 of 2 verified, 2 replay-identity\)/u) + assert.match(message, /states 2 of 2 milestone\(s\) as a hash/u) + assert.match(message, /0 legible, 100% opaque/u) + assert.match(message, /on legible milestones alone: reference 0, best baseline 0, of 0 legible/u) assert.match(message, /does not separate/u) return true }, ) // Declaring both is honest about the checks and still not a benchmark. assert.throws( - () => assertContractSeparates(report, { identityChecks: ['frame-at-three', 'save-at-five'] }), + () => assertContractSeparates(report, { opaqueChecks: ['frame-at-three', 'save-at-five'] }), (error: unknown) => { const message = (error as Error).message assert.match(message, /does not separate/u) - assert.doesNotMatch(message, /no policy can earn/u) + assert.doesNotMatch(message, /as a hash/u) return true }, ) - assertMilestonesEarnable(report, { identityChecks: ['frame-at-three', 'save-at-five'] }) - - // A milestone gated behind an identity check is unearnable too, whatever its - // own check kind: the tracker admits it only after its prerequisite passed. - const gated = contractEarnability({ + assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-at-three', 'save-at-five'] }) + + // A hash over the whole input chain admits one trajectory, measured. + assert.deepEqual(report.collisions.map((row) => [row.milestone, row.substitutions, row.collisions, row.family]), [ + ['frame-at-three', 6, 0, 1], + ['save-at-five', 10, 0, 1], + ]) + + // Opacity propagates through `requires`: the tracker admits a milestone only + // after its prerequisites passed, so a legible check gated behind a hash + // still demands something a reader cannot see. + const gated = contractLegibility({ ...traceContract, milestones: [ ...traceContract.milestones, @@ -491,50 +529,142 @@ const traceContract = deriveContract(traceGame, 0, [...TRACE_REFERENCE], [ }, ], }) - assert.deepEqual(gated.earnable, []) - assert.equal(gated.reasons['steps-after-save'], 'requires save-at-five, which no independent policy can earn') + assert.deepEqual(gated.legible, []) + assert.equal(gated.reasons['steps-after-save'], 'requires save-at-five, whose requirement is opaque') } // (i) the same split on the two packaged toy adapters, measured rather than -// assumed. Both are demonstration targets for the evidence tiers, and both -// turn out to carry no earnable milestone at all. +// assumed. Both are demonstration targets for the evidence tiers, and neither +// states a single milestone a reader can check. { - const levels = contractEarnability(saveLevelsContract()) - assert.deepEqual(levels.earnable, []) - assert.deepEqual(levels.unearnable, ['level-2-saved', 'level-2-logged']) - // The log-event check is semantic; it is unearnable only through its + const levels = contractLegibility(saveLevelsContract()) + assert.deepEqual(levels.legible, []) + assert.deepEqual(levels.opaque, ['level-2-saved', 'level-2-logged']) + // The log-event check reads in the open; it is opaque only through its // prerequisite, which is the dependency rule doing real work. - assert.equal(levels.reasons['level-2-logged'], 'requires level-2-saved, which no independent policy can earn') + assert.equal(levels.reasons['level-2-logged'], 'requires level-2-saved, whose requirement is opaque') const report = calibrateContract(saveLevels, saveLevelsContract(), { reference: SAVE_LEVELS_REFERENCE, vocabulary: ['clear', 'grind'], }) assert.equal(report.separates, false) - assert.deepEqual(report.referenceScore, { verified: 2, earned: 0, earnable: 0, total: 2 }) + assert.deepEqual(report.referenceScore, { verified: 2, total: 2 }) // screen-puzzle renders from one coordinate, so `constant:r` walks to the - // same square and reproduces both pinned frames. A hash another trajectory - // reproduces identifies no run, and the gate says so even when the author + // same square and reproduces both pinned frames. A hash one constant button + // press satisfies demands nothing, and the gate says so even when the author // declares it. + // + // The two probers are complementary, and this is the case that proves it: + // the substitution sweep finds nothing here, because every substitution + // shortens the walk, while a baseline that walks further reaches both. const puzzle = calibrateContract(screenPuzzle, screenPuzzleContract(), { reference: SCREEN_PUZZLE_REFERENCE, vocabulary: ['l', 'r'], }) - assert.deepEqual(puzzle.earnable, []) - assert.deepEqual(puzzle.unearnableReproduced, ['midway-frame', 'east-gate-frame']) + assert.deepEqual(puzzle.legible, []) + assert.deepEqual(puzzle.opaqueReproduced, ['midway-frame', 'east-gate-frame']) + assert.deepEqual(puzzle.collisions.map((row) => row.collisions), [0, 0]) assert.throws( - () => assertMilestonesEarnable(puzzle, { identityChecks: ['midway-frame', 'east-gate-frame'] }), + () => assertOpaqueChecksDeclared(puzzle, { opaqueChecks: ['midway-frame', 'east-gate-frame'] }), (error: unknown) => { const message = (error as Error).message - assert.match(message, /2 replay-identity milestone\(s\) were reproduced by a trivial baseline/u) + assert.match(message, /2 opaque milestone\(s\) were reproduced by a trivial baseline/u) assert.match(message, /midway-frame — reproduced by constant:r/u) return true }, ) } +// --- the collision sweep, and what only it can find -------------------------- +// +// #22 tested an opaque check against trivial baselines alone, and concluded +// from a clean result that the check identified one run. It does not follow. +// The ledge walk is the Breakout shape in miniature: pressing into a wall is a +// state no-op, so two logs that differ at those turns reach the same state and +// the same hash. No trivial baseline finds that; a one-input substitution of +// the reference finds it immediately. + +const LEDGE_VOCABULARY = ['l', 'r'] +const LEDGE_REFERENCE = ['l', 'l', 'r', 'r', 'r', 'r', 'r', 'l', 'l'] +const LEDGE_END = 5 + +interface LedgeState { + x: number + lit: boolean + steps: number +} + +const ledgeWalk: Game = { + id: 'ledge-walk', + init: () => ({ x: 0, lit: false, steps: 0 }), + step: (s, input) => { + // Walking into either wall is a no-op, which is the whole mechanism. + const x = input === 'r' ? Math.min(s.x + 1, LEDGE_END) : input === 'l' ? Math.max(s.x - 1, 0) : s.x + return { x, lit: s.lit || x === LEDGE_END, steps: s.steps + 1 } + }, + frame: (s) => `x ${s.x}${s.lit ? ' · torch lit' : ''}`, + evidence: (s) => ({ + engineState: { x: s.x, lit: s.lit ? 1 : 0, steps: s.steps }, + frameHash: hashString(`ledge:${s.x}:${s.lit ? 1 : 0}`), + }), +} + +const ledgeContract = deriveContract(ledgeWalk, 0, [...LEDGE_REFERENCE], [ + { + afterInputs: 9, + id: 'frame-back-from-the-ledge', + tier: 'screen-frame', + glitchClass: 'legal', + sample: (e) => ({ kind: 'frame-hash', hash: e.frameHash ?? '' }), + }, +]) + +{ + const report = calibrateContract(ledgeWalk, ledgeContract, { + reference: LEDGE_REFERENCE, + vocabulary: LEDGE_VOCABULARY, + }) + assert.deepEqual(report.opaque, ['frame-back-from-the-ledge']) + // The old prober says the hash is clean: no constant, no cycle and no walk + // over this vocabulary lands on the pinned frame. + assert.deepEqual(report.opaqueReproduced, []) + + // The new one says otherwise, with the number. The two turns that press + // into the left wall at x = 0 are free: the state is the same either way. + assert.deepEqual(report.collisions, [{ + milestone: 'frame-back-from-the-ledge', + firesAfter: 9, + probedTurns: 9, + substitutions: 9, + collisions: 2, + freeTurns: 2, + jointCollision: true, + family: 4, + }]) + + // Declaring the check opaque is no longer enough: the gate refuses until the + // author has read the measured weakness and accepted it by id. + assert.throws( + () => assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-back-from-the-ledge'] }), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /1 opaque milestone\(s\) are satisfied by input logs other than the reference/u) + assert.match(message, /2 of 9 single-input substitutions/u) + assert.match(message, /at 2 of 9 probed turn\(s\) before it fires \(after 9 input\(s\)\)/u) + assert.match(message, /also satisfies it, so at least 4 distinct 9-input log\(s\) satisfy it/u) + assert.match(message, /weakChecks: \['frame-back-from-the-ledge'\]/u) + return true + }, + ) + assertOpaqueChecksDeclared(report, { + opaqueChecks: ['frame-back-from-the-ledge'], + weakChecks: ['frame-back-from-the-ledge'], + }) +} + // (j) the split is derived, so every existing contract keeps its bytes. These -// hashes were recorded before earnability existed; a contract whose hash moves +// hashes were recorded before legibility existed; a contract whose hash moves // invalidates every artifact that pinned it. { assert.equal(contractHash(engineCrawlerContract()), '29a9ff9f3bb296a898493589d6bcdb539b5be15f7f53931de7f3725916274426') @@ -549,4 +679,4 @@ const traceContract = deriveContract(traceGame, 0, [...TRACE_REFERENCE], [ } } -console.log('playproof calibration: separating and non-separating contracts, earnable/identity split, policy determinism, edge cases OK') +console.log('playproof calibration: separating and non-separating contracts, legible/opaque split, opaque-collision sweep, policy determinism, edge cases OK') diff --git a/calibration.ts b/calibration.ts index d0693c8..c804f99 100644 --- a/calibration.ts +++ b/calibration.ts @@ -12,22 +12,33 @@ * comparison. `assertContractSeparates` is the fail-closed gate a target author * calls before publishing a contract. * - * A trivial baseline is not the only way a contract fails to measure. A - * milestone can also be out of reach of EVERY policy, because a hash check - * pins the reference run's exact bytes. Such a milestone is never earned by a - * baseline, so the separation test alone reads it as the contract's strongest - * evidence, when in fact nothing but a replay of the reference reproduces it. - * The report therefore splits the contract into earnable and unearnable - * milestones, and the gate refuses a contract whose unearnable milestones the - * author has not declared. + * A trivial baseline is not the only way a contract fails to measure. A hash + * check names one exact game state without saying which state, so a reader + * cannot judge what it demands. Every milestone is earnable — a hash + * identifies a STATE, not a trajectory, and many trajectories reach one state + * — but an opaque milestone cannot carry a separation claim, because "the + * reference reached it and no baseline did" states nothing a reader can check. + * The report therefore splits the contract into legible and opaque milestones, + * and the gate refuses a contract whose opaque checks the author has not + * declared. + * + * The second measurement is how many input logs satisfy an opaque check. + * `probeOpaqueCollisions` perturbs the reference one input at a time over the + * prefix that ends where the check first passes, and counts how many + * perturbed logs still satisfy it. A hash a large family of logs satisfies is + * a weak check, and the gate reports the measured number. + * + * None of this touches replay attestation. The input-log hash chain is the + * mechanism that proves a replay reproduced a recorded run, and it is + * unaffected: a milestone hash was never that proof. * * Everything here is pure and synchronous, and depends only on the runtime, * schema, and attestation planes. It imports no adapter and no model provider. */ -import { attestRun } from './attestation' +import { attestRun, MilestoneTracker } from './attestation' import { logFrom } from './runtime' import type { Game } from './runtime' -import { contractEarnability, formatMilestoneScore, scoreMilestones } from './schema' +import { contractLegibility, formatMilestoneScore, scoreMilestones } from './schema' import type { MilestoneContract, MilestoneScore } from './schema' /** @@ -105,6 +116,166 @@ export interface BaselineOutcome { verdict: 'clean' | 'rejected' } +/** + * How many distinct input logs satisfy one opaque check, measured by + * substitution rather than assumed. + * + * A hash pins a state. The question a contract author needs answered is how + * many ways there are into that state: one, and the check is a real pin; many, + * and the check is a weak restatement of a progression a path check could have + * named in the open. + */ +export interface OpaqueCollision { + /** The opaque milestone this row measures. */ + milestone: string + /** Reference inputs consumed before the check first passed; -1 if it never did. */ + firesAfter: number + /** Turns of that prefix the sweep probed. */ + probedTurns: number + /** Single-input substitutions of the reference the sweep replayed. */ + substitutions: number + /** Substitutions whose log still satisfied the check. */ + collisions: number + /** Probed turns where at least one substitution still satisfied it. */ + freeTurns: number + /** Whether one surviving substitution at EVERY free turn at once still satisfies it. */ + jointCollision: boolean + /** + * A lower bound on the family of `firesAfter`-input logs that satisfy the + * check: the product over probed turns of (1 + surviving alternatives). + * + * It is a bound and not a count, because the sweep changes one input at a + * time. `jointCollision` is the evidence that the free turns combine. + */ + family: number +} + +/** Default cap on probed turns, so a check that fires late stays cheap to measure. */ +export const DEFAULT_COLLISION_TURNS = 32 + +/** Turn indices to probe: every turn of the prefix, thinned evenly to `cap`. */ +function probePoints(prefixLength: number, cap: number): number[] { + if (prefixLength <= cap) return Array.from({ length: prefixLength }, (_unused, i) => i) + return Array.from({ length: cap }, (_unused, i) => Math.floor((i * prefixLength) / cap)) +} + +/** Reference inputs consumed before each milestone first passed, keyed by id. */ +function firstPassTurn( + game: Game, + contract: MilestoneContract, + seed: number, + reference: readonly string[], +): Map { + const tracker = new MilestoneTracker(contract) + const at = new Map() + let state = game.init(seed) + for (const id of tracker.consider(game.evidence(state))) at.set(id, 0) + for (let i = 0; i < reference.length; i++) { + state = game.step(state, reference[i]!) + for (const id of tracker.consider(game.evidence(state))) at.set(id, i + 1) + } + return at +} + +/** + * Measure how many perturbations of the reference still satisfy each opaque + * check of a contract. + * + * The sweep is deterministic and needs no independent policy: it replays the + * reference with one input replaced, over the prefix that ends where the check + * first passed. Milestones that fire at the same turn share one sweep. + * + * Measured on ALE Breakout, whose two hashes fire after 32 inputs over the + * vocabulary NOOP/FIRE/RIGHT/LEFT: 40 of the 96 single-input substitutions + * still reproduced both hashes, at 16 of the 32 turns, and all 16 applied at + * once reproduced them too. `FIRE` while the ball is already in flight is a + * state no-op, so those logs reach a bit-identical emulator state. + */ +export function probeOpaqueCollisions( + game: Game, + contract: MilestoneContract, + options: { reference: readonly string[]; seed: number; vocabulary: readonly string[]; collisionTurns?: number }, +): OpaqueCollision[] { + const { opaque } = contractLegibility(contract) + if (opaque.length === 0) return [] + const cap = options.collisionTurns ?? DEFAULT_COLLISION_TURNS + if (!Number.isInteger(cap) || cap < 0) throw new Error(`collisionTurns must be a non-negative integer, got ${cap}`) + const seed = options.seed + const firesAfter = firstPassTurn(game, contract, seed, options.reference) + const satisfied = (inputs: readonly string[]): Set => { + const tracker = new MilestoneTracker(contract) + let state = game.init(seed) + tracker.consider(game.evidence(state)) + for (const input of inputs) { + state = game.step(state, input) + tracker.consider(game.evidence(state)) + } + return new Set(tracker.verified()) + } + + // Milestones that fire at the same turn share a prefix, so they share a + // sweep: one replay answers for every one of them. + const groups = new Map() + for (const id of opaque) { + const turn = firesAfter.get(id) ?? -1 + groups.set(turn, [...(groups.get(turn) ?? []), id]) + } + + const rows: OpaqueCollision[] = [] + for (const [turn, ids] of groups) { + if (turn <= 0) { + for (const id of ids) { + rows.push({ + milestone: id, firesAfter: turn, probedTurns: 0, substitutions: 0, + collisions: 0, freeTurns: 0, jointCollision: false, family: 1, + }) + } + continue + } + const prefix = options.reference.slice(0, turn) + const points = probePoints(turn, cap) + const survivors = new Map>(ids.map((id) => [id, new Map()])) + let substitutions = 0 + for (const i of points) { + for (const word of options.vocabulary) { + if (word === prefix[i]) continue + substitutions++ + const perturbed = [...prefix] + perturbed[i] = word + const verified = satisfied(perturbed) + for (const id of ids) { + if (!verified.has(id)) continue + const byTurn = survivors.get(id)! + byTurn.set(i, [...(byTurn.get(i) ?? []), word]) + } + } + } + for (const id of ids) { + const byTurn = survivors.get(id)! + const collisions = [...byTurn.values()].reduce((n, words) => n + words.length, 0) + let family = 1 + for (const words of byTurn.values()) family *= 1 + words.length + let jointCollision = false + if (byTurn.size > 0) { + const joint = [...prefix] + for (const [i, words] of byTurn) joint[i] = words[0]! + jointCollision = satisfied(joint).has(id) + } + rows.push({ + milestone: id, + firesAfter: turn, + probedTurns: points.length, + substitutions, + collisions, + freeTurns: byTurn.size, + jointCollision, + family, + }) + } + } + return rows.sort((a, b) => opaque.indexOf(a.milestone) - opaque.indexOf(b.milestone)) +} + export interface CalibrationReport { turns: number seed: number @@ -112,41 +283,42 @@ export interface CalibrationReport { reference: BaselineOutcome baselines: BaselineOutcome[] /** - * Earnable milestones the reference reached and no baseline did — the whole - * of the contract's discriminating power. A replay-identity milestone is - * excluded even though no baseline earned it, because being out of reach of - * every policy is not separation. + * Legible milestones the reference reached and no baseline did — the whole + * of the contract's discriminating power. An opaque milestone is excluded + * even when no baseline earned it: a reader cannot see what it demands, so + * it cannot be the evidence that the contract measures anything. */ separating: string[] /** milestones at least one baseline earned */ trivial: string[] - /** contract milestones an independent policy can earn by playing */ - earnable: string[] + /** contract milestones whose requirement a reader can read off the contract */ + legible: string[] + /** contract milestones stated as a hash, or gated behind one */ + opaque: string[] + /** why each opaque milestone cannot be read, keyed by milestone id */ + opacityReasons: Record /** - * Contract milestones only a replay of the reference earns: a hash check on - * the reference run's exact bytes, or a milestone gated behind one. + * Opaque milestones a trivial baseline reproduced. + * + * A non-empty set is a measured finding about the game: the check is + * satisfied by mashing one button, so it demands nothing. Such a milestone + * is also listed in `trivial`. */ - unearnable: string[] - /** why each unearnable milestone is out of reach, keyed by milestone id */ - unearnableReasons: Record + opaqueReproduced: string[] /** - * Unearnable milestones a trivial baseline reproduced anyway. - * - * A non-empty set is a measured finding about the game, not about the - * policy: the hash covers so little entropy that another trajectory collides - * with it, so it identifies nothing. Such a milestone is also listed in - * `trivial`. + * How many perturbed logs satisfy each opaque check, one row per opaque + * milestone. This is the non-trivial prober: it finds the collisions the + * baseline suite is far too weak to find. */ - unearnableReproduced: string[] + collisions: OpaqueCollision[] /** the strongest baseline's verified count */ bestBaselineCount: number /** - * The strongest baseline's EARNABLE count. This is the number a reference - * must beat: a raw verified count flatters the reference, which reproduces - * every replay-identity check by construction. + * The strongest baseline's count over LEGIBLE milestones. This is the number + * a reference must beat, because an opaque milestone cannot carry the claim. */ - bestBaselineEarnedCount: number - /** the reference's own progress, with the earnable denominator separated */ + bestBaselineLegibleCount: number + /** the reference's own progress */ referenceScore: MilestoneScore separates: boolean } @@ -162,6 +334,12 @@ export interface CalibrateOptions { baselines?: readonly BaselinePolicy[] /** Inputs per run. Default: the reference length, so the comparison is length-matched. */ turns?: number + /** + * Cap on the turns the opaque-collision sweep probes per check. Default + * `DEFAULT_COLLISION_TURNS`. The sweep costs `probedTurns * (vocabulary - 1)` + * replays of the prefix, so lower it for a check that fires very late. + */ + collisionTurns?: number } /** @@ -194,7 +372,8 @@ export function calibrateContract( return { id, verified: attestation.verified, verdict: attestation.verdict } } - const reference = play('reference', options.reference.slice(0, turns)) + const referenceInputs = options.reference.slice(0, turns) + const reference = play('reference', referenceInputs) const baselines = policies.map((policy) => { const inputs = policy.inputs(vocabulary, turns, seed) if (inputs.length !== turns) { @@ -205,14 +384,15 @@ export function calibrateContract( const earnedByBaseline = new Set(baselines.flatMap((b) => b.verified)) const order = contract.milestones.map((m) => m.id) - const { earnable, unearnable, reasons } = contractEarnability(contract) - const earnableSet = new Set(earnable) - const separating = reference.verified.filter((id) => !earnedByBaseline.has(id) && earnableSet.has(id)) + const { legible, opaque, reasons } = contractLegibility(contract) + const legibleSet = new Set(legible) + const separating = reference.verified.filter((id) => !earnedByBaseline.has(id) && legibleSet.has(id)) const trivial = order.filter((id) => earnedByBaseline.has(id)) const bestBaselineCount = baselines.reduce((best, b) => Math.max(best, b.verified.length), 0) - const earnedCount = (outcome: BaselineOutcome): number => outcome.verified.filter((id) => earnableSet.has(id)).length - const bestBaselineEarnedCount = baselines.reduce((best, b) => Math.max(best, earnedCount(b)), 0) + const legibleCount = (outcome: BaselineOutcome): number => outcome.verified.filter((id) => legibleSet.has(id)).length + const bestBaselineLegibleCount = baselines.reduce((best, b) => Math.max(best, legibleCount(b)), 0) const referenceScore = scoreMilestones(contract, reference.verified) + const referenceLegible = legibleCount(reference) return { turns, @@ -222,142 +402,190 @@ export function calibrateContract( baselines, separating, trivial, - earnable, - unearnable, - unearnableReasons: reasons, - unearnableReproduced: unearnable.filter((id) => earnedByBaseline.has(id)), + legible, + opaque, + opacityReasons: reasons, + opaqueReproduced: opaque.filter((id) => earnedByBaseline.has(id)), + collisions: probeOpaqueCollisions(game, contract, { + reference: referenceInputs, + seed, + vocabulary, + ...(options.collisionTurns === undefined ? {} : { collisionTurns: options.collisionTurns }), + }), bestBaselineCount, - bestBaselineEarnedCount, + bestBaselineLegibleCount, referenceScore, - separates: separating.length > 0 && referenceScore.earned > bestBaselineEarnedCount, + separates: separating.length > 0 && referenceLegible > bestBaselineLegibleCount, } } /** - * The replay-identity checks a contract is allowed to carry. + * The opaque checks a contract is allowed to carry, and which of them the + * author accepts as weak. * - * The declaration is an exact set, not a switch. An author who pins a hash + * A declaration is an exact set, not a switch. An author who pins a hash * writes the id down, so a hash milestone that a later derivation adds cannot * enter a published contract unnoticed. */ -export interface EarnabilityDeclaration { - /** Milestone ids the author accepts as replay-identity pins. */ - identityChecks?: readonly string[] +export interface OpacityDeclaration { + /** Milestone ids the author accepts as opaque: stated as a hash, or gated behind one. */ + opaqueChecks?: readonly string[] + /** + * Opaque milestone ids the author accepts as WEAK, having read the measured + * number: the substitution sweep found other input logs that satisfy them, + * so the hash names a state many trajectories reach rather than one. + */ + weakChecks?: readonly string[] } function earners(report: CalibrationReport, milestone: string): string { return report.baselines.filter((b) => b.verified.includes(milestone)).map((b) => b.id).join(', ') } +/** One measured line about an opaque check, with every number the sweep produced. */ +function collisionLine(row: OpaqueCollision): string { + return ( + ` ${row.milestone} — ${row.collisions} of ${row.substitutions} single-input substitutions of the ` + + `reference still satisfy it, at ${row.freeTurns} of ${row.probedTurns} probed turn(s) before it fires ` + + `(after ${row.firesAfter} input(s)); applying one at every free turn at once ` + + `${row.jointCollision ? 'also satisfies it' : 'does not'}, so at least ${row.family} distinct ` + + `${row.firesAfter}-input log(s) satisfy it` + ) +} + /** - * Every way a contract's earnable/unearnable split can be wrong, as message - * blocks. An empty array means the split is sound and declared. + * Every way a contract's legible/opaque split can be wrong, as message blocks. + * An empty array means the split is sound and declared. */ -function earnabilityProblems(report: CalibrationReport, declaration: EarnabilityDeclaration): string[] { +function opacityProblems(report: CalibrationReport, declaration: OpacityDeclaration): string[] { const problems: string[] = [] const { total } = report.referenceScore - const declared = declaration.identityChecks - const share = total === 0 ? 0 : Math.round((report.unearnable.length / total) * 100) - const detail = report.unearnable.map((id) => ` unearnable: ${id} — ${report.unearnableReasons[id]}`) + const declared = declaration.opaqueChecks + const share = total === 0 ? 0 : Math.round((report.opaque.length / total) * 100) + const detail = report.opaque.map((id) => ` opaque: ${id} — ${report.opacityReasons[id]}`) if (declared === undefined) { - if (report.unearnable.length > 0) { + if (report.opaque.length > 0) { problems.push( [ - `contract has ${report.unearnable.length} of ${total} milestone(s) no policy can earn ` + - `(${report.earnable.length} earnable, ${share}% unearnable)`, + `contract states ${report.opaque.length} of ${total} milestone(s) as a hash ` + + `(${report.legible.length} legible, ${share}% opaque)`, ...detail, - ` the reference itself scored ${formatMilestoneScore(report.referenceScore)}, so a score out of ` + - `${total} is not reachable by an independent policy`, - 'A replay-identity check proves that a replay reproduced the recorded run. It is not progress.', - `Keep it and declare it — assertContractSeparates(report, { identityChecks: ` + - `[${report.unearnable.map((id) => `'${id}'`).join(', ')}] }) — or state the progression with a ` + + ` the reference itself scored ${formatMilestoneScore(report.referenceScore)}, and a reader cannot ` + + 'tell what the opaque points asked for', + 'A hash check identifies a state, not a trajectory, so it is earnable — but nobody can read it.', + `Keep it and declare it — assertContractSeparates(report, { opaqueChecks: ` + + `[${report.opaque.map((id) => `'${id}'`).join(', ')}] }) — or state the progression with a ` + 'state-path, save-path, frame-path, or log-contains check.', ].join('\n'), ) } } else { const accepted = new Set(declared) - const pinned = new Set(report.unearnable) - const undeclared = report.unearnable.filter((id) => !accepted.has(id)) + const pinned = new Set(report.opaque) + const undeclared = report.opaque.filter((id) => !accepted.has(id)) const stale = [...accepted].filter((id) => !pinned.has(id)) if (undeclared.length > 0) { problems.push( [ - `contract has ${undeclared.length} undeclared milestone(s) no policy can earn: ${undeclared.join(', ')}`, + `contract has ${undeclared.length} undeclared opaque milestone(s): ${undeclared.join(', ')}`, ...detail, - 'Declare every replay-identity check, or state the progression with a semantic check.', + 'Declare every opaque check, or state the progression with a legible check.', ].join('\n'), ) } if (stale.length > 0) { problems.push( - `identityChecks names ${stale.length} milestone(s) the contract does not pin: ${stale.join(', ')} — ` + + `opaqueChecks names ${stale.length} milestone(s) the contract does not state as a hash: ${stale.join(', ')} — ` + 'the declaration is stale, and it would hide a hash milestone added later', ) } } - if (report.unearnableReproduced.length > 0) { + if (report.opaqueReproduced.length > 0) { problems.push( [ - `${report.unearnableReproduced.length} replay-identity milestone(s) were reproduced by a trivial baseline:`, - ...report.unearnableReproduced.map((id) => ` ${id} — reproduced by ${earners(report, id)}`), - 'A hash that another trajectory reproduces identifies no run. Remove it or hash more state.', + `${report.opaqueReproduced.length} opaque milestone(s) were reproduced by a trivial baseline:`, + ...report.opaqueReproduced.map((id) => ` ${id} — reproduced by ${earners(report, id)}`), + 'A hash one constant button press satisfies demands nothing. Remove it or hash more state.', ].join('\n'), ) } + + const acceptedWeak = new Set(declaration.weakChecks ?? []) + const weak = report.collisions.filter((row) => row.collisions > 0) + const undeclaredWeak = weak.filter((row) => !acceptedWeak.has(row.milestone)) + if (undeclaredWeak.length > 0) { + problems.push( + [ + `${undeclaredWeak.length} opaque milestone(s) are satisfied by input logs other than the reference:`, + ...undeclaredWeak.map(collisionLine), + 'A hash a large family of logs satisfies is a weak check: it names a state many trajectories reach, ' + + 'and says so where no reader can see it. State the progression with a legible check, or accept the ' + + `measured weakness — { weakChecks: [${undeclaredWeak.map((row) => `'${row.milestone}'`).join(', ')}] }.`, + ].join('\n'), + ) + } + const staleWeak = [...acceptedWeak].filter((id) => !weak.some((row) => row.milestone === id)) + if (staleWeak.length > 0) { + problems.push( + `weakChecks names ${staleWeak.length} milestone(s) the sweep found no colliding log for: ${staleWeak.join(', ')} — ` + + 'the declaration is stale, and it would hide a collision measured later', + ) + } return problems } /** - * Fail closed on a contract whose milestones no policy can earn. + * Fail closed on a contract whose milestones a reader cannot understand. * - * Identity checks are correct and useful: replay attestation is exactly the - * claim that one run reproduced another's bytes. They must not be counted as - * achievements, and an undeclared one must not reach a published contract, - * because it puts points in the denominator that only the reference can score. + * Opaque checks are legitimate: a hash names an exact game state, an + * independent policy that reaches that state earns it, and replay attestation + * keeps its own input-log hash chain either way. They must be declared, + * because a point nobody can read is a point nobody can audit, and a hash that + * many logs satisfy must be declared with its measured number. * * Call this for a contract that is not meant to separate — a demonstration * target — where `assertContractSeparates` would fail for the other reason. */ -export function assertMilestonesEarnable( +export function assertOpaqueChecksDeclared( report: CalibrationReport, - declaration: EarnabilityDeclaration = {}, + declaration: OpacityDeclaration = {}, ): void { - const problems = earnabilityProblems(report, declaration) + const problems = opacityProblems(report, declaration) if (problems.length > 0) throw new Error(problems.join('\n')) } /** * Fail closed on a contract that a trivial policy satisfies, and on one that - * carries milestones no policy can earn. + * carries undeclared opaque milestones. * * Call this wherever a target is published. A contract that does not separate * still produces scores; those scores report how many frames elapsed, and * comparing two agents on them compares nothing. A contract whose points are - * partly unearnable still produces scores too, and every one of them is quoted - * against a denominator no agent can reach. + * partly opaque still produces scores too, and no reader can tell what those + * points were awarded for. * * Both failures are reported together, so one run of the gate names everything * an author must fix. */ export function assertContractSeparates( report: CalibrationReport, - declaration: EarnabilityDeclaration = {}, + declaration: OpacityDeclaration = {}, ): void { - const problems = earnabilityProblems(report, declaration) + const problems = opacityProblems(report, declaration) if (!report.separates) { const best = report.baselines.filter((b) => b.verified.length === report.bestBaselineCount).map((b) => b.id) + const referenceLegible = report.reference.verified.filter((id) => report.legible.includes(id)).length problems.push( [ `contract does not separate: the reference verified ${report.reference.verified.length} milestone(s) ` + `and the best trivial baseline verified ${report.bestBaselineCount} over ${report.turns} turns ` + `(seed ${report.seed}, strongest: ${best.join(', ') || 'none'})`, - ` on earnable milestones alone: reference ${report.referenceScore.earned}, ` + - `best baseline ${report.bestBaselineEarnedCount}, of ${report.referenceScore.earnable} earnable`, + ` on legible milestones alone: reference ${referenceLegible}, ` + + `best baseline ${report.bestBaselineLegibleCount}, of ${report.legible.length} legible`, ...report.trivial.map((id) => ` trivial: ${id} — earned by ${earners(report, id)}`), - ` earnable and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, + ` legible and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, 'A derived contract is a hypothesis until it separates. Pin a progression a trivial policy cannot reach.', ].join('\n'), ) diff --git a/campaign.test.mts b/campaign.test.mts index b651c7a..bbe4cb7 100644 --- a/campaign.test.mts +++ b/campaign.test.mts @@ -167,12 +167,9 @@ try { assert.ok(first.recentHistory.length <= 8) assert.ok(first.lastFrame.length > 0) assert.ok(first.verifiedSoFar.includes('first-legal-move')) - // An analyst reads progress against the earnable denominator, not the raw - // milestone count. Every 2048 milestone is a semantic check, so the two - // agree here, and the field says so instead of leaving it to be assumed. + // Mid-run progress carries its denominator, so an analyst never has to + // guess how many points the contract holds. assert.equal(first.scoreSoFar.verified, first.verifiedSoFar.length) - assert.equal(first.scoreSoFar.earned, first.verifiedSoFar.length) - assert.equal(first.scoreSoFar.earnable, contract.milestones.length) assert.equal(first.scoreSoFar.total, contract.milestones.length) assert.equal(first.ledger.decisions.length, 2) } diff --git a/campaign.ts b/campaign.ts index d372339..7cb4f1e 100644 --- a/campaign.ts +++ b/campaign.ts @@ -121,10 +121,7 @@ export interface SegmentReport { newMilestones: MilestoneCostRow[] /** Milestones the live tracker observed so far, in observation order. */ verifiedSoFar: string[] - /** - * Progress so far with the earnable denominator separated from the total, so - * an analyst reads "3 of 4 earnable" instead of "3 of 6". - */ + /** Progress so far as verified over the contract's milestone count. */ scoreSoFar: MilestoneScore lastFrame: string recentHistory: AgentHistoryEntry[] diff --git a/docs/adapters.md b/docs/adapters.md index 7ffd921..cb40d54 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -118,20 +118,23 @@ Over the same 70 turns, on the same ROM and the same contract: The finding is not a Libbet defect and not a PyBoy defect. It is what a derived contract is worth before somebody measures it. -### Earnable milestones and replay-identity checks +### Legible checks and opaque ones A derived contract fails in a second way, and the separation test alone cannot see it. -`deriveContract` samples whatever the mark asks for, so a mark that asks for a frame hash or a save hash pins the exact bytes the reference run produced. -No independent policy earns such a milestone. Reaching it means reproducing the reference, not playing. -Because no trivial baseline earns it either, the separation test used to read it as the contract's strongest evidence. +`deriveContract` samples whatever the mark asks for, so a mark that asks for a frame hash or a save hash writes a check nobody can read. -`contractEarnability(contract)` splits the milestones by check kind, and follows `requires`: a threshold gated behind a hash is unreachable too. -`assertContractSeparates` and `assertMilestonesEarnable` refuse a contract whose unearnable milestones the author has not declared by id. +A hash identifies a **state**, not a trajectory, so it is a point an independent policy earns by reaching that state. +What it cannot do is say which state, or how many ways there are in. +Because no trivial baseline reaches it either, the separation test used to read it as the contract's strongest evidence. + +`contractLegibility(contract)` splits the milestones by check kind, and follows `requires`: a threshold gated behind a hash is opaque too. +`assertContractSeparates` and `assertOpaqueChecksDeclared` refuse a contract whose opaque milestones the author has not declared by id. The declaration is an exact set, so a hash milestone that a later derivation adds cannot enter a published contract unnoticed. -Every packaged contract, by check kind and `requires` graph: +Every packaged contract, by check kind and `requires` graph. +Every milestone is a point, so the **Milestones** column is the denominator of a score on that contract: -| Contract | Milestones | Earnable | Replay-identity | +| Contract | Milestones | Legible | Opaque | |---|---|---|---| | ALE Breakout | 6 | 4 | 2 — `frame-at-first-score`, `save-at-first-score` | | Libbet through `pyboy-generic` | 6 | 4 | 2 — `state-at-first-progression`, `frame-at-first-progression` | @@ -148,16 +151,20 @@ Every packaged contract, by check kind and `requires` graph: The Breakout and Libbet rows are asserted against the running emulator in `ale.test.mts` and `pyboy-libbet.test.mts`. The rest follow from the check kinds their references declare, which is the whole of the rule. -Keep the identity checks. They are the whole of what replay attestation proves. -Declare them, and report the score as earned over earnable: +Keep the opaque checks where a legible one cannot state the progression. +Declare them, and quote the score over the full milestone count: ```ts const report = calibrateContract(game, contract, { reference, vocabulary }) assertContractSeparates(report, { - identityChecks: ['frame-at-first-score', 'save-at-first-score'], + opaqueChecks: ['frame-at-first-score', 'save-at-first-score'], + weakChecks: ['frame-at-first-score', 'save-at-first-score'], }) ``` +`report.collisions` carries the measured strength of each opaque check: how many single-input substitutions of the reference still satisfy it, at how many turns, and a lower bound on the family of logs that do. +On Breakout that bound is 3.82 × 10⁸ distinct 32-input logs, so the two hashes name a state a great many trajectories reach. + ## Libretro consoles through stable-retro ```ts diff --git a/episode-loop.ts b/episode-loop.ts index 23f7708..119d712 100644 --- a/episode-loop.ts +++ b/episode-loop.ts @@ -183,7 +183,6 @@ export function finalizeRecord( budgetUsd, budgetExhausted: rollout.spent >= budgetUsd, verified: attestation.verified, - earned: attestation.earned, score: attestation.score, milestones: [...rollout.milestones], replayDivergence: JSON.stringify(rollout.observed) !== JSON.stringify(attestation.verified), diff --git a/episode.test.mts b/episode.test.mts index 78844ed..9943337 100644 --- a/episode.test.mts +++ b/episode.test.mts @@ -14,11 +14,8 @@ import { saveLevels, saveLevelsContract, SAVE_LEVELS_REFERENCE } from './adapter assert.equal(record.verdict, 'clean') assert.deepEqual(record.verified, ['hp-untouched', 'room-1', 'room-2-plus', 'room-3']) assert.equal(record.spentUsd, 0) - // Every crawler milestone is an engine-state threshold, so the earned set is - // the verified set and the score has no hidden denominator. - assert.deepEqual(record.earned, record.verified) - assert.deepEqual(record.score, { verified: 4, earned: 4, earnable: 4, total: 4 }) - assert.equal(formatMilestoneScore(record.score), '4 of 4 earnable') + assert.deepEqual(record.score, { verified: 4, total: 4 }) + assert.equal(formatMilestoneScore(record.score), '4 of 4') } { @@ -35,19 +32,21 @@ import { saveLevels, saveLevelsContract, SAVE_LEVELS_REFERENCE } from './adapter const { record } = await playEpisode(engineCrawler, engineCrawlerContract(), scriptedDriver(ENGINE_CRAWLER_REFERENCE), 1, 2) assert.equal(record.turns, 2) assert.deepEqual(record.verified, ['hp-untouched', 'room-1']) - assert.deepEqual(record.score, { verified: 2, earned: 2, earnable: 4, total: 4 }) + assert.deepEqual(record.score, { verified: 2, total: 4 }) } -// A run on a contract whose milestones are all hash checks verifies them and -// earns none of them. The record reports both, so nobody quotes the run as two -// out of two. +// A run on a contract whose milestones are all hash checks scores them like +// any other points. A hash names one game state; reaching it is progress a +// reader cannot read, not progress that did not happen. { const { record } = await playEpisode(saveLevels, saveLevelsContract(), scriptedDriver(SAVE_LEVELS_REFERENCE), 1, 8) assert.equal(record.verdict, 'clean') assert.deepEqual(record.verified, ['level-2-saved', 'level-2-logged']) - assert.deepEqual(record.earned, []) - assert.deepEqual(record.score, { verified: 2, earned: 0, earnable: 0, total: 2 }) - assert.equal(formatMilestoneScore(record.score), '0 of 0 earnable (2 of 2 verified, 2 replay-identity)') + // Both milestones are hashes, and both are points a run scores by reaching + // the state they name. The denominator does not shrink because a check is + // opaque; what shrinks is what a reader can tell about it. + assert.deepEqual(record.score, { verified: 2, total: 2 }) + assert.equal(formatMilestoneScore(record.score), '2 of 2') } // A scripted driver is positioned by the harness turn, so a driver created in a diff --git a/episode.ts b/episode.ts index 34de041..6b295f5 100644 --- a/episode.ts +++ b/episode.ts @@ -97,15 +97,9 @@ export interface EpisodeRecord { spentUsd: number budgetUsd: number budgetExhausted: boolean - /** Every milestone the replay reproduced, replay-identity checks included. */ + /** Every milestone the replay reproduced. */ verified: string[] - /** The subset of `verified` an independent policy can earn by playing. */ - earned: string[] - /** - * The run's progress with the earnable denominator separated from the total. - * Report `earned` of `earnable`; `verified` of `total` counts hash checks - * that only a replay of the reference reproduces. - */ + /** `verified` over the contract's milestone count. */ score: MilestoneScore milestones: MilestoneCostRow[] replayDivergence: boolean diff --git a/pyboy-libbet.test.mts b/pyboy-libbet.test.mts index fee494a..ea6ece0 100644 --- a/pyboy-libbet.test.mts +++ b/pyboy-libbet.test.mts @@ -20,9 +20,9 @@ import { existsSync, readFileSync } from 'node:fs' import { fileURLToPath } from 'node:url' import { autoMarks, loadDiscovery, makePyBoyGeneric } from './adapters/pyboy-generic' import { attestRun } from './attestation' -import { assertContractSeparates, assertMilestonesEarnable, calibrateContract, UNKNOWN_BASELINE_WORD } from './calibration' +import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract, UNKNOWN_BASELINE_WORD } from './calibration' import { logFrom, observationOf } from './runtime' -import { contractEarnability, formatMilestoneScore, validateContract } from './schema' +import { contractLegibility, formatMilestoneScore, validateContract } from './schema' import { decodePng, unscale } from './test-png.mts' /** The Game Boy pad, matching `BUTTONS` in pyboy/tetris.py. */ @@ -172,35 +172,42 @@ try { // word earns nothing, so this is not a pure function of elapsed frames. assert.deepEqual(report.baselines.find((b) => b.id === `constant:${UNKNOWN_BASELINE_WORD}`)?.verified, []) - // (e2) The earnable split on the derived contract. `autoMarks` anchors one - // save-hash and one frame-hash milestone at the confirmed channel's first - // progression, so two of the six points can only be scored by a replay of - // this exploration trajectory. - const earnability = contractEarnability(adapter.contract) - assert.deepEqual(earnability.unearnable, ['state-at-first-progression', 'frame-at-first-progression']) - assert.equal(earnability.earnable.length, 4) - assert.ok(earnability.earnable.every((id) => id.endsWith('-progressed'))) - assert.deepEqual(report.earnable, earnability.earnable) - assert.deepEqual(report.referenceScore, { verified: 3, earned: 3, earnable: 4, total: 6 }) - assert.equal(formatMilestoneScore(report.referenceScore), '3 of 4 earnable (3 of 6 verified, 2 replay-identity)') - // No baseline reproduced either hash, so both are real identity checks here. - assert.deepEqual(report.unearnableReproduced, []) - // Undeclared, the contract fails the earnability gate on its own, which is a + // (e2) The legible/opaque split on the derived contract. `autoMarks` anchors + // one save-hash and one frame-hash milestone at the confirmed channel's first + // progression. Both are points a policy can score by reaching that state; + // neither says anything a reader can weigh, so both must be declared. + const legibility = contractLegibility(adapter.contract) + assert.deepEqual(legibility.opaque, ['state-at-first-progression', 'frame-at-first-progression']) + assert.equal(legibility.legible.length, 4) + assert.ok(legibility.legible.every((id: string) => id.endsWith('-progressed'))) + assert.deepEqual(report.legible, legibility.legible) + // Six milestones, six points. The reference stops before the pinned state, + // so it scored three of six. + assert.deepEqual(report.referenceScore, { verified: 3, total: 6 }) + assert.equal(formatMilestoneScore(report.referenceScore), '3 of 6') + assert.deepEqual(report.opaqueReproduced, []) + // The sweep cannot measure a check the reference never reaches, and says so + // with -1 rather than reporting a clean result it did not earn. + assert.deepEqual(report.collisions.map((row) => [row.milestone, row.firesAfter, row.collisions]), [ + ['state-at-first-progression', -1, 0], + ['frame-at-first-progression', -1, 0], + ]) + // Undeclared, the contract fails the opacity gate on its own, which is a // second, independent reason this target must not be published as a score. assert.throws( - () => assertMilestonesEarnable(report), + () => assertOpaqueChecksDeclared(report), (error: unknown) => { const message = (error as Error).message - assert.match(message, /2 of 6 milestone\(s\) no policy can earn/u) - assert.match(message, /4 earnable, 33% unearnable/u) + assert.match(message, /states 2 of 6 milestone\(s\) as a hash/u) + assert.match(message, /4 legible, 33% opaque/u) return true }, ) - const declared = { identityChecks: ['state-at-first-progression', 'frame-at-first-progression'] } - assertMilestonesEarnable(report, declared) + const declared = { opaqueChecks: ['state-at-first-progression', 'frame-at-first-progression'] } + assertOpaqueChecksDeclared(report, declared) assert.throws(() => assertContractSeparates(report, declared), /does not separate/u) - console.log(`pyboy-libbet: calibration regression — reference ${formatMilestoneScore(report.referenceScore)}, best trivial baseline ${report.bestBaselineCount} verified / ${report.bestBaselineEarnedCount} earnable over ${report.turns} turns, separates=${report.separates} OK`) + console.log(`pyboy-libbet: calibration regression — reference ${formatMilestoneScore(report.referenceScore)}, best trivial baseline ${report.bestBaselineCount} verified / ${report.bestBaselineLegibleCount} legible over ${report.turns} turns, separates=${report.separates} OK`) // (f) The observation image channel on the real emulator. PyBoy's own // screen.image needs Pillow, which is absent here — the boot logs say so — diff --git a/schema.ts b/schema.ts index dc95b5c..e199b43 100644 --- a/schema.ts +++ b/schema.ts @@ -1,9 +1,11 @@ /** * Playproof milestone-contract schema — the game-agnostic progression language. * - * Exact hashes remain available for identity checkpoints. Semantic milestones - * should prefer normalized state/save/frame paths so two different valid ways - * of reaching the same progress do not collapse to one reference trajectory. + * A check is a statement about a game STATE. A state/save/frame path or a log + * event states it in the open; a hash states the same kind of thing opaquely, + * naming one exact state without saying which. Both are earnable by playing. + * Prefer a path or an event, because a reader can then judge what the contract + * asks for. */ import { createHash } from 'node:crypto' @@ -109,79 +111,81 @@ export function contractHash(c: MilestoneContract): string { } /** - * What a milestone's check can prove. + * How a check states its requirement. * - * `achievement` — a threshold, a normalized field, or an event that an - * independent policy reaches by playing. Two different valid trajectories can - * both satisfy it. + * `legible` — a threshold, a normalized field, or an event. A reader sees what + * the milestone demands and can judge whether reaching it is progress. * - * `identity` — a hash over the exact bytes one recorded run produced. It proves - * that a replay reproduced that run, which is what replay attestation is for. - * It is not a progression: earning it means reproducing the reference's frame - * or save, not playing well. + * `opaque` — a hash. It demands one exact game state, and a reader cannot tell + * which state, nor how many trajectories reach it. It is an achievement like + * any other: a hash identifies a STATE, not a trajectory, so an independent + * policy that reaches that state earns it without ever seeing the reference. + * Measured on ALE Breakout: of 96 single-input substitutions of the reference + * over the 32-turn prefix, 40 still reproduced both pinned hashes, at 16 of + * the 32 turns; all 16 applied at once reproduced them too. + * + * Do not confuse an opaque check with replay attestation. The input-log hash + * chain is what proves a replay reproduced a recorded run. A milestone hash + * proves only that some run stood in one state. */ -export type MilestoneRole = 'achievement' | 'identity' - -const ROLE_FOR_CHECK: Record = { - 'state-path': 'achievement', - 'save-path': 'achievement', - 'save-hash': 'identity', - 'log-contains': 'achievement', - 'frame-path': 'achievement', - 'frame-hash': 'identity', +export type CheckLegibility = 'legible' | 'opaque' + +const LEGIBILITY_FOR_CHECK: Record = { + 'state-path': 'legible', + 'save-path': 'legible', + 'save-hash': 'opaque', + 'log-contains': 'legible', + 'frame-path': 'legible', + 'frame-hash': 'opaque', } /** - * The role a check carries, derived from its kind. + * Whether a reader can see what a check demands, derived from its kind. * - * The role is derived rather than declared so that an existing contract keeps + * The value is derived rather than declared so that an existing contract keeps * its bytes and its hash, and so that an author cannot forget to set it. A - * hash check is an identity check whatever the author intended. + * hash check is opaque whatever the author intended. */ -export function checkRole(check: MilestoneCheck): MilestoneRole { - return ROLE_FOR_CHECK[check.kind] +export function checkLegibility(check: MilestoneCheck): CheckLegibility { + return LEGIBILITY_FOR_CHECK[check.kind] } -/** Which milestones of a contract an independent policy can earn. */ -export interface ContractEarnability { - /** Milestone ids an independent policy can earn by playing. */ - earnable: string[] - /** Milestone ids only a replay of the reference run earns. */ - unearnable: string[] - /** Why each unearnable milestone is out of reach, keyed by milestone id. */ +/** Which milestones of a contract state their requirement in the open. */ +export interface ContractLegibility { + /** Milestone ids whose requirement a reader can read off the contract. */ + legible: string[] + /** Milestone ids whose requirement is a hash, or is gated behind one. */ + opaque: string[] + /** Why each opaque milestone cannot be read, keyed by milestone id. */ reasons: Record } /** - * Split a contract into the milestones a policy can earn and the ones it cannot. + * Split a contract into the milestones a reader can understand and the ones + * stated as a hash. * - * A milestone is unearnable when its own check pins exact bytes, and also when - * it depends on one that does: `MilestoneTracker` admits a milestone only after - * every prerequisite has passed, so an achievement gated behind a hash is as - * unreachable as the hash. A milestone with a missing or cyclic requirement is - * unearnable for the same reason — no run ever satisfies it. + * Opacity propagates through `requires`: `MilestoneTracker` admits a milestone + * only after every prerequisite passed, so a legible check gated behind a hash + * still demands something a reader cannot see. A missing or cyclic requirement + * is not judged here — `validateContract` reports those. */ -export function contractEarnability(contract: MilestoneContract): ContractEarnability { +export function contractLegibility(contract: MilestoneContract): ContractLegibility { const byId = new Map(contract.milestones.map((m) => [m.id, m])) const decided = new Map() const visiting = new Set() const reasonFor = (m: Milestone): string | null => { const cached = decided.get(m.id) if (cached !== undefined) return cached - if (visiting.has(m.id)) return `sits on a dependency cycle, so no run satisfies it` + if (visiting.has(m.id)) return null visiting.add(m.id) let reason: string | null = null - if (checkRole(m.check) === 'identity') { - reason = `its ${m.check.kind} check pins the reference run's exact bytes` + if (checkLegibility(m.check) === 'opaque') { + reason = `its ${m.check.kind} check states its requirement as a hash, so a reader cannot see what it demands` } else { for (const required of m.requires) { const prerequisite = byId.get(required) - if (prerequisite === undefined) { - reason = `requires ${required}, which the contract does not declare` - break - } - if (reasonFor(prerequisite) !== null) { - reason = `requires ${required}, which no independent policy can earn` + if (prerequisite !== undefined && reasonFor(prerequisite) !== null) { + reason = `requires ${required}, whose requirement is opaque` break } } @@ -191,63 +195,34 @@ export function contractEarnability(contract: MilestoneContract): ContractEarnab return reason } - const earnable: string[] = [] - const unearnable: string[] = [] + const legible: string[] = [] + const opaque: string[] = [] const reasons: Record = {} for (const m of contract.milestones) { const reason = reasonFor(m) - if (reason === null) earnable.push(m.id) + if (reason === null) legible.push(m.id) else { - unearnable.push(m.id) + opaque.push(m.id) reasons[m.id] = reason } } - return { earnable, unearnable, reasons } + return { legible, opaque, reasons } } -/** - * A run's progress against a contract, with the earnable denominator separated - * from the total. - * - * `earned` over `earnable` is the score a run may be compared on. `verified` - * over `total` includes the replay-identity checks, which only a replay of the - * reference reproduces. - */ +/** A run's progress against a contract. Every milestone counts, hashes included. */ export interface MilestoneScore { - /** Milestones the run verified, replay-identity checks included. */ + /** Milestones the run verified. */ verified: number - /** Of those, the ones an independent policy can earn. */ - earned: number - /** Milestones of the contract an independent policy can earn. */ - earnable: number /** Milestones of the contract. */ total: number } -/** The verified milestones an independent policy can earn, in verified order. */ -export function earnedMilestones(contract: MilestoneContract, verified: readonly string[]): string[] { - const earnable = new Set(contractEarnability(contract).earnable) - return verified.filter((id) => earnable.has(id)) -} - /** Score a verified milestone set against its contract. */ export function scoreMilestones(contract: MilestoneContract, verified: readonly string[]): MilestoneScore { - const { earnable } = contractEarnability(contract) - const set = new Set(earnable) - return { - verified: verified.length, - earned: verified.filter((id) => set.has(id)).length, - earnable: earnable.length, - total: contract.milestones.length, - } + return { verified: verified.length, total: contract.milestones.length } } -/** One line for a report or a log. Never states an earned count alone. */ +/** One line for a report or a log. */ export function formatMilestoneScore(score: MilestoneScore): string { - const identity = score.total - score.earnable - if (identity === 0) return `${score.earned} of ${score.earnable} earnable` - return ( - `${score.earned} of ${score.earnable} earnable ` + - `(${score.verified} of ${score.total} verified, ${identity} replay-identity)` - ) + return `${score.verified} of ${score.total}` }