From 87329fa24946eac0454a69aa420f0e10e91c33e1 Mon Sep 17 00:00:00 2001 From: Drew Stone Date: Mon, 24 Aug 2026 05:20:19 -0700 Subject: [PATCH] feat(calibration): score what a run made happen, not what it lost Two hand-written controls on ALE Breakout differ only in what they read: the ASCII frame at four screen pixels per character, or the RAM channels at one pixel each. The RAM control scores 24 to 7 at 600 decisions and never dies. It scored one milestone LOWER, because the packaged life-lost milestone is lives == 4: a point for dying. ProgressionKind splits a milestone into achievement and attrition. The split is measured, not declared, and it reads no field name. measureProgressions watches every numeric channel the evidence publishes across the reference and every baseline, and calls a milestone attrition when its check reads a numeric channel, that channel never rose and fell at least once, and the check does not hold at the initial value. Attrition propagates through requires. scoreAchievements drops attrition from the numerator and the denominator; separating and separates now count legible achievement milestones only. report.collapse states how much of a contract hangs off one milestone, whether a trivial baseline reaches it, and how many milestones first pass at the same reference input. Six of six Breakout milestones require score-opened, and three of the six open at input 32. PackagedContract.calibrate is the only way to build a PackagedContract, and it runs the whole gate. Calibration was optional before: the packaged Airstriker contract reports separates false with an empty separating set, and it shipped. nonSeparating is the escape hatch for a demonstration target, and it is refused when the contract does separate. pnpm test:ale, pnpm test:retro, and pnpm test:gym now calibrate the packaged contracts. No adapter file is edited, and every contract keeps its bytes and its hash. --- CHANGELOG.md | 60 ++++ README.md | 92 +++++- ale.test.mts | 187 +++++++++++- authoring.ts | 10 +- calibration.test.mts | 381 +++++++++++++++++++++++- calibration.ts | 657 +++++++++++++++++++++++++++++++++++++++--- gymnasium.test.mts | 68 +++++ schema.ts | 171 +++++++++++ stable-retro.test.mts | 81 +++++- 9 files changed, 1652 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54faaf0..975d5c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,66 @@ All notable changes to Playproof are documented here. +## Unreleased + +### A milestone earned by dying is not evidence of skill + +- **The measurement.** ALE Breakout, ale-py 0.12.1, seed 0. Two deterministic controls with the same control law and the same deadzone in screen pixels, differing only in what they read: the ASCII frame at four screen pixels per character, or the `ram_ball_x`/`ram_paddle_x` channels at one pixel each. Neither carries state between decisions and neither costs a model call. + + | control | game score @300 | @600 | lives left | milestones | achievements | + |---|---|---|---|---|---| + | `steer-from-ascii` | 6 | 7 | 0 (dead after 375) | **4 of 6** | 3 of 5 | + | `steer-from-ram` | **9** | **24** | **5** | 3 of 6 | 3 of 5 | + + The RAM control wins every column the game itself reports and never dies. It scored one milestone LOWER, because the packaged `life-lost` milestone is `lives == 4`: a point for dying. A program that never dies capped at 3 of 6 at any horizon. +- **`ProgressionKind` is `achievement` or `attrition`.** A milestone is attrition when a resource running down earns it. It marks progress REACHED and never competence shown, because the shortest path to it is to play badly. Recording it is legitimate; scoring it as competence is not. +- **The split is measured, not declared, and it reads no field name.** `measureProgressions` watches every numeric channel the evidence publishes across the reference and every baseline, and calls a milestone attrition when three measured statements hold: its check reads a numeric channel; that channel never rose and fell at least once, over every snapshot of every trajectory; and the check does not hold at the initial value of that channel. Attrition propagates through `requires`, because `MilestoneTracker` admits a milestone only after its prerequisites passed. +- A hardcoded `lives`/`health`/`shields` list would have failed the way a name match already failed one layer down: ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro `episodeDone`. A fixture channel named `lives` that counts rescued divers only rises, and the measurement classifies it as the achievement it is. +- A milestone no numeric channel speaks for — a hash, a log event, a path no trajectory published — is an achievement and is listed in `unmeasured`, so a reader sees where the measurement stopped. Attrition is a positive finding; the absence of one is not evidence of the opposite. +- **`scoreAchievements(contract, profile, verified)`** drops attrition milestones from the numerator AND the denominator. `scoreMilestones` is unchanged and still answers how far a run got. +- `CalibrationReport.separating` now counts legible **achievement** milestones, and `separates` compares the reference to the strongest baseline on those. Attrition milestones out of reach of every baseline move to `attritionSeparating`: they are recorded and never counted as separation. The report also carries `progression`, `referenceAchievementScore`, and `bestBaselineAchievementCount`. + +### A contract that grades on one event + +- `requires` is a partial order, so a contract can state six progressions and demand exactly one event. `report.collapse` states the structure with the number: which milestone the largest share requires, whether a trivial baseline reaches it, and how many milestones first pass at the SAME reference input. + + | packaged contract | gated behind | of | prerequisite reached by a baseline | open at one instant | + |---|---|---|---|---| + | ALE Breakout | `score-opened` | 6 of 6 | no, 0 of 7 | 3 of 6, after 32 inputs | + | stable-retro Airstriker | `score-opened` | 5 of 5 | yes, 2 of 28 | 3 of 5, after 41 inputs | + | Gymnasium CartPole | `survived-25-steps` | 5 of 5 | yes, 2 of 6 | 3 of 5, after 25 steps | + | Gymnasium FrozenLake | `reached-goal` | 3 of 3 | no, 0 of 8 | 3 of 3, after 6 steps | + | native-2048 | `first-legal-move` | 7 of 7 | yes, 6 of 7 | 2 of 7, after 4 inputs | + +- A run that misses the prerequisite scores 0 of N however well it played; where a baseline reaches it, the whole contract opens for free and grades only what follows a free event. The gate refuses until the author declares the structure with `{ gatedBehind: 'score-opened' }`, and refuses a stale declaration the same way `opaqueChecks` does. + +### Calibration is no longer optional for a packaged contract + +- **The measurement.** `calibrateContract` on the packaged stable-retro Airstriker contract reports `separates: false` with an EMPTY separating set. A seeded pseudo-random walk over the 25 advertised button words earns three of the four legible milestones, the same three the reference earns; the fourth is a hash that 271 of 768 single-input substitutions of the reference reproduce. It shipped because nothing between `deriveContract` and a published target ever ran the gate. +- **`PackagedContract.calibrate` is the only way to build a `PackagedContract`.** The class carries a private field and a private constructor, so an object literal of the same shape is not assignable and `new` is not available. It runs `calibrateContract`, refuses every finding the gate reports, and returns the contract with its hash, its report, and the declaration that let it through. A target that hands out a bare `MilestoneContract` has not been calibrated and now says so in its type. +- `{ nonSeparating: '' }` is the escape hatch for a target that is not meant to separate — a tier demonstration, a smoke fixture. It is a sentence rather than a switch, so the reason travels with the package, and it is REFUSED when the contract does separate: a declaration must not outlive the reason for it. +- `assertContractSeparates` now reports the attrition and collapse findings alongside the opacity ones, so one run of the gate names everything an author must fix. `assertOpaqueChecksDeclared` is unchanged and still checks opacity alone. + +### Measured across every packaged contract + +Full calibration, seed as packaged, run on every target this machine can boot. `pnpm test:ale`, `pnpm test:retro`, and `pnpm test:gym` now carry these as gates; nothing but the fixtures ran calibration before. + +| contract | reference | achievements | best baseline | separating | attrition | verdict | +|---|---|---|---|---|---|---| +| ALE Breakout | 6 of 6 | 5 of 5 | 0 of 7 policies scored | `score-opened`, `score-tier-2`, `score-tier-4` | `life-lost` | separates, collapses, 1 undeclared attrition | +| stable-retro Airstriker | 5 of 5 | 4 of 4 | 3 (pseudo-random) | **nothing** | `life-lost` | **does not separate**, collapses | +| Gymnasium CartPole | 5 of 5 | 5 of 5 | 2 (round-robin, pseudo-random) | `survived-50-steps`, `reward-at-50-steps` | none | separates, collapses | +| Gymnasium FrozenLake | 3 of 3 | 3 of 3 | 0 of 8 policies scored | `reached-goal`, `goal-cell` | none | separates, collapses | +| native-2048 | 7 of 7 | 7 of 7 | 7 (pseudo-random) | **nothing** | none | **does not separate**, collapses | +| save-levels | 2 of 2 | 2 of 2 | 2 (round-robin, pseudo-random) | **nothing** | none | **does not separate**, collapses | +| screen-puzzle | 2 of 2 | 2 of 2 | 2 (constant:r) | **nothing** | none | **does not separate**, collapses | +| engine-crawler | 4 of 4 | 4 of 4 | 4 (constant:right) | **nothing** | none | **does not separate**, does not collapse | + +- Breakout's `life-lost` and Airstriker's `life-lost` are the only two attrition milestones in the packaged set. Both were in `separating` before this change, so both contracts advertised a point for dying as part of their discriminating power. +- The two Breakout controls tie at 3 of 5 achievements rather than separating, and the reason is the second finding: the contract's top achievement is `score >= 4`, so nothing in it tells 7 from 24. +- `engineState.hpExact` in the `engine-crawler` fixture only ever falls, and its milestone is `hpExact == 1`, which HOLDS at the initial value. It is an achievement, which is the third condition of the rule doing its work: a check a run satisfies at turn zero needs no resource to run down. +- Existing adapters are unchanged in behaviour. No adapter file was edited, every contract keeps its bytes and its hash, and the pinned hashes for `engine-crawler`, `save-levels`, and `screen-puzzle` still hold. + ## 0.7.0 ### An episode can end because the game ended diff --git a/README.md b/README.md index cb0329e..09a6124 100644 --- a/README.md +++ b/README.md @@ -383,9 +383,11 @@ assertContractSeparates(report) `calibrateContract` replays the reference and a suite of trivial policies through the same attestation path: one constant policy per input word, a word the game cannot interpret, a round-robin cycle over the vocabulary, and a seeded pseudo-random walk over it. Every policy is deterministic in the seed, so a report reproduces from one number. -The report names `separating` (legible milestones no baseline earned), `trivial` (milestones at least one baseline earned), `legible` and `opaque`, `collisions`, and both baseline counts. -`separates` is true only when a **legible** milestone is out of reach of every baseline **and** the reference verifies strictly more legible milestones than the strongest baseline. +The report names `separating` (legible achievement milestones no baseline earned), `attritionSeparating` (out of reach of every baseline, and earned by a resource running down), `trivial` (milestones at least one baseline earned), `legible` and `opaque`, `progression`, `collapse`, `collisions`, and every baseline count. +`separates` is true only when a **legible achievement** milestone is out of reach of every baseline **and** the reference verifies strictly more of them than the strongest baseline. +Two kinds of milestone are excluded, for two different reasons. An opaque milestone cannot carry the claim: "the reference reached it and no baseline did" states nothing a reader can check. +An attrition milestone cannot carry it either: the run that earns it is the run that let a resource run down. `assertContractSeparates` throws otherwise, and the message names every trivial milestone with the baseline that earned it. ### The gate also refuses points nobody can read @@ -418,6 +420,92 @@ Measured on ALE Breakout, whose two hashes fire after 32 inputs over `NOOP/FIRE/ | distinct 32-input logs that satisfy the hashes | at least 3.82 × 10⁸ | `FIRE` while the ball is already in flight is a state no-op, so those logs reach a bit-identical emulator state. + +### A milestone earned by dying is not a milestone earned by playing well + +A milestone is an `achievement` when a run made something happen, and `attrition` when a run let a resource run down. +Lives, health, shields, fuel, time remaining: the shortest path to such a milestone is to play badly. +It is a real fact about a trajectory and it is recorded, but it is never counted as competence. + +The split is **measured, never declared**, and it reads no field name. +`measureProgressions` watches every numeric channel the engine publishes across the reference and every baseline, and calls a milestone attrition when all three of these hold: + +1. its check reads a numeric channel of the evidence; +2. that channel never rose and fell at least once, over every snapshot of every measured trajectory; +3. the check does not hold at the initial value of that channel. + +Anything else is an achievement, including a hash, a log event, and a channel no trajectory published; those are listed in `unmeasured` so a reader sees where the measurement stopped. +Attrition propagates through `requires`, because a milestone gated behind a lost life needs a lost life. + +A name list would have failed the same way `over()` would have: ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro `episodeDone`. +A channel called `lives` that counts rescued hostages goes UP, and the measurement classifies it as the achievement it is. + +```ts +import { measureProgressions, scoreAchievements, scoreMilestones } from '@tangle-network/playproof' + +const profile = report.progression +scoreMilestones(contract, verified) // how far the run got +scoreAchievements(contract, profile, verified) // how well it played +``` + +Measured on ALE Breakout, ale-py 0.12.1, seed 0. Two deterministic controls with the same control law and the same deadzone in screen pixels, differing only in what they read: + +| control | reads | game score @300 | @600 | lives left | milestones | achievements | +|---|---|---|---|---|---|---| +| `steer-from-ascii` | the ASCII frame, 4 px per character | 6 | 7 | 0 | **4 of 6** | 3 of 5 | +| `steer-from-ram` | `ram_ball_x`, `ram_paddle_x` | **9** | **24** | **5** | 3 of 6 | 3 of 5 | + +The RAM control wins every column the game reports and never dies. +Under the whole contract it scored LOWER, because `life-lost` is `lives == 4`: a point for dying. +`separating` and `separates` therefore count legible **achievement** milestones only, and `attritionSeparating` records the rest. +Declare an attrition milestone to keep it: `assertContractSeparates(report, { attritionChecks: ['life-lost'] })`. + +### A contract that grades on one event + +`requires` is a partial order, so a contract can state six progressions and demand exactly one event. +`report.collapse` states how much of the contract hangs off one milestone, whether a trivial baseline reaches it, and how many milestones first pass at the same instant of the reference. + +| contract | gated behind | of | prerequisite reached by a baseline | open at one instant | +|---|---|---|---|---| +| ALE Breakout | `score-opened` | 6 of 6 | no | 3 of 6, after 32 inputs | +| stable-retro Airstriker | `score-opened` | 5 of 5 | yes: `round-robin`, `pseudo-random` | 3 of 5, after 41 inputs | +| Gymnasium CartPole | `survived-25-steps` | 5 of 5 | yes: `round-robin`, `pseudo-random` | 3 of 5, after 25 steps | +| Gymnasium FrozenLake | `reached-goal` | 3 of 3 | no | 3 of 3, after 6 steps | + +A run that misses the prerequisite scores zero however well it played; where a baseline reaches it, the whole contract opens for free. +The gate refuses until the author declares the structure with `{ gatedBehind: 'score-opened' }`. + +### A packaged contract carries the calibration that justified it + +Calibration used to be optional, and an optional gate is a gate nothing has to pass. +Measured on stable-retro Airstriker: the packaged contract reports `separates: false` with an EMPTY separating set, and it shipped, because nothing between `deriveContract` and a published target ever asked. + +`PackagedContract.calibrate` is the only way to build a `PackagedContract`, and it runs the whole gate: + +```ts +const packaged = PackagedContract.calibrate(game, contract, { + reference: referenceInputs, + vocabulary, + declare: { + opaqueChecks: ['frame-at-first-score'], + weakChecks: ['frame-at-first-score'], + attritionChecks: ['life-lost'], + gatedBehind: 'score-opened', + }, +}) +packaged.report.separates // the verdict travels with the contract +``` + +A target that is not meant to separate — a tier demonstration, a smoke fixture — says so in words, and the declaration is refused when it goes stale: + +```ts +PackagedContract.calibrate(game, contract, { + reference, vocabulary, + declare: { nonSeparating: 'tier demonstration: exercises the evidence path, does not grade play' }, +}) +``` + +A contract that hands out a bare `MilestoneContract` has not been calibrated, and now says so in its type. No trivial baseline reproduces either hash, which is exactly why the baseline suite is not enough to judge one. | Contract | Milestones | Legible | Reference score | Best trivial baseline | diff --git a/ale.test.mts b/ale.test.mts index 6576c26..00c9665 100644 --- a/ale.test.mts +++ b/ale.test.mts @@ -16,12 +16,12 @@ import { strict as assert } from 'node:assert' import { spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' -import { attestRun } from './attestation' -import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract } from './calibration' +import { attestRun, MilestoneTracker } from './attestation' +import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract, measureProgressions } from './calibration' import { playEpisode, scriptedDriver } from './episode' -import { logFrom, observationOf } from './runtime' +import { isGameOver, logFrom, observationOf } from './runtime' import { decodePng, unscale } from './test-png.mts' -import { contractLegibility, formatMilestoneScore, scoreMilestones, validateContract } from './schema' +import { contractLegibility, formatMilestoneScore, scoreAchievements, scoreMilestones, validateContract } from './schema' import { AleRpc } from './adapters/ale-rpc' import { bundledReference, makeAle, type Ale, type AleState } from './adapters/ale' @@ -205,26 +205,193 @@ if (!pythonHasAle()) { () => assertOpaqueChecksDeclared(calibration, { opaqueChecks: ['frame-at-first-score', 'save-at-first-score'] }), /2 opaque milestone\(s\) are satisfied by input logs other than the reference/u, ) - const declared = { + const opacity = { opaqueChecks: ['frame-at-first-score', 'save-at-first-score'], weakChecks: ['frame-at-first-score', 'save-at-first-score'], } - assertOpaqueChecksDeclared(calibration, declared) + assertOpaqueChecksDeclared(calibration, opacity) + + // `life-lost` is `lives == 4`: the reference earns it by DYING. The + // classification reads no field name — it reads the motion of the channel, + // which falls and never rises across all eight replayed trajectories, and + // a check that does not hold at the starting value of a channel that only + // falls can only be earned by letting the resource run down. + // + // Before this split it sat in `separating`, so the contract's measured + // discriminating power included a point for dying. Two hand-written + // controls that differ only in how well they steer therefore ranked in the + // wrong order: the one that never died capped at 3 of 6. + assert.deepEqual(calibration.progression.attrition, ['life-lost']) + assert.deepEqual(calibration.progression.unmeasured, ['frame-at-first-score', 'save-at-first-score']) + assert.match( + calibration.progression.reasons['life-lost'] ?? '', + /reads engineState\.lives, which fell \d+ time\(s\) and never rose/u, + ) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.lives')?.rises, 0) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.score')?.falls, 0) + assert.deepEqual(calibration.attritionSeparating, ['life-lost']) + assert.ok(!calibration.separating.includes('life-lost')) + assert.deepEqual(calibration.referenceAchievementScore, { verified: 5, total: 5 }) + + // Every milestone requires `score-opened`, and three of the six open at the + // same instant of the reference. Six milestone ids, one decision point. + assert.equal(calibration.collapse.prerequisite, 'score-opened') + assert.equal(calibration.collapse.gated, 6) + assert.equal(calibration.collapse.total, 6) + assert.equal(calibration.collapse.collapses, true) + assert.deepEqual(calibration.collapse.earnedByBaseline, []) + assert.deepEqual(calibration.collapse.simultaneous, ['score-opened', 'frame-at-first-score', 'save-at-first-score']) + assert.equal(calibration.collapse.simultaneousAfter, 32) + + // Undeclared, the packaged contract does not pass the gate, and the two + // findings are reported together with their numbers. + assert.throws( + () => assertContractSeparates(calibration, opacity), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /states 1 of 6 milestone\(s\) that a resource running down earns/u) + assert.match(message, /attritionChecks: \['life-lost'\]/u) + assert.match(message, /collapses to one event: 6 of 6 milestone\(s\) require score-opened/u) + assert.match(message, /3 of 6 milestone\(s\) first pass at the same reference input \(after 32\)/u) + return true + }, + ) + + const declared = { ...opacity, attritionChecks: ['life-lost'], gatedBehind: 'score-opened' } // Whether Breakout separates is a fact about the ROM, not about this - // change: assert only that the legible comparison is the one being made. - const referenceLegible = calibration.reference.verified.filter((id) => calibration.legible.includes(id)).length + // change: assert only that the legible ACHIEVEMENT comparison is the one + // being made. + const referenceAchievement = calibration.reference.verified + .filter((id) => calibration.legible.includes(id) && calibration.progression.kinds[id] === 'achievement').length assert.equal( calibration.separates, - calibration.separating.length > 0 && referenceLegible > calibration.bestBaselineLegibleCount, + calibration.separating.length > 0 && referenceAchievement > calibration.bestBaselineAchievementCount, ) if (calibration.separates) assertContractSeparates(calibration, declared) else assert.throws(() => assertContractSeparates(calibration, declared), /does not separate/u) console.log( `ale: calibration — reference ${formatMilestoneScore(calibration.referenceScore)}, ` + + `achievements ${formatMilestoneScore(calibration.referenceAchievementScore)}, ` + `best trivial baseline ${calibration.bestBaselineLegibleCount} legible over ${calibration.turns} turns, ` + - `separating=${calibration.separating.join(',') || 'nothing'}, separates=${calibration.separates}`, + `separating=${calibration.separating.join(',') || 'nothing'}, ` + + `attrition-only=${calibration.attritionSeparating.join(',') || 'nothing'}, ` + + `separates=${calibration.separates}, collapses=${calibration.collapse.collapses}`, ) + // --- the ordering the split exists to fix ----------------------------- + // + // Two deterministic controls with the SAME control law and the same + // deadzone in screen pixels, differing only in what they read: the ASCII + // frame the agent sees, at four screen pixels per character, or the RAM + // channels the adapter publishes, at one pixel each. Neither carries state + // between decisions and neither costs a model call. + // + // The RAM control wins every column the game itself reports — score 9 to 6 + // at 300 decisions, 24 to 7 at 600 — and it never dies, while the ASCII + // control runs out of lives after 375. Under the whole contract the ASCII + // control still scores HIGHER, because `life-lost` is a point for dying. + // Under achievements alone the ordering is no longer backwards. + // + // The two achievement scores tie rather than separating, and the reason is + // the second finding this contract carries: its top achievement is + // `score >= 4`, so nothing in it can tell 7 from 24. + const PLAY_ROWS = [11, 20] as const + const PADDLE_ROWS = [21, 22] as const + const marked = (row: string): number[] => { + const columns: number[] = [] + for (let column = 2; column <= 37; column++) if ((row[column] ?? ' ') !== ' ') columns.push(column) + return columns + } + const steerTowards = (ball: number | null, paddle: number, deadzone: number): string => { + if (ball === null) return 'FIRE' + if (ball > paddle + deadzone) return 'RIGHT' + if (ball < paddle - deadzone) return 'LEFT' + return 'NOOP' + } + const steerFromAscii = (state: AleState): string => { + const rows = adapter.game.frame(state).split('\n') + let ball: number | null = null + for (let row = PLAY_ROWS[0]; row <= PLAY_ROWS[1] && ball === null; row++) { + const found = marked(rows[row] ?? '') + if (found.length > 0) ball = (found[0]! + found[found.length - 1]!) / 2 + } + const paddle = [...marked(rows[PADDLE_ROWS[0]] ?? ''), ...marked(rows[PADDLE_ROWS[1]] ?? '')] + if (paddle.length === 0) return 'FIRE' + return steerTowards(ball, (Math.min(...paddle) + Math.max(...paddle)) / 2, 1) + } + const steerFromRam = (state: AleState): string => { + const engine = (adapter.game.evidence(state).engineState ?? {}) as Record + const ball = (engine.ram_ball_x ?? 0) === 0 ? null : engine.ram_ball_x! + // The paddle is 20 RAM units wide, so its centre is its position plus 10, + // and 4 units is the 4-pixel character the ASCII control resolves. + return steerTowards(ball, (engine.ram_paddle_x ?? 0) + 10, 4) + } + + const readCounter = (frame: string, key: string): number => + Number(new RegExp(`${key}=(-?\\d+)`, 'u').exec(frame)?.[1] ?? -1) + const control = (id: string, decide: (state: AleState) => string, turns: number) => { + const tracker = new MilestoneTracker(adapter.contract) + let state = adapter.game.init(adapter.seed) + const inputs: string[] = [] + tracker.consider(adapter.game.evidence(state)) + for (let turn = 0; turn < turns && !isGameOver(adapter.game, state); turn++) { + const input = decide(state) + inputs.push(input) + state = adapter.game.step(state, input) + tracker.consider(adapter.game.evidence(state)) + } + const frame = adapter.game.frame(state) + return { + id, turns, inputs, + verified: tracker.verified(), + gameScore: readCounter(frame, 'score'), + lives: readCounter(frame, 'lives'), + played: inputs.length, + } + } + + const controls = [300, 600].flatMap((turns) => [ + control('steer-from-ascii', steerFromAscii, turns), + control('steer-from-ram', steerFromRam, turns), + ]) + // The classification is measured over the controls too, so the ordering + // claim is not made against a profile fitted to the reference alone. + const profile = measureProgressions(adapter.game, adapter.contract, { + trajectories: [adapter.reference, ...controls.map((run) => run.inputs)], + seed: adapter.seed, + }) + assert.deepEqual(profile.attrition, ['life-lost']) + + for (const run of controls) { + console.log( + ` ${run.id.padEnd(17)} @${String(run.turns).padStart(3)} decisions — ` + + `milestones ${formatMilestoneScore(scoreMilestones(adapter.contract, run.verified))}, ` + + `achievements ${formatMilestoneScore(scoreAchievements(adapter.contract, profile, run.verified))}, ` + + `game score ${run.gameScore}, lives ${run.lives}, played ${run.played}`, + ) + } + for (const turns of [300, 600]) { + const ascii = controls.find((run) => run.id === 'steer-from-ascii' && run.turns === turns)! + const ram = controls.find((run) => run.id === 'steer-from-ram' && run.turns === turns)! + // The RAM control plays better by the game's own report. + assert.ok(ram.gameScore > ascii.gameScore, `ram ${ram.gameScore} must beat ascii ${ascii.gameScore} at ${turns}`) + assert.ok(ram.lives > ascii.lives) + // Only the ASCII control dies, so only it earns the attrition milestone. + assert.ok(ascii.verified.includes('life-lost')) + assert.ok(!ram.verified.includes('life-lost')) + // The defect: the whole-contract score ranks the worse player first. + const wholeAscii = scoreMilestones(adapter.contract, ascii.verified) + const wholeRam = scoreMilestones(adapter.contract, ram.verified) + assert.ok(wholeAscii.verified > wholeRam.verified, + `the whole-contract score is expected to rank ascii first at ${turns}; if it no longer does, the reference moved`) + // The fix: under achievements alone it does not. + const achieveAscii = scoreAchievements(adapter.contract, profile, ascii.verified) + const achieveRam = scoreAchievements(adapter.contract, profile, ram.verified) + assert.ok(achieveRam.verified >= achieveAscii.verified, + `achievement score must not rank the control that died first at ${turns}: ` + + `ram ${formatMilestoneScore(achieveRam)}, ascii ${formatMilestoneScore(achieveAscii)}`) + } + // Determinism: two replays in this worker and one in a freshly spawned // worker must agree on every frame hash, every save-state hash, and every // privileged variable. Cross-process is the load-bearing case, because a diff --git a/authoring.ts b/authoring.ts index 689bac9..8427296 100644 --- a/authoring.ts +++ b/authoring.ts @@ -46,9 +46,15 @@ function checkHolds(check: MilestoneCheck, evidence: Evidence): boolean { * Blind-discovery marks are the sharp case, because nothing in the pipeline * ever asserts that a discovered channel means progress. * + * It also cannot know what a mark DEMONSTRATES. A mark on a life counter fires + * when the reference dies, and a contract that scores it rewards the run that + * played worse. + * * Run `calibrateContract` from calibration.ts on the derived contract and gate - * publication with `assertContractSeparates`. A contract that no trivial policy - * can satisfy is a benchmark; an uncalibrated one is a guess. + * publication with `PackagedContract.calibrate`, which refuses a contract that + * does not separate, an undeclared hash, an undeclared attrition milestone, and + * a contract every milestone of which requires one event. A contract that no + * trivial policy can satisfy is a benchmark; an uncalibrated one is a guess. */ export function deriveContract( game: Game, diff --git a/calibration.test.mts b/calibration.test.mts index 86eb28c..9719e82 100644 --- a/calibration.test.mts +++ b/calibration.test.mts @@ -21,6 +21,8 @@ import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract, + measureProgressions, + PackagedContract, trivialBaselines, UNKNOWN_BASELINE_WORD, } from './calibration' @@ -30,10 +32,11 @@ import { contractHash, contractLegibility, formatMilestoneScore, + scoreAchievements, scoreMilestones, } from './schema' import { makeNative2048, NATIVE_2048_INPUTS, NATIVE_2048_REFERENCE } from './adapters/native-2048' -import { engineCrawlerContract } from './adapters/engine-crawler' +import { engineCrawler, engineCrawlerContract, ENGINE_CRAWLER_REFERENCE } from './adapters/engine-crawler' import { saveLevels, saveLevelsContract, SAVE_LEVELS_REFERENCE } from './adapters/save-levels' import { screenPuzzle, screenPuzzleContract, SCREEN_PUZZLE_REFERENCE } from './adapters/screen-puzzle' @@ -132,7 +135,7 @@ const mashContract = deriveContract(mashGame, 0, [...MASH_REFERENCE], [ assert.deepEqual(report.trivial, ['moved']) assert.equal(report.bestBaselineCount, 1) assert.deepEqual(report.reference.verified, ['moved', 'lock-opened']) - assertContractSeparates(report) + assertContractSeparates(report, { gatedBehind: 'moved' }) // (d) the counts are internally consistent. assert.equal(report.turns, LOCK_REFERENCE.length) @@ -309,7 +312,7 @@ try { assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2') assert.equal(report.bestBaselineLegibleCount, 1) assert.equal(report.separates, true) - assertContractSeparates(report) + assertContractSeparates(report, { gatedBehind: 'moved' }) assertOpaqueChecksDeclared(report) } @@ -388,14 +391,17 @@ const lockOpaqueContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ // Declared, the same contract passes both gates. Nothing needs declaring as // weak, because the sweep found no other log that satisfies the hash. - assertContractSeparates(report, { opaqueChecks: ['frame-at-open'] }) + assertContractSeparates(report, { opaqueChecks: ['frame-at-open'], gatedBehind: 'moved' }) assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-at-open'] }) // A declaration is an exact set. A missing id and a stale id both fail, so a // hash milestone added by a later derivation cannot hide behind it. - assert.throws(() => assertContractSeparates(report, { opaqueChecks: [] }), /1 undeclared opaque milestone\(s\)/u) assert.throws( - () => assertContractSeparates(report, { opaqueChecks: ['frame-at-open', 'lock-opened'] }), + () => assertContractSeparates(report, { opaqueChecks: [], gatedBehind: 'moved' }), + /1 undeclared opaque milestone\(s\)/u, + ) + assert.throws( + () => assertContractSeparates(report, { opaqueChecks: ['frame-at-open', 'lock-opened'], gatedBehind: 'moved' }), /the contract does not state as a hash: lock-opened/u, ) @@ -679,4 +685,367 @@ const ledgeContract = deriveContract(ledgeWalk, 0, [...LEDGE_REFERENCE], [ } } +// --- what a milestone says about the run that earned it ---------------------- +// +// The pressure dive publishes three channels that move in three different +// ways, and the classification never reads their names. `hull` only ever +// cracks, so a milestone that needs it below its starting value is earned by +// letting the resource run down. `lives` only ever rises — it counts rescued +// divers — and it is named `lives` on purpose: a hardcoded list of resource +// names would classify it backwards, which is the mistake `episode-terminal` +// made one layer down when it matched the literal string `terminal` and +// reported `observed: false` for every adapter that spells it otherwise. + +const DIVE_VOCABULARY = ['down', 'up', 'wait'] +const DIVE_REFERENCE = [ + 'wait', 'down', 'wait', 'down', 'wait', 'down', + 'wait', 'down', 'wait', 'down', 'wait', 'down', +] + +interface DiveState { + depth: number + hull: number + /** Rescued divers. Named `lives`, and it only goes UP. */ + lives: number + equalized: number + steps: number +} + +const descend = (s: DiveState, repairs: boolean, input: string): DiveState => { + const steps = s.steps + 1 + if (input === 'wait') return { ...s, equalized: 1, steps } + if (repairs && input === 'up') return { ...s, hull: Math.min(s.hull + 1, 3), equalized: 0, steps } + if (input === 'down' && s.equalized === 1) { + const depth = s.depth + 1 + return { + depth, + hull: depth >= 4 ? s.hull - 1 : s.hull, + lives: depth >= 2 ? s.lives + 1 : s.lives, + equalized: 0, + steps, + } + } + return { ...s, equalized: 0, steps } +} + +const diveState = (): DiveState => ({ depth: 0, hull: 3, lives: 0, equalized: 0, steps: 0 }) +const diveEvidence = (s: DiveState) => ({ + engineState: { depth: s.depth, hull: s.hull, lives: s.lives, steps: s.steps }, +}) + +const pressureDive: Game = { + id: 'pressure-dive', + init: diveState, + step: (s, input) => descend(s, false, input), + frame: (s) => `depth ${s.depth} · hull ${s.hull} · rescued ${s.lives}`, + evidence: diveEvidence, +} + +/** + * The same game and the same contract, with one input added: `up` patches the + * hull. Nothing about the contract, the field name, or the check changes — only + * what the channel is measured doing. + */ +const repairableDive: Game = { + id: 'pressure-dive', + init: diveState, + step: (s, input) => descend(s, true, input), + frame: pressureDive.frame, + evidence: diveEvidence, +} + +const diveContract = deriveContract(pressureDive, 0, [...DIVE_REFERENCE], [ + { + id: 'depth-4', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.depth ?? 0) >= 4, + sample: (e) => ({ kind: 'state-path', path: 'depth', op: '>=', value: e.engineState?.depth ?? 4 }), + }, + { + id: 'lives-2', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.lives ?? 0) >= 2, + sample: (e) => ({ kind: 'state-path', path: 'lives', op: '>=', value: e.engineState?.lives ?? 2 }), + }, + { + id: 'hull-cracked', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.hull ?? 3) < 3, + sample: (e) => ({ kind: 'state-path', path: 'hull', op: '==', value: e.engineState?.hull ?? 2 }), + }, + { + id: 'depth-6', + tier: 'engine-state', + glitchClass: 'legal', + requires: ['depth-4'], + when: (e) => (e.engineState?.depth ?? 0) >= 6, + sample: (e) => ({ kind: 'state-path', path: 'depth', op: '>=', value: e.engineState?.depth ?? 6 }), + }, + { + id: 'steps-after-crack', + tier: 'engine-state', + glitchClass: 'legal', + requires: ['hull-cracked'], + when: (e) => (e.engineState?.steps ?? 0) >= 8, + sample: (e) => ({ kind: 'state-path', path: 'steps', op: '>=', value: e.engineState?.steps ?? 8 }), + }, +]) + +// (k) achievement and attrition, derived from measured motion. +{ + const report = calibrateContract(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + }) + const { progression } = report + assert.deepEqual(report.reference.verified, ['lives-2', 'depth-4', 'hull-cracked', 'steps-after-crack', 'depth-6']) + assert.deepEqual(progression.attrition, ['hull-cracked', 'steps-after-crack']) + assert.deepEqual(progression.achievement, ['depth-4', 'lives-2', 'depth-6']) + assert.deepEqual(progression.unmeasured, []) + assert.equal(progression.gameId, 'pressure-dive') + + // The channel named `lives` rises, so it is an achievement. The channel named + // `hull` only falls, so it is attrition. Neither name was read. + assert.match(progression.reasons['lives-2'] ?? '', /reads engineState\.lives, which rose \d+ time\(s\)/u) + assert.match( + progression.reasons['hull-cracked'] ?? '', + /reads engineState\.hull, which fell \d+ time\(s\) and never rose .*"== 2" does not hold at its initial value 3/u, + ) + // Attrition propagates through `requires`, because the tracker admits a + // milestone only after its prerequisites passed. + assert.equal(progression.reasons['steps-after-crack'], 'requires hull-cracked, which a resource running down earns') + + // Every channel the measurement watched, with its motion. + const motion = Object.fromEntries(progression.motion.map((row) => [row.channel, row])) + assert.deepEqual(Object.keys(motion).sort(), ['engineState.depth', 'engineState.hull', 'engineState.lives', 'engineState.steps']) + assert.equal(motion['engineState.hull']?.rises, 0) + assert.ok((motion['engineState.hull']?.falls ?? 0) > 0) + assert.equal(motion['engineState.hull']?.initial, 3) + assert.equal(motion['engineState.lives']?.falls, 0) + assert.equal(motion['engineState.depth']?.falls, 0) + + // An attrition milestone that no baseline reached is recorded, and it is not + // separation: "the reference lost hull and the baselines did not" says + // nothing about skill. + assert.deepEqual(report.separating, ['depth-4', 'depth-6']) + assert.deepEqual(report.attritionSeparating, ['hull-cracked', 'steps-after-crack']) + assert.equal(report.separates, true) + assert.deepEqual(report.referenceScore, { verified: 5, total: 5 }) + assert.deepEqual(report.referenceAchievementScore, { verified: 3, total: 3 }) + + // The gate refuses until the author has read the finding and written the ids + // down, and the declaration is an exact set like every other one. + assert.throws( + () => assertContractSeparates(report), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /states 2 of 5 milestone\(s\) that a resource running down earns \(2 undeclared\)/u) + assert.match(message, /attrition: hull-cracked/u) + assert.match(message, /the reference scored 5 of 5 over the whole contract and 3 of 3 over its achievements alone/u) + assert.match(message, /hull-cracked, steps-after-crack separated the reference from every baseline/u) + assert.match(message, /attritionChecks: \['hull-cracked', 'steps-after-crack'\]/u) + assert.doesNotMatch(message, /does not separate/u) + return true + }, + ) + assert.throws( + () => assertContractSeparates(report, { attritionChecks: ['hull-cracked'] }), + /states 2 of 5 milestone\(s\) that a resource running down earns \(1 undeclared\)/u, + ) + assert.throws( + () => assertContractSeparates(report, { attritionChecks: ['hull-cracked', 'steps-after-crack', 'depth-6'] }), + /attritionChecks names 1 milestone\(s\) the measurement did not classify as attrition: depth-6/u, + ) + assertContractSeparates(report, { attritionChecks: ['hull-cracked', 'steps-after-crack'] }) + + // The mutation that must flip the classification: the same contract, the same + // milestone, the same field name, on a game where one input patches the hull. + // One trajectory that repairs is enough — which is why the split is measured + // over the reference AND every baseline, not over the reference alone. + const REPAIR = ['wait', 'down', 'wait', 'down', 'wait', 'down', 'wait', 'down', 'up', 'up', 'wait', 'down'] + const never = measureProgressions(repairableDive, diveContract, { trajectories: [DIVE_REFERENCE] }) + assert.deepEqual(never.attrition, ['hull-cracked', 'steps-after-crack']) + const repaired = measureProgressions(repairableDive, diveContract, { trajectories: [DIVE_REFERENCE, REPAIR] }) + assert.deepEqual(repaired.attrition, []) + assert.deepEqual(repaired.achievement, ['depth-4', 'lives-2', 'hull-cracked', 'depth-6', 'steps-after-crack']) + assert.match(repaired.reasons['hull-cracked'] ?? '', /reads engineState\.hull, which rose 1 time\(s\)/u) + + // A milestone no numeric channel speaks for is an achievement, and it says so. + const opaqueProfile = measureProgressions(comboLock, lockOpaqueContract, { trajectories: [LOCK_REFERENCE] }) + assert.deepEqual(opaqueProfile.unmeasured, ['frame-at-open']) + assert.equal( + opaqueProfile.reasons['frame-at-open'], + 'its frame-hash check reads no numeric channel, so no resource motion could be measured', + ) + + // The inversion this whole split exists to stop, in miniature. Run A got + // further and broke; run B played better and did not. The whole-contract + // score ranks A first; the achievement score ranks B first. + const brokeThrough = ['lives-2', 'hull-cracked', 'steps-after-crack'] + const playedClean = ['depth-4', 'lives-2'] + assert.deepEqual(scoreMilestones(diveContract, brokeThrough), { verified: 3, total: 5 }) + assert.deepEqual(scoreMilestones(diveContract, playedClean), { verified: 2, total: 5 }) + assert.deepEqual(scoreAchievements(diveContract, progression, brokeThrough), { verified: 1, total: 3 }) + assert.deepEqual(scoreAchievements(diveContract, progression, playedClean), { verified: 2, total: 3 }) + + // A profile measured against another contract is refused rather than applied. + assert.throws( + () => scoreAchievements(lockContract, progression, ['moved']), + /progression profile is for "pressure-dive" but the contract is for "combo-lock"/u, + ) + assert.throws( + () => scoreAchievements(diveContract, { ...progression, kinds: {} }, []), + /does not classify depth-4, lives-2, hull-cracked, depth-6, steps-after-crack/u, + ) +} + +// (l) a contract that resolves runs on one event. +{ + const report = calibrateContract(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + }) + assert.deepEqual(report.collapse.prerequisite, 'moved') + assert.equal(report.collapse.gated, 2) + assert.equal(report.collapse.total, 2) + assert.deepEqual(report.collapse.gatedMilestones, ['moved', 'lock-opened']) + assert.equal(report.collapse.collapses, true) + assert.equal(report.collapse.earnedByReference, true) + assert.equal(report.collapse.earnedByBaseline.length, report.baselines.length) + assert.deepEqual(report.collapse.firstPassAt, { moved: 1, 'lock-opened': 8 }) + assert.deepEqual(report.collapse.simultaneous, []) + assert.equal(report.collapse.simultaneousAfter, -1) + + assert.throws( + () => assertContractSeparates(report), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /collapses to one event: 2 of 2 milestone\(s\) require moved/u) + assert.match(message, /gated by moved: lock-opened/u) + assert.match(message, /moved is earned by 9 of 9 trivial baseline\(s\)/u) + assert.match(message, /the whole contract opens for free/u) + assert.match(message, /gatedBehind: 'moved'/u) + return true + }, + ) + assert.throws( + () => assertContractSeparates(report, { gatedBehind: 'lock-opened' }), + /gatedBehind names lock-opened, which is not the milestone the contract hangs off \(moved gates 2 of 2\)/u, + ) + assertContractSeparates(report, { gatedBehind: 'moved' }) + + // A prerequisite no baseline reaches collapses the other way: a run that + // misses it scores nothing, however well it played. + const dive = calibrateContract(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + turns: 12, + }) + // The dive contract has two independent roots, so it does not collapse. + assert.equal(dive.collapse.collapses, false) + assert.equal(dive.collapse.gated, 2) + assert.equal(dive.collapse.total, 5) + + // The engine crawler states a milestone that requires nothing, so it is the + // shape a collapsed contract is measured against. + const crawler = calibrateContract(engineCrawler, engineCrawlerContract(), { + reference: ENGINE_CRAWLER_REFERENCE, + vocabulary: ['right', 'rest'], + }) + assert.equal(crawler.collapse.prerequisite, 'room-1') + assert.equal(crawler.collapse.gated, 3) + assert.equal(crawler.collapse.total, 4) + assert.equal(crawler.collapse.collapses, false) + + // Milestones that first pass at the same input are one event wearing two + // ids, and that is measured separately from `requires`. + const levels = calibrateContract(saveLevels, saveLevelsContract(), { + reference: SAVE_LEVELS_REFERENCE, + vocabulary: ['clear', 'grind'], + }) + assert.deepEqual(levels.collapse.simultaneous, ['level-2-saved', 'level-2-logged']) + assert.equal(levels.collapse.simultaneousAfter, 3) + assert.deepEqual(levels.collapse.firstPassAt, { 'level-2-saved': 3, 'level-2-logged': 3 }) +} + +// (m) a packaged contract carries the calibration that justified it. +{ + const packaged = PackagedContract.calibrate(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved' }, + }) + assert.equal(packaged.contract, lockContract) + assert.equal(packaged.hash, contractHash(lockContract)) + assert.equal(packaged.report.separates, true) + assert.deepEqual(packaged.report.separating, ['lock-opened']) + assert.deepEqual(packaged.declaration, { gatedBehind: 'moved' }) + + // A contract that does not separate cannot be packaged by omission. This is + // the Airstriker failure: `separates: false`, an empty separating set, and a + // published target anyway, because running calibration was optional. + const mash = { + reference: MASH_REFERENCE, + vocabulary: MASH_VOCABULARY, + } + assert.throws( + () => PackagedContract.calibrate(mashGame, mashContract, mash), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /does not separate/u) + assert.match(message, /nonSeparating: ''/u) + return true + }, + ) + + // The escape hatch is explicit, and it is a reason rather than a switch. + const demo = PackagedContract.calibrate(mashGame, mashContract, { + ...mash, + declare: { nonSeparating: 'tier demonstration: this target exercises the engine-state path, it does not grade play' }, + }) + assert.equal(demo.report.separates, false) + assert.equal(demo.report.bestBaselineCount, 1) + + assert.throws( + () => PackagedContract.calibrate(mashGame, mashContract, { ...mash, declare: { nonSeparating: ' ' } }), + /nonSeparating must state why this target is not meant to separate/u, + ) + + // And it is refused when it is stale, exactly as `opaqueChecks` is: a target + // that grew a real progression must not keep the excuse it shipped with. + assert.throws( + () => PackagedContract.calibrate(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved', nonSeparating: 'demonstration only' }, + }), + /nonSeparating says "demonstration only", but the contract separates: the reference reached lock-opened/u, + ) + + // Every other finding is refused here too, so packaging is the one gate an + // author cannot pass by calling a narrower one. + assert.throws( + () => PackagedContract.calibrate(comboLock, lockContract, { reference: LOCK_REFERENCE, vocabulary: LOCK_VOCABULARY }), + /collapses to one event/u, + ) + assert.throws( + () => PackagedContract.calibrate(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + }), + /that a resource running down earns/u, + ) + assert.throws( + () => PackagedContract.calibrate(comboLock, lockOpaqueContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved' }, + }), + /states 1 of 3 milestone\(s\) as a hash/u, + ) +} + console.log('playproof calibration: separating and non-separating contracts, legible/opaque split, opaque-collision sweep, policy determinism, edge cases OK') diff --git a/calibration.ts b/calibration.ts index c804f99..ccf3065 100644 --- a/calibration.ts +++ b/calibration.ts @@ -28,6 +28,28 @@ * perturbed logs still satisfy it. A hash a large family of logs satisfies is * a weak check, and the gate reports the measured number. * + * The third measurement is what a milestone says about the run that earned it. + * A milestone earned by a monotone-decreasing resource going down marks + * progress REACHED and never competence shown, because the shortest path to it + * is to play badly. `measureProgressions` derives that split from the motion of + * the channels the engine already publishes; it reads no field name, so an + * adapter that spells its life counter `shields` is classified on the same + * evidence as one that spells it `lives`. `separating` and `separates` count + * achievement milestones only, and the rest is recorded in + * `attritionSeparating`. + * + * The fourth is how much of a contract hangs off one milestone. `requires` is a + * partial order, so a contract can state six progressions and still demand + * exactly one event. `report.collapse` states the structure with the number, + * whether a trivial baseline reaches the prerequisite, and how many milestones + * first pass at one instant of the reference. + * + * `PackagedContract` closes the last hole. Running any of this was optional, + * and an optional gate is a gate nothing has to pass: the packaged Airstriker + * contract reported `separates: false` with an empty separating set and shipped + * anyway. `PackagedContract.calibrate` is the only way to build that type, and + * it runs the whole gate. + * * None of this touches replay attestation. The input-log hash chain is the * mechanism that proves a replay reproduced a recorded run, and it is * unaffected: a milestone hash was never that proof. @@ -37,9 +59,17 @@ */ import { attestRun, MilestoneTracker } from './attestation' import { logFrom } from './runtime' -import type { Game } from './runtime' -import { contractLegibility, formatMilestoneScore, scoreMilestones } from './schema' -import type { MilestoneContract, MilestoneScore } from './schema' +import type { Evidence, Game } from './runtime' +import { contractGate, contractHash, contractLegibility, formatMilestoneScore, scoreAchievements, scoreMilestones } from './schema' +import type { + ContractGate, + MilestoneCheck, + MilestoneContract, + MilestoneScore, + NumericOperator, + ProgressionKind, + ProgressionProfile, +} from './schema' /** * A deterministic input policy that needs no observation of the game. @@ -276,6 +306,265 @@ export function probeOpaqueCollisions( return rows.sort((a, b) => opaque.indexOf(a.milestone) - opaque.indexOf(b.milestone)) } +/** + * The evidence map each numeric check reads. A check outside this table names + * no numeric channel, so no resource motion can be measured for it. + */ +const NUMERIC_MAP_FOR_CHECK: Partial> = { + 'state-path': 'engineState', + 'save-path': 'saveState', + 'frame-path': 'frameState', +} + +/** `.` for a numeric check, or null when the check reads no channel. */ +function checkChannel(check: MilestoneCheck): string | null { + const map = NUMERIC_MAP_FOR_CHECK[check.kind] + if (map === undefined) return null + return `${map}.${'path' in check ? check.path : ''}` +} + +/** Every numeric channel one evidence snapshot publishes, flattened to `.`. */ +function channelValues(evidence: Evidence): Map { + const values = new Map() + for (const map of ['engineState', 'saveState', 'frameState'] as const) { + const fields = evidence[map] + if (fields === undefined) continue + for (const [key, value] of Object.entries(fields)) { + if (typeof value === 'number' && Number.isFinite(value)) values.set(`${map}.${key}`, value) + } + } + return values +} + +interface MutableMotion { + channel: string + initial: number | null + initialSeen: boolean + rises: number + falls: number + min: number + max: number + trajectories: number + samples: number +} + +function numericHolds(op: NumericOperator, value: number, expected: number): boolean { + if (op === '>=') return value >= expected + if (op === '>') return value > expected + return value === expected +} + +export interface ProgressionOptions { + /** + * Input scripts to replay and watch. The useful set is the reference plus + * every trivial baseline, which is what `calibrateContract` passes: a channel + * that only falls on the reference but rises under a baseline is not a + * resource, and one trajectory would not have shown it. + */ + trajectories: readonly (readonly string[])[] + /** Replay seed for every trajectory. Default 0. */ + seed?: number +} + +/** + * Classify every milestone of a contract as an achievement or as attrition, by + * watching how the channels move. + * + * The rule reads no field name. A milestone is attrition when all three of + * these measured statements hold: + * + * 1. its check reads a numeric channel of the evidence; + * 2. that channel never rose and fell at least once, over every snapshot of + * every measured trajectory — it is a resource that only runs down; + * 3. the check does not hold at the initial value of that channel, so the + * only way to earn it is to let the resource run down. + * + * Anything else is an achievement, including a hash, a log event, and a channel + * the trajectories never published: attrition is a positive finding, and the + * absence of one is not evidence of the opposite. Those milestones are listed + * in `unmeasured` so a reader sees where the measurement stopped. + * + * Attrition propagates through `requires`, because `MilestoneTracker` admits a + * milestone only after every prerequisite passed: a milestone gated behind a + * lost life needs a lost life, whatever its own check reads. + * + * A hardcoded list of names would have missed the same way `over()` once did. + * ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro + * `episodeDone`; a life counter is `lives` on Atari and something else on the + * next substrate. Motion is the thing every engine has in common. + */ +export function measureProgressions( + game: Game, + contract: MilestoneContract, + options: ProgressionOptions, +): ProgressionProfile { + const seed = options.seed ?? 0 + const motion = new Map() + const see = (channel: string, value: number, first: boolean): MutableMotion => { + let row = motion.get(channel) + if (row === undefined) { + row = { + channel, initial: null, initialSeen: false, rises: 0, falls: 0, + min: value, max: value, trajectories: 0, samples: 0, + } + motion.set(channel, row) + } + row.samples += 1 + row.min = Math.min(row.min, value) + row.max = Math.max(row.max, value) + if (first) { + row.trajectories += 1 + // The initial value must be one number for the classification to mean + // anything. A game whose init() is not pure disagrees with itself here, + // and every milestone on that channel falls back to unmeasured. + if (!row.initialSeen) { + row.initial = value + row.initialSeen = true + } else if (row.initial !== value) { + row.initial = null + } + } + return row + } + + for (const inputs of options.trajectories) { + let state = game.init(seed) + let previous = channelValues(game.evidence(state)) + for (const [channel, value] of previous) see(channel, value, true) + for (const input of inputs) { + state = game.step(state, input) + const current = channelValues(game.evidence(state)) + for (const [channel, value] of current) { + const row = see(channel, value, false) + const before = previous.get(channel) + if (before === undefined) continue + if (value > before) row.rises += 1 + if (value < before) row.falls += 1 + } + previous = current + } + } + + const kinds: Record = {} + const reasons: Record = {} + const unmeasured: string[] = [] + const runs = options.trajectories.length + for (const milestone of contract.milestones) { + const channel = checkChannel(milestone.check) + const row = channel === null ? undefined : motion.get(channel) + if (channel === null) { + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = `its ${milestone.check.kind} check reads no numeric channel, so no resource motion could be measured` + unmeasured.push(milestone.id) + continue + } + if (row === undefined || row.initial === null) { + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = row === undefined + ? `no measured trajectory published ${channel}, so no resource motion could be measured` + : `${channel} started at a different value on different trajectories, so no resource motion could be measured` + unmeasured.push(milestone.id) + continue + } + const check = milestone.check + const expected = 'value' in check ? check.value : 0 + const op: NumericOperator = 'op' in check ? check.op : '>=' + const holdsAtStart = numericHolds(op, row.initial, expected) + const onlyFalls = row.rises === 0 && row.falls > 0 + if (onlyFalls && !holdsAtStart) { + kinds[milestone.id] = 'attrition' + reasons[milestone.id] = + `its ${check.kind} check reads ${channel}, which fell ${row.falls} time(s) and never rose over ` + + `${row.trajectories} of ${runs} measured trajectory(ies); "${op} ${expected}" does not hold at its ` + + `initial value ${row.initial}, so the only way to earn it is to let the resource run down` + continue + } + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = onlyFalls + ? `its ${check.kind} check on ${channel} already holds at the initial value ${row.initial}, so it needs no resource to run down` + : row.rises > 0 + ? `its ${check.kind} check reads ${channel}, which rose ${row.rises} time(s) over ${row.trajectories} of ${runs} measured trajectory(ies)` + : `its ${check.kind} check reads ${channel}, which never moved over ${row.trajectories} of ${runs} measured trajectory(ies)` + } + + // Attrition propagates the way opacity does: the tracker admits a milestone + // only after every prerequisite passed. + const byId = new Map(contract.milestones.map((m) => [m.id, m])) + const visiting = new Set() + const inherit = (id: string): void => { + if (kinds[id] === 'attrition' || visiting.has(id)) return + visiting.add(id) + for (const required of byId.get(id)?.requires ?? []) { + if (!byId.has(required)) continue + inherit(required) + if (kinds[required] === 'attrition') { + kinds[id] = 'attrition' + reasons[id] = `requires ${required}, which a resource running down earns` + break + } + } + visiting.delete(id) + } + for (const milestone of contract.milestones) inherit(milestone.id) + + const order = contract.milestones.map((m) => m.id) + return { + gameId: contract.gameId, + kinds, + achievement: order.filter((id) => kinds[id] === 'achievement'), + attrition: order.filter((id) => kinds[id] === 'attrition'), + unmeasured: unmeasured.filter((id) => kinds[id] !== 'attrition'), + reasons, + motion: [...motion.values()].map((row) => ({ + channel: row.channel, + initial: row.initial, + rises: row.rises, + falls: row.falls, + min: row.min, + max: row.max, + trajectories: row.trajectories, + samples: row.samples, + })), + } +} + +/** + * How much of a contract hangs off one milestone, and who reaches that + * milestone. + * + * `ContractGate` states the structure; these fields add the measurement that + * says what the structure costs. A contract every milestone of which requires + * one event resolves runs into two classes and no more: those that reached the + * event, and those that scored zero. + */ +export interface ContractCollapse extends ContractGate { + /** Baseline ids that earned the prerequisite. Empty means no trivial policy reached it. */ + earnedByBaseline: string[] + /** Whether the reference itself earned the prerequisite. */ + earnedByReference: boolean + /** Reference inputs consumed before each milestone first passed; -1 when it never passed. */ + firstPassAt: Record + /** + * The largest set of milestones that first passed at the SAME reference + * input, in contract order. + * + * This is the second way a contract counts one event more than once, and it + * is independent of `requires`. Measured on the packaged contracts: ALE + * Breakout opens `score-opened`, `frame-at-first-score`, and + * `save-at-first-score` at input 32, and stable-retro Airstriker opens three + * of its five at one input too. Three milestone ids for one moment of play is + * three points of a score that a run either has all of or none of. + */ + simultaneous: string[] + /** The input index those milestones share, or -1 when no two share one. */ + simultaneousAfter: number + /** + * Whether every milestone of the contract requires one milestone. A contract + * of one milestone cannot collapse, because there is nothing to chain. + */ + collapses: boolean +} + export interface CalibrationReport { turns: number seed: number @@ -283,12 +572,23 @@ export interface CalibrationReport { reference: BaselineOutcome baselines: BaselineOutcome[] /** - * Legible milestones the reference reached and no baseline did — the whole - * of the contract's discriminating power. An opaque milestone is excluded - * even when no baseline earned it: a reader cannot see what it demands, so - * it cannot be the evidence that the contract measures anything. + * Legible ACHIEVEMENT milestones the reference reached and no baseline did — + * the whole of the contract's discriminating power. + * + * Two exclusions, for two different reasons. An opaque milestone is excluded + * even when no baseline earned it, because a reader cannot see what it + * demands. An attrition milestone is excluded even when no baseline earned + * it, because the run that earns it is the run that let a resource run down, + * and "the reference lost a life and the baselines did not" is not a claim + * about skill. Attrition milestones that would otherwise have separated are + * kept in `attritionSeparating`, so nothing measured is thrown away. */ separating: string[] + /** + * Milestones the reference reached and no baseline did, that a resource + * running down earns. They are recorded, never counted as separation. + */ + attritionSeparating: string[] /** milestones at least one baseline earned */ trivial: string[] /** contract milestones whose requirement a reader can read off the contract */ @@ -318,8 +618,23 @@ export interface CalibrationReport { * a reference must beat, because an opaque milestone cannot carry the claim. */ bestBaselineLegibleCount: number - /** the reference's own progress */ + /** the reference's own progress over the whole contract */ referenceScore: MilestoneScore + /** + * The strongest baseline's count over LEGIBLE ACHIEVEMENT milestones. This is + * the number a reference must beat, because neither an opaque milestone nor + * an attrition milestone can carry the claim. + */ + bestBaselineAchievementCount: number + /** the reference's progress over the achievement milestones alone */ + referenceAchievementScore: MilestoneScore + /** + * Which milestones demonstrate something and which a resource running down + * earns, measured over the reference and every baseline. + */ + progression: ProgressionProfile + /** How much of the contract hangs off one milestone, and who reaches it. */ + collapse: ContractCollapse separates: boolean } @@ -373,26 +688,72 @@ export function calibrateContract( } const referenceInputs = options.reference.slice(0, turns) - const reference = play('reference', referenceInputs) - const baselines = policies.map((policy) => { + const scripts = policies.map((policy) => { const inputs = policy.inputs(vocabulary, turns, seed) if (inputs.length !== turns) { throw new Error(`baseline ${policy.id} produced ${inputs.length} inputs for ${turns} turns`) } - return play(policy.id, inputs) + return { policy, inputs } + }) + const reference = play('reference', referenceInputs) + const baselines = scripts.map(({ policy, inputs }) => play(policy.id, inputs)) + + // The achievement/attrition split is measured over every trajectory this + // report already names, not over the reference alone: a channel that only + // falls under the reference but rises under one baseline is not a resource, + // and a single trajectory would not have shown it. + const progression = measureProgressions(game, contract, { + trajectories: [referenceInputs, ...scripts.map(({ inputs }) => inputs)], + seed, }) const earnedByBaseline = new Set(baselines.flatMap((b) => b.verified)) const order = contract.milestones.map((m) => m.id) const { legible, opaque, reasons } = contractLegibility(contract) const legibleSet = new Set(legible) - const separating = reference.verified.filter((id) => !earnedByBaseline.has(id) && legibleSet.has(id)) + const isAchievement = (id: string): boolean => progression.kinds[id] === 'achievement' + const outOfReach = reference.verified.filter((id) => !earnedByBaseline.has(id) && legibleSet.has(id)) + const separating = outOfReach.filter(isAchievement) + const attritionSeparating = reference.verified.filter((id) => !earnedByBaseline.has(id) && !isAchievement(id)) const trivial = order.filter((id) => earnedByBaseline.has(id)) const bestBaselineCount = baselines.reduce((best, b) => Math.max(best, b.verified.length), 0) - const legibleCount = (outcome: BaselineOutcome): number => outcome.verified.filter((id) => legibleSet.has(id)).length - const bestBaselineLegibleCount = baselines.reduce((best, b) => Math.max(best, legibleCount(b)), 0) + const countIn = (outcome: BaselineOutcome, keep: (id: string) => boolean): number => outcome.verified.filter(keep).length + const isLegible = (id: string): boolean => legibleSet.has(id) + const isLegibleAchievement = (id: string): boolean => isLegible(id) && isAchievement(id) + const bestBaselineLegibleCount = baselines.reduce((best, b) => Math.max(best, countIn(b, isLegible)), 0) + const bestBaselineAchievementCount = baselines.reduce((best, b) => Math.max(best, countIn(b, isLegibleAchievement)), 0) const referenceScore = scoreMilestones(contract, reference.verified) - const referenceLegible = legibleCount(reference) + const referenceAchievementScore = scoreAchievements(contract, progression, reference.verified) + const referenceAchievement = countIn(reference, isLegibleAchievement) + + const gate = contractGate(contract) + const firstPass = firstPassTurn(game, contract, seed, referenceInputs) + const firstPassAt: Record = {} + const sharing = new Map() + for (const id of order) { + const turn = firstPass.get(id) ?? -1 + firstPassAt[id] = turn + if (turn >= 0) sharing.set(turn, [...(sharing.get(turn) ?? []), id]) + } + let simultaneous: string[] = [] + let simultaneousAfter = -1 + for (const [turn, ids] of sharing) { + if (ids.length > simultaneous.length) { + simultaneous = ids + simultaneousAfter = turn + } + } + const collapse: ContractCollapse = { + ...gate, + earnedByBaseline: gate.prerequisite === null + ? [] + : baselines.filter((b) => b.verified.includes(gate.prerequisite!)).map((b) => b.id), + earnedByReference: gate.prerequisite !== null && reference.verified.includes(gate.prerequisite), + firstPassAt, + simultaneous: simultaneous.length > 1 ? simultaneous : [], + simultaneousAfter: simultaneous.length > 1 ? simultaneousAfter : -1, + collapses: gate.prerequisite !== null && gate.gated === gate.total && gate.total > 1, + } return { turns, @@ -401,6 +762,7 @@ export function calibrateContract( reference, baselines, separating, + attritionSeparating, trivial, legible, opaque, @@ -414,8 +776,12 @@ export function calibrateContract( }), bestBaselineCount, bestBaselineLegibleCount, + bestBaselineAchievementCount, referenceScore, - separates: separating.length > 0 && referenceLegible > bestBaselineLegibleCount, + referenceAchievementScore, + progression, + collapse, + separates: separating.length > 0 && referenceAchievement > bestBaselineAchievementCount, } } @@ -438,6 +804,44 @@ export interface OpacityDeclaration { weakChecks?: readonly string[] } +/** + * Everything a contract author must have read and accepted before the contract + * is published, beyond the opaque checks. + * + * Every field is an exact set or an exact id, never a switch, for the reason + * `opaqueChecks` is: a finding that a later derivation introduces must not + * enter a published contract under a declaration written for an earlier one. + * A stale declaration fails the gate as loudly as a missing one. + */ +export interface ContractDeclaration extends OpacityDeclaration { + /** + * Milestone ids the author accepts as attrition: a resource running down + * earns them, so they are recorded and never scored as competence. + */ + attritionChecks?: readonly string[] + /** + * The milestone the author accepts the whole contract hangs off. Declaring it + * states that one event decides whether a run scores anything at all. + */ + gatedBehind?: string +} + +/** + * What `PackagedContract.calibrate` accepts on top of the contract declaration. + */ +export interface PackageDeclaration extends ContractDeclaration { + /** + * Why this target is not meant to separate, in the author's words. + * + * A tier demonstration and a smoke fixture exist to exercise the machinery, + * not to grade an agent, and forcing them to separate would mean inventing a + * difficulty they do not have. The field is a sentence rather than a boolean + * so the reason travels with the package, and it is refused when the contract + * DOES separate, so it cannot sit in a target that outgrew it. + */ + nonSeparating?: string +} + function earners(report: CalibrationReport, milestone: string): string { return report.baselines.filter((b) => b.verified.includes(milestone)).map((b) => b.id).join(', ') } @@ -536,6 +940,124 @@ function opacityProblems(report: CalibrationReport, declaration: OpacityDeclarat return problems } +/** + * Every way a contract's progression split can be wrong, as message blocks. + * An empty array means every attrition milestone is declared. + */ +function progressionProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + const problems: string[] = [] + const { attrition, reasons } = report.progression + const declared = declaration.attritionChecks + const accepted = new Set(declared ?? []) + const undeclared = attrition.filter((id) => !accepted.has(id)) + const { total } = report.referenceScore + if (undeclared.length > 0) { + problems.push( + [ + `contract states ${attrition.length} of ${total} milestone(s) that a resource running down earns ` + + `(${undeclared.length} undeclared):`, + ...undeclared.map((id) => ` attrition: ${id} — ${reasons[id]}`), + ` the reference scored ${formatMilestoneScore(report.referenceScore)} over the whole contract and ` + + `${formatMilestoneScore(report.referenceAchievementScore)} over its achievements alone`, + ...(report.attritionSeparating.length > 0 + ? [ + ` ${report.attritionSeparating.join(', ')} separated the reference from every baseline, and no ` + + 'longer counts towards separation', + ] + : []), + 'Such a milestone marks progress REACHED and never competence shown: the shortest path to it is to play badly.', + `Keep it and declare it — { attritionChecks: [${undeclared.map((id) => `'${id}'`).join(', ')}] } — so it is ` + + 'recorded and scored by scoreMilestones but not by scoreAchievements, or pin a progression that a ' + + 'better run reaches.', + ].join('\n'), + ) + } + const stale = [...accepted].filter((id) => !attrition.includes(id)) + if (stale.length > 0) { + problems.push( + `attritionChecks names ${stale.length} milestone(s) the measurement did not classify as attrition: ` + + `${stale.join(', ')} — the declaration is stale, and it would hide an attrition milestone measured later`, + ) + } + return problems +} + +/** + * Whether the contract resolves runs on one event, as message blocks. + * An empty array means nothing collapses, or the author declared the structure. + */ +function collapseProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + const problems: string[] = [] + const { collapse } = report + const declared = declaration.gatedBehind + if (collapse.collapses && declared !== collapse.prerequisite) { + const free = collapse.earnedByBaseline + problems.push( + [ + `contract collapses to one event: ${collapse.gated} of ${collapse.total} milestone(s) require ` + + `${collapse.prerequisite}, so the contract resolves a run into two classes and no more`, + ` gated by ${collapse.prerequisite}: ${collapse.gatedMilestones.filter((id) => id !== collapse.prerequisite).join(', ')}`, + free.length > 0 + ? ` ${collapse.prerequisite} is earned by ${free.length} of ${report.baselines.length} trivial baseline(s) ` + + `(${free.join(', ')}), so the whole contract opens for free and grades only what follows a free event` + : ` ${collapse.prerequisite} is out of reach of all ${report.baselines.length} trivial baseline(s), so a run ` + + `that misses it scores 0 of ${collapse.total} however well it played`, + ...(collapse.simultaneous.length > 1 + ? [ + ` ${collapse.simultaneous.length} of ${collapse.total} milestone(s) first pass at the same reference ` + + `input (after ${collapse.simultaneousAfter}): ${collapse.simultaneous.join(', ')} — one moment of play ` + + 'wearing several milestone ids', + ] + : []), + 'A contract that grades on one event has one bit of resolution, whatever its milestone count says.', + `Pin a progression that does not require ${collapse.prerequisite}, or declare the structure — ` + + `{ gatedBehind: '${collapse.prerequisite}' }.`, + ].join('\n'), + ) + } + if (declared !== undefined && declared !== collapse.prerequisite) { + problems.push( + `gatedBehind names ${declared}, which ${collapse.prerequisite === null + ? 'no milestone of this contract requires' + : `is not the milestone the contract hangs off (${collapse.prerequisite} gates ${collapse.gated} of ${collapse.total})` + } — the declaration is stale`, + ) + } + return problems +} + +/** Every finding an author must have read, whatever gate they called. */ +function contractProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + return [ + ...opacityProblems(report, declaration), + ...progressionProblems(report, declaration), + ...collapseProblems(report, declaration), + ] +} + +/** The message block for a contract a trivial policy keeps up with. */ +function separationProblem(report: CalibrationReport): string { + const best = report.baselines.filter((b) => b.verified.length === report.bestBaselineCount).map((b) => b.id) + const referenceLegible = report.reference.verified.filter((id) => report.legible.includes(id)).length + const legibleAchievements = report.legible.filter((id) => report.progression.kinds[id] === 'achievement') + const referenceAchievement = report.reference.verified.filter((id) => legibleAchievements.includes(id)).length + return [ + `contract does not separate: the reference verified ${report.reference.verified.length} milestone(s) ` + + `and the best trivial baseline verified ${report.bestBaselineCount} over ${report.turns} turns ` + + `(seed ${report.seed}, strongest: ${best.join(', ') || 'none'})`, + ` on legible milestones alone: reference ${referenceLegible}, ` + + `best baseline ${report.bestBaselineLegibleCount}, of ${report.legible.length} legible`, + ` on legible achievements alone: reference ${referenceAchievement}, ` + + `best baseline ${report.bestBaselineAchievementCount}, of ${legibleAchievements.length} legible achievement(s)`, + ...report.trivial.map((id) => ` trivial: ${id} — earned by ${earners(report, id)}`), + ...report.attritionSeparating.map( + (id) => ` out of reach of every baseline but not evidence of skill: ${id} — ${report.progression.reasons[id]}`, + ), + ` legible, an achievement, and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, + 'A derived contract is a hypothesis until it separates. Pin a progression a trivial policy cannot reach.', + ].join('\n') +} + /** * Fail closed on a contract whose milestones a reader cannot understand. * @@ -571,24 +1093,91 @@ export function assertOpaqueChecksDeclared( */ export function assertContractSeparates( report: CalibrationReport, - declaration: OpacityDeclaration = {}, + declaration: ContractDeclaration = {}, ): void { - const problems = opacityProblems(report, declaration) - if (!report.separates) { - const best = report.baselines.filter((b) => b.verified.length === report.bestBaselineCount).map((b) => b.id) - const referenceLegible = report.reference.verified.filter((id) => report.legible.includes(id)).length - problems.push( - [ - `contract does not separate: the reference verified ${report.reference.verified.length} milestone(s) ` + - `and the best trivial baseline verified ${report.bestBaselineCount} over ${report.turns} turns ` + - `(seed ${report.seed}, strongest: ${best.join(', ') || 'none'})`, - ` on legible milestones alone: reference ${referenceLegible}, ` + - `best baseline ${report.bestBaselineLegibleCount}, of ${report.legible.length} legible`, - ...report.trivial.map((id) => ` trivial: ${id} — earned by ${earners(report, id)}`), - ` legible and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, - 'A derived contract is a hypothesis until it separates. Pin a progression a trivial policy cannot reach.', - ].join('\n'), - ) - } + const problems = contractProblems(report, declaration) + if (!report.separates) problems.push(separationProblem(report)) if (problems.length > 0) throw new Error(problems.join('\n')) } + +/** + * A contract that has passed the gate, with the numbers that let it through. + * + * The type exists because running calibration was optional, and an optional + * gate is a gate nothing has to pass. Measured on stable-retro Airstriker: the + * packaged contract reported `separates: false` with an empty separating set, + * and it shipped anyway, because nothing in the path from `deriveContract` to a + * published target ever asked. A screen-blind `B, NOOP, B, NOOP` program earned + * all four legible milestones, the same four the reference earned. + * + * There is one way to build this value, and it runs the gate. The class carries + * a private field, so an object literal of the right shape is not assignable to + * it, and the constructor is private, so `new` is not available either. A + * target that hands out a `PackagedContract` has been calibrated; a target that + * hands out a bare `MilestoneContract` has not, and now says so in its type. + * + * The report travels with the contract on purpose. A published number should + * carry the measurement that justified it, and a reader of a target can ask + * what the baselines scored without re-running an emulator. + */ +export class PackagedContract { + /** + * Nominal brand. It has no other purpose: a private field makes an object + * literal of the same shape unassignable, so this type cannot be forged. + */ + readonly #calibrated = true + + private constructor( + /** The contract itself, byte for byte as it was derived. */ + readonly contract: MilestoneContract, + /** `contractHash(contract)` at the moment the gate passed. */ + readonly hash: string, + /** Every number the gate read. */ + readonly report: CalibrationReport, + /** What the author read and accepted. */ + readonly declaration: PackageDeclaration, + ) {} + + /** + * Calibrate a contract and refuse to package it unless it is justified. + * + * Every finding `assertContractSeparates` reports is refused here too: + * undeclared opaque checks, an opaque check a baseline reproduced, a hash a + * family of logs satisfies, an undeclared attrition milestone, and a contract + * every milestone of which requires one event. + * + * `nonSeparating` is the escape hatch, and it is the same shape as + * `opaqueChecks`: an explicit statement, refused when it is stale. A target + * that declares itself non-separating and then separates fails, because the + * declaration would otherwise outlive the reason for it. + */ + static calibrate( + game: Game, + contract: MilestoneContract, + options: CalibrateOptions & { declare?: PackageDeclaration }, + ): PackagedContract { + const declaration = options.declare ?? {} + const report = calibrateContract(game, contract, options) + const problems = contractProblems(report, declaration) + const excuse = declaration.nonSeparating + if (excuse === undefined) { + if (!report.separates) { + problems.push( + `${separationProblem(report)}\n` + + 'A target that is not meant to separate — a tier demonstration, a smoke fixture — says so: ' + + "{ nonSeparating: '' }.", + ) + } + } else if (excuse.trim().length === 0) { + problems.push('nonSeparating must state why this target is not meant to separate; an empty reason declares nothing') + } else if (report.separates) { + problems.push( + `nonSeparating says "${excuse}", but the contract separates: the reference reached ` + + `${report.separating.join(', ')} and no trivial baseline did — the declaration is stale, and it ` + + 'would hide a later regression that stopped it separating', + ) + } + if (problems.length > 0) throw new Error(problems.join('\n')) + return new PackagedContract(contract, contractHash(contract), report, declaration) + } +} diff --git a/gymnasium.test.mts b/gymnasium.test.mts index 973427f..ca23939 100644 --- a/gymnasium.test.mts +++ b/gymnasium.test.mts @@ -15,6 +15,7 @@ import { strict as assert } from 'node:assert' import { spawnSync } from 'node:child_process' import { tmpdir } from 'node:os' import { attestRun } from './attestation' +import { calibrateContract, PackagedContract } from './calibration' import { playEpisode, scriptedDriver } from './episode' import { logFrom } from './runtime' import { validateContract } from './schema' @@ -206,6 +207,73 @@ if (!pythonHasGymnasium()) { second = makeGymnasium({ envId: FROZENLAKE }) assert.deepEqual(trace(second, lake.reference), lakeFirst, 'frozenlake cross-process replay diverged') + // Calibration on both packaged Gymnasium contracts. + // + // CartPole separates: a random walk drops the pole long before 50 steps. + // Neither contract carries an attrition milestone, which is the negative + // control for the split — a channel that only falls is a finding, and + // these environments publish none. + const pole = calibrateContract(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + }) + assert.equal(pole.separates, true) + assert.deepEqual(pole.separating, ['survived-50-steps', 'reward-at-50-steps']) + assert.deepEqual(pole.trivial, ['survived-25-steps', 'reward-at-25-steps']) + assert.deepEqual(pole.progression.attrition, []) + assert.deepEqual(pole.progression.unmeasured, ['frame-at-25-steps']) + assert.deepEqual(pole.referenceAchievementScore, { verified: 5, total: 5 }) + + // It still collapses: every milestone requires `survived-25-steps`, which + // two baselines reach, and three of the five open at that same step. + assert.equal(pole.collapse.prerequisite, 'survived-25-steps') + assert.equal(pole.collapse.gated, 5) + assert.equal(pole.collapse.collapses, true) + assert.deepEqual(pole.collapse.earnedByBaseline, ['round-robin', 'pseudo-random']) + assert.deepEqual(pole.collapse.simultaneous, ['survived-25-steps', 'reward-at-25-steps', 'frame-at-25-steps']) + assert.equal(pole.collapse.simultaneousAfter, 25) + + const poleDeclared = { + opaqueChecks: ['frame-at-25-steps'], + gatedBehind: 'survived-25-steps', + } + assert.throws(() => PackagedContract.calibrate(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + }), /collapses to one event/u) + const packagedPole = PackagedContract.calibrate(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + declare: poleDeclared, + }) + assert.equal(packagedPole.report.separates, true) + + // FrozenLake is the sharpest packaged contract: no baseline reaches the + // goal at all, so its three milestones are all out of reach. All three + // also fire at the same instant, which is what a goal-only contract is. + const frozen = calibrateContract(frozenLake.game, frozenLake.contract, { + reference: frozenLake.reference, + vocabulary: frozenLake.inputs, + seed: frozenLake.seed, + }) + assert.equal(frozen.separates, true) + assert.deepEqual(frozen.trivial, []) + assert.deepEqual(frozen.separating, ['reached-goal', 'goal-cell']) + assert.deepEqual(frozen.progression.attrition, []) + assert.equal(frozen.collapse.collapses, true) + assert.deepEqual(frozen.collapse.simultaneous, ['reached-goal', 'goal-frame', 'goal-cell']) + assert.deepEqual(frozen.collapse.earnedByBaseline, []) + console.log( + `gymnasium: calibration — ${CARTPOLE} separating=${pole.separating.join(',')} ` + + `(best baseline ${pole.bestBaselineAchievementCount} achievements, collapses=${pole.collapse.collapses}); ` + + `${FROZENLAKE} separating=${frozen.separating.join(',')} ` + + `(best baseline ${frozen.bestBaselineAchievementCount}, collapses=${frozen.collapse.collapses}); ` + + 'no attrition milestone in either', + ) + // Unsupported action spaces fail at boot with a readable message. assert.throws( () => makeGymnasium({ envId: 'Pendulum-v1', reference: { ...bundledReference(CARTPOLE)!, envId: 'Pendulum-v1' } }), diff --git a/schema.ts b/schema.ts index e199b43..0ebdb05 100644 --- a/schema.ts +++ b/schema.ts @@ -226,3 +226,174 @@ export function scoreMilestones(contract: MilestoneContract, verified: readonly export function formatMilestoneScore(score: MilestoneScore): string { return `${score.verified} of ${score.total}` } + +/** + * What a milestone says about the run that earned it. + * + * `achievement` — the run made something happen. A score threshold, a level + * reached, a state a player has to play towards. + * + * `attrition` — the run let a resource run down. Lives, health, shields, time + * remaining. Such a milestone marks progress REACHED and never competence + * shown, because the shortest path to it is to play badly. Measured on ALE + * Breakout: the packaged `life-lost` milestone is `lives == 4`, so a program + * that never dies cannot score it, and two hand-written controls that differ + * only in how well they steer rank in the wrong order because of it. + * + * Recording an attrition milestone is legitimate — "the reference got far + * enough to lose a life" is a real fact about a trajectory. Scoring it as + * competence is not, which is why `scoreAchievements` exists next to + * `scoreMilestones`. + */ +export type ProgressionKind = 'achievement' | 'attrition' + +/** + * How one numeric evidence channel moved across the measured trajectories. + * + * A channel is one key of one evidence map, named `engineState.lives` or + * `frameState.activeCells`. The counts are over every step of every measured + * trajectory, so `rises` 0 with `falls` above 0 is the measured statement + * "this channel is a resource that only runs down". + */ +export interface ChannelMotion { + /** `.`, for example `engineState.lives`. */ + channel: string + /** Value at the initial state, or null when the trajectories disagreed. */ + initial: number | null + /** Steps where the value was greater than the step before. */ + rises: number + /** Steps where the value was less than the step before. */ + falls: number + min: number + max: number + /** Trajectories that published the channel at least once. */ + trajectories: number + /** Snapshots the counts are over. */ + samples: number +} + +/** + * Every milestone of one contract, split by what earning it demonstrates. + * + * The split is MEASURED, never declared in the contract, for the same reason + * legibility is derived: an existing contract keeps its bytes and its hash, and + * an author cannot forget to set it. `measureProgressions` in calibration.ts + * produces this from the motion of the channels the engine already publishes. + * Nothing here reads a field name, so an adapter that spells its life counter + * `shields` or `hull` is classified on the same evidence as one that spells it + * `lives`. + */ +export interface ProgressionProfile { + gameId: string + /** Every milestone of the contract, classified. */ + kinds: Record + /** Milestones that demonstrate something, in contract order. */ + achievement: string[] + /** Milestones earned by a resource going down, in contract order. */ + attrition: string[] + /** + * Milestones no numeric channel could speak for — a hash, a log event, or a + * path the measured trajectories never published. They count as achievements, + * because attrition is a positive finding and this is the absence of one. + * The list exists so a reader sees the limit of the measurement. + */ + unmeasured: string[] + /** Why each milestone is classified as it is, keyed by milestone id. */ + reasons: Record + /** Every channel the measurement watched, in first-seen order. */ + motion: ChannelMotion[] +} + +function assertProfileMatches(contract: MilestoneContract, profile: ProgressionProfile): void { + if (profile.gameId !== contract.gameId) { + throw new Error(`progression profile is for "${profile.gameId}" but the contract is for "${contract.gameId}"`) + } + const missing = contract.milestones.filter((m) => profile.kinds[m.id] === undefined).map((m) => m.id) + if (missing.length > 0) { + throw new Error(`progression profile does not classify ${missing.join(', ')} — re-measure it against this contract`) + } +} + +/** + * Score a run over the achievement milestones alone. + * + * Both the numerator and the denominator drop the attrition milestones, so two + * runs compared on this number are compared on what they made happen. A run + * that also lost a life is neither rewarded nor punished for it. Read + * `scoreMilestones` for the whole contract when the question is how far the run + * got rather than how well it played. + */ +export function scoreAchievements( + contract: MilestoneContract, + profile: ProgressionProfile, + verified: readonly string[], +): MilestoneScore { + assertProfileMatches(contract, profile) + const achievement = new Set(contract.milestones.map((m) => m.id).filter((id) => profile.kinds[id] === 'achievement')) + return { verified: verified.filter((id) => achievement.has(id)).length, total: achievement.size } +} + +/** + * How much of a contract hangs off one milestone. + * + * `requires` is a partial order, so a contract can state five progressions and + * still demand exactly one event: if every other milestone requires the first, + * a run that misses the first scores zero however well it played. That is one + * bit of resolution wearing five milestones. Measured on the packaged + * contracts: ALE Breakout gates 6 of 6 milestones behind `score-opened`, and + * stable-retro Airstriker gates 5 of 5 behind its own `score-opened`. + * + * This structure says nothing about how hard the prerequisite is. Whether a + * trivial baseline earns it is a measurement, and `calibrateContract` reports + * that as `collapse`. + */ +export interface ContractGate { + /** The milestone the largest share of the contract requires, transitively. */ + prerequisite: string | null + /** Milestones that require it, transitively, counting the prerequisite itself. */ + gated: number + /** Those milestone ids, in contract order. */ + gatedMilestones: string[] + total: number +} + +/** + * The milestone the largest share of a contract depends on. + * + * Ties are broken by contract order, so the answer is stable. A contract where + * no milestone requires another has no prerequisite and `gated` 0: nothing is + * chained, so nothing can collapse. A missing or cyclic requirement is not + * judged here — `validateContract` reports those, and the walk below stops on a + * cycle instead of looping. + */ +export function contractGate(contract: MilestoneContract): ContractGate { + const total = contract.milestones.length + const byId = new Map(contract.milestones.map((m) => [m.id, m])) + const closure = new Map>() + const requiredBy = (id: string, seen: Set): Set => { + const cached = closure.get(id) + if (cached !== undefined) return cached + if (seen.has(id)) return new Set() + seen.add(id) + const all = new Set() + for (const required of byId.get(id)?.requires ?? []) { + if (!byId.has(required)) continue + all.add(required) + for (const deeper of requiredBy(required, seen)) all.add(deeper) + } + seen.delete(id) + closure.set(id, all) + return all + } + + let best: ContractGate = { prerequisite: null, gated: 0, gatedMilestones: [], total } + for (const candidate of contract.milestones) { + const dependents = contract.milestones + .filter((m) => m.id === candidate.id || requiredBy(m.id, new Set()).has(candidate.id)) + .map((m) => m.id) + if (dependents.length > best.gated && dependents.length > 1) { + best = { prerequisite: candidate.id, gated: dependents.length, gatedMilestones: dependents, total } + } + } + return best +} diff --git a/stable-retro.test.mts b/stable-retro.test.mts index 81b1fb2..f28175a 100644 --- a/stable-retro.test.mts +++ b/stable-retro.test.mts @@ -16,9 +16,10 @@ import { spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' import { tmpdir } from 'node:os' import { attestRun } from './attestation' +import { calibrateContract, PackagedContract } from './calibration' import { logFrom, observationOf } from './runtime' import { decodePng, unscale } from './test-png.mts' -import { validateContract } from './schema' +import { formatMilestoneScore, validateContract } from './schema' import { RetroRpc } from './adapters/retro-rpc' import { bundledReference, makeStableRetro, type RetroState, type StableRetro } from './adapters/stable-retro' @@ -183,6 +184,84 @@ if (!pythonHasRetro()) { vision.dispose() } + // Calibration on the packaged Airstriker contract. + // + // Nothing in the path from `deriveContract` to a published target used to + // ask for this, and the answer is that the contract does not separate: the + // separating set is EMPTY. A seeded pseudo-random walk over the 25 advertised + // button words earns three of the four legible milestones, the same three + // the reference earns, and the fourth is a hash. Reporting it is the point: + // a contract that grades nothing must not look like one that grades play. + // The sweep is capped at 8 probed turns here. Genesis is the slowest + // substrate in the suite and this console advertises 25 input words, so a + // full 32-turn sweep costs 24,000 emulator steps for a number the first + // eight turns already establish. + const calibration = calibrateContract(adapter.game, adapter.contract, { + reference: adapter.reference, + vocabulary: adapter.inputs, + seed: adapter.seed, + collisionTurns: 8, + }) + assert.deepEqual(calibration.reference.verified, adapter.contract.milestones.map((m) => m.id)) + assert.deepEqual(calibration.separating, []) + assert.equal(calibration.separates, false) + assert.equal(calibration.bestBaselineAchievementCount, 3) + assert.deepEqual(calibration.trivial, ['score-opened', 'screen-active-at-first-score', 'score-tier-2']) + + // `life-lost` is `lives == 2` from a start of 3: earned by dying, on a + // channel measured falling 18 times and never rising across 29 replayed + // trajectories. It is the one milestone no baseline reached, and before + // this split it was the contract's whole claim to discriminating power. + assert.deepEqual(calibration.progression.attrition, ['life-lost']) + assert.deepEqual(calibration.attritionSeparating, ['life-lost']) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.lives')?.rises, 0) + + // Every milestone requires `score-opened`, which two baselines earn, so the + // whole contract opens for free; and three of the five open at one instant. + assert.equal(calibration.collapse.prerequisite, 'score-opened') + assert.equal(calibration.collapse.gated, 5) + assert.equal(calibration.collapse.total, 5) + assert.equal(calibration.collapse.collapses, true) + assert.deepEqual(calibration.collapse.earnedByBaseline, ['round-robin', 'pseudo-random']) + assert.deepEqual( + calibration.collapse.simultaneous, + ['score-opened', 'frame-at-first-score', 'screen-active-at-first-score'], + ) + + // The hash is weak by a wide margin: a large family of logs reaches the + // state it names, measured rather than assumed. + const collision = calibration.collisions[0]! + assert.equal(collision.milestone, 'frame-at-first-score') + assert.ok(collision.collisions > 0, 'the sweep found no colliding log for the pinned frame') + assert.equal(collision.jointCollision, true) + + // A packaged contract cannot be published without a verdict, and this one + // has no verdict to publish. + assert.throws( + () => PackagedContract.calibrate(adapter.game, adapter.contract, { + reference: adapter.reference, + vocabulary: adapter.inputs, + seed: adapter.seed, + collisionTurns: 8, + declare: { + opaqueChecks: ['frame-at-first-score'], + weakChecks: ['frame-at-first-score'], + attritionChecks: ['life-lost'], + gatedBehind: 'score-opened', + }, + }), + /does not separate/u, + ) + console.log( + `stable-retro: calibration — reference ${formatMilestoneScore(calibration.referenceScore)}, ` + + `achievements ${formatMilestoneScore(calibration.referenceAchievementScore)}, ` + + `best baseline ${calibration.bestBaselineCount} over ${calibration.turns} turns, ` + + `separating=${calibration.separating.join(',') || 'NOTHING'}, ` + + `attrition-only=${calibration.attritionSeparating.join(',') || 'nothing'}, ` + + `separates=${calibration.separates}, collapses=${calibration.collapse.collapses}, ` + + `${collision.collisions} of ${collision.substitutions} substitutions reproduce the pinned frame`, + ) + // Teardown: dispose kills the worker and every later call fails loudly // instead of silently reading a dead transport. second.dispose()