diff --git a/CHANGELOG.md b/CHANGELOG.md index 54faaf0..975d5c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,66 @@ All notable changes to Playproof are documented here. +## Unreleased + +### A milestone earned by dying is not evidence of skill + +- **The measurement.** ALE Breakout, ale-py 0.12.1, seed 0. Two deterministic controls with the same control law and the same deadzone in screen pixels, differing only in what they read: the ASCII frame at four screen pixels per character, or the `ram_ball_x`/`ram_paddle_x` channels at one pixel each. Neither carries state between decisions and neither costs a model call. + + | control | game score @300 | @600 | lives left | milestones | achievements | + |---|---|---|---|---|---| + | `steer-from-ascii` | 6 | 7 | 0 (dead after 375) | **4 of 6** | 3 of 5 | + | `steer-from-ram` | **9** | **24** | **5** | 3 of 6 | 3 of 5 | + + The RAM control wins every column the game itself reports and never dies. It scored one milestone LOWER, because the packaged `life-lost` milestone is `lives == 4`: a point for dying. A program that never dies capped at 3 of 6 at any horizon. +- **`ProgressionKind` is `achievement` or `attrition`.** A milestone is attrition when a resource running down earns it. It marks progress REACHED and never competence shown, because the shortest path to it is to play badly. Recording it is legitimate; scoring it as competence is not. +- **The split is measured, not declared, and it reads no field name.** `measureProgressions` watches every numeric channel the evidence publishes across the reference and every baseline, and calls a milestone attrition when three measured statements hold: its check reads a numeric channel; that channel never rose and fell at least once, over every snapshot of every trajectory; and the check does not hold at the initial value of that channel. Attrition propagates through `requires`, because `MilestoneTracker` admits a milestone only after its prerequisites passed. +- A hardcoded `lives`/`health`/`shields` list would have failed the way a name match already failed one layer down: ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro `episodeDone`. A fixture channel named `lives` that counts rescued divers only rises, and the measurement classifies it as the achievement it is. +- A milestone no numeric channel speaks for — a hash, a log event, a path no trajectory published — is an achievement and is listed in `unmeasured`, so a reader sees where the measurement stopped. Attrition is a positive finding; the absence of one is not evidence of the opposite. +- **`scoreAchievements(contract, profile, verified)`** drops attrition milestones from the numerator AND the denominator. `scoreMilestones` is unchanged and still answers how far a run got. +- `CalibrationReport.separating` now counts legible **achievement** milestones, and `separates` compares the reference to the strongest baseline on those. Attrition milestones out of reach of every baseline move to `attritionSeparating`: they are recorded and never counted as separation. The report also carries `progression`, `referenceAchievementScore`, and `bestBaselineAchievementCount`. + +### A contract that grades on one event + +- `requires` is a partial order, so a contract can state six progressions and demand exactly one event. `report.collapse` states the structure with the number: which milestone the largest share requires, whether a trivial baseline reaches it, and how many milestones first pass at the SAME reference input. + + | packaged contract | gated behind | of | prerequisite reached by a baseline | open at one instant | + |---|---|---|---|---| + | ALE Breakout | `score-opened` | 6 of 6 | no, 0 of 7 | 3 of 6, after 32 inputs | + | stable-retro Airstriker | `score-opened` | 5 of 5 | yes, 2 of 28 | 3 of 5, after 41 inputs | + | Gymnasium CartPole | `survived-25-steps` | 5 of 5 | yes, 2 of 6 | 3 of 5, after 25 steps | + | Gymnasium FrozenLake | `reached-goal` | 3 of 3 | no, 0 of 8 | 3 of 3, after 6 steps | + | native-2048 | `first-legal-move` | 7 of 7 | yes, 6 of 7 | 2 of 7, after 4 inputs | + +- A run that misses the prerequisite scores 0 of N however well it played; where a baseline reaches it, the whole contract opens for free and grades only what follows a free event. The gate refuses until the author declares the structure with `{ gatedBehind: 'score-opened' }`, and refuses a stale declaration the same way `opaqueChecks` does. + +### Calibration is no longer optional for a packaged contract + +- **The measurement.** `calibrateContract` on the packaged stable-retro Airstriker contract reports `separates: false` with an EMPTY separating set. A seeded pseudo-random walk over the 25 advertised button words earns three of the four legible milestones, the same three the reference earns; the fourth is a hash that 271 of 768 single-input substitutions of the reference reproduce. It shipped because nothing between `deriveContract` and a published target ever ran the gate. +- **`PackagedContract.calibrate` is the only way to build a `PackagedContract`.** The class carries a private field and a private constructor, so an object literal of the same shape is not assignable and `new` is not available. It runs `calibrateContract`, refuses every finding the gate reports, and returns the contract with its hash, its report, and the declaration that let it through. A target that hands out a bare `MilestoneContract` has not been calibrated and now says so in its type. +- `{ nonSeparating: '' }` is the escape hatch for a target that is not meant to separate — a tier demonstration, a smoke fixture. It is a sentence rather than a switch, so the reason travels with the package, and it is REFUSED when the contract does separate: a declaration must not outlive the reason for it. +- `assertContractSeparates` now reports the attrition and collapse findings alongside the opacity ones, so one run of the gate names everything an author must fix. `assertOpaqueChecksDeclared` is unchanged and still checks opacity alone. + +### Measured across every packaged contract + +Full calibration, seed as packaged, run on every target this machine can boot. `pnpm test:ale`, `pnpm test:retro`, and `pnpm test:gym` now carry these as gates; nothing but the fixtures ran calibration before. + +| contract | reference | achievements | best baseline | separating | attrition | verdict | +|---|---|---|---|---|---|---| +| ALE Breakout | 6 of 6 | 5 of 5 | 0 of 7 policies scored | `score-opened`, `score-tier-2`, `score-tier-4` | `life-lost` | separates, collapses, 1 undeclared attrition | +| stable-retro Airstriker | 5 of 5 | 4 of 4 | 3 (pseudo-random) | **nothing** | `life-lost` | **does not separate**, collapses | +| Gymnasium CartPole | 5 of 5 | 5 of 5 | 2 (round-robin, pseudo-random) | `survived-50-steps`, `reward-at-50-steps` | none | separates, collapses | +| Gymnasium FrozenLake | 3 of 3 | 3 of 3 | 0 of 8 policies scored | `reached-goal`, `goal-cell` | none | separates, collapses | +| native-2048 | 7 of 7 | 7 of 7 | 7 (pseudo-random) | **nothing** | none | **does not separate**, collapses | +| save-levels | 2 of 2 | 2 of 2 | 2 (round-robin, pseudo-random) | **nothing** | none | **does not separate**, collapses | +| screen-puzzle | 2 of 2 | 2 of 2 | 2 (constant:r) | **nothing** | none | **does not separate**, collapses | +| engine-crawler | 4 of 4 | 4 of 4 | 4 (constant:right) | **nothing** | none | **does not separate**, does not collapse | + +- Breakout's `life-lost` and Airstriker's `life-lost` are the only two attrition milestones in the packaged set. Both were in `separating` before this change, so both contracts advertised a point for dying as part of their discriminating power. +- The two Breakout controls tie at 3 of 5 achievements rather than separating, and the reason is the second finding: the contract's top achievement is `score >= 4`, so nothing in it tells 7 from 24. +- `engineState.hpExact` in the `engine-crawler` fixture only ever falls, and its milestone is `hpExact == 1`, which HOLDS at the initial value. It is an achievement, which is the third condition of the rule doing its work: a check a run satisfies at turn zero needs no resource to run down. +- Existing adapters are unchanged in behaviour. No adapter file was edited, every contract keeps its bytes and its hash, and the pinned hashes for `engine-crawler`, `save-levels`, and `screen-puzzle` still hold. + ## 0.7.0 ### An episode can end because the game ended diff --git a/README.md b/README.md index cb0329e..09a6124 100644 --- a/README.md +++ b/README.md @@ -383,9 +383,11 @@ assertContractSeparates(report) `calibrateContract` replays the reference and a suite of trivial policies through the same attestation path: one constant policy per input word, a word the game cannot interpret, a round-robin cycle over the vocabulary, and a seeded pseudo-random walk over it. Every policy is deterministic in the seed, so a report reproduces from one number. -The report names `separating` (legible milestones no baseline earned), `trivial` (milestones at least one baseline earned), `legible` and `opaque`, `collisions`, and both baseline counts. -`separates` is true only when a **legible** milestone is out of reach of every baseline **and** the reference verifies strictly more legible milestones than the strongest baseline. +The report names `separating` (legible achievement milestones no baseline earned), `attritionSeparating` (out of reach of every baseline, and earned by a resource running down), `trivial` (milestones at least one baseline earned), `legible` and `opaque`, `progression`, `collapse`, `collisions`, and every baseline count. +`separates` is true only when a **legible achievement** milestone is out of reach of every baseline **and** the reference verifies strictly more of them than the strongest baseline. +Two kinds of milestone are excluded, for two different reasons. An opaque milestone cannot carry the claim: "the reference reached it and no baseline did" states nothing a reader can check. +An attrition milestone cannot carry it either: the run that earns it is the run that let a resource run down. `assertContractSeparates` throws otherwise, and the message names every trivial milestone with the baseline that earned it. ### The gate also refuses points nobody can read @@ -418,6 +420,92 @@ Measured on ALE Breakout, whose two hashes fire after 32 inputs over `NOOP/FIRE/ | distinct 32-input logs that satisfy the hashes | at least 3.82 × 10⁸ | `FIRE` while the ball is already in flight is a state no-op, so those logs reach a bit-identical emulator state. + +### A milestone earned by dying is not a milestone earned by playing well + +A milestone is an `achievement` when a run made something happen, and `attrition` when a run let a resource run down. +Lives, health, shields, fuel, time remaining: the shortest path to such a milestone is to play badly. +It is a real fact about a trajectory and it is recorded, but it is never counted as competence. + +The split is **measured, never declared**, and it reads no field name. +`measureProgressions` watches every numeric channel the engine publishes across the reference and every baseline, and calls a milestone attrition when all three of these hold: + +1. its check reads a numeric channel of the evidence; +2. that channel never rose and fell at least once, over every snapshot of every measured trajectory; +3. the check does not hold at the initial value of that channel. + +Anything else is an achievement, including a hash, a log event, and a channel no trajectory published; those are listed in `unmeasured` so a reader sees where the measurement stopped. +Attrition propagates through `requires`, because a milestone gated behind a lost life needs a lost life. + +A name list would have failed the same way `over()` would have: ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro `episodeDone`. +A channel called `lives` that counts rescued hostages goes UP, and the measurement classifies it as the achievement it is. + +```ts +import { measureProgressions, scoreAchievements, scoreMilestones } from '@tangle-network/playproof' + +const profile = report.progression +scoreMilestones(contract, verified) // how far the run got +scoreAchievements(contract, profile, verified) // how well it played +``` + +Measured on ALE Breakout, ale-py 0.12.1, seed 0. Two deterministic controls with the same control law and the same deadzone in screen pixels, differing only in what they read: + +| control | reads | game score @300 | @600 | lives left | milestones | achievements | +|---|---|---|---|---|---|---| +| `steer-from-ascii` | the ASCII frame, 4 px per character | 6 | 7 | 0 | **4 of 6** | 3 of 5 | +| `steer-from-ram` | `ram_ball_x`, `ram_paddle_x` | **9** | **24** | **5** | 3 of 6 | 3 of 5 | + +The RAM control wins every column the game reports and never dies. +Under the whole contract it scored LOWER, because `life-lost` is `lives == 4`: a point for dying. +`separating` and `separates` therefore count legible **achievement** milestones only, and `attritionSeparating` records the rest. +Declare an attrition milestone to keep it: `assertContractSeparates(report, { attritionChecks: ['life-lost'] })`. + +### A contract that grades on one event + +`requires` is a partial order, so a contract can state six progressions and demand exactly one event. +`report.collapse` states how much of the contract hangs off one milestone, whether a trivial baseline reaches it, and how many milestones first pass at the same instant of the reference. + +| contract | gated behind | of | prerequisite reached by a baseline | open at one instant | +|---|---|---|---|---| +| ALE Breakout | `score-opened` | 6 of 6 | no | 3 of 6, after 32 inputs | +| stable-retro Airstriker | `score-opened` | 5 of 5 | yes: `round-robin`, `pseudo-random` | 3 of 5, after 41 inputs | +| Gymnasium CartPole | `survived-25-steps` | 5 of 5 | yes: `round-robin`, `pseudo-random` | 3 of 5, after 25 steps | +| Gymnasium FrozenLake | `reached-goal` | 3 of 3 | no | 3 of 3, after 6 steps | + +A run that misses the prerequisite scores zero however well it played; where a baseline reaches it, the whole contract opens for free. +The gate refuses until the author declares the structure with `{ gatedBehind: 'score-opened' }`. + +### A packaged contract carries the calibration that justified it + +Calibration used to be optional, and an optional gate is a gate nothing has to pass. +Measured on stable-retro Airstriker: the packaged contract reports `separates: false` with an EMPTY separating set, and it shipped, because nothing between `deriveContract` and a published target ever asked. + +`PackagedContract.calibrate` is the only way to build a `PackagedContract`, and it runs the whole gate: + +```ts +const packaged = PackagedContract.calibrate(game, contract, { + reference: referenceInputs, + vocabulary, + declare: { + opaqueChecks: ['frame-at-first-score'], + weakChecks: ['frame-at-first-score'], + attritionChecks: ['life-lost'], + gatedBehind: 'score-opened', + }, +}) +packaged.report.separates // the verdict travels with the contract +``` + +A target that is not meant to separate — a tier demonstration, a smoke fixture — says so in words, and the declaration is refused when it goes stale: + +```ts +PackagedContract.calibrate(game, contract, { + reference, vocabulary, + declare: { nonSeparating: 'tier demonstration: exercises the evidence path, does not grade play' }, +}) +``` + +A contract that hands out a bare `MilestoneContract` has not been calibrated, and now says so in its type. No trivial baseline reproduces either hash, which is exactly why the baseline suite is not enough to judge one. | Contract | Milestones | Legible | Reference score | Best trivial baseline | diff --git a/ale.test.mts b/ale.test.mts index 6576c26..00c9665 100644 --- a/ale.test.mts +++ b/ale.test.mts @@ -16,12 +16,12 @@ import { strict as assert } from 'node:assert' import { spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' -import { attestRun } from './attestation' -import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract } from './calibration' +import { attestRun, MilestoneTracker } from './attestation' +import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract, measureProgressions } from './calibration' import { playEpisode, scriptedDriver } from './episode' -import { logFrom, observationOf } from './runtime' +import { isGameOver, logFrom, observationOf } from './runtime' import { decodePng, unscale } from './test-png.mts' -import { contractLegibility, formatMilestoneScore, scoreMilestones, validateContract } from './schema' +import { contractLegibility, formatMilestoneScore, scoreAchievements, scoreMilestones, validateContract } from './schema' import { AleRpc } from './adapters/ale-rpc' import { bundledReference, makeAle, type Ale, type AleState } from './adapters/ale' @@ -205,26 +205,193 @@ if (!pythonHasAle()) { () => assertOpaqueChecksDeclared(calibration, { opaqueChecks: ['frame-at-first-score', 'save-at-first-score'] }), /2 opaque milestone\(s\) are satisfied by input logs other than the reference/u, ) - const declared = { + const opacity = { opaqueChecks: ['frame-at-first-score', 'save-at-first-score'], weakChecks: ['frame-at-first-score', 'save-at-first-score'], } - assertOpaqueChecksDeclared(calibration, declared) + assertOpaqueChecksDeclared(calibration, opacity) + + // `life-lost` is `lives == 4`: the reference earns it by DYING. The + // classification reads no field name — it reads the motion of the channel, + // which falls and never rises across all eight replayed trajectories, and + // a check that does not hold at the starting value of a channel that only + // falls can only be earned by letting the resource run down. + // + // Before this split it sat in `separating`, so the contract's measured + // discriminating power included a point for dying. Two hand-written + // controls that differ only in how well they steer therefore ranked in the + // wrong order: the one that never died capped at 3 of 6. + assert.deepEqual(calibration.progression.attrition, ['life-lost']) + assert.deepEqual(calibration.progression.unmeasured, ['frame-at-first-score', 'save-at-first-score']) + assert.match( + calibration.progression.reasons['life-lost'] ?? '', + /reads engineState\.lives, which fell \d+ time\(s\) and never rose/u, + ) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.lives')?.rises, 0) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.score')?.falls, 0) + assert.deepEqual(calibration.attritionSeparating, ['life-lost']) + assert.ok(!calibration.separating.includes('life-lost')) + assert.deepEqual(calibration.referenceAchievementScore, { verified: 5, total: 5 }) + + // Every milestone requires `score-opened`, and three of the six open at the + // same instant of the reference. Six milestone ids, one decision point. + assert.equal(calibration.collapse.prerequisite, 'score-opened') + assert.equal(calibration.collapse.gated, 6) + assert.equal(calibration.collapse.total, 6) + assert.equal(calibration.collapse.collapses, true) + assert.deepEqual(calibration.collapse.earnedByBaseline, []) + assert.deepEqual(calibration.collapse.simultaneous, ['score-opened', 'frame-at-first-score', 'save-at-first-score']) + assert.equal(calibration.collapse.simultaneousAfter, 32) + + // Undeclared, the packaged contract does not pass the gate, and the two + // findings are reported together with their numbers. + assert.throws( + () => assertContractSeparates(calibration, opacity), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /states 1 of 6 milestone\(s\) that a resource running down earns/u) + assert.match(message, /attritionChecks: \['life-lost'\]/u) + assert.match(message, /collapses to one event: 6 of 6 milestone\(s\) require score-opened/u) + assert.match(message, /3 of 6 milestone\(s\) first pass at the same reference input \(after 32\)/u) + return true + }, + ) + + const declared = { ...opacity, attritionChecks: ['life-lost'], gatedBehind: 'score-opened' } // Whether Breakout separates is a fact about the ROM, not about this - // change: assert only that the legible comparison is the one being made. - const referenceLegible = calibration.reference.verified.filter((id) => calibration.legible.includes(id)).length + // change: assert only that the legible ACHIEVEMENT comparison is the one + // being made. + const referenceAchievement = calibration.reference.verified + .filter((id) => calibration.legible.includes(id) && calibration.progression.kinds[id] === 'achievement').length assert.equal( calibration.separates, - calibration.separating.length > 0 && referenceLegible > calibration.bestBaselineLegibleCount, + calibration.separating.length > 0 && referenceAchievement > calibration.bestBaselineAchievementCount, ) if (calibration.separates) assertContractSeparates(calibration, declared) else assert.throws(() => assertContractSeparates(calibration, declared), /does not separate/u) console.log( `ale: calibration — reference ${formatMilestoneScore(calibration.referenceScore)}, ` + + `achievements ${formatMilestoneScore(calibration.referenceAchievementScore)}, ` + `best trivial baseline ${calibration.bestBaselineLegibleCount} legible over ${calibration.turns} turns, ` + - `separating=${calibration.separating.join(',') || 'nothing'}, separates=${calibration.separates}`, + `separating=${calibration.separating.join(',') || 'nothing'}, ` + + `attrition-only=${calibration.attritionSeparating.join(',') || 'nothing'}, ` + + `separates=${calibration.separates}, collapses=${calibration.collapse.collapses}`, ) + // --- the ordering the split exists to fix ----------------------------- + // + // Two deterministic controls with the SAME control law and the same + // deadzone in screen pixels, differing only in what they read: the ASCII + // frame the agent sees, at four screen pixels per character, or the RAM + // channels the adapter publishes, at one pixel each. Neither carries state + // between decisions and neither costs a model call. + // + // The RAM control wins every column the game itself reports — score 9 to 6 + // at 300 decisions, 24 to 7 at 600 — and it never dies, while the ASCII + // control runs out of lives after 375. Under the whole contract the ASCII + // control still scores HIGHER, because `life-lost` is a point for dying. + // Under achievements alone the ordering is no longer backwards. + // + // The two achievement scores tie rather than separating, and the reason is + // the second finding this contract carries: its top achievement is + // `score >= 4`, so nothing in it can tell 7 from 24. + const PLAY_ROWS = [11, 20] as const + const PADDLE_ROWS = [21, 22] as const + const marked = (row: string): number[] => { + const columns: number[] = [] + for (let column = 2; column <= 37; column++) if ((row[column] ?? ' ') !== ' ') columns.push(column) + return columns + } + const steerTowards = (ball: number | null, paddle: number, deadzone: number): string => { + if (ball === null) return 'FIRE' + if (ball > paddle + deadzone) return 'RIGHT' + if (ball < paddle - deadzone) return 'LEFT' + return 'NOOP' + } + const steerFromAscii = (state: AleState): string => { + const rows = adapter.game.frame(state).split('\n') + let ball: number | null = null + for (let row = PLAY_ROWS[0]; row <= PLAY_ROWS[1] && ball === null; row++) { + const found = marked(rows[row] ?? '') + if (found.length > 0) ball = (found[0]! + found[found.length - 1]!) / 2 + } + const paddle = [...marked(rows[PADDLE_ROWS[0]] ?? ''), ...marked(rows[PADDLE_ROWS[1]] ?? '')] + if (paddle.length === 0) return 'FIRE' + return steerTowards(ball, (Math.min(...paddle) + Math.max(...paddle)) / 2, 1) + } + const steerFromRam = (state: AleState): string => { + const engine = (adapter.game.evidence(state).engineState ?? {}) as Record + const ball = (engine.ram_ball_x ?? 0) === 0 ? null : engine.ram_ball_x! + // The paddle is 20 RAM units wide, so its centre is its position plus 10, + // and 4 units is the 4-pixel character the ASCII control resolves. + return steerTowards(ball, (engine.ram_paddle_x ?? 0) + 10, 4) + } + + const readCounter = (frame: string, key: string): number => + Number(new RegExp(`${key}=(-?\\d+)`, 'u').exec(frame)?.[1] ?? -1) + const control = (id: string, decide: (state: AleState) => string, turns: number) => { + const tracker = new MilestoneTracker(adapter.contract) + let state = adapter.game.init(adapter.seed) + const inputs: string[] = [] + tracker.consider(adapter.game.evidence(state)) + for (let turn = 0; turn < turns && !isGameOver(adapter.game, state); turn++) { + const input = decide(state) + inputs.push(input) + state = adapter.game.step(state, input) + tracker.consider(adapter.game.evidence(state)) + } + const frame = adapter.game.frame(state) + return { + id, turns, inputs, + verified: tracker.verified(), + gameScore: readCounter(frame, 'score'), + lives: readCounter(frame, 'lives'), + played: inputs.length, + } + } + + const controls = [300, 600].flatMap((turns) => [ + control('steer-from-ascii', steerFromAscii, turns), + control('steer-from-ram', steerFromRam, turns), + ]) + // The classification is measured over the controls too, so the ordering + // claim is not made against a profile fitted to the reference alone. + const profile = measureProgressions(adapter.game, adapter.contract, { + trajectories: [adapter.reference, ...controls.map((run) => run.inputs)], + seed: adapter.seed, + }) + assert.deepEqual(profile.attrition, ['life-lost']) + + for (const run of controls) { + console.log( + ` ${run.id.padEnd(17)} @${String(run.turns).padStart(3)} decisions — ` + + `milestones ${formatMilestoneScore(scoreMilestones(adapter.contract, run.verified))}, ` + + `achievements ${formatMilestoneScore(scoreAchievements(adapter.contract, profile, run.verified))}, ` + + `game score ${run.gameScore}, lives ${run.lives}, played ${run.played}`, + ) + } + for (const turns of [300, 600]) { + const ascii = controls.find((run) => run.id === 'steer-from-ascii' && run.turns === turns)! + const ram = controls.find((run) => run.id === 'steer-from-ram' && run.turns === turns)! + // The RAM control plays better by the game's own report. + assert.ok(ram.gameScore > ascii.gameScore, `ram ${ram.gameScore} must beat ascii ${ascii.gameScore} at ${turns}`) + assert.ok(ram.lives > ascii.lives) + // Only the ASCII control dies, so only it earns the attrition milestone. + assert.ok(ascii.verified.includes('life-lost')) + assert.ok(!ram.verified.includes('life-lost')) + // The defect: the whole-contract score ranks the worse player first. + const wholeAscii = scoreMilestones(adapter.contract, ascii.verified) + const wholeRam = scoreMilestones(adapter.contract, ram.verified) + assert.ok(wholeAscii.verified > wholeRam.verified, + `the whole-contract score is expected to rank ascii first at ${turns}; if it no longer does, the reference moved`) + // The fix: under achievements alone it does not. + const achieveAscii = scoreAchievements(adapter.contract, profile, ascii.verified) + const achieveRam = scoreAchievements(adapter.contract, profile, ram.verified) + assert.ok(achieveRam.verified >= achieveAscii.verified, + `achievement score must not rank the control that died first at ${turns}: ` + + `ram ${formatMilestoneScore(achieveRam)}, ascii ${formatMilestoneScore(achieveAscii)}`) + } + // Determinism: two replays in this worker and one in a freshly spawned // worker must agree on every frame hash, every save-state hash, and every // privileged variable. Cross-process is the load-bearing case, because a diff --git a/authoring.ts b/authoring.ts index 689bac9..8427296 100644 --- a/authoring.ts +++ b/authoring.ts @@ -46,9 +46,15 @@ function checkHolds(check: MilestoneCheck, evidence: Evidence): boolean { * Blind-discovery marks are the sharp case, because nothing in the pipeline * ever asserts that a discovered channel means progress. * + * It also cannot know what a mark DEMONSTRATES. A mark on a life counter fires + * when the reference dies, and a contract that scores it rewards the run that + * played worse. + * * Run `calibrateContract` from calibration.ts on the derived contract and gate - * publication with `assertContractSeparates`. A contract that no trivial policy - * can satisfy is a benchmark; an uncalibrated one is a guess. + * publication with `PackagedContract.calibrate`, which refuses a contract that + * does not separate, an undeclared hash, an undeclared attrition milestone, and + * a contract every milestone of which requires one event. A contract that no + * trivial policy can satisfy is a benchmark; an uncalibrated one is a guess. */ export function deriveContract( game: Game, diff --git a/calibration.test.mts b/calibration.test.mts index 86eb28c..9719e82 100644 --- a/calibration.test.mts +++ b/calibration.test.mts @@ -21,6 +21,8 @@ import { assertContractSeparates, assertOpaqueChecksDeclared, calibrateContract, + measureProgressions, + PackagedContract, trivialBaselines, UNKNOWN_BASELINE_WORD, } from './calibration' @@ -30,10 +32,11 @@ import { contractHash, contractLegibility, formatMilestoneScore, + scoreAchievements, scoreMilestones, } from './schema' import { makeNative2048, NATIVE_2048_INPUTS, NATIVE_2048_REFERENCE } from './adapters/native-2048' -import { engineCrawlerContract } from './adapters/engine-crawler' +import { engineCrawler, engineCrawlerContract, ENGINE_CRAWLER_REFERENCE } from './adapters/engine-crawler' import { saveLevels, saveLevelsContract, SAVE_LEVELS_REFERENCE } from './adapters/save-levels' import { screenPuzzle, screenPuzzleContract, SCREEN_PUZZLE_REFERENCE } from './adapters/screen-puzzle' @@ -132,7 +135,7 @@ const mashContract = deriveContract(mashGame, 0, [...MASH_REFERENCE], [ assert.deepEqual(report.trivial, ['moved']) assert.equal(report.bestBaselineCount, 1) assert.deepEqual(report.reference.verified, ['moved', 'lock-opened']) - assertContractSeparates(report) + assertContractSeparates(report, { gatedBehind: 'moved' }) // (d) the counts are internally consistent. assert.equal(report.turns, LOCK_REFERENCE.length) @@ -309,7 +312,7 @@ try { assert.equal(formatMilestoneScore(report.referenceScore), '2 of 2') assert.equal(report.bestBaselineLegibleCount, 1) assert.equal(report.separates, true) - assertContractSeparates(report) + assertContractSeparates(report, { gatedBehind: 'moved' }) assertOpaqueChecksDeclared(report) } @@ -388,14 +391,17 @@ const lockOpaqueContract = deriveContract(comboLock, 0, [...LOCK_REFERENCE], [ // Declared, the same contract passes both gates. Nothing needs declaring as // weak, because the sweep found no other log that satisfies the hash. - assertContractSeparates(report, { opaqueChecks: ['frame-at-open'] }) + assertContractSeparates(report, { opaqueChecks: ['frame-at-open'], gatedBehind: 'moved' }) assertOpaqueChecksDeclared(report, { opaqueChecks: ['frame-at-open'] }) // A declaration is an exact set. A missing id and a stale id both fail, so a // hash milestone added by a later derivation cannot hide behind it. - assert.throws(() => assertContractSeparates(report, { opaqueChecks: [] }), /1 undeclared opaque milestone\(s\)/u) assert.throws( - () => assertContractSeparates(report, { opaqueChecks: ['frame-at-open', 'lock-opened'] }), + () => assertContractSeparates(report, { opaqueChecks: [], gatedBehind: 'moved' }), + /1 undeclared opaque milestone\(s\)/u, + ) + assert.throws( + () => assertContractSeparates(report, { opaqueChecks: ['frame-at-open', 'lock-opened'], gatedBehind: 'moved' }), /the contract does not state as a hash: lock-opened/u, ) @@ -679,4 +685,367 @@ const ledgeContract = deriveContract(ledgeWalk, 0, [...LEDGE_REFERENCE], [ } } +// --- what a milestone says about the run that earned it ---------------------- +// +// The pressure dive publishes three channels that move in three different +// ways, and the classification never reads their names. `hull` only ever +// cracks, so a milestone that needs it below its starting value is earned by +// letting the resource run down. `lives` only ever rises — it counts rescued +// divers — and it is named `lives` on purpose: a hardcoded list of resource +// names would classify it backwards, which is the mistake `episode-terminal` +// made one layer down when it matched the literal string `terminal` and +// reported `observed: false` for every adapter that spells it otherwise. + +const DIVE_VOCABULARY = ['down', 'up', 'wait'] +const DIVE_REFERENCE = [ + 'wait', 'down', 'wait', 'down', 'wait', 'down', + 'wait', 'down', 'wait', 'down', 'wait', 'down', +] + +interface DiveState { + depth: number + hull: number + /** Rescued divers. Named `lives`, and it only goes UP. */ + lives: number + equalized: number + steps: number +} + +const descend = (s: DiveState, repairs: boolean, input: string): DiveState => { + const steps = s.steps + 1 + if (input === 'wait') return { ...s, equalized: 1, steps } + if (repairs && input === 'up') return { ...s, hull: Math.min(s.hull + 1, 3), equalized: 0, steps } + if (input === 'down' && s.equalized === 1) { + const depth = s.depth + 1 + return { + depth, + hull: depth >= 4 ? s.hull - 1 : s.hull, + lives: depth >= 2 ? s.lives + 1 : s.lives, + equalized: 0, + steps, + } + } + return { ...s, equalized: 0, steps } +} + +const diveState = (): DiveState => ({ depth: 0, hull: 3, lives: 0, equalized: 0, steps: 0 }) +const diveEvidence = (s: DiveState) => ({ + engineState: { depth: s.depth, hull: s.hull, lives: s.lives, steps: s.steps }, +}) + +const pressureDive: Game = { + id: 'pressure-dive', + init: diveState, + step: (s, input) => descend(s, false, input), + frame: (s) => `depth ${s.depth} · hull ${s.hull} · rescued ${s.lives}`, + evidence: diveEvidence, +} + +/** + * The same game and the same contract, with one input added: `up` patches the + * hull. Nothing about the contract, the field name, or the check changes — only + * what the channel is measured doing. + */ +const repairableDive: Game = { + id: 'pressure-dive', + init: diveState, + step: (s, input) => descend(s, true, input), + frame: pressureDive.frame, + evidence: diveEvidence, +} + +const diveContract = deriveContract(pressureDive, 0, [...DIVE_REFERENCE], [ + { + id: 'depth-4', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.depth ?? 0) >= 4, + sample: (e) => ({ kind: 'state-path', path: 'depth', op: '>=', value: e.engineState?.depth ?? 4 }), + }, + { + id: 'lives-2', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.lives ?? 0) >= 2, + sample: (e) => ({ kind: 'state-path', path: 'lives', op: '>=', value: e.engineState?.lives ?? 2 }), + }, + { + id: 'hull-cracked', + tier: 'engine-state', + glitchClass: 'legal', + when: (e) => (e.engineState?.hull ?? 3) < 3, + sample: (e) => ({ kind: 'state-path', path: 'hull', op: '==', value: e.engineState?.hull ?? 2 }), + }, + { + id: 'depth-6', + tier: 'engine-state', + glitchClass: 'legal', + requires: ['depth-4'], + when: (e) => (e.engineState?.depth ?? 0) >= 6, + sample: (e) => ({ kind: 'state-path', path: 'depth', op: '>=', value: e.engineState?.depth ?? 6 }), + }, + { + id: 'steps-after-crack', + tier: 'engine-state', + glitchClass: 'legal', + requires: ['hull-cracked'], + when: (e) => (e.engineState?.steps ?? 0) >= 8, + sample: (e) => ({ kind: 'state-path', path: 'steps', op: '>=', value: e.engineState?.steps ?? 8 }), + }, +]) + +// (k) achievement and attrition, derived from measured motion. +{ + const report = calibrateContract(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + }) + const { progression } = report + assert.deepEqual(report.reference.verified, ['lives-2', 'depth-4', 'hull-cracked', 'steps-after-crack', 'depth-6']) + assert.deepEqual(progression.attrition, ['hull-cracked', 'steps-after-crack']) + assert.deepEqual(progression.achievement, ['depth-4', 'lives-2', 'depth-6']) + assert.deepEqual(progression.unmeasured, []) + assert.equal(progression.gameId, 'pressure-dive') + + // The channel named `lives` rises, so it is an achievement. The channel named + // `hull` only falls, so it is attrition. Neither name was read. + assert.match(progression.reasons['lives-2'] ?? '', /reads engineState\.lives, which rose \d+ time\(s\)/u) + assert.match( + progression.reasons['hull-cracked'] ?? '', + /reads engineState\.hull, which fell \d+ time\(s\) and never rose .*"== 2" does not hold at its initial value 3/u, + ) + // Attrition propagates through `requires`, because the tracker admits a + // milestone only after its prerequisites passed. + assert.equal(progression.reasons['steps-after-crack'], 'requires hull-cracked, which a resource running down earns') + + // Every channel the measurement watched, with its motion. + const motion = Object.fromEntries(progression.motion.map((row) => [row.channel, row])) + assert.deepEqual(Object.keys(motion).sort(), ['engineState.depth', 'engineState.hull', 'engineState.lives', 'engineState.steps']) + assert.equal(motion['engineState.hull']?.rises, 0) + assert.ok((motion['engineState.hull']?.falls ?? 0) > 0) + assert.equal(motion['engineState.hull']?.initial, 3) + assert.equal(motion['engineState.lives']?.falls, 0) + assert.equal(motion['engineState.depth']?.falls, 0) + + // An attrition milestone that no baseline reached is recorded, and it is not + // separation: "the reference lost hull and the baselines did not" says + // nothing about skill. + assert.deepEqual(report.separating, ['depth-4', 'depth-6']) + assert.deepEqual(report.attritionSeparating, ['hull-cracked', 'steps-after-crack']) + assert.equal(report.separates, true) + assert.deepEqual(report.referenceScore, { verified: 5, total: 5 }) + assert.deepEqual(report.referenceAchievementScore, { verified: 3, total: 3 }) + + // The gate refuses until the author has read the finding and written the ids + // down, and the declaration is an exact set like every other one. + assert.throws( + () => assertContractSeparates(report), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /states 2 of 5 milestone\(s\) that a resource running down earns \(2 undeclared\)/u) + assert.match(message, /attrition: hull-cracked/u) + assert.match(message, /the reference scored 5 of 5 over the whole contract and 3 of 3 over its achievements alone/u) + assert.match(message, /hull-cracked, steps-after-crack separated the reference from every baseline/u) + assert.match(message, /attritionChecks: \['hull-cracked', 'steps-after-crack'\]/u) + assert.doesNotMatch(message, /does not separate/u) + return true + }, + ) + assert.throws( + () => assertContractSeparates(report, { attritionChecks: ['hull-cracked'] }), + /states 2 of 5 milestone\(s\) that a resource running down earns \(1 undeclared\)/u, + ) + assert.throws( + () => assertContractSeparates(report, { attritionChecks: ['hull-cracked', 'steps-after-crack', 'depth-6'] }), + /attritionChecks names 1 milestone\(s\) the measurement did not classify as attrition: depth-6/u, + ) + assertContractSeparates(report, { attritionChecks: ['hull-cracked', 'steps-after-crack'] }) + + // The mutation that must flip the classification: the same contract, the same + // milestone, the same field name, on a game where one input patches the hull. + // One trajectory that repairs is enough — which is why the split is measured + // over the reference AND every baseline, not over the reference alone. + const REPAIR = ['wait', 'down', 'wait', 'down', 'wait', 'down', 'wait', 'down', 'up', 'up', 'wait', 'down'] + const never = measureProgressions(repairableDive, diveContract, { trajectories: [DIVE_REFERENCE] }) + assert.deepEqual(never.attrition, ['hull-cracked', 'steps-after-crack']) + const repaired = measureProgressions(repairableDive, diveContract, { trajectories: [DIVE_REFERENCE, REPAIR] }) + assert.deepEqual(repaired.attrition, []) + assert.deepEqual(repaired.achievement, ['depth-4', 'lives-2', 'hull-cracked', 'depth-6', 'steps-after-crack']) + assert.match(repaired.reasons['hull-cracked'] ?? '', /reads engineState\.hull, which rose 1 time\(s\)/u) + + // A milestone no numeric channel speaks for is an achievement, and it says so. + const opaqueProfile = measureProgressions(comboLock, lockOpaqueContract, { trajectories: [LOCK_REFERENCE] }) + assert.deepEqual(opaqueProfile.unmeasured, ['frame-at-open']) + assert.equal( + opaqueProfile.reasons['frame-at-open'], + 'its frame-hash check reads no numeric channel, so no resource motion could be measured', + ) + + // The inversion this whole split exists to stop, in miniature. Run A got + // further and broke; run B played better and did not. The whole-contract + // score ranks A first; the achievement score ranks B first. + const brokeThrough = ['lives-2', 'hull-cracked', 'steps-after-crack'] + const playedClean = ['depth-4', 'lives-2'] + assert.deepEqual(scoreMilestones(diveContract, brokeThrough), { verified: 3, total: 5 }) + assert.deepEqual(scoreMilestones(diveContract, playedClean), { verified: 2, total: 5 }) + assert.deepEqual(scoreAchievements(diveContract, progression, brokeThrough), { verified: 1, total: 3 }) + assert.deepEqual(scoreAchievements(diveContract, progression, playedClean), { verified: 2, total: 3 }) + + // A profile measured against another contract is refused rather than applied. + assert.throws( + () => scoreAchievements(lockContract, progression, ['moved']), + /progression profile is for "pressure-dive" but the contract is for "combo-lock"/u, + ) + assert.throws( + () => scoreAchievements(diveContract, { ...progression, kinds: {} }, []), + /does not classify depth-4, lives-2, hull-cracked, depth-6, steps-after-crack/u, + ) +} + +// (l) a contract that resolves runs on one event. +{ + const report = calibrateContract(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + }) + assert.deepEqual(report.collapse.prerequisite, 'moved') + assert.equal(report.collapse.gated, 2) + assert.equal(report.collapse.total, 2) + assert.deepEqual(report.collapse.gatedMilestones, ['moved', 'lock-opened']) + assert.equal(report.collapse.collapses, true) + assert.equal(report.collapse.earnedByReference, true) + assert.equal(report.collapse.earnedByBaseline.length, report.baselines.length) + assert.deepEqual(report.collapse.firstPassAt, { moved: 1, 'lock-opened': 8 }) + assert.deepEqual(report.collapse.simultaneous, []) + assert.equal(report.collapse.simultaneousAfter, -1) + + assert.throws( + () => assertContractSeparates(report), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /collapses to one event: 2 of 2 milestone\(s\) require moved/u) + assert.match(message, /gated by moved: lock-opened/u) + assert.match(message, /moved is earned by 9 of 9 trivial baseline\(s\)/u) + assert.match(message, /the whole contract opens for free/u) + assert.match(message, /gatedBehind: 'moved'/u) + return true + }, + ) + assert.throws( + () => assertContractSeparates(report, { gatedBehind: 'lock-opened' }), + /gatedBehind names lock-opened, which is not the milestone the contract hangs off \(moved gates 2 of 2\)/u, + ) + assertContractSeparates(report, { gatedBehind: 'moved' }) + + // A prerequisite no baseline reaches collapses the other way: a run that + // misses it scores nothing, however well it played. + const dive = calibrateContract(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + turns: 12, + }) + // The dive contract has two independent roots, so it does not collapse. + assert.equal(dive.collapse.collapses, false) + assert.equal(dive.collapse.gated, 2) + assert.equal(dive.collapse.total, 5) + + // The engine crawler states a milestone that requires nothing, so it is the + // shape a collapsed contract is measured against. + const crawler = calibrateContract(engineCrawler, engineCrawlerContract(), { + reference: ENGINE_CRAWLER_REFERENCE, + vocabulary: ['right', 'rest'], + }) + assert.equal(crawler.collapse.prerequisite, 'room-1') + assert.equal(crawler.collapse.gated, 3) + assert.equal(crawler.collapse.total, 4) + assert.equal(crawler.collapse.collapses, false) + + // Milestones that first pass at the same input are one event wearing two + // ids, and that is measured separately from `requires`. + const levels = calibrateContract(saveLevels, saveLevelsContract(), { + reference: SAVE_LEVELS_REFERENCE, + vocabulary: ['clear', 'grind'], + }) + assert.deepEqual(levels.collapse.simultaneous, ['level-2-saved', 'level-2-logged']) + assert.equal(levels.collapse.simultaneousAfter, 3) + assert.deepEqual(levels.collapse.firstPassAt, { 'level-2-saved': 3, 'level-2-logged': 3 }) +} + +// (m) a packaged contract carries the calibration that justified it. +{ + const packaged = PackagedContract.calibrate(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved' }, + }) + assert.equal(packaged.contract, lockContract) + assert.equal(packaged.hash, contractHash(lockContract)) + assert.equal(packaged.report.separates, true) + assert.deepEqual(packaged.report.separating, ['lock-opened']) + assert.deepEqual(packaged.declaration, { gatedBehind: 'moved' }) + + // A contract that does not separate cannot be packaged by omission. This is + // the Airstriker failure: `separates: false`, an empty separating set, and a + // published target anyway, because running calibration was optional. + const mash = { + reference: MASH_REFERENCE, + vocabulary: MASH_VOCABULARY, + } + assert.throws( + () => PackagedContract.calibrate(mashGame, mashContract, mash), + (error: unknown) => { + const message = (error as Error).message + assert.match(message, /does not separate/u) + assert.match(message, /nonSeparating: ''/u) + return true + }, + ) + + // The escape hatch is explicit, and it is a reason rather than a switch. + const demo = PackagedContract.calibrate(mashGame, mashContract, { + ...mash, + declare: { nonSeparating: 'tier demonstration: this target exercises the engine-state path, it does not grade play' }, + }) + assert.equal(demo.report.separates, false) + assert.equal(demo.report.bestBaselineCount, 1) + + assert.throws( + () => PackagedContract.calibrate(mashGame, mashContract, { ...mash, declare: { nonSeparating: ' ' } }), + /nonSeparating must state why this target is not meant to separate/u, + ) + + // And it is refused when it is stale, exactly as `opaqueChecks` is: a target + // that grew a real progression must not keep the excuse it shipped with. + assert.throws( + () => PackagedContract.calibrate(comboLock, lockContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved', nonSeparating: 'demonstration only' }, + }), + /nonSeparating says "demonstration only", but the contract separates: the reference reached lock-opened/u, + ) + + // Every other finding is refused here too, so packaging is the one gate an + // author cannot pass by calling a narrower one. + assert.throws( + () => PackagedContract.calibrate(comboLock, lockContract, { reference: LOCK_REFERENCE, vocabulary: LOCK_VOCABULARY }), + /collapses to one event/u, + ) + assert.throws( + () => PackagedContract.calibrate(pressureDive, diveContract, { + reference: DIVE_REFERENCE, + vocabulary: DIVE_VOCABULARY, + }), + /that a resource running down earns/u, + ) + assert.throws( + () => PackagedContract.calibrate(comboLock, lockOpaqueContract, { + reference: LOCK_REFERENCE, + vocabulary: LOCK_VOCABULARY, + declare: { gatedBehind: 'moved' }, + }), + /states 1 of 3 milestone\(s\) as a hash/u, + ) +} + console.log('playproof calibration: separating and non-separating contracts, legible/opaque split, opaque-collision sweep, policy determinism, edge cases OK') diff --git a/calibration.ts b/calibration.ts index c804f99..ccf3065 100644 --- a/calibration.ts +++ b/calibration.ts @@ -28,6 +28,28 @@ * perturbed logs still satisfy it. A hash a large family of logs satisfies is * a weak check, and the gate reports the measured number. * + * The third measurement is what a milestone says about the run that earned it. + * A milestone earned by a monotone-decreasing resource going down marks + * progress REACHED and never competence shown, because the shortest path to it + * is to play badly. `measureProgressions` derives that split from the motion of + * the channels the engine already publishes; it reads no field name, so an + * adapter that spells its life counter `shields` is classified on the same + * evidence as one that spells it `lives`. `separating` and `separates` count + * achievement milestones only, and the rest is recorded in + * `attritionSeparating`. + * + * The fourth is how much of a contract hangs off one milestone. `requires` is a + * partial order, so a contract can state six progressions and still demand + * exactly one event. `report.collapse` states the structure with the number, + * whether a trivial baseline reaches the prerequisite, and how many milestones + * first pass at one instant of the reference. + * + * `PackagedContract` closes the last hole. Running any of this was optional, + * and an optional gate is a gate nothing has to pass: the packaged Airstriker + * contract reported `separates: false` with an empty separating set and shipped + * anyway. `PackagedContract.calibrate` is the only way to build that type, and + * it runs the whole gate. + * * None of this touches replay attestation. The input-log hash chain is the * mechanism that proves a replay reproduced a recorded run, and it is * unaffected: a milestone hash was never that proof. @@ -37,9 +59,17 @@ */ import { attestRun, MilestoneTracker } from './attestation' import { logFrom } from './runtime' -import type { Game } from './runtime' -import { contractLegibility, formatMilestoneScore, scoreMilestones } from './schema' -import type { MilestoneContract, MilestoneScore } from './schema' +import type { Evidence, Game } from './runtime' +import { contractGate, contractHash, contractLegibility, formatMilestoneScore, scoreAchievements, scoreMilestones } from './schema' +import type { + ContractGate, + MilestoneCheck, + MilestoneContract, + MilestoneScore, + NumericOperator, + ProgressionKind, + ProgressionProfile, +} from './schema' /** * A deterministic input policy that needs no observation of the game. @@ -276,6 +306,265 @@ export function probeOpaqueCollisions( return rows.sort((a, b) => opaque.indexOf(a.milestone) - opaque.indexOf(b.milestone)) } +/** + * The evidence map each numeric check reads. A check outside this table names + * no numeric channel, so no resource motion can be measured for it. + */ +const NUMERIC_MAP_FOR_CHECK: Partial> = { + 'state-path': 'engineState', + 'save-path': 'saveState', + 'frame-path': 'frameState', +} + +/** `.` for a numeric check, or null when the check reads no channel. */ +function checkChannel(check: MilestoneCheck): string | null { + const map = NUMERIC_MAP_FOR_CHECK[check.kind] + if (map === undefined) return null + return `${map}.${'path' in check ? check.path : ''}` +} + +/** Every numeric channel one evidence snapshot publishes, flattened to `.`. */ +function channelValues(evidence: Evidence): Map { + const values = new Map() + for (const map of ['engineState', 'saveState', 'frameState'] as const) { + const fields = evidence[map] + if (fields === undefined) continue + for (const [key, value] of Object.entries(fields)) { + if (typeof value === 'number' && Number.isFinite(value)) values.set(`${map}.${key}`, value) + } + } + return values +} + +interface MutableMotion { + channel: string + initial: number | null + initialSeen: boolean + rises: number + falls: number + min: number + max: number + trajectories: number + samples: number +} + +function numericHolds(op: NumericOperator, value: number, expected: number): boolean { + if (op === '>=') return value >= expected + if (op === '>') return value > expected + return value === expected +} + +export interface ProgressionOptions { + /** + * Input scripts to replay and watch. The useful set is the reference plus + * every trivial baseline, which is what `calibrateContract` passes: a channel + * that only falls on the reference but rises under a baseline is not a + * resource, and one trajectory would not have shown it. + */ + trajectories: readonly (readonly string[])[] + /** Replay seed for every trajectory. Default 0. */ + seed?: number +} + +/** + * Classify every milestone of a contract as an achievement or as attrition, by + * watching how the channels move. + * + * The rule reads no field name. A milestone is attrition when all three of + * these measured statements hold: + * + * 1. its check reads a numeric channel of the evidence; + * 2. that channel never rose and fell at least once, over every snapshot of + * every measured trajectory — it is a resource that only runs down; + * 3. the check does not hold at the initial value of that channel, so the + * only way to earn it is to let the resource run down. + * + * Anything else is an achievement, including a hash, a log event, and a channel + * the trajectories never published: attrition is a positive finding, and the + * absence of one is not evidence of the opposite. Those milestones are listed + * in `unmeasured` so a reader sees where the measurement stopped. + * + * Attrition propagates through `requires`, because `MilestoneTracker` admits a + * milestone only after every prerequisite passed: a milestone gated behind a + * lost life needs a lost life, whatever its own check reads. + * + * A hardcoded list of names would have missed the same way `over()` once did. + * ALE spells its terminal flag `terminal`, Gymnasium `terminated`, stable-retro + * `episodeDone`; a life counter is `lives` on Atari and something else on the + * next substrate. Motion is the thing every engine has in common. + */ +export function measureProgressions( + game: Game, + contract: MilestoneContract, + options: ProgressionOptions, +): ProgressionProfile { + const seed = options.seed ?? 0 + const motion = new Map() + const see = (channel: string, value: number, first: boolean): MutableMotion => { + let row = motion.get(channel) + if (row === undefined) { + row = { + channel, initial: null, initialSeen: false, rises: 0, falls: 0, + min: value, max: value, trajectories: 0, samples: 0, + } + motion.set(channel, row) + } + row.samples += 1 + row.min = Math.min(row.min, value) + row.max = Math.max(row.max, value) + if (first) { + row.trajectories += 1 + // The initial value must be one number for the classification to mean + // anything. A game whose init() is not pure disagrees with itself here, + // and every milestone on that channel falls back to unmeasured. + if (!row.initialSeen) { + row.initial = value + row.initialSeen = true + } else if (row.initial !== value) { + row.initial = null + } + } + return row + } + + for (const inputs of options.trajectories) { + let state = game.init(seed) + let previous = channelValues(game.evidence(state)) + for (const [channel, value] of previous) see(channel, value, true) + for (const input of inputs) { + state = game.step(state, input) + const current = channelValues(game.evidence(state)) + for (const [channel, value] of current) { + const row = see(channel, value, false) + const before = previous.get(channel) + if (before === undefined) continue + if (value > before) row.rises += 1 + if (value < before) row.falls += 1 + } + previous = current + } + } + + const kinds: Record = {} + const reasons: Record = {} + const unmeasured: string[] = [] + const runs = options.trajectories.length + for (const milestone of contract.milestones) { + const channel = checkChannel(milestone.check) + const row = channel === null ? undefined : motion.get(channel) + if (channel === null) { + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = `its ${milestone.check.kind} check reads no numeric channel, so no resource motion could be measured` + unmeasured.push(milestone.id) + continue + } + if (row === undefined || row.initial === null) { + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = row === undefined + ? `no measured trajectory published ${channel}, so no resource motion could be measured` + : `${channel} started at a different value on different trajectories, so no resource motion could be measured` + unmeasured.push(milestone.id) + continue + } + const check = milestone.check + const expected = 'value' in check ? check.value : 0 + const op: NumericOperator = 'op' in check ? check.op : '>=' + const holdsAtStart = numericHolds(op, row.initial, expected) + const onlyFalls = row.rises === 0 && row.falls > 0 + if (onlyFalls && !holdsAtStart) { + kinds[milestone.id] = 'attrition' + reasons[milestone.id] = + `its ${check.kind} check reads ${channel}, which fell ${row.falls} time(s) and never rose over ` + + `${row.trajectories} of ${runs} measured trajectory(ies); "${op} ${expected}" does not hold at its ` + + `initial value ${row.initial}, so the only way to earn it is to let the resource run down` + continue + } + kinds[milestone.id] = 'achievement' + reasons[milestone.id] = onlyFalls + ? `its ${check.kind} check on ${channel} already holds at the initial value ${row.initial}, so it needs no resource to run down` + : row.rises > 0 + ? `its ${check.kind} check reads ${channel}, which rose ${row.rises} time(s) over ${row.trajectories} of ${runs} measured trajectory(ies)` + : `its ${check.kind} check reads ${channel}, which never moved over ${row.trajectories} of ${runs} measured trajectory(ies)` + } + + // Attrition propagates the way opacity does: the tracker admits a milestone + // only after every prerequisite passed. + const byId = new Map(contract.milestones.map((m) => [m.id, m])) + const visiting = new Set() + const inherit = (id: string): void => { + if (kinds[id] === 'attrition' || visiting.has(id)) return + visiting.add(id) + for (const required of byId.get(id)?.requires ?? []) { + if (!byId.has(required)) continue + inherit(required) + if (kinds[required] === 'attrition') { + kinds[id] = 'attrition' + reasons[id] = `requires ${required}, which a resource running down earns` + break + } + } + visiting.delete(id) + } + for (const milestone of contract.milestones) inherit(milestone.id) + + const order = contract.milestones.map((m) => m.id) + return { + gameId: contract.gameId, + kinds, + achievement: order.filter((id) => kinds[id] === 'achievement'), + attrition: order.filter((id) => kinds[id] === 'attrition'), + unmeasured: unmeasured.filter((id) => kinds[id] !== 'attrition'), + reasons, + motion: [...motion.values()].map((row) => ({ + channel: row.channel, + initial: row.initial, + rises: row.rises, + falls: row.falls, + min: row.min, + max: row.max, + trajectories: row.trajectories, + samples: row.samples, + })), + } +} + +/** + * How much of a contract hangs off one milestone, and who reaches that + * milestone. + * + * `ContractGate` states the structure; these fields add the measurement that + * says what the structure costs. A contract every milestone of which requires + * one event resolves runs into two classes and no more: those that reached the + * event, and those that scored zero. + */ +export interface ContractCollapse extends ContractGate { + /** Baseline ids that earned the prerequisite. Empty means no trivial policy reached it. */ + earnedByBaseline: string[] + /** Whether the reference itself earned the prerequisite. */ + earnedByReference: boolean + /** Reference inputs consumed before each milestone first passed; -1 when it never passed. */ + firstPassAt: Record + /** + * The largest set of milestones that first passed at the SAME reference + * input, in contract order. + * + * This is the second way a contract counts one event more than once, and it + * is independent of `requires`. Measured on the packaged contracts: ALE + * Breakout opens `score-opened`, `frame-at-first-score`, and + * `save-at-first-score` at input 32, and stable-retro Airstriker opens three + * of its five at one input too. Three milestone ids for one moment of play is + * three points of a score that a run either has all of or none of. + */ + simultaneous: string[] + /** The input index those milestones share, or -1 when no two share one. */ + simultaneousAfter: number + /** + * Whether every milestone of the contract requires one milestone. A contract + * of one milestone cannot collapse, because there is nothing to chain. + */ + collapses: boolean +} + export interface CalibrationReport { turns: number seed: number @@ -283,12 +572,23 @@ export interface CalibrationReport { reference: BaselineOutcome baselines: BaselineOutcome[] /** - * Legible milestones the reference reached and no baseline did — the whole - * of the contract's discriminating power. An opaque milestone is excluded - * even when no baseline earned it: a reader cannot see what it demands, so - * it cannot be the evidence that the contract measures anything. + * Legible ACHIEVEMENT milestones the reference reached and no baseline did — + * the whole of the contract's discriminating power. + * + * Two exclusions, for two different reasons. An opaque milestone is excluded + * even when no baseline earned it, because a reader cannot see what it + * demands. An attrition milestone is excluded even when no baseline earned + * it, because the run that earns it is the run that let a resource run down, + * and "the reference lost a life and the baselines did not" is not a claim + * about skill. Attrition milestones that would otherwise have separated are + * kept in `attritionSeparating`, so nothing measured is thrown away. */ separating: string[] + /** + * Milestones the reference reached and no baseline did, that a resource + * running down earns. They are recorded, never counted as separation. + */ + attritionSeparating: string[] /** milestones at least one baseline earned */ trivial: string[] /** contract milestones whose requirement a reader can read off the contract */ @@ -318,8 +618,23 @@ export interface CalibrationReport { * a reference must beat, because an opaque milestone cannot carry the claim. */ bestBaselineLegibleCount: number - /** the reference's own progress */ + /** the reference's own progress over the whole contract */ referenceScore: MilestoneScore + /** + * The strongest baseline's count over LEGIBLE ACHIEVEMENT milestones. This is + * the number a reference must beat, because neither an opaque milestone nor + * an attrition milestone can carry the claim. + */ + bestBaselineAchievementCount: number + /** the reference's progress over the achievement milestones alone */ + referenceAchievementScore: MilestoneScore + /** + * Which milestones demonstrate something and which a resource running down + * earns, measured over the reference and every baseline. + */ + progression: ProgressionProfile + /** How much of the contract hangs off one milestone, and who reaches it. */ + collapse: ContractCollapse separates: boolean } @@ -373,26 +688,72 @@ export function calibrateContract( } const referenceInputs = options.reference.slice(0, turns) - const reference = play('reference', referenceInputs) - const baselines = policies.map((policy) => { + const scripts = policies.map((policy) => { const inputs = policy.inputs(vocabulary, turns, seed) if (inputs.length !== turns) { throw new Error(`baseline ${policy.id} produced ${inputs.length} inputs for ${turns} turns`) } - return play(policy.id, inputs) + return { policy, inputs } + }) + const reference = play('reference', referenceInputs) + const baselines = scripts.map(({ policy, inputs }) => play(policy.id, inputs)) + + // The achievement/attrition split is measured over every trajectory this + // report already names, not over the reference alone: a channel that only + // falls under the reference but rises under one baseline is not a resource, + // and a single trajectory would not have shown it. + const progression = measureProgressions(game, contract, { + trajectories: [referenceInputs, ...scripts.map(({ inputs }) => inputs)], + seed, }) const earnedByBaseline = new Set(baselines.flatMap((b) => b.verified)) const order = contract.milestones.map((m) => m.id) const { legible, opaque, reasons } = contractLegibility(contract) const legibleSet = new Set(legible) - const separating = reference.verified.filter((id) => !earnedByBaseline.has(id) && legibleSet.has(id)) + const isAchievement = (id: string): boolean => progression.kinds[id] === 'achievement' + const outOfReach = reference.verified.filter((id) => !earnedByBaseline.has(id) && legibleSet.has(id)) + const separating = outOfReach.filter(isAchievement) + const attritionSeparating = reference.verified.filter((id) => !earnedByBaseline.has(id) && !isAchievement(id)) const trivial = order.filter((id) => earnedByBaseline.has(id)) const bestBaselineCount = baselines.reduce((best, b) => Math.max(best, b.verified.length), 0) - const legibleCount = (outcome: BaselineOutcome): number => outcome.verified.filter((id) => legibleSet.has(id)).length - const bestBaselineLegibleCount = baselines.reduce((best, b) => Math.max(best, legibleCount(b)), 0) + const countIn = (outcome: BaselineOutcome, keep: (id: string) => boolean): number => outcome.verified.filter(keep).length + const isLegible = (id: string): boolean => legibleSet.has(id) + const isLegibleAchievement = (id: string): boolean => isLegible(id) && isAchievement(id) + const bestBaselineLegibleCount = baselines.reduce((best, b) => Math.max(best, countIn(b, isLegible)), 0) + const bestBaselineAchievementCount = baselines.reduce((best, b) => Math.max(best, countIn(b, isLegibleAchievement)), 0) const referenceScore = scoreMilestones(contract, reference.verified) - const referenceLegible = legibleCount(reference) + const referenceAchievementScore = scoreAchievements(contract, progression, reference.verified) + const referenceAchievement = countIn(reference, isLegibleAchievement) + + const gate = contractGate(contract) + const firstPass = firstPassTurn(game, contract, seed, referenceInputs) + const firstPassAt: Record = {} + const sharing = new Map() + for (const id of order) { + const turn = firstPass.get(id) ?? -1 + firstPassAt[id] = turn + if (turn >= 0) sharing.set(turn, [...(sharing.get(turn) ?? []), id]) + } + let simultaneous: string[] = [] + let simultaneousAfter = -1 + for (const [turn, ids] of sharing) { + if (ids.length > simultaneous.length) { + simultaneous = ids + simultaneousAfter = turn + } + } + const collapse: ContractCollapse = { + ...gate, + earnedByBaseline: gate.prerequisite === null + ? [] + : baselines.filter((b) => b.verified.includes(gate.prerequisite!)).map((b) => b.id), + earnedByReference: gate.prerequisite !== null && reference.verified.includes(gate.prerequisite), + firstPassAt, + simultaneous: simultaneous.length > 1 ? simultaneous : [], + simultaneousAfter: simultaneous.length > 1 ? simultaneousAfter : -1, + collapses: gate.prerequisite !== null && gate.gated === gate.total && gate.total > 1, + } return { turns, @@ -401,6 +762,7 @@ export function calibrateContract( reference, baselines, separating, + attritionSeparating, trivial, legible, opaque, @@ -414,8 +776,12 @@ export function calibrateContract( }), bestBaselineCount, bestBaselineLegibleCount, + bestBaselineAchievementCount, referenceScore, - separates: separating.length > 0 && referenceLegible > bestBaselineLegibleCount, + referenceAchievementScore, + progression, + collapse, + separates: separating.length > 0 && referenceAchievement > bestBaselineAchievementCount, } } @@ -438,6 +804,44 @@ export interface OpacityDeclaration { weakChecks?: readonly string[] } +/** + * Everything a contract author must have read and accepted before the contract + * is published, beyond the opaque checks. + * + * Every field is an exact set or an exact id, never a switch, for the reason + * `opaqueChecks` is: a finding that a later derivation introduces must not + * enter a published contract under a declaration written for an earlier one. + * A stale declaration fails the gate as loudly as a missing one. + */ +export interface ContractDeclaration extends OpacityDeclaration { + /** + * Milestone ids the author accepts as attrition: a resource running down + * earns them, so they are recorded and never scored as competence. + */ + attritionChecks?: readonly string[] + /** + * The milestone the author accepts the whole contract hangs off. Declaring it + * states that one event decides whether a run scores anything at all. + */ + gatedBehind?: string +} + +/** + * What `PackagedContract.calibrate` accepts on top of the contract declaration. + */ +export interface PackageDeclaration extends ContractDeclaration { + /** + * Why this target is not meant to separate, in the author's words. + * + * A tier demonstration and a smoke fixture exist to exercise the machinery, + * not to grade an agent, and forcing them to separate would mean inventing a + * difficulty they do not have. The field is a sentence rather than a boolean + * so the reason travels with the package, and it is refused when the contract + * DOES separate, so it cannot sit in a target that outgrew it. + */ + nonSeparating?: string +} + function earners(report: CalibrationReport, milestone: string): string { return report.baselines.filter((b) => b.verified.includes(milestone)).map((b) => b.id).join(', ') } @@ -536,6 +940,124 @@ function opacityProblems(report: CalibrationReport, declaration: OpacityDeclarat return problems } +/** + * Every way a contract's progression split can be wrong, as message blocks. + * An empty array means every attrition milestone is declared. + */ +function progressionProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + const problems: string[] = [] + const { attrition, reasons } = report.progression + const declared = declaration.attritionChecks + const accepted = new Set(declared ?? []) + const undeclared = attrition.filter((id) => !accepted.has(id)) + const { total } = report.referenceScore + if (undeclared.length > 0) { + problems.push( + [ + `contract states ${attrition.length} of ${total} milestone(s) that a resource running down earns ` + + `(${undeclared.length} undeclared):`, + ...undeclared.map((id) => ` attrition: ${id} — ${reasons[id]}`), + ` the reference scored ${formatMilestoneScore(report.referenceScore)} over the whole contract and ` + + `${formatMilestoneScore(report.referenceAchievementScore)} over its achievements alone`, + ...(report.attritionSeparating.length > 0 + ? [ + ` ${report.attritionSeparating.join(', ')} separated the reference from every baseline, and no ` + + 'longer counts towards separation', + ] + : []), + 'Such a milestone marks progress REACHED and never competence shown: the shortest path to it is to play badly.', + `Keep it and declare it — { attritionChecks: [${undeclared.map((id) => `'${id}'`).join(', ')}] } — so it is ` + + 'recorded and scored by scoreMilestones but not by scoreAchievements, or pin a progression that a ' + + 'better run reaches.', + ].join('\n'), + ) + } + const stale = [...accepted].filter((id) => !attrition.includes(id)) + if (stale.length > 0) { + problems.push( + `attritionChecks names ${stale.length} milestone(s) the measurement did not classify as attrition: ` + + `${stale.join(', ')} — the declaration is stale, and it would hide an attrition milestone measured later`, + ) + } + return problems +} + +/** + * Whether the contract resolves runs on one event, as message blocks. + * An empty array means nothing collapses, or the author declared the structure. + */ +function collapseProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + const problems: string[] = [] + const { collapse } = report + const declared = declaration.gatedBehind + if (collapse.collapses && declared !== collapse.prerequisite) { + const free = collapse.earnedByBaseline + problems.push( + [ + `contract collapses to one event: ${collapse.gated} of ${collapse.total} milestone(s) require ` + + `${collapse.prerequisite}, so the contract resolves a run into two classes and no more`, + ` gated by ${collapse.prerequisite}: ${collapse.gatedMilestones.filter((id) => id !== collapse.prerequisite).join(', ')}`, + free.length > 0 + ? ` ${collapse.prerequisite} is earned by ${free.length} of ${report.baselines.length} trivial baseline(s) ` + + `(${free.join(', ')}), so the whole contract opens for free and grades only what follows a free event` + : ` ${collapse.prerequisite} is out of reach of all ${report.baselines.length} trivial baseline(s), so a run ` + + `that misses it scores 0 of ${collapse.total} however well it played`, + ...(collapse.simultaneous.length > 1 + ? [ + ` ${collapse.simultaneous.length} of ${collapse.total} milestone(s) first pass at the same reference ` + + `input (after ${collapse.simultaneousAfter}): ${collapse.simultaneous.join(', ')} — one moment of play ` + + 'wearing several milestone ids', + ] + : []), + 'A contract that grades on one event has one bit of resolution, whatever its milestone count says.', + `Pin a progression that does not require ${collapse.prerequisite}, or declare the structure — ` + + `{ gatedBehind: '${collapse.prerequisite}' }.`, + ].join('\n'), + ) + } + if (declared !== undefined && declared !== collapse.prerequisite) { + problems.push( + `gatedBehind names ${declared}, which ${collapse.prerequisite === null + ? 'no milestone of this contract requires' + : `is not the milestone the contract hangs off (${collapse.prerequisite} gates ${collapse.gated} of ${collapse.total})` + } — the declaration is stale`, + ) + } + return problems +} + +/** Every finding an author must have read, whatever gate they called. */ +function contractProblems(report: CalibrationReport, declaration: ContractDeclaration): string[] { + return [ + ...opacityProblems(report, declaration), + ...progressionProblems(report, declaration), + ...collapseProblems(report, declaration), + ] +} + +/** The message block for a contract a trivial policy keeps up with. */ +function separationProblem(report: CalibrationReport): string { + const best = report.baselines.filter((b) => b.verified.length === report.bestBaselineCount).map((b) => b.id) + const referenceLegible = report.reference.verified.filter((id) => report.legible.includes(id)).length + const legibleAchievements = report.legible.filter((id) => report.progression.kinds[id] === 'achievement') + const referenceAchievement = report.reference.verified.filter((id) => legibleAchievements.includes(id)).length + return [ + `contract does not separate: the reference verified ${report.reference.verified.length} milestone(s) ` + + `and the best trivial baseline verified ${report.bestBaselineCount} over ${report.turns} turns ` + + `(seed ${report.seed}, strongest: ${best.join(', ') || 'none'})`, + ` on legible milestones alone: reference ${referenceLegible}, ` + + `best baseline ${report.bestBaselineLegibleCount}, of ${report.legible.length} legible`, + ` on legible achievements alone: reference ${referenceAchievement}, ` + + `best baseline ${report.bestBaselineAchievementCount}, of ${legibleAchievements.length} legible achievement(s)`, + ...report.trivial.map((id) => ` trivial: ${id} — earned by ${earners(report, id)}`), + ...report.attritionSeparating.map( + (id) => ` out of reach of every baseline but not evidence of skill: ${id} — ${report.progression.reasons[id]}`, + ), + ` legible, an achievement, and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, + 'A derived contract is a hypothesis until it separates. Pin a progression a trivial policy cannot reach.', + ].join('\n') +} + /** * Fail closed on a contract whose milestones a reader cannot understand. * @@ -571,24 +1093,91 @@ export function assertOpaqueChecksDeclared( */ export function assertContractSeparates( report: CalibrationReport, - declaration: OpacityDeclaration = {}, + declaration: ContractDeclaration = {}, ): void { - const problems = opacityProblems(report, declaration) - if (!report.separates) { - const best = report.baselines.filter((b) => b.verified.length === report.bestBaselineCount).map((b) => b.id) - const referenceLegible = report.reference.verified.filter((id) => report.legible.includes(id)).length - problems.push( - [ - `contract does not separate: the reference verified ${report.reference.verified.length} milestone(s) ` + - `and the best trivial baseline verified ${report.bestBaselineCount} over ${report.turns} turns ` + - `(seed ${report.seed}, strongest: ${best.join(', ') || 'none'})`, - ` on legible milestones alone: reference ${referenceLegible}, ` + - `best baseline ${report.bestBaselineLegibleCount}, of ${report.legible.length} legible`, - ...report.trivial.map((id) => ` trivial: ${id} — earned by ${earners(report, id)}`), - ` legible and out of reach of every baseline: ${report.separating.join(', ') || 'nothing'}`, - 'A derived contract is a hypothesis until it separates. Pin a progression a trivial policy cannot reach.', - ].join('\n'), - ) - } + const problems = contractProblems(report, declaration) + if (!report.separates) problems.push(separationProblem(report)) if (problems.length > 0) throw new Error(problems.join('\n')) } + +/** + * A contract that has passed the gate, with the numbers that let it through. + * + * The type exists because running calibration was optional, and an optional + * gate is a gate nothing has to pass. Measured on stable-retro Airstriker: the + * packaged contract reported `separates: false` with an empty separating set, + * and it shipped anyway, because nothing in the path from `deriveContract` to a + * published target ever asked. A screen-blind `B, NOOP, B, NOOP` program earned + * all four legible milestones, the same four the reference earned. + * + * There is one way to build this value, and it runs the gate. The class carries + * a private field, so an object literal of the right shape is not assignable to + * it, and the constructor is private, so `new` is not available either. A + * target that hands out a `PackagedContract` has been calibrated; a target that + * hands out a bare `MilestoneContract` has not, and now says so in its type. + * + * The report travels with the contract on purpose. A published number should + * carry the measurement that justified it, and a reader of a target can ask + * what the baselines scored without re-running an emulator. + */ +export class PackagedContract { + /** + * Nominal brand. It has no other purpose: a private field makes an object + * literal of the same shape unassignable, so this type cannot be forged. + */ + readonly #calibrated = true + + private constructor( + /** The contract itself, byte for byte as it was derived. */ + readonly contract: MilestoneContract, + /** `contractHash(contract)` at the moment the gate passed. */ + readonly hash: string, + /** Every number the gate read. */ + readonly report: CalibrationReport, + /** What the author read and accepted. */ + readonly declaration: PackageDeclaration, + ) {} + + /** + * Calibrate a contract and refuse to package it unless it is justified. + * + * Every finding `assertContractSeparates` reports is refused here too: + * undeclared opaque checks, an opaque check a baseline reproduced, a hash a + * family of logs satisfies, an undeclared attrition milestone, and a contract + * every milestone of which requires one event. + * + * `nonSeparating` is the escape hatch, and it is the same shape as + * `opaqueChecks`: an explicit statement, refused when it is stale. A target + * that declares itself non-separating and then separates fails, because the + * declaration would otherwise outlive the reason for it. + */ + static calibrate( + game: Game, + contract: MilestoneContract, + options: CalibrateOptions & { declare?: PackageDeclaration }, + ): PackagedContract { + const declaration = options.declare ?? {} + const report = calibrateContract(game, contract, options) + const problems = contractProblems(report, declaration) + const excuse = declaration.nonSeparating + if (excuse === undefined) { + if (!report.separates) { + problems.push( + `${separationProblem(report)}\n` + + 'A target that is not meant to separate — a tier demonstration, a smoke fixture — says so: ' + + "{ nonSeparating: '' }.", + ) + } + } else if (excuse.trim().length === 0) { + problems.push('nonSeparating must state why this target is not meant to separate; an empty reason declares nothing') + } else if (report.separates) { + problems.push( + `nonSeparating says "${excuse}", but the contract separates: the reference reached ` + + `${report.separating.join(', ')} and no trivial baseline did — the declaration is stale, and it ` + + 'would hide a later regression that stopped it separating', + ) + } + if (problems.length > 0) throw new Error(problems.join('\n')) + return new PackagedContract(contract, contractHash(contract), report, declaration) + } +} diff --git a/gymnasium.test.mts b/gymnasium.test.mts index 973427f..ca23939 100644 --- a/gymnasium.test.mts +++ b/gymnasium.test.mts @@ -15,6 +15,7 @@ import { strict as assert } from 'node:assert' import { spawnSync } from 'node:child_process' import { tmpdir } from 'node:os' import { attestRun } from './attestation' +import { calibrateContract, PackagedContract } from './calibration' import { playEpisode, scriptedDriver } from './episode' import { logFrom } from './runtime' import { validateContract } from './schema' @@ -206,6 +207,73 @@ if (!pythonHasGymnasium()) { second = makeGymnasium({ envId: FROZENLAKE }) assert.deepEqual(trace(second, lake.reference), lakeFirst, 'frozenlake cross-process replay diverged') + // Calibration on both packaged Gymnasium contracts. + // + // CartPole separates: a random walk drops the pole long before 50 steps. + // Neither contract carries an attrition milestone, which is the negative + // control for the split — a channel that only falls is a finding, and + // these environments publish none. + const pole = calibrateContract(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + }) + assert.equal(pole.separates, true) + assert.deepEqual(pole.separating, ['survived-50-steps', 'reward-at-50-steps']) + assert.deepEqual(pole.trivial, ['survived-25-steps', 'reward-at-25-steps']) + assert.deepEqual(pole.progression.attrition, []) + assert.deepEqual(pole.progression.unmeasured, ['frame-at-25-steps']) + assert.deepEqual(pole.referenceAchievementScore, { verified: 5, total: 5 }) + + // It still collapses: every milestone requires `survived-25-steps`, which + // two baselines reach, and three of the five open at that same step. + assert.equal(pole.collapse.prerequisite, 'survived-25-steps') + assert.equal(pole.collapse.gated, 5) + assert.equal(pole.collapse.collapses, true) + assert.deepEqual(pole.collapse.earnedByBaseline, ['round-robin', 'pseudo-random']) + assert.deepEqual(pole.collapse.simultaneous, ['survived-25-steps', 'reward-at-25-steps', 'frame-at-25-steps']) + assert.equal(pole.collapse.simultaneousAfter, 25) + + const poleDeclared = { + opaqueChecks: ['frame-at-25-steps'], + gatedBehind: 'survived-25-steps', + } + assert.throws(() => PackagedContract.calibrate(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + }), /collapses to one event/u) + const packagedPole = PackagedContract.calibrate(cartpole.game, cartpole.contract, { + reference: cartpole.reference, + vocabulary: cartpole.inputs, + seed: cartpole.seed, + declare: poleDeclared, + }) + assert.equal(packagedPole.report.separates, true) + + // FrozenLake is the sharpest packaged contract: no baseline reaches the + // goal at all, so its three milestones are all out of reach. All three + // also fire at the same instant, which is what a goal-only contract is. + const frozen = calibrateContract(frozenLake.game, frozenLake.contract, { + reference: frozenLake.reference, + vocabulary: frozenLake.inputs, + seed: frozenLake.seed, + }) + assert.equal(frozen.separates, true) + assert.deepEqual(frozen.trivial, []) + assert.deepEqual(frozen.separating, ['reached-goal', 'goal-cell']) + assert.deepEqual(frozen.progression.attrition, []) + assert.equal(frozen.collapse.collapses, true) + assert.deepEqual(frozen.collapse.simultaneous, ['reached-goal', 'goal-frame', 'goal-cell']) + assert.deepEqual(frozen.collapse.earnedByBaseline, []) + console.log( + `gymnasium: calibration — ${CARTPOLE} separating=${pole.separating.join(',')} ` + + `(best baseline ${pole.bestBaselineAchievementCount} achievements, collapses=${pole.collapse.collapses}); ` + + `${FROZENLAKE} separating=${frozen.separating.join(',')} ` + + `(best baseline ${frozen.bestBaselineAchievementCount}, collapses=${frozen.collapse.collapses}); ` + + 'no attrition milestone in either', + ) + // Unsupported action spaces fail at boot with a readable message. assert.throws( () => makeGymnasium({ envId: 'Pendulum-v1', reference: { ...bundledReference(CARTPOLE)!, envId: 'Pendulum-v1' } }), diff --git a/schema.ts b/schema.ts index e199b43..0ebdb05 100644 --- a/schema.ts +++ b/schema.ts @@ -226,3 +226,174 @@ export function scoreMilestones(contract: MilestoneContract, verified: readonly export function formatMilestoneScore(score: MilestoneScore): string { return `${score.verified} of ${score.total}` } + +/** + * What a milestone says about the run that earned it. + * + * `achievement` — the run made something happen. A score threshold, a level + * reached, a state a player has to play towards. + * + * `attrition` — the run let a resource run down. Lives, health, shields, time + * remaining. Such a milestone marks progress REACHED and never competence + * shown, because the shortest path to it is to play badly. Measured on ALE + * Breakout: the packaged `life-lost` milestone is `lives == 4`, so a program + * that never dies cannot score it, and two hand-written controls that differ + * only in how well they steer rank in the wrong order because of it. + * + * Recording an attrition milestone is legitimate — "the reference got far + * enough to lose a life" is a real fact about a trajectory. Scoring it as + * competence is not, which is why `scoreAchievements` exists next to + * `scoreMilestones`. + */ +export type ProgressionKind = 'achievement' | 'attrition' + +/** + * How one numeric evidence channel moved across the measured trajectories. + * + * A channel is one key of one evidence map, named `engineState.lives` or + * `frameState.activeCells`. The counts are over every step of every measured + * trajectory, so `rises` 0 with `falls` above 0 is the measured statement + * "this channel is a resource that only runs down". + */ +export interface ChannelMotion { + /** `.`, for example `engineState.lives`. */ + channel: string + /** Value at the initial state, or null when the trajectories disagreed. */ + initial: number | null + /** Steps where the value was greater than the step before. */ + rises: number + /** Steps where the value was less than the step before. */ + falls: number + min: number + max: number + /** Trajectories that published the channel at least once. */ + trajectories: number + /** Snapshots the counts are over. */ + samples: number +} + +/** + * Every milestone of one contract, split by what earning it demonstrates. + * + * The split is MEASURED, never declared in the contract, for the same reason + * legibility is derived: an existing contract keeps its bytes and its hash, and + * an author cannot forget to set it. `measureProgressions` in calibration.ts + * produces this from the motion of the channels the engine already publishes. + * Nothing here reads a field name, so an adapter that spells its life counter + * `shields` or `hull` is classified on the same evidence as one that spells it + * `lives`. + */ +export interface ProgressionProfile { + gameId: string + /** Every milestone of the contract, classified. */ + kinds: Record + /** Milestones that demonstrate something, in contract order. */ + achievement: string[] + /** Milestones earned by a resource going down, in contract order. */ + attrition: string[] + /** + * Milestones no numeric channel could speak for — a hash, a log event, or a + * path the measured trajectories never published. They count as achievements, + * because attrition is a positive finding and this is the absence of one. + * The list exists so a reader sees the limit of the measurement. + */ + unmeasured: string[] + /** Why each milestone is classified as it is, keyed by milestone id. */ + reasons: Record + /** Every channel the measurement watched, in first-seen order. */ + motion: ChannelMotion[] +} + +function assertProfileMatches(contract: MilestoneContract, profile: ProgressionProfile): void { + if (profile.gameId !== contract.gameId) { + throw new Error(`progression profile is for "${profile.gameId}" but the contract is for "${contract.gameId}"`) + } + const missing = contract.milestones.filter((m) => profile.kinds[m.id] === undefined).map((m) => m.id) + if (missing.length > 0) { + throw new Error(`progression profile does not classify ${missing.join(', ')} — re-measure it against this contract`) + } +} + +/** + * Score a run over the achievement milestones alone. + * + * Both the numerator and the denominator drop the attrition milestones, so two + * runs compared on this number are compared on what they made happen. A run + * that also lost a life is neither rewarded nor punished for it. Read + * `scoreMilestones` for the whole contract when the question is how far the run + * got rather than how well it played. + */ +export function scoreAchievements( + contract: MilestoneContract, + profile: ProgressionProfile, + verified: readonly string[], +): MilestoneScore { + assertProfileMatches(contract, profile) + const achievement = new Set(contract.milestones.map((m) => m.id).filter((id) => profile.kinds[id] === 'achievement')) + return { verified: verified.filter((id) => achievement.has(id)).length, total: achievement.size } +} + +/** + * How much of a contract hangs off one milestone. + * + * `requires` is a partial order, so a contract can state five progressions and + * still demand exactly one event: if every other milestone requires the first, + * a run that misses the first scores zero however well it played. That is one + * bit of resolution wearing five milestones. Measured on the packaged + * contracts: ALE Breakout gates 6 of 6 milestones behind `score-opened`, and + * stable-retro Airstriker gates 5 of 5 behind its own `score-opened`. + * + * This structure says nothing about how hard the prerequisite is. Whether a + * trivial baseline earns it is a measurement, and `calibrateContract` reports + * that as `collapse`. + */ +export interface ContractGate { + /** The milestone the largest share of the contract requires, transitively. */ + prerequisite: string | null + /** Milestones that require it, transitively, counting the prerequisite itself. */ + gated: number + /** Those milestone ids, in contract order. */ + gatedMilestones: string[] + total: number +} + +/** + * The milestone the largest share of a contract depends on. + * + * Ties are broken by contract order, so the answer is stable. A contract where + * no milestone requires another has no prerequisite and `gated` 0: nothing is + * chained, so nothing can collapse. A missing or cyclic requirement is not + * judged here — `validateContract` reports those, and the walk below stops on a + * cycle instead of looping. + */ +export function contractGate(contract: MilestoneContract): ContractGate { + const total = contract.milestones.length + const byId = new Map(contract.milestones.map((m) => [m.id, m])) + const closure = new Map>() + const requiredBy = (id: string, seen: Set): Set => { + const cached = closure.get(id) + if (cached !== undefined) return cached + if (seen.has(id)) return new Set() + seen.add(id) + const all = new Set() + for (const required of byId.get(id)?.requires ?? []) { + if (!byId.has(required)) continue + all.add(required) + for (const deeper of requiredBy(required, seen)) all.add(deeper) + } + seen.delete(id) + closure.set(id, all) + return all + } + + let best: ContractGate = { prerequisite: null, gated: 0, gatedMilestones: [], total } + for (const candidate of contract.milestones) { + const dependents = contract.milestones + .filter((m) => m.id === candidate.id || requiredBy(m.id, new Set()).has(candidate.id)) + .map((m) => m.id) + if (dependents.length > best.gated && dependents.length > 1) { + best = { prerequisite: candidate.id, gated: dependents.length, gatedMilestones: dependents, total } + } + } + return best +} diff --git a/stable-retro.test.mts b/stable-retro.test.mts index 81b1fb2..f28175a 100644 --- a/stable-retro.test.mts +++ b/stable-retro.test.mts @@ -16,9 +16,10 @@ import { spawnSync } from 'node:child_process' import { createHash } from 'node:crypto' import { tmpdir } from 'node:os' import { attestRun } from './attestation' +import { calibrateContract, PackagedContract } from './calibration' import { logFrom, observationOf } from './runtime' import { decodePng, unscale } from './test-png.mts' -import { validateContract } from './schema' +import { formatMilestoneScore, validateContract } from './schema' import { RetroRpc } from './adapters/retro-rpc' import { bundledReference, makeStableRetro, type RetroState, type StableRetro } from './adapters/stable-retro' @@ -183,6 +184,84 @@ if (!pythonHasRetro()) { vision.dispose() } + // Calibration on the packaged Airstriker contract. + // + // Nothing in the path from `deriveContract` to a published target used to + // ask for this, and the answer is that the contract does not separate: the + // separating set is EMPTY. A seeded pseudo-random walk over the 25 advertised + // button words earns three of the four legible milestones, the same three + // the reference earns, and the fourth is a hash. Reporting it is the point: + // a contract that grades nothing must not look like one that grades play. + // The sweep is capped at 8 probed turns here. Genesis is the slowest + // substrate in the suite and this console advertises 25 input words, so a + // full 32-turn sweep costs 24,000 emulator steps for a number the first + // eight turns already establish. + const calibration = calibrateContract(adapter.game, adapter.contract, { + reference: adapter.reference, + vocabulary: adapter.inputs, + seed: adapter.seed, + collisionTurns: 8, + }) + assert.deepEqual(calibration.reference.verified, adapter.contract.milestones.map((m) => m.id)) + assert.deepEqual(calibration.separating, []) + assert.equal(calibration.separates, false) + assert.equal(calibration.bestBaselineAchievementCount, 3) + assert.deepEqual(calibration.trivial, ['score-opened', 'screen-active-at-first-score', 'score-tier-2']) + + // `life-lost` is `lives == 2` from a start of 3: earned by dying, on a + // channel measured falling 18 times and never rising across 29 replayed + // trajectories. It is the one milestone no baseline reached, and before + // this split it was the contract's whole claim to discriminating power. + assert.deepEqual(calibration.progression.attrition, ['life-lost']) + assert.deepEqual(calibration.attritionSeparating, ['life-lost']) + assert.equal(calibration.progression.motion.find((row) => row.channel === 'engineState.lives')?.rises, 0) + + // Every milestone requires `score-opened`, which two baselines earn, so the + // whole contract opens for free; and three of the five open at one instant. + assert.equal(calibration.collapse.prerequisite, 'score-opened') + assert.equal(calibration.collapse.gated, 5) + assert.equal(calibration.collapse.total, 5) + assert.equal(calibration.collapse.collapses, true) + assert.deepEqual(calibration.collapse.earnedByBaseline, ['round-robin', 'pseudo-random']) + assert.deepEqual( + calibration.collapse.simultaneous, + ['score-opened', 'frame-at-first-score', 'screen-active-at-first-score'], + ) + + // The hash is weak by a wide margin: a large family of logs reaches the + // state it names, measured rather than assumed. + const collision = calibration.collisions[0]! + assert.equal(collision.milestone, 'frame-at-first-score') + assert.ok(collision.collisions > 0, 'the sweep found no colliding log for the pinned frame') + assert.equal(collision.jointCollision, true) + + // A packaged contract cannot be published without a verdict, and this one + // has no verdict to publish. + assert.throws( + () => PackagedContract.calibrate(adapter.game, adapter.contract, { + reference: adapter.reference, + vocabulary: adapter.inputs, + seed: adapter.seed, + collisionTurns: 8, + declare: { + opaqueChecks: ['frame-at-first-score'], + weakChecks: ['frame-at-first-score'], + attritionChecks: ['life-lost'], + gatedBehind: 'score-opened', + }, + }), + /does not separate/u, + ) + console.log( + `stable-retro: calibration — reference ${formatMilestoneScore(calibration.referenceScore)}, ` + + `achievements ${formatMilestoneScore(calibration.referenceAchievementScore)}, ` + + `best baseline ${calibration.bestBaselineCount} over ${calibration.turns} turns, ` + + `separating=${calibration.separating.join(',') || 'NOTHING'}, ` + + `attrition-only=${calibration.attritionSeparating.join(',') || 'nothing'}, ` + + `separates=${calibration.separates}, collapses=${calibration.collapse.collapses}, ` + + `${collision.collisions} of ${collision.substitutions} substitutions reproduce the pinned frame`, + ) + // Teardown: dispose kills the worker and every later call fails loudly // instead of silently reading a dead transport. second.dispose()