feat(calibration): score what a run made happen, not what it lost

[drewstone]

Why

deriveContract samples milestones from one reference playthrough. Three failures followed, and all three are measured here.

1. A milestone anti-correlated with skill

ALE Breakout's life-lost is lives == 4: earned by dying. Two deterministic controls, same control law, same deadzone in screen pixels, differing only in what they read. Seed 0, no model spend, both now gates in pnpm test:ale.

control	reads	game score @300	@600	lives left	milestones	achievements
steer-from-ascii	the ASCII frame, 4 px per character	6	7	0 (dead after 375)	4 of 6	3 of 5
steer-from-ram	ram_ball_x, ram_paddle_x	9	24	5	3 of 6	3 of 5

The RAM control wins every column the game itself reports and never dies. Under the whole contract it scored LOWER. Under achievements alone the ordering is no longer backwards.

2. A contract that does not separate, shipped

calibrateContract on the packaged stable-retro Airstriker contract: separates: false, separating set empty. A seeded pseudo-random walk over the 25 advertised button words earns three of the four legible milestones, the same three the reference earns. It shipped because running calibration was optional.

3. Milestones chained behind one event

Breakout gates 6 of 6 behind score-opened; three of the six open at the same input.

What changed

A. Achievement vs attrition, derived from measured motion. measureProgressions watches every numeric channel the evidence publishes, across the reference and every baseline, and calls a milestone attrition when three measured statements hold: its check reads a numeric channel; that channel never rose and fell at least once, over every snapshot of every trajectory; and the check does not hold at the initial value of that channel. Attrition propagates through requires.

No field-name list. A sibling matched the literal string terminal and reported observed: false for every adapter spelling it otherwise. A fixture channel deliberately named lives that counts rescued divers only rises, and the measurement classifies it as the achievement it is. The mutation test flips the classification of the same milestone, on the same contract, by adding one input that repairs the resource.

scoreAchievements drops attrition from the numerator AND the denominator. separating and separates count legible achievement milestones only; attritionSeparating records the rest.

B. PackagedContract.calibrate is the only way to build a PackagedContract — private constructor, private field, so an object literal is not assignable and new is not available. It runs calibrateContract and refuses every finding. { nonSeparating: '<why>' } is the escape hatch for a tier demonstration or a smoke fixture; it is a sentence, not a switch, and it is refused when the contract DOES separate.

C. report.collapse names the milestone the largest share requires, whether a trivial baseline reaches it, and how many milestones first pass at the same reference input. Declared with { gatedBehind: 'score-opened' }, refused when stale.

Calibration across every packaged contract

Run on every target this machine can boot. pnpm test:ale, pnpm test:retro, and pnpm test:gym now carry these as gates; only the offline fixtures calibrated before.

contract	reference	achievements	best baseline	separating	attrition	verdict
ALE Breakout	6 of 6	5 of 5	0 of 7 policies scored	score-opened, score-tier-2, score-tier-4	life-lost	separates; collapses; 1 undeclared attrition
stable-retro Airstriker	5 of 5	4 of 4	3 (pseudo-random)	nothing	life-lost	does not separate; collapses
Gymnasium CartPole	5 of 5	5 of 5	2 (round-robin, pseudo-random)	survived-50-steps, reward-at-50-steps	none	separates; collapses
Gymnasium FrozenLake	3 of 3	3 of 3	0 of 8 policies scored	reached-goal, goal-cell	none	separates; collapses
native-2048	7 of 7	7 of 7	7 (pseudo-random)	nothing	none	does not separate; collapses
save-levels	2 of 2	2 of 2	2 (round-robin, pseudo-random)	nothing	none	does not separate; collapses
screen-puzzle	2 of 2	2 of 2	2 (constant:r)	nothing	none	does not separate; collapses
engine-crawler	4 of 4	4 of 4	4 (constant:right)	nothing	none	does not separate; does not collapse

Collapse structure:

contract	gated behind	of	prerequisite reached by a baseline	open at one instant
ALE Breakout	score-opened	6 of 6	no, 0 of 7	3 of 6, after 32 inputs
stable-retro Airstriker	score-opened	5 of 5	yes, 2 of 28	3 of 5, after 41 inputs
Gymnasium CartPole	survived-25-steps	5 of 5	yes, 2 of 6	3 of 5, after 25 steps
Gymnasium FrozenLake	reached-goal	3 of 3	no, 0 of 8	3 of 3, after 6 steps
native-2048	first-legal-move	7 of 7	yes, 6 of 7	2 of 7, after 4 inputs

PyBoy Tetris and RetroArch are not measured here: PyBoy is not installed on this machine and RetroArch needs a host binary.

Guards against a gate that passes everything

• engine-crawler's hpExact only ever falls, and its milestone is hpExact == 1, which HOLDS at the initial value. It is classified an achievement — the third condition of the rule doing its work.
• Gymnasium publishes no falling channel, so neither contract carries an attrition milestone. That is the negative control.
• engine-crawler states a milestone that requires nothing, so collapses is false at 3 of 4 gated. The collapse flag is not universal.
• Every declaration is refused when stale: attritionChecks naming a non-attrition milestone, gatedBehind naming the wrong prerequisite, nonSeparating on a contract that separates.

Existing adapters

Untouched in behaviour. No file under adapters/ is edited, no reference JSON is edited, every contract keeps its bytes, and the pinned engine-crawler, save-levels, and screen-puzzle contract hashes still hold.

Proof

$ pnpm run ci
playproof-boundary: 104 files checked; framework is research-harness- and provider-neutral
playproof calibration: separating and non-separating contracts, legible/opaque split, opaque-collision sweep, policy determinism, edge cases OK
playproof episode: loop, budget, and partial-progress semantics OK
playproof game-over stop: 8 moves to the gate, three distinct stop reasons, compatibility and replay OK
playproof observation: text default, image round-trip, bounds, evidence boundary, replay parity, and both drivers OK
playproof-platform: native process, frontier replay, explicit trust modes, signed execution, and red-team gates green
playproof-desktop-platforms: hardened native process, Steam, Xbox, bounded IO, and signed recorder composition green
playproof-drivers: CLI and OpenAI-compatible adapters green
playproof campaign: segment, steer, resume, and tamper semantics OK
playproof-package: 96 packed entries verified
ci exit=0

$ pnpm test:ale         (ale-py 0.12.1, 26s)
ale: calibration — reference 6 of 6, achievements 5 of 5, best trivial baseline 0 legible over 210 turns,
     separating=score-opened,score-tier-2,score-tier-4, attrition-only=life-lost, separates=true, collapses=true
  steer-from-ascii  @300 decisions — milestones 4 of 6, achievements 3 of 5, game score 6, lives 2, played 300
  steer-from-ram    @300 decisions — milestones 3 of 6, achievements 3 of 5, game score 9, lives 5, played 300
  steer-from-ascii  @600 decisions — milestones 4 of 6, achievements 3 of 5, game score 7, lives 0, played 375
  steer-from-ram    @600 decisions — milestones 3 of 6, achievements 3 of 5, game score 24, lives 5, played 600

$ pnpm test:retro       (stable-retro 1.0.1, 74s)
stable-retro: calibration — reference 5 of 5, achievements 4 of 4, best baseline 3 over 96 turns,
     separating=NOTHING, attrition-only=life-lost, separates=false, collapses=true,
     71 of 192 substitutions reproduce the pinned frame

$ pnpm test:gym         (gymnasium 1.3.0)
gymnasium: calibration — CartPole-v1 separating=survived-50-steps,reward-at-50-steps (best baseline 2 achievements,
     collapses=true); FrozenLake-v1 separating=reached-goal,goal-cell (best baseline 0, collapses=true);
     no attrition milestone in either

pnpm run ci was run from a copy of the tree outside /Users/drew, because a stale /Users/drew/node_modules/@tangle-network on this machine shadows @tangle-network/* into a sibling repository and breaks tsc on main too. That is a local environment fault, not a repository one.

Where the code disagreed with the brief

The brief attributes failure 3 to "milestones chained behind a trivial first one" and names Breakout. Measured: no trivial baseline scores anything at all on Breakout, so score-opened is not free there. The collapse flag therefore fires on the STRUCTURE (every milestone requires one), and the report states separately whether a baseline reaches the prerequisite — free on Airstriker, CartPole and 2048; out of reach on Breakout and FrozenLake. Both are worth refusing, and the message says which one it found.

The falsifiable test asks the RAM control to rank at or above the ASCII control under an achievement-only score. It ties at 3 of 5, and the reason is a second finding rather than a partial fix: the contract's top achievement is score >= 4, so nothing in it can tell 7 from 24. Raising that ceiling would change the packaged reference and every artifact that pinned its hash, so it is not done here.

Not merged

Not merged, per the brief.