Skip to content

feat(gates): neutralizationGate — footprint-matched placebo control (0.107.0)#324

Merged
drewstone merged 1 commit into
mainfrom
feat/neutralization-gate
Jul 8, 2026
Merged

feat(gates): neutralizationGate — footprint-matched placebo control (0.107.0)#324
drewstone merged 1 commit into
mainfrom
feat/neutralization-gate

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What

Adds the one primitive the substrate genuinely lacks: a placebo control that proves a held-out win comes from the candidate's content, not from the prompt/mount footprint the content happened to add.

A held-out gate (heldOutGate/defaultProductionGate) proves candidate beat baseline. It cannot tell apart (a) informative content from (b) simply more bytes/lines the model spends attention on. neutralizationGate closes that hole.

How

  • neutralizeText (src/campaign/neutralize.ts) — blanks every non-whitespace char to a 1-byte filler while preserving layout/length. Byte-exact for ASCII; character/layout-preserving for multibyte (what the tokenizer footprint tracks).
  • neutralizationGate (src/campaign/gates/neutralization-gate.ts) — composable Gate. Compares the candidate's held-out lift against a footprint-matched neutralized variant's lift (reusing pairHoldout). Holds when the neutralized variant reproduces ≥ maxDecorativeFraction (default 0.5) of the lift — the "R46" decorative-reject rule — however large the raw lift. Fail-closed on no positive lift; fail-loud if composed without the loop wiring.
  • GateContext gains optional neutralizedJudgeScores / neutralizedArtifacts.
  • runImprovementLoop gains an optional neutralize fn; when supplied and the winner changed, it scores a third holdout arm and exposes it to the gate. Opt-in — one extra holdout campaign only when wired.

Compose after significance: composeGate(heldOutGate({...}), neutralizationGate({...})) — significance says the lift is real; this says the content caused it.

Provenance

Lifted from a hand-rolled tax-benchmark fork (compile-and-prove/ablation-gate). A three-agent audit confirmed the fork's other halves — its permutation lift test, <12 power check, JSONL ledger, content-hash promotion, and mine→gate→promote loop — are duplicates of existing substrate primitives (heldOutGate, powerPreflight, LoopProvenanceRecord, surfaceContentHash, selfImprove) and dissolve into them. This placebo control was the only piece with no substrate equivalent (grep-confirmed: agent-eval's "decorative" uses are dead-code detection + rubric validity, not a footprint-matched content control).

Verify

  • pnpm typecheck → 0 errors; pnpm test272 files, 2797 passed; pnpm build → clean.
  • New gate suite: 8 tests (ships-on-causal, holds-on-decorative, threshold equality, fail-closed no-lift, fail-loud missing-arm, neutralizer layout/byte behavior). Full gate suite 65/65; loop/distillation 37/38 (1 pre-existing skip) — wiring didn't regress the loop.
  • Additive only (new exports + optional fields) → minor bump 0.106.3 → 0.107.0, no consumer lockstep. Version trio bumped.

…0.107.0)

A held-out gate proves a candidate beat baseline; it cannot prove the lift
came from the candidate's CONTENT rather than the prompt/mount FOOTPRINT the
content added. neutralizationGate closes that hole: runImprovementLoop scores
a third holdout arm — the winner with its content footprint-matched-blanked
(neutralizeText) — and the gate holds any win whose lift survives blanking
(decorative), however large or significant its raw lift.

- src/campaign/neutralize.ts: neutralizeText — layout-preserving, byte-exact
  (ASCII) content blanking.
- src/campaign/gates/neutralization-gate.ts: composable Gate; R46 rule (reject
  when neutralized lift >= maxDecorativeFraction of candidate lift, default 0.5);
  fail-closed on no lift, fail-loud on a missing neutralized arm.
- GateContext gains optional neutralizedJudgeScores/neutralizedArtifacts.
- runImprovementLoop gains an optional neutralize fn; when supplied and the
  winner changed, it runs the placebo arm and exposes it to the gate.

Additive: new exports + optional fields only; no consumer bump required.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 68ad00eb

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-08T00:04:16Z

@drewstone drewstone merged commit 84e8791 into main Jul 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants