feat(gates): neutralizationGate — footprint-matched placebo control (0.107.0)#324
Merged
Conversation
…0.107.0) A held-out gate proves a candidate beat baseline; it cannot prove the lift came from the candidate's CONTENT rather than the prompt/mount FOOTPRINT the content added. neutralizationGate closes that hole: runImprovementLoop scores a third holdout arm — the winner with its content footprint-matched-blanked (neutralizeText) — and the gate holds any win whose lift survives blanking (decorative), however large or significant its raw lift. - src/campaign/neutralize.ts: neutralizeText — layout-preserving, byte-exact (ASCII) content blanking. - src/campaign/gates/neutralization-gate.ts: composable Gate; R46 rule (reject when neutralized lift >= maxDecorativeFraction of candidate lift, default 0.5); fail-closed on no lift, fail-loud on a missing neutralized arm. - GateContext gains optional neutralizedJudgeScores/neutralizedArtifacts. - runImprovementLoop gains an optional neutralize fn; when supplied and the winner changed, it runs the placebo arm and exposes it to the gate. Additive: new exports + optional fields only; no consumer bump required.
tangletools
approved these changes
Jul 8, 2026
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 68ad00eb
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-08T00:04:16Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the one primitive the substrate genuinely lacks: a placebo control that proves a held-out win comes from the candidate's content, not from the prompt/mount footprint the content happened to add.
A held-out gate (
heldOutGate/defaultProductionGate) proves candidate beat baseline. It cannot tell apart (a) informative content from (b) simply more bytes/lines the model spends attention on.neutralizationGatecloses that hole.How
neutralizeText(src/campaign/neutralize.ts) — blanks every non-whitespace char to a 1-byte filler while preserving layout/length. Byte-exact for ASCII; character/layout-preserving for multibyte (what the tokenizer footprint tracks).neutralizationGate(src/campaign/gates/neutralization-gate.ts) — composableGate. Compares the candidate's held-out lift against a footprint-matched neutralized variant's lift (reusingpairHoldout). Holds when the neutralized variant reproduces ≥maxDecorativeFraction(default 0.5) of the lift — the "R46" decorative-reject rule — however large the raw lift. Fail-closed on no positive lift; fail-loud if composed without the loop wiring.GateContextgains optionalneutralizedJudgeScores/neutralizedArtifacts.runImprovementLoopgains an optionalneutralizefn; when supplied and the winner changed, it scores a third holdout arm and exposes it to the gate. Opt-in — one extra holdout campaign only when wired.Compose after significance:
composeGate(heldOutGate({...}), neutralizationGate({...}))— significance says the lift is real; this says the content caused it.Provenance
Lifted from a hand-rolled tax-benchmark fork (
compile-and-prove/ablation-gate). A three-agent audit confirmed the fork's other halves — its permutation lift test,<12power check, JSONL ledger, content-hash promotion, and mine→gate→promote loop — are duplicates of existing substrate primitives (heldOutGate,powerPreflight,LoopProvenanceRecord,surfaceContentHash,selfImprove) and dissolve into them. This placebo control was the only piece with no substrate equivalent (grep-confirmed: agent-eval's "decorative" uses are dead-code detection + rubric validity, not a footprint-matched content control).Verify
pnpm typecheck→ 0 errors;pnpm test→ 272 files, 2797 passed;pnpm build→ clean.