From 8b9422e29d60207cb05985950c7069ae374ef677 Mon Sep 17 00:00:00 2001 From: drewstone Date: Tue, 7 Jul 2026 21:03:59 -0600 Subject: [PATCH] docs(gates): document neutralizationGate (placebo/content-causality control) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds neutralizationGate to the README gate enumeration and an improvement-glossary entry — proves content (not added prompt bytes) caused a lift; shipped 0.107.0. --- README.md | 2 +- docs/improvement-glossary.md | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3d7e98de..9b30abf7 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,7 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints | Subpath | What it gives you | |---|---| -| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. | +| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `neutralizationGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. | | `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) | | `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency | | `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep | diff --git a/docs/improvement-glossary.md b/docs/improvement-glossary.md index 800c33f8..71a9bca7 100644 --- a/docs/improvement-glossary.md +++ b/docs/improvement-glossary.md @@ -196,4 +196,8 @@ Every entrant is re-scored on the **same** holdout with the **same** judges, so - Do not put eval logic inside a proposer — scoring lives in `dispatch` + `judges`, proposing lives in the proposer. - Do not let a proposer read held-out judge scores — `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer. - Do not read `lift` without `result.power`/MDE — a "+4" on a valset too small to detect +4 is noise wearing a number. -- Do not look for a `compositeProposer` — `fapoProposer` is the composite; `compareProposers` is the race; `runImprovementLoop`/`improve` is the plug. \ No newline at end of file +- Do not look for a `compositeProposer` — `fapoProposer` is the composite; `compareProposers` is the race; `runImprovementLoop`/`improve` is the plug. + +### neutralizationGate — the placebo / content-causality control + +Standard gates prove a candidate *beat baseline*. `neutralizationGate` proves the candidate's **content** caused the lift, not the extra prompt bytes: it blanks the candidate's added content to byte-length-matched filler, holds everything else fixed, and requires the lift to vanish. A fully-neutralized candidate that still scores is decorative and is rejected. Exports: `neutralizationGate` (`src/campaign/gates/neutralization-gate.ts`), `neutralizeText` (`src/campaign/neutralize.ts`). Since 0.107.0. Pair it with `heldOutGate` in any loop that promotes authored artifacts (prompts, tool docs, knowledge) so a lift that is really just added prompt size cannot be shipped.