Skip to content

fix(drift): route every provider surface to auto-fix via a surface registry#311

Merged
jpr5 merged 3 commits into
mainfrom
harden/collector-provider-key
Jul 17, 2026
Merged

fix(drift): route every provider surface to auto-fix via a surface registry#311
jpr5 merged 3 commits into
mainfrom
harden/collector-provider-key

Conversation

@jpr5

@jpr5 jpr5 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

The hole

extractProviderName matched drift-block titles against ~9 hardcoded
PROVIDER_MAP keys via text.includes(). ~15 additional provider surfaces
(Cohere, Bedrock, Ollama, fal, ElevenLabs, images, video, moderation,
transcription, rerank, vertex-ai, ...) emit valid, fully-parseable
API DRIFT DETECTED: blocks whose PROSE titles are not keys → extractProviderName
returns null → the block is quarantined (exit 5) → fix-drift.yml gates
auto-remediation on exit 2 only, so those surfaces were never auto-fixed
(mislabeled "manual triage"). models.drift.ts is left as-is (deliberate
PROVIDER_MAP-matching canaries).

The design — structural keying via a single-source-of-truth registry

  • SURFACE_REGISTRY (src/__tests__/drift/surface-registry.ts), slug-keyed,
    imported by BOTH schema.ts (validate a slug at emit time) and the collector
    (resolve slug → builderFile/builderFunctions). One table, two consumers —
    adding a surface is a single edit both see.
  • formatDriftReport(context, diffs, surface?) emits a machine-readable
    Surface: <slug> marker line. An unknown slug throws at emit time.
  • Collector resolution:
    • marker present + known slug → exit-2 entry (auto-fixable) — the fix.
    • marker present + UNKNOWN slug → THROW (loud, exit 1), never silent
      quarantine — a future unkeyed surface fails loudly instead of silently
      routing to manual triage.
    • no-marker legacy block → today's provider-label fallback (exit 5 only for
      truly un-attributable output).
  • Every currently-unmapped *.drift.ts emitter now passes its slug;
    builderFunctions filled from the real source exports (not invented).
    Bedrock is split per the spec: bedrock-invoke/bedrock-invoke-stream
    bedrock.ts, bedrock-converse/bedrock-converse-stream
    bedrock-converse.ts. Vertex AI keys onto the Gemini path (gemini.ts).
  • A registry-coverage test asserts every entry resolves to an existing
    builderFile with non-empty builderFunctions (mirrors fix-drift.ts
    validation, so a bad entry fails locally, not in CI).

Base-report reuse fix (own commit — efficiency, not safety)

The collector wrote only { timestamp, entries, quarantine? }, but the reuse
guard reads report.generatedAt + report.conclusion, so reuse always
failed
and every PR ran a fresh ~30-min live base. The collector now writes
generatedAt (ISO alias of timestamp, kept for back-compat) and conclusion
(derived from the exit code: 0→clean, 2→critical, 5→quarantine).

Red-green (fixtures only, no live SDK)

RED (original code) — a Cohere/fal/ElevenLabs drift block routes to
quarantine (len 1, exit 5), never an entry:

× "Cohere /v2/chat (non-streaming)" → exit-2 auto-fix entry (EXPECTED to FAIL)
  → expected [ { …(4) } ] to have a length of +0 but got 1
× "fal.ai sync-run (image payload)" → exit-2 auto-fix entry (EXPECTED to FAIL)
  → expected [ { …(4) } ] to have a length of +0 but got 1
× "ElevenLabs /v1/sound-generation 400 error" → exit-2 auto-fix entry (EXPECTED to FAIL)
  → expected [ { …(4) } ] to have a length of +0 but got 1
Tests  3 failed (3)

GREEN (new code, with the Surface: marker) — all three route to an
exit-2 entry, quarantine empty:

✓ src/__tests__/ws5-red-proof.test.ts (3 tests) 3ms
Tests  3 passed (3)

Surfaces flipped 5 → 2: cohere-chat, fal-sync, elevenlabs (plus every
other previously-unmapped surface: rerank, bedrock-*, ollama, images, video,
moderation, transcription, vertex-ai, fal-queue).

Additional in-suite coverage: unknown-slug marker throws (not quarantine);
isBaseReportReusable accepts a written clean report; registry-coverage test.
Full suite: 4637 passed. format:check, lint, tsc (both tsconfigs),
build, commitlint all clean.

🤖 Generated with Claude Code

jpr5 added 2 commits July 17, 2026 13:11
…gistry

extractProviderName matched drift-block titles against ~9 hardcoded
PROVIDER_MAP keys via text.includes(). ~15 additional provider surfaces
(Cohere, Bedrock, Ollama, fal, ElevenLabs, images, video, moderation,
transcription, rerank, vertex-ai, ...) emit valid, fully-parseable drift
blocks whose PROSE titles are not keys, so extractProviderName returned
null, the block was quarantined (exit 5), and fix-drift.yml (which gates
auto-remediation on exit 2) never ran for them.

Replace the title-substring match with structural keying:

- New single-source-of-truth SURFACE_REGISTRY (surface-registry.ts),
  slug-keyed, imported by BOTH schema.ts (validate a slug at emit time)
  and the collector (resolve slug -> builderFile/functions).
- formatDriftReport gains an optional `surface` slug that emits a
  machine-readable `Surface: <slug>` marker line; an unknown slug throws
  at emit time.
- Collector resolution: marker present + known slug -> exit-2 entry
  (auto-fixable); marker present + UNKNOWN slug -> THROW (loud, exit 1),
  never silent quarantine; no-marker legacy block -> today's provider-label
  fallback (exit 5 only for truly un-attributable output).
- Wire every currently-unmapped *.drift.ts emitter to pass its slug;
  builderFunctions filled from the real source exports.
- Registry-coverage test asserts every entry resolves to an existing
  builderFile with non-empty builderFunctions.

models.drift.ts is left as-is (deliberate PROVIDER_MAP-matching canaries).
The collector wrote only { timestamp, entries, quarantine? }, but the
base-report reuse guard reads report.generatedAt and report.conclusion
(isBaseReportReusable requires a same-UTC-day generatedAt plus a known-good
conclusion). Since neither field was ever written, reuse always failed and
every PR ran a fresh ~30-min live base.

Write both fields on the report:
- generatedAt: ISO alias of the existing timestamp (timestamp kept for
  back-compat with existing consumers).
- conclusion: derived from the collector exit code via conclusionForExitCode
  (0 -> "clean", 2 -> "critical", 5 -> "quarantine").

The exit code is now computed before the report write so conclusion can be
derived from it. Efficiency only — a failed reuse just runs a fresh base, so
safety is unaffected.
@pkg-pr-new

pkg-pr-new Bot commented Jul 17, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@311

commit: 37733e6

…r registration

The collector indexed SURFACE_REGISTRY[surfaceKey] with untrusted marker text.
A plain-object bracket lookup walks the prototype chain, so a slug like
constructor/toString/hasOwnProperty/valueOf/__proto__ resolved to a truthy
inherited member, skipped the unknown-surface throw, and emitted a garbage
DriftEntry (builderFile: undefined) that fix-drift.ts chokes on far from the
root cause. Gate the lookup on isKnownSurface (Object.prototype.hasOwnProperty
.call) before indexing, matching the emit-side guard, so an unknown or
prototype-member slug THROWS loudly. Apply the same Object.hasOwn discipline to
the legacy PROVIDER_LABEL_MAP[label] fallback for parity.

Strengthen the WS-5 red-green so >=3 cells genuinely require the surface seam.
The prior cases (fal-sync, elevenlabs) stayed green on revert because their
titles contained legacy provider-label substrings that extractProviderName
matched anyway. Replace with four surfaces (moderation, video, transcription,
rerank) whose emitted titles contain NO legacy label, so the marker is the only
route; each asserts extractProviderName is null and goes RED when the seam is
reverted.

Replace the tautological coverage test (which iterated the registry's own keys)
with a static TypeScript-AST scan of every *.drift.ts formatDriftReport third
argument, asserting each emitted slug is registered (and no orphan slugs). This
locks the every-emitter-is-registered invariant at TEST time, so an
unregistered emitter fails CI even on a credential-less run where the runtime
throw is never reached. Add a generatedAt->sameUtcDay staleness test proving a
prior-day base report is not reusable. Keep the existsSync builder-file
integrity test.
@jpr5
jpr5 marked this pull request as ready for review July 17, 2026 20:30
@jpr5
jpr5 merged commit a21e786 into main Jul 17, 2026
27 checks passed
@jpr5
jpr5 deleted the harden/collector-provider-key branch July 17, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant