test: cover Anthropic provider override across all text-complexity evaluators

[adnanrhussain]

Fourth of four foundational SDK PRs. Independent — based on main.

Why: the batch CLI is about to offer user-selectable providers. Today Anthropic is only reachable via --model-override anthropic:<model>, because requiredProviders never returns it otherwise. Once the CLI advertises provider choice, users will run every evaluator on Anthropic assuming it's supported.

What: unit tests that all six text-complexity evaluators accept an Anthropic override with only an anthropicApiKey and route every provider construction to Anthropic. This path matters because validateApiKeys short-circuits on modelOverride and checks only that provider's key — untested per evaluator until now. Plus a gated live-API test and caveats documented on ModelOverride.

Two pre-existing surprises, now test-covered so the CLI can warn:

• vocabulary deliberately uses three models (Gemini 2.5 Pro for grades 3-4, GPT-4.1 for 5-12, GPT-4o for background knowledge); a run-level override collapses all three.
• purpose takes its model from the shared cross-language eval config, so overriding diverges from that rather than from a hardcoded default.

Verified against the live API

Unit plumbing: 333 passing, lint 0 errors, typecheck clean.

Live Anthropic: all six evaluators passed against the real API on 2026-08-07 with claude-haiku-4-5-20251001, in an ~80s run. The risk this PR existed to check — Anthropic's tool-call structured output against the larger nested schemas (vocabulary, sentence-structure, purpose) — did not materialise; every schema round-tripped.

ANTHROPIC_API_KEY=... RUN_INTEGRATION_TESTS=true npm run test:integration

What this does not establish: evaluation quality on Anthropic. Scores were asserted for shape, not correctness, and evaluators remain validated against their recommended models. The vocabulary three-model collapse and the purpose config divergence still apply to anyone setting an override.

On the gating: this suite fails loudly rather than skipping when RUN_INTEGRATION_TESTS=true without a key, matching batch.integration.test.ts — a silent skip would have left the gap above permanently unchecked while reporting success.

The same run exposed the inverse bug in model-override.integration.test.ts, fixed here: its gate used &&, so it ran whenever RUN_INTEGRATION_TESTS was set and then failed on its own missing OPENAI_API_KEY instead of skipping. All three integration files now agree on the convention.

[Copilot]

Pull request overview

Adds verification coverage to ensure all TypeScript text-complexity evaluators correctly route a modelOverride to Anthropic (using only anthropicApiKey), plus a gated live integration suite to smoke-test structured-output round trips against Anthropic’s API. This supports upcoming CLI provider/model selection by preventing regressions where evaluators would accidentally require non-Anthropic keys or fail when forced onto Anthropic.

Changes:

• Add unit tests asserting all six text-complexity evaluators accept an Anthropic modelOverride and route provider construction to Anthropic with the override model.
• Add gated live integration tests that run all six evaluators against Anthropic when RUN_INTEGRATION_TESTS=true and ANTHROPIC_API_KEY is set (and fail loudly when misconfigured).
• Document key override caveats directly on ModelOverride (vocabulary multi-model collapse; purpose model sourced from shared config).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
sdks/typescript/tests/unit/evaluators/anthropic-override.test.ts	Unit coverage for Anthropic override plumbing across all six text-complexity evaluators, plus explicit tests for known override caveats.
sdks/typescript/tests/integration/anthropic-provider.integration.test.ts	Gated live Anthropic integration suite validating that each evaluator completes and returns a structured result under override.
sdks/typescript/src/evaluators/base.ts	Documentation updates to ModelOverride warning about evaluator-wide overrides and specific evaluator caveats.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[adnanrhussain]

Live verification done: all six evaluators passed against the real Anthropic API (2026-08-07, claude-haiku-4-5-20251001, ~80s run). The nested-schema risk this PR existed to check did not materialise — vocabulary, sentence-structure and purpose all round-tripped through the tool-call path. Body updated; the PR no longer carries an unproven claim.

That run also exposed the inverse gating bug in model-override.integration.test.ts, fixed here in 7ddb62a9: its gate used &&, so it ran whenever RUN_INTEGRATION_TESTS was set and then failed on its own missing OPENAI_API_KEY rather than skipping. All three integration files now agree on the convention.

The other failures in that run were pre-existing suites needing GOOGLE_API_KEY/OPENAI_API_KEY, which were not supplied — unrelated to Anthropic.