feat: add agent evals#229
Draft
angeloashmore wants to merge 8 commits into
Draft
Conversation
Plain vitest evals that drive a real Claude Code agent (via the Agent SDK) against the built CLI in a generic, hermetic config. Adds the `agent` fixture, an `toHaveRun` matcher, an LLM-judge matcher with a never-regress ratchet, and a per-run reporter that records efficiency metrics and the agent's prismic calls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merge the agent fixture, matchers, judge, config isolation, and result recording into eval/it.ts; delete claude.ts, report.ts, and setup.ts. Move the judge ratchet from a read-back of results.jsonl into an inline threshold literal that vitest -u rewrites like a snapshot, so results.jsonl is write-only. Trim recorded fields to effectiveness and effort (turns, tokens, duration); drop cost, which swings with provider pricing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Judge uses the SDK's native structured output instead of hand-parsing JSON; score is range-checked to [0, 1] - Create a throwaway Claude config dir per agent/judge call and clean it up; set persistSession: false so runs stay ephemeral - Detect -u from vitest's update state; the worker's process.argv is empty - Fix raiseLiteral rewriting the wrong literal on the callsite line, and support inline bars - Fix the evals npm script so vitest also receives RUN_EVALS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each eval now runs EVAL_TRIALS times (default 3) via it.for; a custom reporter prints per-eval pass rates with infra failures excluded and appends per-trial rows (pass, cost, tokens, turns, commands) to results.jsonl. The judge is binary like agent-eval's: prose criteria, pass/fail verdicts, no scores, no ratcheting bars. The full eval catalog is implemented as it.skip drafts; only the rich-text smoke eval runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`node --run evals:report` prints one block per eval with one line per run (pass rate, cost, mean tokens/turns/duration) and a latest-run footer with best-of and per-trial rates. Rows now carry a per-run id so the digest groups runs exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rows record the git SHA that produced them, shown in the digest so rate changes point at a commit range. The file keeps the newest 100 runs; the digest shows the last 10 per eval. results.jsonl merges with the union driver so branch runs never conflict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The judge now calls the Messages API directly with structured outputs instead of spawning a Claude Code session. The agent fixture writes its own task.meta via the built-in task context, trial rows no longer carry a parsed trial number, and eval env vars are parsed with a zod schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves:
Description
Adds an eval suite that measures how well AI agents (like Claude Code) can use the CLI.
Each eval gives an agent a task in a temp project. The agent runs the real CLI against a real repo, and the test checks the outcome. A few notes on the design:
node --run evals:report.node --run evals) and refuse to run outside a container or CI, because the agent runs with permissions bypassed.Checklist
Preview
How to QA 1
Inside a disposable container with
.env.test.localavailable:🤖 Generated with Claude Code
Footnotes
Please use these labels when submitting a review:
⚠️ #issue: Strongly suggest a change.
❓ #ask: Ask a question.
💡 #idea: Suggest an idea.
🎉 #nice: Share a compliment. ↩