feat: add regression eval for finding Edge Function console output

[claude]

Requested via Slack thread

A user asks where their Edge Function's console.log output went. The agent's only logs tool is query_logs, and its ClickHouse schemaHint lists edge_logs, postgres_logs and function_edge_logs — but not function_logs, which is the source console output actually lands in (debugging-tools.ts, logs.ts). An agent that trusts the hint queries function_edge_logs, finds nothing but request envelopes, and tells the user their function is fine. This adds one scenario that catches that.

The seed makes the two streams cleanly separable. investigate-functions-002-edge-function-console-output/remote/logs.jsonl gives checkout-quote ten request envelopes that are all POST | 200 with no console content, plus six real console lines under a different source: an expired SPRING24 coupon being silently dropped, and a pricing-gateway timeout that falls back to zero tax. Both are things a user would want to know and neither is visible from the envelopes, so "reported only 200s" and "found the console output" are distinguishable outcomes. A second, quiet function keeps the scenario from being single-function trivial.

The eval needs query_logs, which the repo-wide pin (MCP_SERVER_VERSION in packages/core/src/index.ts) now provides at 0.11.0, so the scenario runs under the standard experiments.

Scope is one eval; non-goals are changing the pin, changing platform-lite (it already serves the ClickHouse route and the function_logs source), and hand-editing apps/web/src/data. The refreshed apps/web/src/data/regression-eval-results.json in the diff is a run-evals-changed workflow commit (15a71c3), not part of the hand-authored change. Evidence: pnpm format:check clean, pnpm typecheck clean, pnpm eval:dry parses all 49 scenarios and plans this one at mode=tools under the standard experiments, and a throwaway platform-lite test (run, then deleted) confirmed the seed's console rows land under source = 'function_logs' while the envelope rows land only under function_edge_logs/edge_logs.

pnpm format:check
pnpm typecheck
pnpm eval:dry -- --eval investigate-functions-002-edge-function-console-output
pnpm --filter @supabase-evals/core test

Requested by Matt Rossman.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk

---

Generated by Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview	Aug 26, 2026 1:18pm

[barryroodt]

Requesting changes for one blocker: the query check can accept the assistant’s narration instead of an actual query_logs query. Please scope it to real SQL inputs and make the intended policy explicit, including whether a broad unified-logs query that finds the correct rows should pass.

The scenario, seed, and version isolation otherwise look sound.

For agents

Exact evidence + requested invariant

Pinned head: 15a71c30169813ee80afeef94f378cf3890c5fea.

In evals/investigate-functions-002-edge-function-console-output/EVAL.ts, lines 14-24 combine every string tool-call input with serializeTranscript(..., { includeToolCallInputs: true }). The pinned serializer includes every message’s text unconditionally at packages/core/src/index.ts:580-582. Line 39 then matches /function_logs/i anywhere in that combined text, line 50 gates check 2 on the match, and line 62 gates the eval through checks.every. Raw tool outputs are excluded unless the assistant restates them.

Requested invariant: check only SQL inputs from actual query_logs calls, then apply the chosen source policy to those inputs. Please decide whether a broad unified-logs query that surfaces the correct rows satisfies that policy.

Verification status: Vercel Preview Comments passed; CodeSmith skipped. GitHub exposes no format, typecheck, or dry-run check results.

Non-blocking wording note: experiments/claude-code-sonnet-5-mcp-0-11.ts:17 says “past 0.10.0”, though query_logs exists at 0.10.0.

[barryroodt]

The original blocker is fixed: queryLogsSql now reads only real query_logs calls. couldReturnConsoleRows leaves one replacement blocker, so REQUEST_CHANGES stays. The rest of the delta looks sound.

For agents

Delta review details

• Head: 3e37545a2494c16b4fe91ff88166182a69ecb09f
• Mechanism: a metadata-only grouped logs query passes SOURCE_DISCOVERY without satisfying SOURCE_NARROWED.
• Required invariant: actual tool-result evidence must show that the query returned the scenario’s known console rows.
• Related shallow matches include function_logs or source = in comments and strings, plus lower(source).
• CI: Vercel PASS, Vercel Preview Comments PASS, CodeSmith SKIPPED, no failed checks.
• The later workflow commit changed only generated result text. EVAL.ts did not change.

[claude]

Your hardening was right and is kept as-is: findings must co-occur in one row, and only row values count. Both close real holes. Pushed 85b1011 to fix the row extractor under it.

queryResultRows only accepted an array result. At the pinned 0.11.0, query_logs returns { result: wrapWithUntrustedDataBoundary(body) }, and that helper returns a string — JSON.stringify(body) between <untrusted-data-{uuid}> tags, prose either side (tools/debugging-tools.ts:254, tools/util.ts:89-101 @ mcp-server-supabase-v0.11.0). So Array.isArray is false and zero rows come out.

It landed. At 703356b the eval flipped:

- "passed": true                                   + "passed": false
- attempts: 1                                      + attempts: 2
- markers returned: SPRING24, pricing-gateway,     + markers returned: none
    cart_8f21ac, timed out after 3 retries

The judge check still PASSED on that same run, naming SPRING24 and the pricing-gateway timeout — so the agent did read and report the rows. False negative.

The unit test stayed green because the fixture built { result: rows } directly, without the wrapper — a shape the server never returns.

Fix: unwrap a string result once, then apply your existing { result: [rows] } rule to the parsed value (the embedded JSON is the management API body, itself { result: [rows] }). Direct array path stays. Plain JSON.parse runs first so every working shape keeps its exact behaviour, and the tag scan goes backwards because the prose names the tag on both sides of the real block — only bytes between the real tags count, so a marker quoted in the prose is not evidence.

Fixtures now build the real wrapped envelope. Verified all six of your reject cases still reject against both implementations, so nothing was relaxed; added two more (wrapped error body, markers quoted in the boundary prose). test:eval-scorers 14 passed. Driving check 2 with the real envelope returns SPRING24, cart_8f21ac, pricing-gateway, timed out after 3 retries, and split-across-rows still fails.

---

Generated by Claude Code

[barryroodt]

Both scorer blockers are resolved, and the final delta looks good to merge.

[claude]

Merged main in and resolved the conflict in apps/framework/package.json — kept main's updated test:vercel-runner list alongside this branch's test:eval-scorers script. main now pins MCP_SERVER_VERSION to 0.11.0, so the version-pinning scaffolding here is obsolete: dropped the claude-code-sonnet-5-mcp-0-11 experiment, removed the skipEval clause from claude-code-sonnet-5, and restored the single skills predicate in claude-code-sonnet-5-no-skills. Format check, typechecks, and the eval scorer tests all pass locally.

---

Generated by Claude Code

[barryroodt]

Approved