chore: bump MCP_SERVER_VERSION to 0.11.0 and adapt references - #236
Merged
Conversation
Adds investigate-functions-002-edge-function-console-output, which asks an agent to pull up the console.log output of a deployed Edge Function. The only logs tool it has is query_logs, whose ClickHouse schema hint lists edge_logs, postgres_logs and function_edge_logs but not function_logs, which is where console output actually lands. The seed makes the two streams separable: the request envelopes are all 200s with no console content, so an agent that stops at function_edge_logs can only report that the invocations succeeded. query_logs first ships in @supabase/mcp-server-supabase 0.10.0, which is newer than the repo-wide MCP_SERVER_VERSION pin, so the scenario runs under a new version-pinned experiment (claude-code-sonnet-5-mcp-0-11) and is skipped by the two shared regression experiments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
…_logs The eight log-seeding evals returned nothing: get_logs at the pinned 0.8.1 emits BigQuery-dialect SQL against per-service tables, which platform-lite rejects. Past 0.10.0 the server hides get_logs and exposes query_logs, whose ClickHouse dialect platform-lite already serves. Move the platform-lite devDependency to ^0.11.0 so the repo's only type-checked contact with the package tracks what the harness runs, and regenerate the lockfile; the new peer is auto-installed as the old one was. investigate-logs-001 hard-asserts exact counts and each invocation is seeded onto two log sources, so an unfiltered aggregate doubles them. Its prompt now says which rows to count; the scorer is untouched.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Check 2 read `queriedSources`, which appended
`serializeTranscript(..., { includeToolCallInputs: true })`. That helper
emits every message's text unconditionally, before it looks at the flag
(packages/core/src/index.ts:580-582), so an assistant sentence like "the
logs are in function_logs" satisfied `/function_logs/i` and passed the
check with no query behind it — and check 2 gates the eval via
`checks.every`.
Read the SQL only from `ctx.toolCalls` entries whose `tool.toolName` is
`query_logs`, taking the `sql` argument (the parameter name in the pinned
0.11.0 server). No `query_logs` call at all now fails. The
source-discovery diagnostic note reads the same real SQL, so narration
cannot satisfy it either.
Policy, stated in the code: a query passes if it names `function_logs`,
or if it reads the unified `logs` stream without narrowing `source`. A
broad unified-logs query that surfaces the console rows passes on
purpose — the check exists to stop a pass built on narration alone, not
to demand a particular WHERE clause, and check 1 (the judge) already
decides whether the console content was reported.
Also fixes the experiment comment: `query_logs` exists at 0.10.0, so the
experiment can go once MCP_SERVER_VERSION reaches 0.10.0 or newer, not
"past" it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
The second check inspected the shape of the agent's SQL: accept it if it
named `function_logs`, or if it read the unified `logs` stream without
narrowing `source`. That is unbounded, and it was wrong in both
directions.
`SELECT source, count(*) FROM logs GROUP BY source` reads no log row at
all, yet it matched `logs`, avoided the `select distinct source`
rejection and never narrowed `source` — so it passed as a "broad query",
letting an agent supply the expected narration without ever reading a
console row. In the other direction the regexes matched `function_logs`
or `source =` inside comments and string literals, and could not see
through wrappers like `lower(source)`.
Assert on what came back instead. The check still filters to real
`query_logs` tool calls, then requires at least one call whose result
contains two distinct markers that occur only in the scenario's
`edge-function-runtime` console rows (SPRING24, pricing-gateway,
cart_8f21ac, exec-3b91d2f0, "timed out after 3 retries") and never in
the `edge-function` request-envelope rows. Two markers rather than one
so a marker echoed back inside agent-authored SQL is not by itself
evidence; every console row carrying a finding the judge asks for clears
that bar on message text alone.
`ToolCallRecord.result` is populated for CLI harnesses: the Claude Code
parser emits `tool_result` events with the raw content, and
`adaptTranscript` pairs them onto the call by `tool.id`. Its shape is
harness-specific, so the result is stringified rather than assumed to be
any one of string / MCP content-block array / `{ result: rows }`
envelope.
Drops `SOURCE_NARROWED` and `couldReturnConsoleRows`. Keeps the
non-gating source-discovery diagnostic, still derived from real
tool-call SQL, and adds the markers actually returned to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
The per-row, values-only matching added in d82b8a7 is right and is kept unchanged. Its row extractor is not: it only accepted a payload whose `result` was an array. At the pinned `@supabase/mcp-server-supabase@0.11.0`, `query_logs` returns `{ result: wrapWithUntrustedDataBoundary(body) }`, and that helper returns a STRING with `JSON.stringify(body)` embedded between `<untrusted-data-{uuid}>` tags and surrounded by prose (tools/debugging-tools.ts:254 and tools/util.ts:89-101 at tag mcp-server-supabase-v0.11.0). `Array.isArray` is therefore false, no rows are extracted, and the check reports "markers returned: none". That is a false negative, and it landed: at 703356b the eval flipped to passed=false / attempts=2 with check 2 reporting no markers, while the judge check still PASSED and named SPRING24 and the pricing-gateway timeout. The previous refresh, on the same scenario, reported all four markers. The unit test stayed green because its fixture built `{ result: rows }` directly, omitting the boundary wrapper — a shape the real server never returns. So: unwrap a string `result` once, then apply the existing `{ result: [rows] }` rule to the parsed value (the JSON the server embeds is the management API body, which is itself `{ result: [rows] }`). The direct array path stays. Plain `JSON.parse` is tried first so every already-working shape keeps its exact behaviour, and the tag scan runs backwards because the wrapper prose names the tag on both sides of the real block — only bytes between the real tags count, so a marker quoted in the prose is not evidence. Fixtures now build the real wrapped envelope. All of d82b8a7's reject cases still reject (verified against both implementations), plus new ones for a wrapped error body and markers quoted in the boundary prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
Left over from replacing the boundary regex with a position scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
With MCP_SERVER_VERSION now pinned to 0.11.0, the workaround that #232 added to run the console-output eval against a newer MCP server is dead. Worse than redundant: its skipEval clauses would wrongly exclude investigate-functions-002-edge-function-console-output from the default experiments. - Delete experiments/claude-code-sonnet-5-mcp-0-11.ts. - Drop the investigate-functions-002 skipEval from claude-code-sonnet-5. - Reduce claude-code-sonnet-5-no-skills' skipEval to the pre-existing `skills?.length === 0` clause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxPoSQH7JY8rjV3CVHFbZW
barryroodt
changed the base branch from
main
to
feat/eval-edge-function-console-output
August 25, 2026 13:37
Adds investigate-functions-002-edge-function-console-output, which asks an agent to pull up the console.log output of a deployed Edge Function. The only logs tool it has is query_logs, whose ClickHouse schema hint lists edge_logs, postgres_logs and function_edge_logs but not function_logs, which is where console output actually lands. The seed makes the two streams separable: the request envelopes are all 200s with no console content, so an agent that stops at function_edge_logs can only report that the invocations succeeded. query_logs first ships in @supabase/mcp-server-supabase 0.10.0, which is newer than the repo-wide MCP_SERVER_VERSION pin, so the scenario runs under a new version-pinned experiment (claude-code-sonnet-5-mcp-0-11) and is skipped by the two shared regression experiments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
Check 2 read `queriedSources`, which appended
`serializeTranscript(..., { includeToolCallInputs: true })`. That helper
emits every message's text unconditionally, before it looks at the flag
(packages/core/src/index.ts:580-582), so an assistant sentence like "the
logs are in function_logs" satisfied `/function_logs/i` and passed the
check with no query behind it — and check 2 gates the eval via
`checks.every`.
Read the SQL only from `ctx.toolCalls` entries whose `tool.toolName` is
`query_logs`, taking the `sql` argument (the parameter name in the pinned
0.11.0 server). No `query_logs` call at all now fails. The
source-discovery diagnostic note reads the same real SQL, so narration
cannot satisfy it either.
Policy, stated in the code: a query passes if it names `function_logs`,
or if it reads the unified `logs` stream without narrowing `source`. A
broad unified-logs query that surfaces the console rows passes on
purpose — the check exists to stop a pass built on narration alone, not
to demand a particular WHERE clause, and check 1 (the judge) already
decides whether the console content was reported.
Also fixes the experiment comment: `query_logs` exists at 0.10.0, so the
experiment can go once MCP_SERVER_VERSION reaches 0.10.0 or newer, not
"past" it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
The second check inspected the shape of the agent's SQL: accept it if it
named `function_logs`, or if it read the unified `logs` stream without
narrowing `source`. That is unbounded, and it was wrong in both
directions.
`SELECT source, count(*) FROM logs GROUP BY source` reads no log row at
all, yet it matched `logs`, avoided the `select distinct source`
rejection and never narrowed `source` — so it passed as a "broad query",
letting an agent supply the expected narration without ever reading a
console row. In the other direction the regexes matched `function_logs`
or `source =` inside comments and string literals, and could not see
through wrappers like `lower(source)`.
Assert on what came back instead. The check still filters to real
`query_logs` tool calls, then requires at least one call whose result
contains two distinct markers that occur only in the scenario's
`edge-function-runtime` console rows (SPRING24, pricing-gateway,
cart_8f21ac, exec-3b91d2f0, "timed out after 3 retries") and never in
the `edge-function` request-envelope rows. Two markers rather than one
so a marker echoed back inside agent-authored SQL is not by itself
evidence; every console row carrying a finding the judge asks for clears
that bar on message text alone.
`ToolCallRecord.result` is populated for CLI harnesses: the Claude Code
parser emits `tool_result` events with the raw content, and
`adaptTranscript` pairs them onto the call by `tool.id`. Its shape is
harness-specific, so the result is stringified rather than assumed to be
any one of string / MCP content-block array / `{ result: rows }`
envelope.
Drops `SOURCE_NARROWED` and `couldReturnConsoleRows`. Keeps the
non-gating source-discovery diagnostic, still derived from real
tool-call SQL, and adds the markers actually returned to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
The per-row, values-only matching added in d82b8a7 is right and is kept unchanged. Its row extractor is not: it only accepted a payload whose `result` was an array. At the pinned `@supabase/mcp-server-supabase@0.11.0`, `query_logs` returns `{ result: wrapWithUntrustedDataBoundary(body) }`, and that helper returns a STRING with `JSON.stringify(body)` embedded between `<untrusted-data-{uuid}>` tags and surrounded by prose (tools/debugging-tools.ts:254 and tools/util.ts:89-101 at tag mcp-server-supabase-v0.11.0). `Array.isArray` is therefore false, no rows are extracted, and the check reports "markers returned: none". That is a false negative, and it landed: at 703356b the eval flipped to passed=false / attempts=2 with check 2 reporting no markers, while the judge check still PASSED and named SPRING24 and the pricing-gateway timeout. The previous refresh, on the same scenario, reported all four markers. The unit test stayed green because its fixture built `{ result: rows }` directly, omitting the boundary wrapper — a shape the real server never returns. So: unwrap a string `result` once, then apply the existing `{ result: [rows] }` rule to the parsed value (the JSON the server embeds is the management API body, which is itself `{ result: [rows] }`). The direct array path stays. Plain `JSON.parse` is tried first so every already-working shape keeps its exact behaviour, and the tag scan runs backwards because the wrapper prose names the tag on both sides of the real block — only bytes between the real tags count, so a marker quoted in the prose is not evidence. Fixtures now build the real wrapped envelope. All of d82b8a7's reject cases still reject (verified against both implementations), plus new ones for a wrapped error body and markers quoted in the boundary prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
Left over from replacing the boundary regex with a position scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JACD1KWUAJ4G5BD7KfwTtk
barryroodt
force-pushed
the
feat/eval-edge-function-console-output
branch
from
August 25, 2026 13:41
b7f0e26 to
00c17bf
Compare
…-mcp-server-0-11-0 The base branch was rebased onto main (a58ee5b, PR #228's `interface:` frontmatter requirement), which moved the merge base back to old main and made this PR un-mergeable. Resolutions: - evals/investigate-functions-002-edge-function-console-output/README.md (add/add): kept this branch's wording. The base side still points at experiments/claude-code-sonnet-5-mcp-0-11.ts, which this branch deletes. - Re-applied this branch's removal of the MCP 0.11 version-isolation scaffolding (d7164d7), which the rebase resurrected as a merge-base artifact: deleted experiments/claude-code-sonnet-5-mcp-0-11.ts again and dropped the investigate-functions-002 skipEval clauses from claude-code-sonnet-5.ts and claude-code-sonnet-5-no-skills.ts. Both base versions were byte-identical to the pre-rebase ones, so nothing new was lost. The unrelated `skills?.length === 0` clause is kept. - apps/web/src/data/*eval-results.json: took the base branch's versions verbatim; these are bot-generated snapshots owned by the refresh workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxPoSQH7JY8rjV3CVHFbZW
…l README" This reverts commit 5e824d7.
This reverts commit d7164d7.
…ation The two `git revert -m 1` commits that unstacked PR #232 also removed the `interface:` frontmatter work that landed on main in #228, because the rebased #232 branch carried it in as its second parent. Restore those paths to main's versions so the branch diff is the pin bump alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxPoSQH7JY8rjV3CVHFbZW
The provenance comments in the ClickHouse log fixture and its test named specific mcp-server-supabase releases (0.8.1/0.9.0/0.10.0), so every pin bump had to resync prose that carries no constraint. Reword them to describe the pinned server generically instead. The only intentional hardcodes left are the devDependency range and MCP_SERVER_VERSION. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxPoSQH7JY8rjV3CVHFbZW
mattrossman
changed the base branch from
feat/eval-edge-function-console-output
to
main
August 25, 2026 20:39
mattrossman
approved these changes
Aug 25, 2026
mattrossman
changed the title
barryroodt
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Pedro Rodrigues · Slack thread
Updates MCP server pin to
0.11.0to enable the newquery_logstool.Also rewords / trims references to the old pin.
Closes AI-1122