Skip to content

LLM Benchmark Fixes#5552

Merged
bradleyshep merged 4 commits into
masterfrom
bradley/llm-benchmarks-0715
Jul 15, 2026
Merged

LLM Benchmark Fixes#5552
bradleyshep merged 4 commits into
masterfrom
bradley/llm-benchmarks-0715

Conversation

@bradleyshep

Copy link
Copy Markdown
Contributor

Description of Changes

Fixes the LLM benchmark so C# and TypeScript can actually pass. Both scored 0% in every periodic run while all golden answers publish; the causes were generally in the harness and task suite.

  • The prompt hardcoded "Use valid SpacetimeDB 1.6 syntax" since the original benchmark PR (LLM Benchmarking #3486), so models were instructed to write the pre-2.0 API. The version now comes from spacetimedb-lib at compile time.
  • TypeScript task specs named tables camelCase while goldens use snake_case, and the eval layer camelized TS table names in SQL. All three now agree on snake_case (15 specs, sql_fmt.rs).
  • Scorers read tables through anonymous spacetime sql, but no golden marked its tables public, so every data-parity check failed in every language and every run. The 17 tasks whose scorers read tables now say (public) in the spec and declare it in the goldens, in all three languages.
  • Golden cleanup: schema keys match table names, and a few skill lines for failure patterns the runs exposed (C# nullable unwrap after Find, Insert returns the row, ScheduleAt import in the TS scheduled-table snippet).

Local results with gpt-5.4-mini in guidelines mode, full 47-task suite: C# 0% to ~94%, TypeScript 0% to ~99%, Rust 95.5% (its data-parity checks had been silently failing too).

API and ABI breaking changes

None.

Expected complexity level and risk

  1. The tool changes are small (version lookup, one casing arm). The bulk of the diff is benchmark data: task specs and golden answers. Risk is that benchmark scores shift meaningfully after merge, which is the point.

Testing

  • Goldens build and publish for all three languages after the spec/golden edits.
  • Full 47-task suite run locally per language with gpt-5.4-mini
  • cargo test -p xtask-llm-benchmark; new test asserts the prompt carries the workspace version.
  • Reviewer: next periodic run shows C#/TS publishing and scoring instead of 0%.

@bradleyshep bradleyshep enabled auto-merge July 15, 2026 18:31
@bradleyshep bradleyshep added this pull request to the merge queue Jul 15, 2026
Merged via the queue into master with commit ea3402a Jul 15, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants