Skip to content

test(benchmarks): add all-family smoke pack#52

Merged
drewstone merged 1 commit into
mainfrom
feat/knowledge-benchmark-smoke-pack
Jul 8, 2026
Merged

test(benchmarks): add all-family smoke pack#52
drewstone merged 1 commit into
mainfrom
feat/knowledge-benchmark-smoke-pack

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • Adds buildIndustryRagBenchmarkSmokeCases() and respondToIndustryRagBenchmarkSmokeCase() to run one CI-safe case for every declared RAG/KB benchmark family.
  • Covers BEIR, MTEB retrieval, MS MARCO, TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA, KILT, RAGTruth, FaithBench, and first-party KB improvement in one persisted report.
  • Updates the benchmark README and adds a test that proves all families execute through runKnowledgeBenchmarkSuite().

Verification

  • pnpm exec tsc --noEmit
  • pnpm exec vitest run tests/benchmarks.test.ts — 5 passed
  • pnpm lint
  • git diff --check
  • pnpm exec vitest run — 264 passed, 12 skipped
  • pnpm build
  • File-backed proof: /tmp/agent-knowledge-industry-smoke-proof-2/knowledge-benchmark-report.json has 13 cases, 13 cells, 0 failures, mean score 1.0, and every family represented.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — a95bb687

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-08T01:24:48Z

@drewstone drewstone merged commit 894154f into main Jul 8, 2026
1 check passed
@drewstone drewstone deleted the feat/knowledge-benchmark-smoke-pack branch July 8, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants