Skip to content

feat(benchmarks): add knowledge benchmark suite#51

Merged
drewstone merged 1 commit into
mainfrom
feat/knowledge-benchmark-suite
Jul 8, 2026
Merged

feat(benchmarks): add knowledge benchmark suite#51
drewstone merged 1 commit into
mainfrom
feat/knowledge-benchmark-suite

Conversation

@drewstone

@drewstone drewstone commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Adds a reusable @tangle-network/agent-knowledge/benchmarks entrypoint for retrieval, RAG answer quality, hallucination, and KB-improvement benchmark cases.
  • Declares the industry benchmark families we care about: BEIR, MTEB retrieval, MS MARCO/TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA/KILT, RAGTruth/FaithBench, and first-party KB-improvement suites.
  • Wraps the existing campaign runner so benchmark runs inherit resumability, traces, cost, latency, split summaries, and persisted JSON/Markdown reports.
  • Adds qrels/JSONL parsers, normalized scoring dimensions, docs, and smoke tests.

Verification

  • pnpm exec tsc --noEmit
  • pnpm exec vitest run
  • pnpm build
  • pnpm lint
  • git diff --check
  • git merge-tree --write-tree origin/main HEAD
  • Proof artifact written at /tmp/agent-knowledge-benchmark-proof-clean/knowledge-benchmark-report.json and .md with 2 cells, 0 failures, mean score 0.5.

Agent-eval reusable benchmark primitives landed in tangle-network/agent-eval#326.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 2166fd1d

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-08T01:18:55Z

@drewstone drewstone merged commit 212cbd6 into main Jul 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants