Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,12 @@ The module also exports `INDUSTRY_RAG_BENCHMARKS`, a compact manifest for BEIR,

```ts
import {
buildIndustryMemoryBenchmarkSmokeCases,
buildIndustryRagBenchmarkSmokeCases,
buildRetrievalBenchmarkCasesFromQrels,
isKnowledgeMemoryBenchmarkCase,
parseKnowledgeBenchmarkQrels,
respondToIndustryMemoryBenchmarkSmokeCase,
respondToIndustryRagBenchmarkSmokeCase,
runKnowledgeBenchmarkSuite,
} from '@tangle-network/agent-knowledge/benchmarks'
Expand All @@ -155,6 +158,15 @@ await runKnowledgeBenchmarkSuite({
respond: respondToIndustryRagBenchmarkSmokeCase,
})

await runKnowledgeBenchmarkSuite({
cases: buildIndustryMemoryBenchmarkSmokeCases(),
runDir: '.agent-knowledge/benchmark-runs/memory-smoke',
respond: ({ case: testCase }) => {
if (!isKnowledgeMemoryBenchmarkCase(testCase)) return { answer: '' }
return respondToIndustryMemoryBenchmarkSmokeCase({ case: testCase })
},
})

const cases = buildRetrievalBenchmarkCasesFromQrels({
benchmarkId: 'beir/nfcorpus',
family: 'beir',
Expand All @@ -181,6 +193,9 @@ Use `buildRetrievalBenchmarkCasesFromQrels()` for qrels-backed retrieval dataset
The smoke pack proves that every declared benchmark family is wired through the runner; full BEIR, MTEB, MS MARCO, TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA, KILT, RAGTruth, and FaithBench runs should pass real dataset rows through the same case shapes.
Use `KnowledgeAnswerBenchmarkCase` for CRAG/HotpotQA/KILT-style answer checks and RAGTruth/FaithBench-style hallucination checks by encoding required claims, forbidden claims, and expected source IDs.
Use `taskKind: 'kb-improvement'` when the artifact is candidate KB text produced by `improveKnowledgeBase()`.
Use `KnowledgeMemoryBenchmarkCase` for memory systems.
Memory cases carry ordered events plus current required facts, stale forbidden facts, expected event IDs, and expected actor IDs.
The built-in memory scorer reports current fact recall, stale-memory safety, event recall, and actor recall, which maps to LoCoMo, LongMemEval, Memora, MemoryAgentBench, MemoryBank-style personalization, GroupMemBench, and first-party memory lifecycle checks.

## Agent-Eval Integration

Expand Down
Loading
Loading