Skip to content

feat(benchmarks): add memory benchmark suite#53

Merged
drewstone merged 1 commit into
mainfrom
feat/memory-benchmark-suite
Jul 8, 2026
Merged

feat(benchmarks): add memory benchmark suite#53
drewstone merged 1 commit into
mainfrom
feat/memory-benchmark-suite

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • Adds memory benchmark task kinds for ingest, recall, temporal update, forgetting, reasoning, summarization, recommendation, and multi-party attribution.
  • Adds INDUSTRY_MEMORY_BENCHMARKS, KnowledgeMemoryBenchmarkCase, buildIndustryMemoryBenchmarkSmokeCases(), respondToIndustryMemoryBenchmarkSmokeCase(), scoreMemoryBenchmarkArtifact(), and isKnowledgeMemoryBenchmarkCase().
  • Scores memory separately from RAG: current fact recall, stale-memory safety, event/source recall, actor recall, and stale rate.
  • Adds calibration coverage showing a current-memory answer scores high while a stale-memory answer scores low.

Verification

  • pnpm install --frozen-lockfile --prefer-offline
  • pnpm exec tsc --noEmit
  • pnpm exec vitest run tests/benchmarks.test.ts — 7 passed
  • pnpm lint
  • git diff --check
  • pnpm exec vitest run — 266 passed, 12 skipped
  • pnpm build
  • File-backed proof: /tmp/agent-knowledge-memory-smoke-proof-final/knowledge-benchmark-report.json has 9 cases, 9 cells, 0 failures, mean score 1.0, 8 memory families, and 9 memory task kinds.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — fa4ce184

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-08T16:07:17Z

@drewstone drewstone merged commit 3eadd50 into main Jul 8, 2026
1 check passed
@drewstone drewstone deleted the feat/memory-benchmark-suite branch July 8, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants