Skip to content

feat(benchmarks): rank real memory adapters#54

Merged
drewstone merged 1 commit into
mainfrom
feat/real-memory-benchmark-run
Jul 8, 2026
Merged

feat(benchmarks): rank real memory adapters#54
drewstone merged 1 commit into
mainfrom
feat/real-memory-benchmark-run

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • Adds runMemoryAdapterBenchmark() to run memory cases through actual AgentMemoryAdapter implementations and persist a ranking report.
  • Adds a 12-case first-party memory lifecycle benchmark covering ingest, recall, temporal updates, stale forgetting, reasoning, summarization, recommendations, and multi-party attribution.
  • Adds adapter-backed local contenders: createNoopMemoryBenchmarkAdapter() and createInMemoryBenchmarkAdapter().
  • Documents how Neo4j Agent Memory, Mem0, Zep, Letta, Graphiti, vector stores, or product-owned memory systems enter through the same adapter interface.

Real run

  • Command: pnpm exec tsx -e ... runMemoryAdapterBenchmark(...)
  • Artifact: /tmp/agent-knowledge-real-memory-ranking-final/memory-adapter-ranking.json
  • Result: in-memory ranked feat(readiness): bridge knowledge index to agent-eval #1 with mean score 0.9375 and pass rate 0.9167 over 12/12 cells; no-memory ranked feat: add source freshness readiness audit #2 with mean score 0.25 and pass rate 0 over 12/12 cells.
  • Confirmed raw miss: /tmp/agent-knowledge-real-memory-ranking-final/in-memory/first-party_memory-lifecycle_allergy_0/cached-result.json shows the in-memory adapter retrieved stale “no food allergies” memory, scoring 0.25 on that case.

Verification

  • pnpm install --frozen-lockfile --prefer-offline
  • pnpm exec tsc --noEmit
  • pnpm exec vitest run tests/benchmarks.test.ts — 8 passed
  • pnpm lint
  • git diff --check
  • pnpm exec vitest run — 267 passed, 12 skipped
  • pnpm build

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — a008ddcd

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-08T18:05:31Z

@drewstone drewstone merged commit d9be64f into main Jul 8, 2026
1 check passed
@drewstone drewstone deleted the feat/real-memory-benchmark-run branch July 8, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants