feat(retrieval): add RAG eval improvement loop

README.md

@@ -10,7 +10,7 @@ This package turns raw sources and generated markdown knowledge into a versionab
 - [Start here](#start-here) — pick CLI vs programmatic
 - [CLI](#cli) — `init` → `source-add` → `index` → `search` → `lint`
 - [Design](#design) — the invariants (immutable sources, cited claims, deterministic graph)
-- [Agent-Eval integration](#agent-eval-integration) — readiness bundles + release reports
+- [Agent-Eval integration](#agent-eval-integration) — retrieval eval + readiness bundles + release reports
 - [Memory adapters](#memory-adapters) — generic memory contract + Neo4j Agent Memory bridge
 - [Research loop](#research-loop) — `runKnowledgeResearchLoop` + control-loop adapter
 - [Researcher profile](#researcher-profile) — sandbox `AgentProfile` for `runLoop`
@@ -31,6 +31,7 @@ Two ways in, depending on what you're doing:
   - *"Does the agent have enough context to run?"* → [`buildEvalKnowledgeBundle`](#agent-eval-integration) (block / ask / acquire before execution).
   - *"Grow the KB as a researcher"* → [`runKnowledgeResearchLoop`](#research-loop) (deterministic mechanics; your agent owns judgment), [`runTwoAgentResearchLoop`](#two-agent-research-loop) (researcher proposes, verifier checks + fills gaps, offline), or the sandbox [researcher profile](#researcher-profile) for `runLoop`.
   - *"Spawn one researcher per sub-topic and stop when the KB is ready"* → [`runResearchSupervisor`](#research-supervisor) (a supervisor brain sizes the topology over a `Scope`; LIVE, needs creds).
+  - *"Tune retrieval for a knowledge base"* → `runRetrievalImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section.
   - *"Does this candidate KB actually improve task success?"* → run an [agent-eval improvement loop](#agent-eval-integration) over KB variants, then `knowledgeReleaseReport` for the promotion decision.
   - *"Keep live authorities fresh"* → [pluggable sources](#pluggable-knowledge-sources) + `detectChanges` → eval re-runs.
 
@@ -101,6 +102,7 @@ from `@tangle-network/agent-knowledge`.
   when comparing against natural confidence thresholds. The normalization is
   within-set ranking, not a cross-query absolute confidence.
 - Release confidence uses `@tangle-network/agent-eval` release gates (`evaluateReleaseConfidence`) instead of reimplementing them.
+- Retrieval eval turns retrieval/RAG configs into `agent-eval` surfaces, auto-searches candidate configs, and scores them against page, source, source-anchor, or source-span targets.
 - `buildEvalKnowledgeBundle()` maps wiki/search evidence into
   `agent-eval` `KnowledgeRequirement`, `KnowledgeBundle`, and
   `KnowledgeReadinessReport` contracts so control loops can block, ask, or
@@ -114,6 +116,36 @@ readiness/eval machinery without making `agent-knowledge` own the database.
 
 ## Agent-Eval Integration
 
+Use retrieval eval when the question is whether a retrieval/RAG config can find the right knowledge before an agent reasons over it.
+The labels should name stable pages, source records, anchors, or source spans, not ephemeral chunk IDs.
+The completion roadmap is in [`docs/eval/rag-eval-roadmap.md`](docs/eval/rag-eval-roadmap.md).
+
+```ts
+import { runRetrievalImprovementLoop } from '@tangle-network/agent-knowledge'
+
+const result = await runRetrievalImprovementLoop({
+  baseline: { k: 5, hybrid: false, reranker: null },
+  scenarios: trainRetrievalScenarios,
+  holdoutScenarios: holdoutRetrievalScenarios,
+  index,
+  searchSpace: {
+    k: [5, 10, 20],
+    hybrid: [false, true],
+    reranker: [null, 'bge-reranker'],
+  },
+  targetRecall: 0.9,
+  deltaThreshold: 0.02,
+  costCeiling: 15,
+  runDir: '.agent-knowledge/retrieval-runs',
+})
+
+console.log(result.winnerConfig)
+```
+
+Pass a custom `retrieve` function to `buildRetrievalEvalDispatch` when the config controls an external vector store, reranker, hybrid search service, or chunker.
+The built-in fallback uses `searchKnowledge` over the local deterministic index.
+Use `buildRetrievalEvalDispatch`, `retrievalRecallJudge`, and `retrievalParameterSweepProposer` directly only when you need custom `agent-eval` wiring.
+
 To answer whether a candidate knowledge base actually improves agent task success, run an `@tangle-network/agent-eval` improvement loop (`runImprovementLoop`) over your KB variants on a real task corpus; each run is scored into a `RunRecord`.
 
 Use `knowledgeReleaseReport()` before promotion: pass the candidate and baseline `RunRecord[]` (plus optional `ReleaseTraceEvidence` and the gate decision) and it folds them into a `ReleaseConfidenceScorecard` and a `KnowledgeRelease` using `agent-eval`'s release gates and `RunRecord` validation.

docs/architecture.md

@@ -1,13 +1,14 @@
 # Architecture
 
-`@tangle-network/agent-knowledge` is a domain-agnostic knowledge growth layer for agents.
+`@tangle-network/agent-knowledge` is a domain-agnostic knowledge-base construction layer for agents.
 
-It does not try to be a vector database, a RAG framework, or a product-specific wiki. It owns the small set of primitives every serious agent knowledge system needs:
+It owns the small set of primitives every serious agent knowledge system needs:
 
 - immutable source records
 - generated knowledge pages and units
 - claims with source references
 - deterministic indexing, graph construction, search, and lint
+- retrieval/RAG candidate surfaces, gold-target scoring, and eval-loop adapters
 - safe LLM write proposals
 - eval-gated release confidence through `@tangle-network/agent-eval`
 - visualization DTOs under the `/viz` subpath
@@ -20,9 +21,10 @@ It does not try to be a vector database, a RAG framework, or a product-specific
 
 `agent-eval` owns traces, ASI, improvement loops, run records, and promotion gates.
 
-`agent-knowledge` owns sources, claims, pages, graph/search/lint, and knowledge base candidates. It calls `agent-eval` instead of reimplementing evaluation.
+`agent-knowledge` owns sources, claims, pages, graph/search/lint, retrieval/RAG construction surfaces, and knowledge base candidates.
+It calls `agent-eval` instead of reimplementing improvement loops or promotion math.
 
-Product apps own domain policies, source adapters, task corpora, and promotion decisions.
+Product apps own domain policies, provider accounts, vector stores, source adapters, task corpora, and promotion decisions.
 
 Core does not own a D1 schema or fleet dispatcher. Apps wire `KbStore` and `KnowledgeDiscoveryDispatcher` to their tenancy, queue, budget, auth, and sandbox systems.
 
@@ -34,7 +36,7 @@ Core does not own a D1 schema or fleet dispatcher. Apps wire `KbStore` and `Know
 4. Validate paths, citations, links, and schema.
 5. Index generated knowledge pages.
 6. Search and graph-lint the knowledge base.
-7. Evaluate candidate KB variants with an `agent-eval` improvement loop, then fold the resulting run records into release confidence with `knowledgeReleaseReport`.
+7. Evaluate candidate KB and retrieval variants with an `agent-eval` improvement loop, then fold the resulting run records into release confidence with `knowledgeReleaseReport`.
 8. Promote only variants that pass downstream gates.
 
 ## CLI

docs/eval/rag-eval-roadmap.md

@@ -0,0 +1,114 @@
+# RAG Eval Completion Roadmap
+
+Verdict: `runRetrievalImprovementLoop()` is the right first loop, but it is only the retrieval layer.
+SOTA RAG evaluation requires retrieval quality, context quality, generated-answer quality, abstention behavior, robustness, and operating budgets.
+
+## Research Basis
+
+| Source | What matters for us |
+| --- | --- |
+| [Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) / [paper](https://arxiv.org/abs/2309.15217) | Standard RAG eval splits retrieval and generation into context precision, context recall, faithfulness, and answer relevance. |
+| [ARES](https://arxiv.org/abs/2311.09476) | Strong RAG eval scores context relevance, answer faithfulness, and answer relevance, and uses small human-labeled sets to calibrate automated judges. |
+| [TruLens RAG Triad](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | The minimal end-to-end triad is context relevance, groundedness, and answer relevance. |
+| [RAGChecker](https://papers.nips.cc/paper_files/paper/2024/hash/27245589131d17368cccdfa990cbf16e-Abstract-Datasets_and_Benchmarks_Track.html) | Fine-grained diagnosis should separate retrieval misses, noisy context, and unsupported generated claims. |
+| [BEIR](https://arxiv.org/abs/2104.08663) / TREC-style retrieval | Retrieval still needs classical rank metrics: Recall@k, Precision@k, MRR, MAP, and nDCG. |
+| [CRAG](https://arxiv.org/abs/2406.04744) | Real RAG evals must include long-tail, dynamic, multi-hop, and unanswerable questions, not only easy static facts. |
+| [DeepEval RAG metrics](https://deepeval.com/docs/metrics-faithfulness) | Production tools converge on faithfulness, answer relevance, and context relevance as generator/retriever checks. |
+
+## Current Repo Status
+
+Done:
+
+- `runRetrievalImprovementLoop()` auto-searches retrieval configs through `agent-eval`.
+- Retrieval scenarios can label pages, page paths, sources, source anchors, and source spans.
+- The retrieval judge reports recall, MRR, nDCG, precision@k, cost, and held-out promotion.
+- The loop is tested with a real `agent-eval` run where `{ k: 2 }` beats `{ k: 1 }`.
+
+Not done:
+
+- Generated-answer evaluation.
+- Context relevance and context sufficiency judges.
+- Citation support and claim-level groundedness.
+- Abstention and unanswerable-question scoring.
+- Slice-level reporting for freshness, distractors, multi-hop, and long-tail cases.
+
+## Completion Criteria
+
+### Phase 1: Retrieval Quality
+
+Build a retrieval eval pack with at least 100 labeled scenarios.
+Use source-span labels wherever possible.
+
+Required slices:
+
+- 25 known-answer questions.
+- 25 paraphrase questions.
+- 20 distractor questions.
+- 10 freshness/version questions.
+- 10 multi-source questions.
+- 10 unanswerable or forbidden-source questions.
+
+Ship criteria:
+
+- Holdout source-span Recall@5 is at least 0.90.
+- Holdout nDCG@5 is at least 0.80.
+- Train-to-holdout recall gap is at most 0.08.
+- Stale or forbidden source hit rate is at most 0.02.
+- p95 retrieval latency and cost do not regress by more than 10 percent versus baseline.
+
+### Phase 2: Answer Quality
+
+Add a generated-answer eval artifact that includes query, retrieved context, answer text, citations, cost, latency, and trace ids.
+Score it with deterministic checks first and LLM judges only for semantic quality.
+
+Ship criteria:
+
+- Faithfulness or groundedness is at least 0.95 on holdout.
+- Answer relevance is at least 0.90 on holdout.
+- Answer correctness is at least 0.85 on human-labeled holdout.
+- Citation support is at least 0.95 for claims that cite sources.
+- Unsupported-answer rate on unanswerable questions is at most 0.05.
+
+### Phase 3: Diagnosis
+
+Add RAGChecker-style failure attribution.
+Every failed case must classify as one primary cause.
+
+Required failure classes:
+
+- Retrieval miss.
+- Retrieval noisy context.
+- Stale retrieval.
+- Missing multi-hop evidence.
+- Generator ignored evidence.
+- Generator hallucinated unsupported claim.
+- Citation mismatch.
+- Correct abstention.
+- Incorrect abstention.
+
+Ship criteria:
+
+- Every failed eval has one primary failure class.
+- At least 95 percent of generated claims can be mapped to supporting context, contradicted context, or no context.
+- Reports show metrics by slice and by failure class, not only the aggregate score.
+
+### Phase 4: Production Loop
+
+Run the same eval pack on every retrieval or prompt change.
+Keep train/dev/holdout isolated.
+Never tune on holdout.
+
+Ship criteria:
+
+- `runRetrievalImprovementLoop()` gates retrieval config changes.
+- Answer-quality eval gates prompt and synthesis changes.
+- Reports persist run id, commit, config hash, dataset hash, metric versions, cost, latency, and traces.
+- A promoted candidate must improve the target metric without violating faithfulness, abstention, cost, or latency limits.
+
+## Next Implementation Steps
+
+1. Add `RagAnswerEvalScenario`, `RagAnswerEvalArtifact`, and `ragAnswerQualityJudge()`.
+2. Add context relevance and context sufficiency judges over retrieved hits.
+3. Add forbidden/stale source targets to retrieval scenarios.
+4. Add slice-level aggregation helpers for the required six eval slices.
+5. Add a CLI command that runs the retrieval loop and writes a reproducible report under `.agent-knowledge/eval/`.

src/index.ts

@@ -25,6 +25,7 @@ export * from './release'
 export * from './research-driving-driver'
 export * from './research-loop'
 export * from './research-supervisor'
+export * from './retrieval-eval'
 export * from './schemas'
 export * from './search'
 export * from './sources'