answer-engine

answer-engine is a pre-1.0 FastAPI RAG reference backend that routes simple queries, combines BM25 with configurable vector ranking, and returns citations, lexical grounding diagnostics, and content-hashed evidence records.

Retrieval benchmark

Configuration	Recall@1	Recall@5	MRR	nDCG@10	p50 ms	p95 ms	p99 ms
BM25 only	0.667	0.667	0.667	0.667	3.471	5.023	7.827
Frozen dense only	0.667	1.000	0.833	0.877	6.054	7.162	9.178
BM25 + dense via RRF	1.000	1.000	1.000	1.000	6.288	7.361	8.506

Measured on commit 7585c2b, CPython 3.11.15 on Windows, using 12 judged queries, a 256-chunk in-memory SQLite corpus, two warmups, and 100 timed repetitions per query (1,200 observations per row). The controlled frozen vectors isolate channel fusion: four queries need exact lexical evidence, four need the dense fixture, and four reward agreement. They demonstrate the RRF implementation, not the quality of a learned embedding model. Raw result · fixture and methodology

The shipped HashingEmbedder is deterministic hashed bag-of-words, not a semantic model. On the same fixture its fused row scored 0.667 Recall@1 / 0.667 MRR with p95 61.609 ms; RRF matched BM25 but added no paraphrase lift. Raw hashing result

Quickstart

The default demo is deterministic, local, and needs no API key:

python -m venv .venv
python -m pip install -e ".[dev]"
python examples/demo.py

Run the persistent HTTP service with the same offline defaults:

python -m uvicorn answer_engine.server:build_application --factory --reload

Then index and search one document:

curl -X POST http://127.0.0.1:8000/documents \
  -H "Content-Type: application/json" \
  -d '{"title":"Launch note","text":"Answer Engine launched its pilot in Dublin."}'

curl -X POST http://127.0.0.1:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query":"Where did Answer Engine launch its pilot?","limit":5}'

What the repository proves

Capability	Implementation	Evidence in this repository
Adaptive execution	A deterministic router selects direct, calculator/tool, or retrieval mode.	Router and engine tests cover greeting, arithmetic, and corpus-query paths.
Hybrid retrieval	BM25 and cosine-ranked candidates are fused with Reciprocal Rank Fusion; MMR can rerank for diversity.	Unit tests cover BM25, RRF, and MMR; the committed ablation runs all three retrieval modes through HybridRetriever.
Durable corpus	Documents, chunks, metadata, and embeddings live in SQLite; BM25 is rebuilt from stored chunks.	Store tests cover CRUD, persistence, requested chunk order, and index rebuild.
Citation diagnostics	Generated retrieval answers carry retrieved sources and a lexical grounding report.	Grounding tests flag uncited claims, dangling citations, and some low-overlap citations.
Evidence self-consistency	A canonical SHA-256 manifest binds the query, answer, sources, ranks, tool calls, and grounding report.	Provenance tests detect edits when hashes are not recomputed.
Runnable interfaces	FastAPI exposes document, search, ask, stream, and verification routes; an optional MCP factory reuses the engine.	API tests cover health, CRUD, search, ask, verification, and the terminal SSE event. MCP itself is not integration-tested.

The current suite runs with FakeProvider and local embeddings, so tests do not call the network.

Architecture

query
  -> rule-based router (direct | tool | retrieve)
  -> BM25 + vector candidates
  -> Reciprocal Rank Fusion
  -> optional Maximal Marginal Relevance reranking
  -> provider answer with retrieved context
  -> lexical grounding report
  -> content-hashed evidence manifest

Not retrieving is useful for the small set of cases the router recognizes, such as greetings and pure arithmetic. Other queries route to the corpus; this is a rules engine, not a learned relevance classifier.

Benchmark methodology

Run the committed benchmark without credentials or downloads:

python -m benchmarks.retrieval_benchmark
python -m benchmarks.retrieval_benchmark --profile hashing
python -m benchmarks.retrieval_benchmark --json

The harness reports macro Recall@1, Recall@5, mean reciprocal rank, graded nDCG@10, and nearest-rank p50/p95/p99 latency. Index construction and warmups are excluded. Each latency observation includes SQLite chunk loading, BM25, query embedding when enabled, the dense full scan, ranking, RRF when enabled, and result construction.

The ablation sets diversity=0 and final_limit=10 so it isolates RRF and can observe nDCG@10. The application default is diversity=0.3 and final_limit=5; MMR quality and scale sensitivity are not established by this table. CI asserts quality metrics, not latency thresholds.

Retrieval design

BM25 handles rare exact strings such as error codes, invoice IDs, and function names. A genuine semantic embedder can retrieve no-overlap paraphrases. RRF combines rank positions instead of mixing unbounded corpus-dependent BM25 scores with bounded cosine scores.

The supplied server intentionally chooses HashingEmbedder so local runs remain credential-free. It hashes exact lexical tokens into 384 dimensions and should be treated as a deterministic vector plumbing implementation, not semantic search. OpenAIEmbedder is available through manual composition but is not selected by the supplied server.

Grounding and provenance boundaries

Grounding checks citation syntax and token overlap. It can flag a citation that points to an unrelated chunk, but it is not entailment, contradiction detection, or a truth score. The engine returns the report and a trustworthy flag so a caller can reject, retry, or escalate.

The provenance manifest is unsigned. It detects internal mismatches when content changes without recomputing hashes; anyone able to rewrite the manifest can also recompute them. It proves self-consistency, not authorship, source authenticity, or semantic correctness.

The agent

answer_engine.agent is a bounded, multi-step tool-calling loop. It can list documents, retrieve evidence, measure lexical grounding, and calculate — a working set for exploring the corpus over several turns, which the single-pass trigger-keyword router cannot do.

An agent loop is where a grounding guarantee usually stops being true. Hand a model tools over a RAG system and it will compose a final answer directly and attach a manifest it wrote itself, because answering looks like the job and the prompt asked it to cite sources.

So the boundary is structural rather than instructed. finalize_answer, answer_without_evidence, build_manifest, sign_manifest, forge_manifest, and skip_manifest have no handler; the agent package imports neither AnswerEngine nor provenance, so there is no code path to reach them. Dispatch fails closed on any unregistered name, and registering a tool under a forbidden name raises at construction — a boundary violation is a reviewable code failure, not a capability a prompt is expected to withhold.

Authoritative answers stay on the grounded engine path, and that path alone constructs manifests. Refusals are handed back to the model so it can work inside the boundary; the loop is bounded by a step budget and records every attempt, refusals included.

HTTP API

Method	Path	Description
GET	/healthz	Process health check.
POST	/documents	Add {title, text, source?, metadata?}.
GET	/documents	List documents.
GET	/documents/{doc_id}	Fetch a document and its chunks.
DELETE	/documents/{doc_id}	Delete a document and its chunks.
POST	/search	Return ranked chunks with scores and why() provenance.
POST	/ask	Return answer text, citations, grounding, trust flag, and manifest.
POST	/ask/stream	Emit answer text and then final metadata as SSE.
POST	/provenance/verify	Recompute and verify a submitted manifest.

Python composition

from answer_engine.api import create_app
from answer_engine.embeddings import HashingEmbedder
from answer_engine.engine import AnswerEngine
from answer_engine.providers.fake import FakeProvider
from answer_engine.retrieval.retriever import HybridRetriever
from answer_engine.router import QueryRouter
from answer_engine.store import DocumentStore
from answer_engine.tools import default_registry

embedder = HashingEmbedder()
store = DocumentStore("answer-engine.db", embedder=embedder)
tools = default_registry()
engine = AnswerEngine(
    store,
    HybridRetriever(store, embedder),
    FakeProvider(),
    QueryRouter(tools.triggers()),
    tools,
)
app = create_app(engine, store)

Provider and deployment configuration

docker compose up --build

Variable	Purpose
ANSWER_ENGINE_PROVIDER	fake (default), openai, or grok.
ANSWER_ENGINE_DB	SQLite path used by the supplied factory.
OPENAI_API_KEY	Required for OpenAI generation or a manually composed OpenAIEmbedder.
OPENAI_MODEL	Optional OpenAI chat model override.
XAI_API_KEY	Required for Grok generation.
XAI_MODEL	Optional Grok model override.

answer_engine.mcp_server.create_mcp_server(engine) exposes ask, search, add_document, and list_documents when installed with pip install -e ".[mcp]". It reuses the engine and store but is not the same wire contract as the HTTP API.

Tests

python -m pip install -e ".[dev]"
ruff check .
pytest -q -p no:cacheprovider

Current local result: 55 passed, 1 warning in 1.18s on CPython 3.11.15. The warning is the existing Starlette/httpx test-client deprecation.

Not implemented or not proven

• No authentication, authorization, tenancy, rate limiting, request-size limits, migrations, tracing, pagination, batch ingestion, cache, or multi-process scalability proof.
• No public-dataset retrieval benchmark, learned-embedding quality evaluation, generation-quality evaluation, or load test. The committed fixture is controlled and deliberately small.
• No learned router; ordinary questions outside a few explicit rules are sent to retrieval whether or not the corpus can answer them.
• No built-in semantic local embedder. The default hashed vectors cannot represent no-overlap paraphrases.
• No approximate-nearest-neighbor index. SQLite stores vectors as JSON and dense retrieval scans every chunk.
• No live OpenAI or Grok integration tests, retries, provider fallback, or model-output quality guarantees.
• /ask/stream computes the complete answer before emitting events; it is not token streaming.
• trustworthy is not truth. Direct/tool responses are trusted by policy, while retrieval responses use a lexical heuristic.
• The manifest is not signed and does not bind code version, embedder identity, corpus snapshot, prompts, SDK versions, or generation parameters.
• MCP behavior, prompt-injection resistance, concurrent writes, Docker runtime behavior, and the default MMR setting are not covered by end-to-end tests.

License

MIT