From 8dc3f3032606740dcebed943600ac3e53ee59d7a Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 15:36:05 +0100 Subject: [PATCH 01/25] DOC-6809 Add docs-only MCP server spec + v0 prototype MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigation into a read-only "docs MCP server" — distinct from the existing data-plane redis/mcp-redis — that exposes the documentation corpus as agent-queryable tools over the docs.ndjson feed we already publish. Spec plus a working v0 (search_docs + get_page, self-contained BM25, no datastore). Two findings from running it against the live feed shaped the design: indexing 2,531 pages takes ~0.3s and queries ~100ms, so search compute is a non-issue and Rust/WASM would be premature optimisation; but untuned lexical ranking mis-ranks natural-language queries — with no stemmer, "append" misses XADD's "appends", and canonical command pages lose to release-notes, operator custom-resources, and client-library pages that repeat the same terms. That ranking gap, not speed, is what justifies the vector-search upgrade path in the spec. Learned: measured perf rules out WASM; lexical ranking (not speed) is the real limit Constraint: docs MCP server stays read-only, no DB connection, no creds (unlike mcp-redis data plane) Rejected: Rust/WASM for search speed | ~0.3s index + ~100ms queries on 2,531 pages, compute isn't the bottleneck Directive: don't over-tune lexical BM25 weights (overfits to sample queries); fix ranking via stemming/analyzer or vector search Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 232 ++++++++++++++++++ build/docs-mcp-server/node/.gitignore | 3 + build/docs-mcp-server/node/README.md | 78 ++++++ build/docs-mcp-server/node/package.json | 31 +++ build/docs-mcp-server/node/src/feed.ts | 44 ++++ build/docs-mcp-server/node/src/index.ts | 111 +++++++++ build/docs-mcp-server/node/src/search.ts | 198 +++++++++++++++ build/docs-mcp-server/node/src/smoke.ts | 28 +++ .../node/src/tools/get-page.ts | 40 +++ .../node/src/tools/search-docs.ts | 20 ++ build/docs-mcp-server/node/src/types.ts | 35 +++ .../docs-mcp-server/node/test/fixture.ndjson | 3 + build/docs-mcp-server/node/tsconfig.json | 19 ++ 13 files changed, 842 insertions(+) create mode 100644 build/docs-mcp-server/SPEC.md create mode 100644 build/docs-mcp-server/node/.gitignore create mode 100644 build/docs-mcp-server/node/README.md create mode 100644 build/docs-mcp-server/node/package.json create mode 100644 build/docs-mcp-server/node/src/feed.ts create mode 100644 build/docs-mcp-server/node/src/index.ts create mode 100644 build/docs-mcp-server/node/src/search.ts create mode 100644 build/docs-mcp-server/node/src/smoke.ts create mode 100644 build/docs-mcp-server/node/src/tools/get-page.ts create mode 100644 build/docs-mcp-server/node/src/tools/search-docs.ts create mode 100644 build/docs-mcp-server/node/src/types.ts create mode 100644 build/docs-mcp-server/node/test/fixture.ndjson create mode 100644 build/docs-mcp-server/node/tsconfig.json diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md new file mode 100644 index 0000000000..3a8a739d21 --- /dev/null +++ b/build/docs-mcp-server/SPEC.md @@ -0,0 +1,232 @@ +# Docs MCP server — design spec + +**Status:** Draft (investigation, DOC-6809) +**Owner:** Docs +**Scope:** A read-only MCP server that lets AI coding agents query the Redis +documentation corpus as a tool, backed by the JSON/NDJSON feed we already +publish. + +--- + +## 1. Motivation + +We already publish AI-readable docs three ways: `llms.txt` (index), per-page +Markdown (`index.html.md`), and structured JSON/NDJSON with role-tagged +sections. These are all *passive files* — an agent must know they exist, fetch +them, and do its own retrieval. + +The gap is an *active, queryable* surface: a tool an agent (Cursor, Claude +Code, ChatGPT, VS Code) can call mid-task to get a **current, sourced** answer +instead of relying on stale training data. That is what this server provides. + +### Explicitly *not* this server + +- **`redis/mcp-redis`** is a *data-plane* server: it connects an agent to a + *running Redis instance* to read/write/query data. It needs a connection + string and can mutate data. +- **This server** is a *knowledge-plane* server: it connects an agent to the + *documentation*. It is read-only, needs no database, no credentials, and its + entire value is returning citations to our docs. + +They are orthogonal and should stay separate products/installs. Bundling doc +lookup into the data-plane server forces the reference-only audience to stand +up a data server and hand it credentials — friction that kills adoption for +the exact audience (coding agents) that benefits most. + +## 2. Goals / non-goals + +**Goals** +- Thin retrieval wrapper over the **existing** JSON feed — no new content + pipeline. +- Read-only, no secrets, no live DB connection. +- Every response carries a canonical `url` (citations by construction). +- Token-lean: search returns summaries + refs; agents drill down deliberately. +- Version-aware (our docs are versioned; mixing versions is a correctness bug). + +**Non-goals** +- No writes, no code execution, no live Redis access (that's `mcp-redis`). +- No new authoring format — we consume `sections[]` / `examples[]` as-is. +- WebMCP / in-browser tool registration — out of scope for now (different + layer, different audience; see DOC-6809 discussion). + +## 3. Data source + +No new pipeline. Reuse the current build output: + +``` +Hugo ──► per-page public/**/index.json ──► generate_ndjson.py ──► docs.ndjson +``` + +Document schema (already published on the *AI Agent Resources* page): + +- **Page**: `id`, `title`, `url`, `summary`, `page_type` (`content` | `index`), + `content_hash`, `sections[]`, `examples[]`, `children[]` +- **Section**: `id`, `title`, `role` (`overview` | `syntax` | `parameters` | + `returns` | `example` | …), `text` +- **Example**: `id`, `language`, `code`, `section_id` + +The server loads `docs.ndjson` (or an index built from it) at startup. Because +`content_hash` is deterministic (`sha256` over summary + section text + +example code), it doubles as a cache/freshness key. + +## 4. Tool surface + +Five tools. Names, inputs, and the field of the existing schema each is +projected from: + +### `search_docs` +Rank pages by relevance. Returns **refs only, no full text**. + +- **In:** `query` (string, required); optional `page_type`, `group`, + `version`, `limit` (default 10) +- **Out:** `[{ id, title, url, summary, matching_section_ids[] }]` +- **From:** NDJSON feed; `page_type` filter lets callers skip `index` pages. + +### `get_page` +Fetch one page, optionally filtered to specific section roles. + +- **In:** `id` **or** `url` (required); optional `roles[]` (e.g. + `["syntax","parameters"]`) +- **Out:** `{ id, title, url, summary, page_type, content_hash, sections[] }` + where `sections` is filtered to `roles[]` if given +- **From:** per-page `index.json`. `roles[]` filtering is only possible because + sections are role-tagged — big token savings (pull `parameters` without the + overview prose). + +### `get_section` +Return a single role-tagged chunk — the retrieval-native unit. + +- **In:** `page_id` (required), `section_id` (required) +- **Out:** `{ page_id, section_id, title, role, text, url }` +- **From:** `sections[]`. + +### `get_examples` +Return runnable code, filterable by language. **The highest-value tool for +coding agents** — their most common need is "the go-redis snippet for `XADD`", +not prose. + +- **In:** `query` **or** `command` (one required); optional `language` + (`python` | `go` | `java` | …) +- **Out:** `[{ id, code, language, url, section_id }]` +- **From:** `examples[]` (carries `language` + `section_id` already). + +### `get_command` +Convenience lookup for the highest-traffic page type. + +- **In:** `name` (e.g. `XADD`) +- **Out:** command page with `syntax`, `parameters`, `returns` sections + + `examples[]` + `url` +- **From:** command pages (specialised `get_page`; commands are first-class). + +## 5. Response conventions + +- **Always include `url`.** Agents cite; users click. +- **Search never returns full text.** Force the drill-down path + (`search_docs` → `get_section` / `get_examples`) so context stays small. +- **Return `content_hash` on page/section responses.** Lets agents and our own + eval harness do `If-None-Match`-style freshness checks for free. +- **Truncate defensively.** Cap `text` length per section in responses; expose + a `truncated: true` flag rather than silently cutting. + +## 6. Transport & deployment + +Follow the existing repo pattern (`build/command_api_mapping/mcp-server/`): +TypeScript, `@modelcontextprotocol/sdk`, Zod input schemas. + +- **Local / stdio:** publish an npx-runnable package so developers can add it to + Cursor/Claude Code config. Zero infra. +- **Remote / hosted:** an HTTP+SSE endpoint on `redis.io` (e.g. + `https://redis.io/mcp`) built from the same handlers, so no install is + required. Advertise it on the *AI Agent Resources* page next to `llms.txt`. + +Both modes share one core: load feed → build index → handle tool calls. The +data is public, so the remote endpoint needs no auth (rate-limit only). + +### Search backend vs. transport — what needs a datastore + +Whether the server needs a Redis (or any) backend depends entirely on the +search implementation, **not** on the transport. The corpus is small +(`docs.ndjson` ≈ 30 MB / ≈ 5 MB gzipped, ≈ 4,100 docs), which is what makes +the lexical path infra-free. + +| | Lexical (BM25) | Vector (semantic) | +|---|---|---| +| **stdio (client-side)** | ✅ self-contained in-memory index | ❌ impractical (would ship an index + embedding model per install) | +| **remote (hosted)** | ✅ in-process index, no datastore | ✅ needs a vector store (RediSearch / RedisVL) | + +- **Lexical (v1):** build a BM25 index in process memory from `docs.ndjson` at + startup — MiniSearch/Lunr (JS) or a simple BM25 (Python). No datastore, even + when hosted; horizontally-scaled instances each just load ~5 MB gzipped at + boot and build their own index. Role-tagged sections and exact command-name + tokens make lexical retrieval unusually strong on this corpus. +- **Vector (v2, hosted only):** needs embeddings computed offline at build + time, a vector index queried at runtime, and query-time embedding on each + request. That implies a real server-side backend — the natural fit is + **RediSearch / RedisVL**, which doubles as a Redis showcase. It cannot run + purely client-side, so it's a hosted-endpoint upgrade, not a stdio feature. + +Recommendation: ship v1 lexical in **both** modes with no backend; treat +vector-on-Redis as a later upgrade to the **hosted** endpoint only. + +## 7. Versioning + +- `search_docs` / `get_page` accept an optional `version`; default to `latest`. +- Responses echo the resolved version so an agent can't silently blend + versions. This is the single most common RAG-over-docs correctness bug. + +## 8. Freshness + +- Rebuild the server's index whenever `docs.ndjson` is regenerated (same build + step). No separate content pipeline to keep in sync. +- `content_hash` per page enables incremental index updates and client caching. + +## 9. Security + +- Read-only. No write tools, no code execution, no connection string. +- Serves only already-public content. +- Remote endpoint: rate-limit, no auth, no PII. + +## 10. Open questions + +- **Search backend:** resolved for v1 — lexical (BM25 over NDJSON), no + datastore, runs in both stdio and hosted modes (see §6). Open: do we add + vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / + RedisVL, and is the retrieval gain worth the added infra given how + well-structured the corpus already is? +- **Ranking quality (found in v0 prototype):** untuned lexical BM25 — even + with stopword removal and title/summary/slug field boosts — mis-ranks + natural-language queries. Two concrete failure modes observed on the live + feed: (1) **no stemming**, so "append" ≠ "appends" and `XADD` won't surface + for "append an entry to a stream"; (2) **canonical command pages compete with + release notes, operator `custom-resources` pages, and client-library + overviews** that repeat the same terms. Fix needs an analyzer + (stemming/lemmatization), possibly a page-type/canonical boost, and/or the + vector-search path. This is the strongest argument for §6's v2 upgrade. +- **`get_command` coverage:** command pages *do* carry `parameters`/`returns`/ + `example` roles (confirmed: `get_page('expire')` → roles + `[content, parameters, example, returns]`). Confirm this holds across all + command pages or add a fallback. +- **Package ownership:** does this live here in `docs`, or graduate to its own + repo like `mcp-redis`? +- **Overlap with `mcp-redis`:** does that server already do any doc lookup we + should pull into here and deprecate there? + +## 11. Phased plan + +1. **v0 (prototype):** stdio server, lexical `search_docs` + `get_page` over a + local `docs.ndjson`. Prove the loop in Claude Code. +2. **v1:** add `get_examples`, `get_section`, `get_command`; `version` support; + token-budget guards. +3. **v2:** hosted remote endpoint on `redis.io`; advertise on AI Agent + Resources. +4. **Ongoing:** wire an **AI-answer eval** — a fixed set of real questions run + through the server, scored for correctness — as a docs-quality regression + gate. (Extends the code-example verification mindset to answer quality.) + +## 12. Success signals + +- A coding agent, given only this server, answers common Redis how-to questions + correctly and with citations. +- Measurable reduction in version-mixing / stale-API answers versus the model's + own training data. +- Adoption: entries in Cursor/Claude Code MCP configs pointing at the endpoint. diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore new file mode 100644 index 0000000000..3c25e1e49c --- /dev/null +++ b/build/docs-mcp-server/node/.gitignore @@ -0,0 +1,3 @@ +node_modules/ +dist/ +*.log diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md new file mode 100644 index 0000000000..0bd4af5209 --- /dev/null +++ b/build/docs-mcp-server/node/README.md @@ -0,0 +1,78 @@ +# redis-docs-mcp (v0 prototype) + +A read-only MCP server that lets an AI coding agent query the Redis +documentation corpus as a tool, backed by the `docs.ndjson` feed we already +publish. See [`../SPEC.md`](../SPEC.md) for the full design. + +**This is a v0 prototype.** It ships two tools (`search_docs`, `get_page`) over +an in-memory lexical (BM25 + field-boost) index. No datastore, no credentials, +no live Redis connection. + +## Install & build + +```bash +cd build/docs-mcp-server/node +npm install +npm run build # compiles to dist/ +``` + +## Try it offline (no network) + +```bash +npm run smoke # runs against test/fixture.ndjson +``` + +Point it at the real feed (or any local `.ndjson` / `.ndjson.gz`): + +```bash +DOCS_NDJSON="https://redis.io/docs/latest/docs.ndjson" npm run smoke +``` + +## Run as an MCP server + +The server speaks MCP over stdio. Feed source is set via `DOCS_NDJSON` +(default: `https://redis.io/docs/latest/docs.ndjson`). + +Add to a Claude Code / Cursor MCP config after `npm run build`: + +```json +{ + "mcpServers": { + "redis-docs": { + "command": "node", + "args": ["/ABSOLUTE/PATH/build/docs-mcp-server/node/dist/index.js"], + "env": { "DOCS_NDJSON": "https://redis.io/docs/latest/docs.ndjson" } + } + } +} +``` + +## Tools + +| Tool | Inputs | Returns | +|------|--------|---------| +| `search_docs` | `query` (req), `page_type`, `version`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | +| `get_page` | `id` **or** `url` (req), `roles[]` | one page with `content_hash` + `sections`, optionally filtered to the given section roles | + +Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` +(e.g. `["parameters","returns"]`) to pull just what's needed. + +## Measured (real feed, 2,531 pages) + +- Index build: ~0.3 s. Query latency: ~75–125 ms. This is why v0 needs no + datastore and why Rust/WASM would be premature (see SPEC §6). + +## Known limitations (v0) + +- **Ranking is untuned lexical search.** Good on distinctive terms + (`publish`, `incrby`, `pexpire`), but weaker where: + - there is **no stemmer** — a query for "append" won't match a summary that + says "appends", so `XADD` is hard to surface from natural language; + - **canonical command pages** compete with release notes, operator + (custom-resources) pages, and client-library overviews that repeat the same + terms. + Fixing this properly means an analyzer (stemming/lemmatization) and/or the + vector-search upgrade tracked in SPEC §6/§10. +- **Version filtering is heuristic** (URL-path based); see SPEC §7. +- Only `search_docs` + `get_page`. `get_examples`, `get_section`, + `get_command` are v1 (SPEC §4/§11). diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json new file mode 100644 index 0000000000..ce615096b0 --- /dev/null +++ b/build/docs-mcp-server/node/package.json @@ -0,0 +1,31 @@ +{ + "name": "redis-docs-mcp", + "version": "0.0.1", + "description": "Read-only MCP server that queries the Redis documentation corpus (docs.ndjson) as a tool. v0 prototype: lexical search_docs + get_page.", + "type": "module", + "main": "dist/index.js", + "bin": { + "redis-docs-mcp": "dist/index.js" + }, + "scripts": { + "build": "tsc", + "start": "tsx src/index.ts", + "dev": "tsx watch src/index.ts", + "smoke": "tsx src/smoke.ts" + }, + "keywords": [ + "redis", + "mcp", + "docs" + ], + "license": "MIT", + "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "zod": "^3.22.0" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "tsx": "^4.0.0", + "typescript": "^5.0.0" + } +} diff --git a/build/docs-mcp-server/node/src/feed.ts b/build/docs-mcp-server/node/src/feed.ts new file mode 100644 index 0000000000..7eb01a0a78 --- /dev/null +++ b/build/docs-mcp-server/node/src/feed.ts @@ -0,0 +1,44 @@ +import { readFile } from "node:fs/promises"; +import { gunzipSync } from "node:zlib"; +import type { Page } from "./types.js"; + +/** Load raw feed bytes from a local path or http(s) URL, gunzipping if .gz. */ +async function loadFeedRaw(source: string): Promise { + let buf: Buffer; + if (/^https?:\/\//i.test(source)) { + const res = await fetch(source); + if (!res.ok) { + throw new Error(`Failed to fetch feed ${source}: ${res.status} ${res.statusText}`); + } + buf = Buffer.from(await res.arrayBuffer()); + } else { + buf = await readFile(source); + } + if (source.toLowerCase().endsWith(".gz")) { + buf = gunzipSync(buf); + } + return buf.toString("utf-8"); +} + +/** Parse NDJSON text into pages, skipping blank/invalid lines and non-doc objects. */ +export function parseNdjson(text: string): Page[] { + const pages: Page[] = []; + for (const line of text.split("\n")) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const obj = JSON.parse(trimmed); + // Match the same guard generate_ndjson.py uses: must have id + title + url. + if (obj && typeof obj.id === "string" && typeof obj.url === "string" && typeof obj.title === "string") { + pages.push(obj as Page); + } + } catch { + // Not our format — skip. + } + } + return pages; +} + +export async function loadFeed(source: string): Promise { + return parseNdjson(await loadFeedRaw(source)); +} diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts new file mode 100644 index 0000000000..ff6fbd87db --- /dev/null +++ b/build/docs-mcp-server/node/src/index.ts @@ -0,0 +1,111 @@ +#!/usr/bin/env node +import { Server } from "@modelcontextprotocol/sdk/server/index.js"; +import { + ListToolsRequestSchema, + CallToolRequestSchema, +} from "@modelcontextprotocol/sdk/types.js"; +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; + +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; +import { getPage, GetPageInput } from "./tools/get-page.js"; + +// Feed source: local path or http(s) URL, gzip-aware. Defaults to production. +const FEED_SOURCE = + process.env.DOCS_NDJSON ?? "https://redis.io/docs/latest/docs.ndjson"; + +const TOOLS = [ + { + name: "search_docs", + description: + "Search the Redis documentation and return the most relevant pages as references (id, title, url, summary, matching section ids). Returns no full text — follow up with get_page to read a result.", + inputSchema: { + type: "object" as const, + properties: { + query: { type: "string", description: "Search query" }, + page_type: { + type: "string", + enum: ["content", "index"], + description: "Restrict to prose ('content') or navigation ('index') pages", + }, + version: { + type: "string", + description: "Redis docs version, e.g. 'latest' (default) or '7.4'", + }, + limit: { + type: "number", + description: "Max results (default 10, max 50)", + }, + }, + required: ["query"], + }, + }, + { + name: "get_page", + description: + "Fetch a single documentation page by 'id' or 'url'. Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", + inputSchema: { + type: "object" as const, + properties: { + id: { type: "string", description: "Page id (URL slug), e.g. 'commands/xadd'" }, + url: { type: "string", description: "Full or partial page URL" }, + roles: { + type: "array", + items: { type: "string" }, + description: "Only return sections with these roles", + }, + }, + }, + }, +]; + +function ok(data: unknown) { + return { content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }] }; +} + +function fail(message: string) { + return { + content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], + isError: true, + }; +} + +async function main() { + const pages = await loadFeed(FEED_SOURCE); + const index = new DocsIndex(pages); + // Log to stderr — stdout is reserved for the MCP protocol. + console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); + + const server = new Server( + { name: "redis-docs-mcp", version: "0.0.1" }, + { capabilities: { tools: {} } }, + ); + + server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS })); + + server.setRequestHandler(CallToolRequestSchema, async (req) => { + const { name, arguments: args } = req.params; + try { + switch (name) { + case "search_docs": + return ok(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + case "get_page": + return ok(getPage(index, GetPageInput.parse(args ?? {}))); + default: + return fail(`Unknown tool: ${name}`); + } + } catch (e) { + return fail(e instanceof Error ? e.message : String(e)); + } + }); + + const transport = new StdioServerTransport(); + await server.connect(transport); + console.error("[redis-docs-mcp] ready on stdio"); +} + +main().catch((e) => { + console.error("[redis-docs-mcp] fatal:", e); + process.exit(1); +}); diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts new file mode 100644 index 0000000000..d177d0556a --- /dev/null +++ b/build/docs-mcp-server/node/src/search.ts @@ -0,0 +1,198 @@ +import type { Page } from "./types.js"; + +// Self-contained BM25 lexical index. No external search dependency: at +// ~4,100 docs the whole index builds in-memory in well under a second, which +// is why v0 needs no datastore (see SPEC.md §6). + +const K1 = 1.5; +const B = 0.75; + +// Field boosts (added on top of the body BM25 score, weighted by term idf). +// A query term appearing in the title/slug/summary is a strong signal that the +// page is *about* that term — this lifts canonical command pages (whose summary +// is a one-line definition) above long pages that merely mention the terms. +const W_SUMMARY = 6; // canonical one-line definition — strongest signal +const W_TITLE = 4; +const W_SLUG = 2; // lowest: slug word-collisions (set-up-redis, key-specs) mislead + +// Common words carry no topical signal and their title/slug/summary collisions +// distort ranking (e.g. "set"/"key"). Dropped from the query only. +const STOPWORDS = new Set([ + "a", "an", "the", "to", "of", "in", "on", "for", "with", "and", "or", "is", + "are", "how", "do", "i", "my", "me", "can", "what", "when", "which", "you", + "your", "it", "this", "that", "from", "by", "as", "at", "be", "using", "use", +]); + +/** Split on non-alphanumerics so "JSON.SET" -> ["json","set"], "XADD" -> ["xadd"]. */ +function tokenize(text: string): string[] { + return text.toLowerCase().match(/[a-z0-9]+/g) ?? []; +} + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +/** Everything worth matching against for a page: slug, title, summary, section text. */ +function searchableText(p: Page): string { + const parts: string[] = [p.id ?? "", p.title ?? "", p.summary ?? ""]; + for (const s of p.sections ?? []) { + parts.push(s.title ?? "", s.text ?? ""); + } + return parts.join(" "); +} + +/** Section ids whose title/text contain any query term (capped for token budget). */ +function matchingSections(p: Page, qterms: Set): string[] { + const out: string[] = []; + for (const s of p.sections ?? []) { + const toks = new Set(tokenize(`${s.title ?? ""} ${s.text ?? ""}`)); + for (const t of qterms) { + if (toks.has(t)) { + out.push(s.id); + break; + } + } + if (out.length >= 5) break; + } + return out; +} + +/** + * Heuristic version filter. Redis docs carry the version in the URL path + * (e.g. /docs/latest/... or /docs/7.4/...). "latest"/unset does not filter. + * NOTE: approximate pending confirmation of how versions appear in the feed + * (SPEC.md §7 / open question). + */ +function matchesVersion(page: Page, version: string): boolean { + if (!version || version.toLowerCase() === "latest") return true; + return page.url.includes(`/${version}/`); +} + +export interface SearchOptions { + limit?: number; + pageType?: string; + version?: string; +} + +export interface SearchHit { + id: string; + title: string; + url: string; + summary: string; + page_type: string; + score: number; + matching_section_ids: string[]; +} + +export class DocsIndex { + readonly pages: Page[]; + private byId = new Map(); + private byUrl = new Map(); + private docs: Array<{ + page: Page; + tf: Map; + len: number; + titleTok: Set; + slugTok: Set; + summaryTok: Set; + }> = []; + private df = new Map(); + private avgdl = 0; + private N = 0; + + constructor(pages: Page[]) { + this.pages = pages; + let totalLen = 0; + for (const p of pages) { + this.byId.set(p.id, p); + if (p.url) this.byUrl.set(normalizeUrl(p.url), p); + + const tokens = tokenize(searchableText(p)); + if (tokens.length === 0) continue; + const tf = new Map(); + for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); + for (const t of tf.keys()) this.df.set(t, (this.df.get(t) ?? 0) + 1); + this.docs.push({ + page: p, + tf, + len: tokens.length, + titleTok: new Set(tokenize(p.title ?? "")), + slugTok: new Set(tokenize(p.id ?? "")), + summaryTok: new Set(tokenize(p.summary ?? "")), + }); + totalLen += tokens.length; + } + this.N = this.docs.length; + this.avgdl = this.N ? totalLen / this.N : 0; + } + + get size(): number { + return this.pages.length; + } + + getById(id: string): Page | undefined { + return this.byId.get(id); + } + + getByUrl(url: string): Page | undefined { + return this.byUrl.get(normalizeUrl(url)); + } + + /** Fallback lookup when a caller passes a path or partial URL. */ + findByUrlSuffix(url: string): Page | undefined { + const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); + if (!target) return undefined; + for (const p of this.pages) { + if (normalizeUrl(p.url).endsWith(target)) return p; + } + return undefined; + } + + search(query: string, opts: SearchOptions = {}): SearchHit[] { + let qterms = [...new Set(tokenize(query))].filter((t) => !STOPWORDS.has(t)); + // If the query was *all* stopwords, fall back to using them rather than + // returning nothing. + if (qterms.length === 0) qterms = [...new Set(tokenize(query))]; + if (qterms.length === 0) return []; + + const idf = new Map(); + for (const t of qterms) { + const df = this.df.get(t) ?? 0; + idf.set(t, Math.log(1 + (this.N - df + 0.5) / (df + 0.5))); + } + + const qset = new Set(qterms); + const hits: SearchHit[] = []; + for (const d of this.docs) { + if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; + if (opts.version && !matchesVersion(d.page, opts.version)) continue; + + let score = 0; + for (const t of qterms) { + const termIdf = idf.get(t) ?? 0; + const tf = d.tf.get(t); + if (tf) { + const denom = tf + K1 * (1 - B + B * (d.len / (this.avgdl || 1))); + score += termIdf * ((tf * (K1 + 1)) / denom); + } + // Field boosts: reward the term appearing in high-signal fields. + if (d.slugTok.has(t)) score += termIdf * W_SLUG; + if (d.titleTok.has(t)) score += termIdf * W_TITLE; + if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; + } + if (score > 0) { + hits.push({ + id: d.page.id, + title: d.page.title, + url: d.page.url, + summary: d.page.summary ?? "", + page_type: d.page.page_type ?? "content", + score: Number(score.toFixed(4)), + matching_section_ids: matchingSections(d.page, qset), + }); + } + } + hits.sort((a, b) => b.score - a.score); + return hits.slice(0, opts.limit ?? 10); + } +} diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts new file mode 100644 index 0000000000..feef8ac118 --- /dev/null +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -0,0 +1,28 @@ +// Offline smoke test: exercises the index + tools directly (no MCP transport). +// npm run smoke # uses test/fixture.ndjson +// DOCS_NDJSON=... npm run smoke # point at a real feed (path or URL) +import { fileURLToPath } from "node:url"; +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs } from "./tools/search-docs.js"; +import { getPage } from "./tools/get-page.js"; + +const feed = + process.env.DOCS_NDJSON ?? + fileURLToPath(new URL("../test/fixture.ndjson", import.meta.url)); + +const pages = await loadFeed(feed); +const index = new DocsIndex(pages); +console.log(`loaded ${index.size} pages from ${feed}\n`); + +console.log("# search_docs('append an entry to a stream')"); +console.log(JSON.stringify(searchDocs(index, { query: "append an entry to a stream" }), null, 2)); + +console.log("\n# search_docs('store json document', page_type=content)"); +console.log(JSON.stringify(searchDocs(index, { query: "store json document", page_type: "content" }), null, 2)); + +console.log("\n# get_page(id='commands/xadd', roles=['parameters'])"); +console.log(JSON.stringify(getPage(index, { id: "commands/xadd", roles: ["parameters"] }), null, 2)); + +console.log("\n# get_page(url='/commands/xadd/') -- partial-URL fallback"); +console.log(JSON.stringify(getPage(index, { url: "/commands/xadd/", roles: ["return"] }).url ?? "not found", null, 2)); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts new file mode 100644 index 0000000000..96b5fbba59 --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -0,0 +1,40 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; + +export const GetPageInput = z + .object({ + id: z.string().optional(), + url: z.string().optional(), + roles: z.array(z.string()).optional(), + }) + .refine((v) => Boolean(v.id || v.url), { + message: "Provide either 'id' or 'url'.", + }); +export type GetPageInput = z.infer; + +/** Fetch one page, optionally filtered to sections with the given roles. */ +export function getPage(index: DocsIndex, input: GetPageInput) { + let page = input.id ? index.getById(input.id) : undefined; + if (!page && input.url) { + page = index.getByUrl(input.url) ?? index.findByUrlSuffix(input.url); + } + if (!page) { + return { error: `Page not found for ${JSON.stringify(input.id ?? input.url)}` }; + } + + let sections = page.sections ?? []; + if (input.roles && input.roles.length) { + const want = new Set(input.roles.map((r) => r.toLowerCase())); + sections = sections.filter((s) => want.has((s.role ?? "").toLowerCase())); + } + + return { + id: page.id, + title: page.title, + url: page.url, + summary: page.summary ?? "", + page_type: page.page_type ?? "content", + content_hash: page.content_hash, + sections, + }; +} diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts new file mode 100644 index 0000000000..32313207ed --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -0,0 +1,20 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; + +export const SearchDocsInput = z.object({ + query: z.string().min(1, "query is required"), + page_type: z.enum(["content", "index"]).optional(), + version: z.string().optional(), + limit: z.number().int().positive().max(50).optional(), +}); +export type SearchDocsInput = z.infer; + +/** Rank pages by relevance. Returns refs + summaries only — never full text. */ +export function searchDocs(index: DocsIndex, input: SearchDocsInput) { + const results = index.search(input.query, { + limit: input.limit ?? 10, + pageType: input.page_type, + version: input.version, + }); + return { query: input.query, count: results.length, results }; +} diff --git a/build/docs-mcp-server/node/src/types.ts b/build/docs-mcp-server/node/src/types.ts new file mode 100644 index 0000000000..82ab8cc32d --- /dev/null +++ b/build/docs-mcp-server/node/src/types.ts @@ -0,0 +1,35 @@ +// Document schema as published in docs.ndjson / per-page index.json. +// See content/ai-agent-resources.md for the authoritative field reference. + +export interface Section { + id: string; + title: string; + /** Semantic role: overview | syntax | parameters | returns | example | ... */ + role?: string; + text: string; +} + +export interface Example { + id: string; + language: string; + code: string; + section_id: string; +} + +export interface Child { + title?: string; + url?: string; +} + +export interface Page { + id: string; + title: string; + url: string; + summary?: string; + /** "content" (has prose) or "index" (navigation only) */ + page_type?: string; + content_hash?: string; + sections?: Section[]; + examples?: Example[]; + children?: Child[]; +} diff --git a/build/docs-mcp-server/node/test/fixture.ndjson b/build/docs-mcp-server/node/test/fixture.ndjson new file mode 100644 index 0000000000..6e0c4ca8b8 --- /dev/null +++ b/build/docs-mcp-server/node/test/fixture.ndjson @@ -0,0 +1,3 @@ +{"id":"commands/xadd","title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/","summary":"Appends a new entry to a stream.","page_type":"content","content_hash":"abc123","sections":[{"id":"overview","title":"XADD","role":"overview","text":"Appends the specified stream entry to the stream at the specified key. If the key does not exist, a new stream is created."},{"id":"parameters","title":"Parameters","role":"parameters","text":"key: the name of the stream. NOMKSTREAM: optional, do not create the stream if it does not exist. field value: one or more field-value pairs to add to the stream entry."},{"id":"return","title":"Return value","role":"returns","text":"Returns the ID of the added stream entry."}],"examples":[{"id":"overview-ex0","language":"python","code":"r.xadd('mystream', {'field': 'value'})","section_id":"overview"}],"children":[]} +{"id":"develop/data-types/json","title":"JSON","url":"https://redis.io/docs/latest/develop/data-types/json/","summary":"Store and query JSON documents in Redis.","page_type":"content","content_hash":"def456","sections":[{"id":"overview","title":"Redis JSON","role":"overview","text":"The JSON data type lets you store, update, and retrieve JSON values in a Redis database."},{"id":"example","title":"Example","role":"example","text":"Use JSON.SET to store a document and JSON.GET to retrieve values from it by path."}],"examples":[{"id":"example-ex0","language":"python","code":"r.json().set('doc', '$', {'a': 1})","section_id":"example"}],"children":[]} +{"id":"commands","title":"Commands","url":"https://redis.io/docs/latest/commands/","summary":"Reference documentation for all Redis commands.","page_type":"index","children":[{"title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/"}]} diff --git a/build/docs-mcp-server/node/tsconfig.json b/build/docs-mcp-server/node/tsconfig.json new file mode 100644 index 0000000000..01b2904d2f --- /dev/null +++ b/build/docs-mcp-server/node/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "esnext", + "lib": ["ES2022"], + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true, + "declaration": true, + "sourceMap": true, + "moduleResolution": "node" + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist", "src/smoke.ts"] +} From ea1b9725482f9a4ce9324baf2c771e47fcf7b4fe Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:05:53 +0100 Subject: [PATCH 02/25] DOC-6809 Fix get_page id/url ambiguity and error signalling (Bugbot + Codex) Address Cursor Bugbot findings on PR #3585 plus a Codex follow-up review. The feed's `id` is the last URL path segment and is NOT unique (~213 ids map to several pages), so `byId` is now a multimap and `get_page` resolves by the unique `url` first; an ambiguous `id` or partial `url` returns candidate urls instead of silently returning the wrong page. The MCP handler now sets `isError` when a tool result carries an `error`. Smoke test rewritten as assertions (incl. a colliding id="install" fixture pair) so this can't regress. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/src/index.ts | 30 +++++----- build/docs-mcp-server/node/src/response.ts | 24 ++++++++ build/docs-mcp-server/node/src/search.ts | 30 ++++++---- build/docs-mcp-server/node/src/smoke.ts | 56 +++++++++++++++---- .../node/src/tools/get-page.ts | 39 +++++++++++-- .../docs-mcp-server/node/test/fixture.ndjson | 2 + 6 files changed, 138 insertions(+), 43 deletions(-) create mode 100644 build/docs-mcp-server/node/src/response.ts diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index ff6fbd87db..43ac439c42 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -10,6 +10,7 @@ import { loadFeed } from "./feed.js"; import { DocsIndex } from "./search.js"; import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; import { getPage, GetPageInput } from "./tools/get-page.js"; +import { toolResult, fail } from "./response.js"; // Feed source: local path or http(s) URL, gzip-aware. Defaults to production. const FEED_SOURCE = @@ -19,7 +20,7 @@ const TOOLS = [ { name: "search_docs", description: - "Search the Redis documentation and return the most relevant pages as references (id, title, url, summary, matching section ids). Returns no full text — follow up with get_page to read a result.", + "Search the Redis documentation and return the most relevant pages as references (title, url, summary, matching section ids). Each hit includes a unique `url` — pass that `url` to get_page (a hit's `id` is NOT unique and may be ambiguous). Returns no full text — follow up with get_page to read a result.", inputSchema: { type: "object" as const, properties: { @@ -44,12 +45,18 @@ const TOOLS = [ { name: "get_page", description: - "Fetch a single documentation page by 'id' or 'url'. Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", + "Fetch a single documentation page. Prefer the unique `url` from a search_docs hit. `id` also works but is NOT unique, so an ambiguous id returns an error listing candidate urls (likewise an ambiguous partial url). Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", inputSchema: { type: "object" as const, properties: { - id: { type: "string", description: "Page id (URL slug), e.g. 'commands/xadd'" }, - url: { type: "string", description: "Full or partial page URL" }, + id: { + type: "string", + description: "Page id (last URL slug); not unique — prefer url", + }, + url: { + type: "string", + description: "Full page URL (preferred — unique). A partial/suffix URL also works when it matches exactly one page.", + }, roles: { type: "array", items: { type: "string" }, @@ -60,17 +67,6 @@ const TOOLS = [ }, ]; -function ok(data: unknown) { - return { content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }] }; -} - -function fail(message: string) { - return { - content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], - isError: true, - }; -} - async function main() { const pages = await loadFeed(FEED_SOURCE); const index = new DocsIndex(pages); @@ -89,9 +85,9 @@ async function main() { try { switch (name) { case "search_docs": - return ok(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + return toolResult(searchDocs(index, SearchDocsInput.parse(args ?? {}))); case "get_page": - return ok(getPage(index, GetPageInput.parse(args ?? {}))); + return toolResult(getPage(index, GetPageInput.parse(args ?? {}))); default: return fail(`Unknown tool: ${name}`); } diff --git a/build/docs-mcp-server/node/src/response.ts b/build/docs-mcp-server/node/src/response.ts new file mode 100644 index 0000000000..6f1db590f1 --- /dev/null +++ b/build/docs-mcp-server/node/src/response.ts @@ -0,0 +1,24 @@ +// MCP tool-response helpers, factored out of index.ts so tests can import them +// without triggering index.ts's main() (which starts the stdio server). + +/** + * Serialise a tool result. If the tool returned an object carrying an `error` + * field (e.g. get_page couldn't resolve the page, or an id/url was ambiguous), + * mark the MCP response as an error so clients don't treat a failed lookup as + * success. + */ +export function toolResult(data: unknown) { + const isError = typeof data === "object" && data !== null && "error" in data; + return { + content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }], + ...(isError ? { isError: true } : {}), + }; +} + +/** For protocol-level failures (unknown tool, input parse errors). */ +export function fail(message: string) { + return { + content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], + isError: true, + }; +} diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index d177d0556a..ca2804e0e8 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -86,7 +86,10 @@ export interface SearchHit { export class DocsIndex { readonly pages: Page[]; - private byId = new Map(); + // The feed's `id` is the last URL path segment (e.g. "config", "acl") and is + // NOT unique — ~200 ids map to several pages. So id -> list, and callers must + // disambiguate by url. `url` IS unique, so byUrl stays 1:1. + private byId = new Map(); private byUrl = new Map(); private docs: Array<{ page: Page; @@ -104,7 +107,9 @@ export class DocsIndex { this.pages = pages; let totalLen = 0; for (const p of pages) { - this.byId.set(p.id, p); + const bucket = this.byId.get(p.id); + if (bucket) bucket.push(p); + else this.byId.set(p.id, [p]); if (p.url) this.byUrl.set(normalizeUrl(p.url), p); const tokens = tokenize(searchableText(p)); @@ -130,22 +135,25 @@ export class DocsIndex { return this.pages.length; } - getById(id: string): Page | undefined { - return this.byId.get(id); + /** All pages sharing this id (usually one, but the feed's id is not unique). */ + getPagesById(id: string): Page[] { + return this.byId.get(id) ?? []; } getByUrl(url: string): Page | undefined { return this.byUrl.get(normalizeUrl(url)); } - /** Fallback lookup when a caller passes a path or partial URL. */ - findByUrlSuffix(url: string): Page | undefined { + /** + * Fallback lookup when a caller passes a path or partial URL. Returns EVERY + * page whose url ends with the given suffix — a suffix can match several + * pages (e.g. "/install/"), so the caller must disambiguate rather than + * silently taking the first. + */ + matchByUrlSuffix(url: string): Page[] { const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); - if (!target) return undefined; - for (const p of this.pages) { - if (normalizeUrl(p.url).endsWith(target)) return p; - } - return undefined; + if (!target) return []; + return this.pages.filter((p) => normalizeUrl(p.url).endsWith(target)); } search(query: string, opts: SearchOptions = {}): SearchHit[] { diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index feef8ac118..4a3cbb30f0 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -1,11 +1,21 @@ -// Offline smoke test: exercises the index + tools directly (no MCP transport). -// npm run smoke # uses test/fixture.ndjson -// DOCS_NDJSON=... npm run smoke # point at a real feed (path or URL) +// Offline smoke test: assertion-based checks of the index + tools + MCP +// response wrapping (no stdio transport). Exits non-zero on any failure. +// npm run smoke +// Runs against test/fixture.ndjson; the assertions encode fixture-specific +// ids/urls (incl. the colliding id="install" pair), so it is not meant to be +// pointed at the live feed. import { fileURLToPath } from "node:url"; import { loadFeed } from "./feed.js"; import { DocsIndex } from "./search.js"; import { searchDocs } from "./tools/search-docs.js"; import { getPage } from "./tools/get-page.js"; +import { toolResult } from "./response.js"; + +let failures = 0; +function check(label: string, cond: boolean) { + console.log(`${cond ? "PASS" : "FAIL"} ${label}`); + if (!cond) failures++; +} const feed = process.env.DOCS_NDJSON ?? @@ -15,14 +25,38 @@ const pages = await loadFeed(feed); const index = new DocsIndex(pages); console.log(`loaded ${index.size} pages from ${feed}\n`); -console.log("# search_docs('append an entry to a stream')"); -console.log(JSON.stringify(searchDocs(index, { query: "append an entry to a stream" }), null, 2)); +// --- search_docs --- +const stream = searchDocs(index, { query: "append an entry to a stream" }); +check("search returns hits", stream.count > 0); +check("search hits carry a url", Boolean(stream.results[0]?.url)); + +// --- get_page happy paths --- +const xadd = getPage(index, { id: "commands/xadd", roles: ["parameters"] }) as any; +check("get_page(unique id) resolves", xadd.id === "commands/xadd"); +check("roles filter returns only 'parameters'", (xadd.sections ?? []).every((s: any) => s.role === "parameters")); + +const exact = getPage(index, { url: "https://redis.io/docs/latest/operate/redisinsight/install/" }) as any; +check("get_page(exact url) resolves the right page", exact.title === "Install Redis Insight"); + +const suffixUnique = getPage(index, { url: "/commands/xadd/" }) as any; +check("get_page(unambiguous partial url) resolves", suffixUnique.id === "commands/xadd"); + +// --- get_page ambiguity (Bugbot High + Codex Medium) --- +const ambId = getPage(index, { id: "install" }) as any; +check("ambiguous id returns error", typeof ambId.error === "string"); +check("ambiguous id lists candidates", (ambId.candidates ?? []).length === 2); + +const ambUrl = getPage(index, { url: "/install/" }) as any; +check("ambiguous partial url returns error (not silent first match)", typeof ambUrl.error === "string"); +check("ambiguous partial url lists candidates", (ambUrl.candidates ?? []).length === 2); -console.log("\n# search_docs('store json document', page_type=content)"); -console.log(JSON.stringify(searchDocs(index, { query: "store json document", page_type: "content" }), null, 2)); +const missing = getPage(index, { id: "does-not-exist-anywhere" }) as any; +check("missing page returns error", typeof missing.error === "string"); -console.log("\n# get_page(id='commands/xadd', roles=['parameters'])"); -console.log(JSON.stringify(getPage(index, { id: "commands/xadd", roles: ["parameters"] }), null, 2)); +// --- MCP response wrapping (Fix 2 / Bugbot Medium) --- +check("toolResult(missing) sets isError", toolResult(missing).isError === true); +check("toolResult(ambiguous id) sets isError", toolResult(ambId).isError === true); +check("toolResult(search) does NOT set isError", toolResult(stream).isError === undefined); -console.log("\n# get_page(url='/commands/xadd/') -- partial-URL fallback"); -console.log(JSON.stringify(getPage(index, { url: "/commands/xadd/", roles: ["return"] }).url ?? "not found", null, 2)); +console.log(`\n${failures === 0 ? "ALL PASSED" : failures + " FAILED"}`); +if (failures > 0) process.exit(1); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index 96b5fbba59..107cc486a3 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import type { DocsIndex } from "../search.js"; +import type { Page } from "../types.js"; export const GetPageInput = z .object({ @@ -12,14 +13,44 @@ export const GetPageInput = z }); export type GetPageInput = z.infer; +/** An ambiguous id/url lookup: report all candidates so the caller can retry. */ +function ambiguous(kind: "id" | "url", value: string, matches: Page[]) { + return { + error: `Ambiguous ${kind} '${value}' matches ${matches.length} pages. Call get_page again with a full, exact url.`, + candidates: matches.map((p) => ({ title: p.title, url: p.url })), + }; +} + /** Fetch one page, optionally filtered to sections with the given roles. */ export function getPage(index: DocsIndex, input: GetPageInput) { - let page = input.id ? index.getById(input.id) : undefined; - if (!page && input.url) { - page = index.getByUrl(input.url) ?? index.findByUrlSuffix(input.url); + // Prefer url: a full url is unique. A partial/suffix url can match several + // pages, so disambiguate rather than silently taking the first. + let page: Page | undefined; + if (input.url) { + page = index.getByUrl(input.url); + if (!page) { + const matches = index.matchByUrlSuffix(input.url); + if (matches.length === 1) { + page = matches[0]; + } else if (matches.length > 1) { + return ambiguous("url", input.url, matches); + } + } } + + // id is the last URL path segment and can match several pages, so resolve by + // id only when unambiguous. + if (!page && input.id) { + const matches = index.getPagesById(input.id); + if (matches.length === 1) { + page = matches[0]; + } else if (matches.length > 1) { + return ambiguous("id", input.id, matches); + } + } + if (!page) { - return { error: `Page not found for ${JSON.stringify(input.id ?? input.url)}` }; + return { error: `Page not found for ${JSON.stringify(input.url ?? input.id)}` }; } let sections = page.sections ?? []; diff --git a/build/docs-mcp-server/node/test/fixture.ndjson b/build/docs-mcp-server/node/test/fixture.ndjson index 6e0c4ca8b8..95b767477c 100644 --- a/build/docs-mcp-server/node/test/fixture.ndjson +++ b/build/docs-mcp-server/node/test/fixture.ndjson @@ -1,3 +1,5 @@ {"id":"commands/xadd","title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/","summary":"Appends a new entry to a stream.","page_type":"content","content_hash":"abc123","sections":[{"id":"overview","title":"XADD","role":"overview","text":"Appends the specified stream entry to the stream at the specified key. If the key does not exist, a new stream is created."},{"id":"parameters","title":"Parameters","role":"parameters","text":"key: the name of the stream. NOMKSTREAM: optional, do not create the stream if it does not exist. field value: one or more field-value pairs to add to the stream entry."},{"id":"return","title":"Return value","role":"returns","text":"Returns the ID of the added stream entry."}],"examples":[{"id":"overview-ex0","language":"python","code":"r.xadd('mystream', {'field': 'value'})","section_id":"overview"}],"children":[]} {"id":"develop/data-types/json","title":"JSON","url":"https://redis.io/docs/latest/develop/data-types/json/","summary":"Store and query JSON documents in Redis.","page_type":"content","content_hash":"def456","sections":[{"id":"overview","title":"Redis JSON","role":"overview","text":"The JSON data type lets you store, update, and retrieve JSON values in a Redis database."},{"id":"example","title":"Example","role":"example","text":"Use JSON.SET to store a document and JSON.GET to retrieve values from it by path."}],"examples":[{"id":"example-ex0","language":"python","code":"r.json().set('doc', '$', {'a': 1})","section_id":"example"}],"children":[]} {"id":"commands","title":"Commands","url":"https://redis.io/docs/latest/commands/","summary":"Reference documentation for all Redis commands.","page_type":"index","children":[{"title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/"}]} +{"id":"install","title":"Install Redis","url":"https://redis.io/docs/latest/operate/oss_and_stack/install/","summary":"Install Redis Open Source on your platform.","page_type":"content","content_hash":"aaa111","sections":[{"id":"overview","title":"Install Redis","role":"overview","text":"Install Redis Open Source on Linux, macOS, or Windows."}],"examples":[],"children":[]} +{"id":"install","title":"Install Redis Insight","url":"https://redis.io/docs/latest/operate/redisinsight/install/","summary":"Install the Redis Insight GUI.","page_type":"content","content_hash":"bbb222","sections":[{"id":"overview","title":"Install Redis Insight","role":"overview","text":"Download and install the Redis Insight desktop application."}],"examples":[],"children":[]} From 97b2dd29d2956b8969807cdf671072e03c52564c Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:09:48 +0100 Subject: [PATCH 03/25] DOC-6809 Add local MCP client harness; note role-vocabulary gap in spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test/mcp-client.mjs drives the built stdio server through the real MCP client (spawn -> initialize -> tools/list -> tools/call) against the fixture, so the transport path and isError signalling are exercised end-to-end, not just the tool logic. Live-feed testing surfaced that the spec's assumed section roles (syntax/examples) don't match the feed (it uses content/parameters/example/ returns, no syntax), so a roles filter can silently return nothing — recorded in SPEC open questions to resolve before building get_examples/get_command. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 9 ++++ .../docs-mcp-server/node/test/mcp-client.mjs | 44 +++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 build/docs-mcp-server/node/test/mcp-client.mjs diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 3a8a739d21..cdc8c1dba3 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -202,6 +202,15 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. overviews** that repeat the same terms. Fix needs an analyzer (stemming/lemmatization), possibly a page-type/canonical boost, and/or the vector-search path. This is the strongest argument for §6's v2 upgrade. +- **Section-role vocabulary (found via live MCP test):** the roles the spec + assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match + the feed. Command pages actually carry `content` / `parameters` / `example` + (singular) / `returns` — there is no `syntax` role, and it's `example` not + `examples`. So a `roles: ["examples","syntax"]` filter returns **zero + sections** against the real feed (verified on EXPIREAT). Before building + `get_examples` / `get_command` and documenting `roles`, enumerate the actual + role set across the corpus and align tool params/docs to it (and decide + whether the server should normalise synonyms like `examples`→`example`). - **`get_command` coverage:** command pages *do* carry `parameters`/`returns`/ `example` roles (confirmed: `get_page('expire')` → roles `[content, parameters, example, returns]`). Confirm this holds across all diff --git a/build/docs-mcp-server/node/test/mcp-client.mjs b/build/docs-mcp-server/node/test/mcp-client.mjs new file mode 100644 index 0000000000..5010723f0e --- /dev/null +++ b/build/docs-mcp-server/node/test/mcp-client.mjs @@ -0,0 +1,44 @@ +// Manual local test: drive the built stdio server through the real MCP client +// (spawn -> initialize -> tools/list -> tools/call), against the local fixture. +// node test/mcp-client.mjs +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"; +import { fileURLToPath } from "node:url"; + +const feed = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./fixture.ndjson", import.meta.url)); +const serverEntry = fileURLToPath(new URL("../dist/index.js", import.meta.url)); + +const transport = new StdioClientTransport({ + command: "node", + args: [serverEntry], + env: { ...process.env, DOCS_NDJSON: feed }, +}); + +const client = new Client({ name: "local-test-client", version: "0.0.1" }, { capabilities: {} }); +await client.connect(transport); +console.log("connected to server\n"); + +const tools = await client.listTools(); +console.log("tools/list ->", tools.tools.map((t) => t.name).join(", "), "\n"); + +const search = await client.callTool({ + name: "search_docs", + arguments: { query: "append an entry to a stream" }, +}); +console.log("tools/call search_docs('append an entry to a stream'):"); +console.log(search.content[0].text, "\n"); + +const amb = await client.callTool({ name: "get_page", arguments: { id: "install" } }); +console.log(`tools/call get_page(id='install') -> isError=${amb.isError}`); +console.log(amb.content[0].text, "\n"); + +const page = await client.callTool({ + name: "get_page", + arguments: { url: "https://redis.io/docs/latest/commands/xadd/", roles: ["parameters"] }, +}); +console.log(`tools/call get_page(url=.../commands/xadd/, roles=['parameters']) -> isError=${page.isError}`); +console.log(page.content[0].text); + +await client.close(); +console.log("\nclosed cleanly"); From e94b8562ee74eff2b2cec4307474654fec96be24 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:30:22 +0100 Subject: [PATCH 04/25] DOC-6809 Consolidate get_page resolution; drop unbacked version param MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 Bugbot findings on the get_page/search resolution logic were churn: the round-1 duplicate-id fix introduced two of them. Rather than three point- patches, consolidate resolution into one convergence model (per Codex design review): collect candidate pages across all supplied handles (exact url, id, boundary-anchored suffix url) and resolve only when they converge on exactly one page — so an ambiguous url still tries id, and url/id pointing at different pages reports conflicting handles instead of silently applying precedence. - Partial-url suffix matching is now anchored to "/" path segments, so "get" matches .../commands/get but not .../config-get or .../arget (was 19 false matches on the live feed, now 2 legitimate ones). [Bugbot High] - Ambiguous url no longer short-circuits the id fallback. [Bugbot Medium] - Removed the `version` param: it advertised a `latest` default it never enforced. Deferred until a committed URL/version model exists. [Bugbot Medium] - Smoke test extended with boundary and conflicting-handles assertions (17 total). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 13 +++- build/docs-mcp-server/node/README.md | 5 +- build/docs-mcp-server/node/src/index.ts | 4 -- build/docs-mcp-server/node/src/search.ts | 26 +++---- build/docs-mcp-server/node/src/smoke.ts | 12 ++++ .../node/src/tools/get-page.ts | 71 +++++++++++-------- .../node/src/tools/search-docs.ts | 2 - 7 files changed, 74 insertions(+), 59 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index cdc8c1dba3..3bbd70272d 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -170,9 +170,16 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. ## 7. Versioning -- `search_docs` / `get_page` accept an optional `version`; default to `latest`. -- Responses echo the resolved version so an agent can't silently blend - versions. This is the single most common RAG-over-docs correctness bug. +- **Planned:** `search_docs` / `get_page` accept an optional `version` (default + `latest`), and responses echo the resolved version so an agent can't silently + blend versions — the single most common RAG-over-docs correctness bug. +- **v0 status: deferred, not implemented.** The prototype does **not** expose a + `version` param. An earlier v0 advertised a `latest` default it didn't + enforce (the filter was a no-op, and the live feed is single-version anyway), + which misleads agents. Rather than bake in a `page.url.includes("//")` + heuristic, the param was removed until a committed URL/version model exists + (Bugbot #3585 + Codex review). Add it back with the real filter when the feed + carries multiple version trees. ## 8. Freshness diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 0bd4af5209..3995ca0121 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -51,7 +51,7 @@ Add to a Claude Code / Cursor MCP config after `npm run build`: | Tool | Inputs | Returns | |------|--------|---------| -| `search_docs` | `query` (req), `page_type`, `version`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | +| `search_docs` | `query` (req), `page_type`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | | `get_page` | `id` **or** `url` (req), `roles[]` | one page with `content_hash` + `sections`, optionally filtered to the given section roles | Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` @@ -73,6 +73,7 @@ Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` terms. Fixing this properly means an analyzer (stemming/lemmatization) and/or the vector-search upgrade tracked in SPEC §6/§10. -- **Version filtering is heuristic** (URL-path based); see SPEC §7. +- **No version filtering.** The `version` param was removed until a committed + URL/version model exists (the feed is single-version today); see SPEC §7. - Only `search_docs` + `get_page`. `get_examples`, `get_section`, `get_command` are v1 (SPEC §4/§11). diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index 43ac439c42..21fbcd51ec 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -30,10 +30,6 @@ const TOOLS = [ enum: ["content", "index"], description: "Restrict to prose ('content') or navigation ('index') pages", }, - version: { - type: "string", - description: "Redis docs version, e.g. 'latest' (default) or '7.4'", - }, limit: { type: "number", description: "Max results (default 10, max 50)", diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index ca2804e0e8..9e54cd0efb 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -57,21 +57,9 @@ function matchingSections(p: Page, qterms: Set): string[] { return out; } -/** - * Heuristic version filter. Redis docs carry the version in the URL path - * (e.g. /docs/latest/... or /docs/7.4/...). "latest"/unset does not filter. - * NOTE: approximate pending confirmation of how versions appear in the feed - * (SPEC.md §7 / open question). - */ -function matchesVersion(page: Page, version: string): boolean { - if (!version || version.toLowerCase() === "latest") return true; - return page.url.includes(`/${version}/`); -} - export interface SearchOptions { limit?: number; pageType?: string; - version?: string; } export interface SearchHit { @@ -146,14 +134,19 @@ export class DocsIndex { /** * Fallback lookup when a caller passes a path or partial URL. Returns EVERY - * page whose url ends with the given suffix — a suffix can match several - * pages (e.g. "/install/"), so the caller must disambiguate rather than - * silently taking the first. + * page whose url ends with the given suffix **at a path-segment boundary**, + * so "get" matches ".../commands/get" but NOT ".../config-get" or + * ".../arget". A suffix can still match several pages (e.g. "/install/" or a + * bare last segment shared by many pages), so the caller must disambiguate. */ matchByUrlSuffix(url: string): Page[] { const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); if (!target) return []; - return this.pages.filter((p) => normalizeUrl(p.url).endsWith(target)); + // Anchor the leading edge to a "/" so we match whole path segments. The + // trailing edge is already anchored: normalizeUrl strips the trailing slash + // and we compare against the end of the string. + const anchored = target.startsWith("/") ? target : `/${target}`; + return this.pages.filter((p) => normalizeUrl(p.url).endsWith(anchored)); } search(query: string, opts: SearchOptions = {}): SearchHit[] { @@ -173,7 +166,6 @@ export class DocsIndex { const hits: SearchHit[] = []; for (const d of this.docs) { if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; - if (opts.version && !matchesVersion(d.page, opts.version)) continue; let score = 0; for (const t of qterms) { diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index 4a3cbb30f0..a44475d6cd 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -50,6 +50,18 @@ const ambUrl = getPage(index, { url: "/install/" }) as any; check("ambiguous partial url returns error (not silent first match)", typeof ambUrl.error === "string"); check("ambiguous partial url lists candidates", (ambUrl.candidates ?? []).length === 2); +// --- boundary-anchored suffix (Bugbot round-2 High): "add" must NOT match "xadd" --- +const boundary = getPage(index, { url: "add" }) as any; +check("partial url matches only on path-segment boundary (add !-> xadd)", typeof boundary.error === "string" && !boundary.candidates); + +// --- conflicting handles (Codex convergence model): url and id point at different pages --- +const conflict = getPage(index, { + url: "https://redis.io/docs/latest/develop/data-types/json/", + id: "commands/xadd", +}) as any; +check("conflicting url+id returns error", typeof conflict.error === "string"); +check("conflicting url+id lists both candidates", (conflict.candidates ?? []).length === 2); + const missing = getPage(index, { id: "does-not-exist-anywhere" }) as any; check("missing page returns error", typeof missing.error === "string"); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index 107cc486a3..cbe9bd7f25 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -13,46 +13,55 @@ export const GetPageInput = z }); export type GetPageInput = z.infer; -/** An ambiguous id/url lookup: report all candidates so the caller can retry. */ -function ambiguous(kind: "id" | "url", value: string, matches: Page[]) { - return { - error: `Ambiguous ${kind} '${value}' matches ${matches.length} pages. Call get_page again with a full, exact url.`, - candidates: matches.map((p) => ({ title: p.title, url: p.url })), +/** + * The distinct pages (deduped by unique url) that the supplied handles resolve + * to. `url` is authoritative when it matches an exact page; otherwise it is + * treated as a path-boundary suffix. `id` is the last URL segment and is not + * unique, so it may contribute several pages. Collecting across all handles + * (rather than short-circuiting) means an ambiguous url still lets `id` help, + * and it surfaces the case where url and id point at *different* pages. + */ +function collectCandidates(index: DocsIndex, input: GetPageInput): Page[] { + const byUrl = new Map(); + const add = (p: Page | undefined) => { + if (p) byUrl.set(p.url, p); }; + + if (input.url) { + const exact = index.getByUrl(input.url); + if (exact) add(exact); + else index.matchByUrlSuffix(input.url).forEach(add); + } + if (input.id) { + index.getPagesById(input.id).forEach(add); + } + return [...byUrl.values()]; +} + +function describeHandles(input: GetPageInput): string { + const parts: string[] = []; + if (input.url) parts.push(`url '${input.url}'`); + if (input.id) parts.push(`id '${input.id}'`); + return parts.join(" and "); } /** Fetch one page, optionally filtered to sections with the given roles. */ export function getPage(index: DocsIndex, input: GetPageInput) { - // Prefer url: a full url is unique. A partial/suffix url can match several - // pages, so disambiguate rather than silently taking the first. - let page: Page | undefined; - if (input.url) { - page = index.getByUrl(input.url); - if (!page) { - const matches = index.matchByUrlSuffix(input.url); - if (matches.length === 1) { - page = matches[0]; - } else if (matches.length > 1) { - return ambiguous("url", input.url, matches); - } - } - } + const candidates = collectCandidates(index, input); - // id is the last URL path segment and can match several pages, so resolve by - // id only when unambiguous. - if (!page && input.id) { - const matches = index.getPagesById(input.id); - if (matches.length === 1) { - page = matches[0]; - } else if (matches.length > 1) { - return ambiguous("id", input.id, matches); - } + if (candidates.length === 0) { + return { error: `Page not found for ${describeHandles(input)}.` }; } - - if (!page) { - return { error: `Page not found for ${JSON.stringify(input.url ?? input.id)}` }; + if (candidates.length > 1) { + // Ambiguous (a non-unique id / boundary suffix) or conflicting (url and id + // point at different pages) — either way, make the caller pick a url. + return { + error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, + candidates: candidates.map((p) => ({ title: p.title, url: p.url })), + }; } + const page = candidates[0]; let sections = page.sections ?? []; if (input.roles && input.roles.length) { const want = new Set(input.roles.map((r) => r.toLowerCase())); diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts index 32313207ed..08f1574feb 100644 --- a/build/docs-mcp-server/node/src/tools/search-docs.ts +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -4,7 +4,6 @@ import type { DocsIndex } from "../search.js"; export const SearchDocsInput = z.object({ query: z.string().min(1, "query is required"), page_type: z.enum(["content", "index"]).optional(), - version: z.string().optional(), limit: z.number().int().positive().max(50).optional(), }); export type SearchDocsInput = z.infer; @@ -14,7 +13,6 @@ export function searchDocs(index: DocsIndex, input: SearchDocsInput) { const results = index.search(input.query, { limit: input.limit ?? 10, pageType: input.page_type, - version: input.version, }); return { query: input.query, count: results.length, results }; } From 7f8ef54f67d64853557d3ab1e0e10a56512868ba Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:38:56 +0100 Subject: [PATCH 05/25] DOC-6809 Add retrieval eval harness; baseline shows lexical is insufficient MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit npm run eval scores search_docs retrieval against a curated question set (test/eval/cases.json — command lookups phrased without the command name), reporting recall@k / MRR with a data-integrity check that flags expected urls missing from the feed. Feed read from DOCS_NDJSON or a gitignored local cache. Baseline on the live feed (22 cases): recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42. Confirms with data what was hypothesised: lexical retrieval alone is not good enough — canonical command pages lose to sibling commands (zadd -> zincrby), operator pages (del -> remove-node), and concept pages (ft.create -> a full-text concept page), and there is no stemming (append !-> appends). This is the measured case for the stemming/boost/vector work in SPEC §6/§10 and lets future ranking changes be compared against a baseline instead of tuned blind. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/.gitignore | 1 + build/docs-mcp-server/node/README.md | 17 ++++- build/docs-mcp-server/node/package.json | 3 +- .../docs-mcp-server/node/test/eval/cases.json | 24 +++++++ build/docs-mcp-server/node/test/eval/run.mjs | 72 +++++++++++++++++++ 5 files changed, 115 insertions(+), 2 deletions(-) create mode 100644 build/docs-mcp-server/node/test/eval/cases.json create mode 100644 build/docs-mcp-server/node/test/eval/run.mjs diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore index 3c25e1e49c..d458242e86 100644 --- a/build/docs-mcp-server/node/.gitignore +++ b/build/docs-mcp-server/node/.gitignore @@ -1,3 +1,4 @@ node_modules/ dist/ *.log +test/eval/docs.ndjson* diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 3995ca0121..f85e4977e7 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -57,11 +57,26 @@ Add to a Claude Code / Cursor MCP config after `npm run build`: Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` (e.g. `["parameters","returns"]`) to pull just what's needed. -## Measured (real feed, 2,531 pages) +## Measured (real feed, ~2,530 pages) - Index build: ~0.3 s. Query latency: ~75–125 ms. This is why v0 needs no datastore and why Rust/WASM would be premature (see SPEC §6). +## Retrieval eval + +`npm run eval` scores retrieval quality: it runs the questions in +`test/eval/cases.json` (command-lookup questions phrased *without* the command +name) through `search_docs` and reports recall@k / MRR, plus a data-integrity +check that flags any expected url missing from the feed. The feed is read from +`DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; +`curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). + +**Baseline (lexical v0):** recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42 — i.e. +lexical retrieval is **not** good enough alone (canonical command pages lose to +sibling commands, operator, and concept pages; no stemming). This is the +measured case for the ranking / vector-search work in SPEC §6/§10. Use it to +compare any ranking change against the baseline rather than tuning blind. + ## Known limitations (v0) - **Ranking is untuned lexical search.** Good on distinctive terms diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json index ce615096b0..90f51e6444 100644 --- a/build/docs-mcp-server/node/package.json +++ b/build/docs-mcp-server/node/package.json @@ -11,7 +11,8 @@ "build": "tsc", "start": "tsx src/index.ts", "dev": "tsx watch src/index.ts", - "smoke": "tsx src/smoke.ts" + "smoke": "tsx src/smoke.ts", + "eval": "npm run build && node test/eval/run.mjs" }, "keywords": [ "redis", diff --git a/build/docs-mcp-server/node/test/eval/cases.json b/build/docs-mcp-server/node/test/eval/cases.json new file mode 100644 index 0000000000..382331819e --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/cases.json @@ -0,0 +1,24 @@ +[ + { "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, + { "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, + { "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, + { "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, + { "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, + { "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, + { "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, + { "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, + { "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, + { "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, + { "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, + { "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, + { "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, + { "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, + { "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, + { "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, + { "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, + { "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, + { "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, + { "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, + { "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, + { "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] } +] diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs new file mode 100644 index 0000000000..c73964d8ca --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -0,0 +1,72 @@ +// AI-answer eval (retrieval quality) for the docs MCP server. +// Runs each question in cases.json through search_docs and measures whether the +// expected canonical page is retrieved (recall@k, MRR). A data-integrity check +// flags any expected url that isn't in the feed, so a bad ground-truth entry is +// reported rather than silently scored as a miss. +// +// node test/eval/run.mjs # uses cached test/eval/docs.ndjson.gz +// DOCS_NDJSON= node test/eval/run.mjs +// +// Imports the BUILT server (dist/), so run `npm run build` first. +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { loadFeed } from "../../dist/feed.js"; +import { DocsIndex } from "../../dist/search.js"; +import { searchDocs } from "../../dist/tools/search-docs.js"; + +const K = [1, 3, 5, 10]; +const LIMIT = 10; +const norm = (u) => u.trim().toLowerCase().replace(/\/+$/, ""); +const short = (u) => (u ? u.replace("https://redis.io/docs/latest", "") : "—"); + +const feedSrc = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./docs.ndjson.gz", import.meta.url)); +const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", import.meta.url)), "utf8")); + +const pages = await loadFeed(feedSrc); +const index = new DocsIndex(pages); +const feedUrls = new Set(pages.map((p) => norm(p.url))); + +const broken = []; +const rows = []; +for (const c of cases) { + const expected = c.expected.map(norm); + if (expected.every((u) => !feedUrls.has(u))) { + broken.push({ q: c.q, missing: expected }); + continue; + } + const results = searchDocs(index, { query: c.q, limit: LIMIT }).results.map((r) => norm(r.url)); + let rank = null; + for (let i = 0; i < results.length; i++) { + if (expected.includes(results[i])) { + rank = i + 1; + break; + } + } + rows.push({ q: c.q, rank, top: results[0] }); +} + +const scored = rows.length; +const recall = Object.fromEntries( + K.map((k) => [k, rows.filter((r) => r.rank && r.rank <= k).length / scored]), +); +const mrr = rows.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / scored; + +console.log( + `Feed: ${pages.length} pages | cases scored: ${scored}` + + (broken.length ? ` | ${broken.length} BROKEN (expected url not in feed)` : "") + + "\n", +); +for (const r of rows) { + const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; + console.log(`${tag} ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); +} + +console.log("\n--- retrieval quality ---"); +for (const k of K) console.log(`recall@${k}: ${(recall[k] * 100).toFixed(0)}%`); +console.log(`MRR: ${mrr.toFixed(3)}`); + +if (broken.length) { + console.log("\n--- BROKEN eval cases (fix ground truth) ---"); + for (const b of broken) console.log(` "${b.q}" -> not in feed: ${b.missing.map(short).join(", ")}`); +} From c5f4ebe296ea8447ca48ee06c01366526277a3e7 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:45:48 +0100 Subject: [PATCH 06/25] DOC-6809 Add stemming + page-type weighting; recall@5 59%->86% MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two principled ranking changes, measured against the eval baseline: - Porter stemmer (src/stem.ts) applied to both index and query tokens so word forms conflate (append/appends, prepend/prepending, expires/expire). Stopwords are filtered on raw tokens before stemming. - Page-type weighting on the final score: demote release-notes/REST-API (x0.5) and operator (x0.7) pages, modestly boost /commands/* (x1.5) — targeting the observed failures where operator/concept pages outranked command pages. Eval (22 command-lookup cases): recall@1 32->50, @3 45->68, @5 59->86, @10 73->95, MRR 0.42->0.65; misses 6->1. Caveat recorded in README/SPEC: the eval is command-heavy so the /commands/* boost partly flatters it — concept/ how-to cases still needed before calling lexical sufficient generally, and the residual miss (del/unlink beaten by flushdb for "remove a key") is a semantic gap that motivates vector search. Smoke still green (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 20 ++-- build/docs-mcp-server/node/README.md | 33 +++--- build/docs-mcp-server/node/src/search.ts | 41 +++++-- build/docs-mcp-server/node/src/stem.ts | 136 +++++++++++++++++++++++ 4 files changed, 198 insertions(+), 32 deletions(-) create mode 100644 build/docs-mcp-server/node/src/stem.ts diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 3bbd70272d..97d26837b9 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -200,15 +200,17 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / RedisVL, and is the retrieval gain worth the added infra given how well-structured the corpus already is? -- **Ranking quality (found in v0 prototype):** untuned lexical BM25 — even - with stopword removal and title/summary/slug field boosts — mis-ranks - natural-language queries. Two concrete failure modes observed on the live - feed: (1) **no stemming**, so "append" ≠ "appends" and `XADD` won't surface - for "append an entry to a stream"; (2) **canonical command pages compete with - release notes, operator `custom-resources` pages, and client-library - overviews** that repeat the same terms. Fix needs an analyzer - (stemming/lemmatization), possibly a page-type/canonical boost, and/or the - vector-search path. This is the strongest argument for §6's v2 upgrade. +- **Ranking quality (measured via the eval harness):** lexical BM25 with + Porter stemming, stopword removal, title/summary/slug field boosts, and + page-type weighting (demote release-notes/REST-API/operator, modestly boost + `/commands/*`) gets **recall@5 86% / MRR 0.65** on 22 command-lookup cases — + up from a **59% / 0.42** un-stemmed, un-weighted baseline. Two caveats: (1) + the eval is command-heavy so the `/commands/*` boost partly flatters it — + **add concept/how-to eval cases** before concluding lexical is sufficient + generally; (2) the residual misses are pure semantic gaps ("remove a key" → + `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting + **weakened but did not eliminate** the §6 vector-search case — the eval now + lets that call be made on numbers. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index f85e4977e7..2cd83d5dc6 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -71,23 +71,28 @@ check that flags any expected url missing from the feed. The feed is read from `DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; `curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). -**Baseline (lexical v0):** recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42 — i.e. -lexical retrieval is **not** good enough alone (canonical command pages lose to -sibling commands, operator, and concept pages; no stemming). This is the -measured case for the ranking / vector-search work in SPEC §6/§10. Use it to -compare any ranking change against the baseline rather than tuning blind. +**Results (22 command-lookup cases):** + +| | recall@1 | recall@3 | recall@5 | recall@10 | MRR | +|---|---|---|---|---|---| +| lexical baseline | 32% | 45% | 59% | 73% | 0.42 | +| + Porter stemming + page-type weighting | 50% | 68% | 86% | 95% | 0.65 | + +Stemming (append↔appends) and demoting secondary pages (release-notes / REST-API / +operator) while modestly boosting `/commands/*` lifted recall@5 from 59% → 86%. +**Caveat:** this eval is command-heavy, so the `/commands/*` boost partly flatters +it — concept/how-to query cases are not yet covered and should be added before +concluding lexical is sufficient generally. The remaining case (`del`/`unlink` +for "remove a key", beaten by `flushdb`) is a genuine lexical limitation and is +the kind of gap vector search would close (SPEC §6/§10). ## Known limitations (v0) -- **Ranking is untuned lexical search.** Good on distinctive terms - (`publish`, `incrby`, `pexpire`), but weaker where: - - there is **no stemmer** — a query for "append" won't match a summary that - says "appends", so `XADD` is hard to surface from natural language; - - **canonical command pages** compete with release notes, operator - (custom-resources) pages, and client-library overviews that repeat the same - terms. - Fixing this properly means an analyzer (stemming/lemmatization) and/or the - vector-search upgrade tracked in SPEC §6/§10. +- **Ranking is lexical (BM25 + Porter stemming + field/page-type weighting).** + Now recall@5 86% on command lookups (see eval above), but still lexical: it + can't bridge pure semantic gaps (e.g. "remove a key" → `flushdb` over `del`), + and concept/how-to queries are unmeasured. Closing the remainder is the case + for vector search (SPEC §6/§10). - **No version filtering.** The `version` param was removed until a committed URL/version model exists (the feed is single-version today); see SPEC §7. - Only `search_docs` + `get_page`. `get_examples`, `get_section`, diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index 9e54cd0efb..d5d913285d 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -1,4 +1,5 @@ import type { Page } from "./types.js"; +import { stem } from "./stem.js"; // Self-contained BM25 lexical index. No external search dependency: at // ~4,100 docs the whole index builds in-memory in well under a second, which @@ -7,6 +8,20 @@ import type { Page } from "./types.js"; const K1 = 1.5; const B = 0.75; +// Page-type weighting applied to the final score. Command reference pages are +// the canonical answer for command-shaped queries; release-notes / REST-API / +// operator pages are secondary and were observed outranking primary docs +// (e.g. "remove a key" -> operate/.../remove-node). Multipliers, not filters. +// NOTE: the retrieval eval is command-heavy, so keep the command boost modest +// to avoid tuning to the eval — the demotions are the more general fix. +function pageWeight(url: string): number { + const u = url.toLowerCase(); + if (u.includes("/release-notes") || u.includes("/rest-api/")) return 0.5; + if (u.includes("/operate/")) return 0.7; + if (u.includes("/commands/")) return 1.5; + return 1; +} + // Field boosts (added on top of the body BM25 score, weighted by term idf). // A query term appearing in the title/slug/summary is a strong signal that the // page is *about* that term — this lifts canonical command pages (whose summary @@ -28,6 +43,12 @@ function tokenize(text: string): string[] { return text.toLowerCase().match(/[a-z0-9]+/g) ?? []; } +/** Tokenize + stem. Used for everything indexed and for query terms, so word + * forms conflate. Stopwords are filtered on RAW tokens before this (see search). */ +function analyze(text: string): string[] { + return tokenize(text).map(stem); +} + function normalizeUrl(u: string): string { return u.trim().toLowerCase().replace(/\/+$/, ""); } @@ -45,7 +66,7 @@ function searchableText(p: Page): string { function matchingSections(p: Page, qterms: Set): string[] { const out: string[] = []; for (const s of p.sections ?? []) { - const toks = new Set(tokenize(`${s.title ?? ""} ${s.text ?? ""}`)); + const toks = new Set(analyze(`${s.title ?? ""} ${s.text ?? ""}`)); for (const t of qterms) { if (toks.has(t)) { out.push(s.id); @@ -100,7 +121,7 @@ export class DocsIndex { else this.byId.set(p.id, [p]); if (p.url) this.byUrl.set(normalizeUrl(p.url), p); - const tokens = tokenize(searchableText(p)); + const tokens = analyze(searchableText(p)); if (tokens.length === 0) continue; const tf = new Map(); for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); @@ -109,9 +130,9 @@ export class DocsIndex { page: p, tf, len: tokens.length, - titleTok: new Set(tokenize(p.title ?? "")), - slugTok: new Set(tokenize(p.id ?? "")), - summaryTok: new Set(tokenize(p.summary ?? "")), + titleTok: new Set(analyze(p.title ?? "")), + slugTok: new Set(analyze(p.id ?? "")), + summaryTok: new Set(analyze(p.summary ?? "")), }); totalLen += tokens.length; } @@ -150,10 +171,11 @@ export class DocsIndex { } search(query: string, opts: SearchOptions = {}): SearchHit[] { - let qterms = [...new Set(tokenize(query))].filter((t) => !STOPWORDS.has(t)); - // If the query was *all* stopwords, fall back to using them rather than - // returning nothing. - if (qterms.length === 0) qterms = [...new Set(tokenize(query))]; + // Filter stopwords on RAW tokens (before stemming), then stem + dedupe. + const raw = [...new Set(tokenize(query))]; + let kept = raw.filter((t) => !STOPWORDS.has(t)); + if (kept.length === 0) kept = raw; // query was all stopwords + const qterms = [...new Set(kept.map(stem))]; if (qterms.length === 0) return []; const idf = new Map(); @@ -181,6 +203,7 @@ export class DocsIndex { if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; } if (score > 0) { + score *= pageWeight(d.page.url); hits.push({ id: d.page.id, title: d.page.title, diff --git a/build/docs-mcp-server/node/src/stem.ts b/build/docs-mcp-server/node/src/stem.ts new file mode 100644 index 0000000000..e94296f3b4 --- /dev/null +++ b/build/docs-mcp-server/node/src/stem.ts @@ -0,0 +1,136 @@ +// Compact Porter stemmer (classic algorithm). Applied to BOTH index and query +// tokens so word-form variants conflate (append/appends, prepend/prepending, +// expire/expires, queries/query). Consistency matters more than linguistic +// perfection here — the retrieval eval validates the net effect. + +const step2list: Record = { + ational: "ate", tional: "tion", enci: "ence", anci: "ance", izer: "ize", + bli: "ble", alli: "al", entli: "ent", eli: "e", ousli: "ous", + ization: "ize", ation: "ate", ator: "ate", alism: "al", iveness: "ive", + fulness: "ful", ousness: "ous", aliti: "al", iviti: "ive", biliti: "ble", + logi: "log", +}; +const step3list: Record = { + icate: "ic", ative: "", alize: "al", iciti: "ic", ical: "ic", ful: "", ness: "", +}; + +const c = "[^aeiou]"; +const v = "[aeiouy]"; +const C = c + "[^aeiouy]*"; +const V = v + "[aeiou]*"; +const mgr0 = "^(" + C + ")?" + V + C; +const meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; +const mgr1 = "^(" + C + ")?" + V + C + V + C; +const s_v = "^(" + C + ")?" + v; + +export function stem(w: string): string { + if (w.length < 3) return w; + + let stemmed: string; + let suffix: string; + let re: RegExp; + let re2: RegExp; + let re3: RegExp; + let re4: RegExp; + + const firstch = w.substr(0, 1); + if (firstch === "y") w = firstch.toUpperCase() + w.substr(1); + + // Step 1a + re = /^(.+?)(ss|i)es$/; + re2 = /^(.+?)([^s])s$/; + if (re.test(w)) w = w.replace(re, "$1$2"); + else if (re2.test(w)) w = w.replace(re2, "$1$2"); + + // Step 1b + re = /^(.+?)eed$/; + re2 = /^(.+?)(ed|ing)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + re = new RegExp(mgr0); + if (re.test(fp[1])) { + re = /.$/; + w = w.replace(re, ""); + } + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1]; + re2 = new RegExp(s_v); + if (re2.test(stemmed)) { + w = stemmed; + re2 = /(at|bl|iz)$/; + re3 = new RegExp("([^aeiouylsz])\\1$"); + re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re2.test(w)) w = w + "e"; + else if (re3.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } else if (re4.test(w)) w = w + "e"; + } + } + + // Step 1c + re = /^(.+?)y$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(s_v); + if (re.test(stemmed)) w = stemmed + "i"; + } + + // Step 2 + re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step2list[suffix]; + } + + // Step 3 + re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step3list[suffix]; + } + + // Step 4 + re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; + re2 = /^(.+?)(s|t)(ion)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + if (re.test(stemmed)) w = stemmed; + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1] + fp[2]; + re2 = new RegExp(mgr1); + if (re2.test(stemmed)) w = stemmed; + } + + // Step 5a + re = /^(.+?)e$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + re2 = new RegExp(meq1); + re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re.test(stemmed) || (re2.test(stemmed) && !re3.test(stemmed))) w = stemmed; + } + + // Step 5b + re = /ll$/; + re2 = new RegExp(mgr1); + if (re.test(w) && re2.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } + + return w.toLowerCase(); +} From eb0e7ca72034021613a55a94f6f2e43b01ee37d8 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 10:47:12 +0100 Subject: [PATCH 07/25] DOC-6809 Make exact url authoritative in get_page (Bugbot round 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bugbot High: collectCandidates always merged id matches even when url already resolved to an exact page, so get_page(url=, id=) — the natural flow when an agent echoes both fields from a search hit — returned an ambiguous error instead of the page. The consolidation's "always converge" rule over-corrected: an exact url is unique and should win. Fix: an exact url short-circuits to that page, EXCEPT when a supplied id points somewhere else entirely (idPages don't include the exact page) — still a genuine conflict, so still reported. This reconciles the Bugbot finding with Codex's earlier conflict-detection ask: exact-url + its-own-id resolves; exact-url + a different id conflicts. Smoke covers both (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/src/smoke.ts | 8 +++ .../node/src/tools/get-page.ts | 69 ++++++++++++------- 2 files changed, 52 insertions(+), 25 deletions(-) diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index a44475d6cd..fedc6c450d 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -38,6 +38,14 @@ check("roles filter returns only 'parameters'", (xadd.sections ?? []).every((s: const exact = getPage(index, { url: "https://redis.io/docs/latest/operate/redisinsight/install/" }) as any; check("get_page(exact url) resolves the right page", exact.title === "Install Redis Insight"); +// exact url is authoritative even when a non-unique id is passed alongside it +// (Bugbot round-3 High): search hits carry both id + url, and id "install" is ambiguous. +const exactPlusId = getPage(index, { + url: "https://redis.io/docs/latest/operate/redisinsight/install/", + id: "install", +}) as any; +check("exact url + non-unique id resolves (not ambiguous)", exactPlusId.title === "Install Redis Insight"); + const suffixUnique = getPage(index, { url: "/commands/xadd/" }) as any; check("get_page(unambiguous partial url) resolves", suffixUnique.id === "commands/xadd"); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index cbe9bd7f25..d3326423b8 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -27,14 +27,9 @@ function collectCandidates(index: DocsIndex, input: GetPageInput): Page[] { if (p) byUrl.set(p.url, p); }; - if (input.url) { - const exact = index.getByUrl(input.url); - if (exact) add(exact); - else index.matchByUrlSuffix(input.url).forEach(add); - } - if (input.id) { - index.getPagesById(input.id).forEach(add); - } + // A boundary-suffix url (only reached when there was no exact match) plus id. + if (input.url) index.matchByUrlSuffix(input.url).forEach(add); + if (input.id) index.getPagesById(input.id).forEach(add); return [...byUrl.values()]; } @@ -45,29 +40,19 @@ function describeHandles(input: GetPageInput): string { return parts.join(" and "); } -/** Fetch one page, optionally filtered to sections with the given roles. */ -export function getPage(index: DocsIndex, input: GetPageInput) { - const candidates = collectCandidates(index, input); - - if (candidates.length === 0) { - return { error: `Page not found for ${describeHandles(input)}.` }; - } - if (candidates.length > 1) { - // Ambiguous (a non-unique id / boundary suffix) or conflicting (url and id - // point at different pages) — either way, make the caller pick a url. - return { - error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, - candidates: candidates.map((p) => ({ title: p.title, url: p.url })), - }; - } +function ambiguous(input: GetPageInput, candidates: Page[]) { + return { + error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, + candidates: candidates.map((p) => ({ title: p.title, url: p.url })), + }; +} - const page = candidates[0]; +function render(page: Page, input: GetPageInput) { let sections = page.sections ?? []; if (input.roles && input.roles.length) { const want = new Set(input.roles.map((r) => r.toLowerCase())); sections = sections.filter((s) => want.has((s.role ?? "").toLowerCase())); } - return { id: page.id, title: page.title, @@ -78,3 +63,37 @@ export function getPage(index: DocsIndex, input: GetPageInput) { sections, }; } + +/** Fetch one page, optionally filtered to sections with the given roles. */ +export function getPage(index: DocsIndex, input: GetPageInput) { + // An exact url is unique and authoritative. Return it directly rather than + // diluting it with a (possibly non-unique) id supplied alongside it — UNLESS + // the id points somewhere else entirely, which is a genuine conflict worth + // surfacing. A search hit's id is that page's own id, so the common + // exact-url + its-own-id case resolves cleanly. + if (input.url) { + const exact = index.getByUrl(input.url); + if (exact) { + if (input.id) { + const idPages = index.getPagesById(input.id); + if (idPages.length > 0 && !idPages.some((p) => p.url === exact.url)) { + const byUrl = new Map(); + [exact, ...idPages].forEach((p) => byUrl.set(p.url, p)); + return ambiguous(input, [...byUrl.values()]); + } + } + return render(exact, input); + } + } + + // Otherwise converge across the remaining handles (boundary-suffix url + id) + // and resolve only when they point at exactly one page. + const candidates = collectCandidates(index, input); + if (candidates.length === 0) { + return { error: `Page not found for ${describeHandles(input)}.` }; + } + if (candidates.length > 1) { + return ambiguous(input, candidates); + } + return render(candidates[0], input); +} From 4b94459261fa1f33c7e0488e4aa83f833dfd3086 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 10:54:44 +0100 Subject: [PATCH 08/25] DOC-6809 Add concept/how-to eval cases; reveals command boost is overfit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit De-bias the retrieval eval (was 22 command-only cases): tag each case command/concept and add 13 concept/how-to cases with feed-verified ground truth (incl. pages under /operate/, to test the demotion). Runner now reports recall per kind. Result under the shipped weighting: command recall@5 86% / MRR 0.65 but **concept recall@5 46% / MRR 0.29** — concept retrieval is ~half as good. The /commands/* x1.5 boost is the cause: it ranks command pages above the canonical concept page when both compete (persistence -> bgrewriteaof, replication -> cluster-replicate, keyspace-notifications -> expire), and the blanket /operate/ demotion drags down legitimate concept pages. Ablation (neutralise command boost, demote only rest-api/release-notes/references): concept @5 46->62, MRR .29->.45; command @5 86->73. Command-optimised vs balanced is a workload call — left as an open decision in SPEC, weighting unchanged for now. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 14 +++++ build/docs-mcp-server/node/README.md | 33 +++++++---- .../docs-mcp-server/node/test/eval/cases.json | 58 ++++++++++++------- build/docs-mcp-server/node/test/eval/run.mjs | 34 +++++++---- 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 97d26837b9..c42f790dfd 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -211,6 +211,20 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting **weakened but did not eliminate** the §6 vector-search case — the eval now lets that call be made on numbers. +- **Command boost is command-overfit (found after adding concept cases).** With + 13 concept/how-to cases added, per-kind numbers diverge sharply: command + recall@5 86% / MRR 0.65 vs **concept recall@5 46% / MRR 0.29**. The + `/commands/*` ×1.5 boost is the cause — it ranks command pages above the + canonical concept page when both compete ("configure persistence" → + `bgrewriteaof`; "set up replication" → `cluster-replicate`; "keyspace + notifications" → `expire`), and the blanket `/operate/` demotion drags down + legitimate concept pages (persistence, replication). Ablation (neutralise the + command boost, demote only REST-API/release-notes/references): concept @5 + 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Open decision:** command- + optimised (current) vs balanced weighting — a workload-mix call. A modest + command boost (×1.2) helps command none vs neutral, so the command signal + should really come from better lexical handling or vectors, not the thumb on + the scale. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 2cd83d5dc6..28b6c1cd75 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -71,20 +71,29 @@ check that flags any expected url missing from the feed. The feed is read from `DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; `curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). -**Results (22 command-lookup cases):** +Cases are tagged `command` (22) or `concept` (13, how-to / concept pages) so the +runner reports recall per kind — because command and concept queries behave very +differently. -| | recall@1 | recall@3 | recall@5 | recall@10 | MRR | +**Current results (shipped config: Porter stemming + field boosts + page-type +weighting with `/commands/*` ×1.5, `/operate/` ×0.7, REST-API/release-notes ×0.5):** + +| group | recall@1 | @3 | @5 | @10 | MRR | |---|---|---|---|---|---| -| lexical baseline | 32% | 45% | 59% | 73% | 0.42 | -| + Porter stemming + page-type weighting | 50% | 68% | 86% | 95% | 0.65 | - -Stemming (append↔appends) and demoting secondary pages (release-notes / REST-API / -operator) while modestly boosting `/commands/*` lifted recall@5 from 59% → 86%. -**Caveat:** this eval is command-heavy, so the `/commands/*` boost partly flatters -it — concept/how-to query cases are not yet covered and should be added before -concluding lexical is sufficient generally. The remaining case (`del`/`unlink` -for "remove a key", beaten by `flushdb`) is a genuine lexical limitation and is -the kind of gap vector search would close (SPEC §6/§10). +| command (22) | 50% | 68% | 86% | 95% | 0.65 | +| concept (13) | 15% | 31% | 46% | 62% | 0.29 | +| overall (35) | 37% | 54% | 71% | 83% | 0.52 | + +(Un-stemmed, un-weighted lexical baseline on the command set was 59%@5 / 0.42.) + +**Finding:** command retrieval is good, but **concept/how-to retrieval is about +half as good** — and the `/commands/*` boost is the cause: it lifts command +queries but pushes command pages *above* the canonical concept page when both +compete (e.g. "configure persistence" → `bgrewriteaof`; "set up replication" → +`cluster-replicate`). An ablation neutralising the command boost moves concept +@5 46%→62% and MRR 0.29→0.45, at the cost of command @5 86%→73%. That trade-off +(command-optimised vs balanced) is a product call; the residual misses are pure +semantic gaps that motivate vector search (SPEC §6/§10). ## Known limitations (v0) diff --git a/build/docs-mcp-server/node/test/eval/cases.json b/build/docs-mcp-server/node/test/eval/cases.json index 382331819e..3e24aa50ce 100644 --- a/build/docs-mcp-server/node/test/eval/cases.json +++ b/build/docs-mcp-server/node/test/eval/cases.json @@ -1,24 +1,38 @@ [ - { "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, - { "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, - { "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, - { "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, - { "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, - { "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, - { "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, - { "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, - { "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, - { "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, - { "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, - { "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, - { "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, - { "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, - { "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, - { "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, - { "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, - { "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, - { "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, - { "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, - { "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, - { "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] } + { "kind": "command", "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, + { "kind": "command", "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, + { "kind": "command", "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, + { "kind": "command", "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, + { "kind": "command", "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, + { "kind": "command", "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, + { "kind": "command", "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, + { "kind": "command", "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, + { "kind": "command", "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, + { "kind": "command", "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, + { "kind": "command", "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, + { "kind": "command", "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, + { "kind": "command", "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, + { "kind": "command", "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, + { "kind": "command", "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, + { "kind": "command", "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, + { "kind": "command", "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, + { "kind": "command", "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, + { "kind": "command", "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, + { "kind": "command", "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, + { "kind": "command", "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, + { "kind": "command", "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] }, + + { "kind": "concept", "q": "connect to Redis from a Python application", "expected": ["https://redis.io/docs/latest/develop/clients/redis-py/connect/"] }, + { "kind": "concept", "q": "connect to Redis using the Jedis Java client", "expected": ["https://redis.io/docs/latest/develop/clients/jedis/connect/"] }, + { "kind": "concept", "q": "what are the differences between the Redis data types", "expected": ["https://redis.io/docs/latest/develop/data-types/compare-data-types/", "https://redis.io/docs/latest/develop/data-types/"] }, + { "kind": "concept", "q": "how do Redis transactions work", "expected": ["https://redis.io/docs/latest/develop/using-commands/transactions/"] }, + { "kind": "concept", "q": "send several commands together to reduce round trips", "expected": ["https://redis.io/docs/latest/develop/using-commands/pipelining/"] }, + { "kind": "concept", "q": "configure persistence with RDB snapshots and the append-only file", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/"] }, + { "kind": "concept", "q": "set up replication between a primary and its replicas", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/replication/"] }, + { "kind": "concept", "q": "run a vector similarity search over embeddings", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/query/vector-search/", "https://redis.io/docs/latest/develop/ai/search-and-query/"] }, + { "kind": "concept", "q": "get notified when keys are changed or expire", "expected": ["https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/"] }, + { "kind": "concept", "q": "cache data on the client side to reduce load", "expected": ["https://redis.io/docs/latest/develop/clients/client-side-caching/", "https://redis.io/docs/latest/develop/reference/client-side-caching/"] }, + { "kind": "concept", "q": "work with the string data type", "expected": ["https://redis.io/docs/latest/develop/data-types/strings/"] }, + { "kind": "concept", "q": "use hashes to store object-like records", "expected": ["https://redis.io/docs/latest/develop/data-types/hashes/"] }, + { "kind": "concept", "q": "full-text search and secondary indexing over Redis data", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/"] } ] diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs index c73964d8ca..6c8da79419 100644 --- a/build/docs-mcp-server/node/test/eval/run.mjs +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -43,28 +43,40 @@ for (const c of cases) { break; } } - rows.push({ q: c.q, rank, top: results[0] }); + rows.push({ kind: c.kind ?? "command", q: c.q, rank, top: results[0] }); } -const scored = rows.length; -const recall = Object.fromEntries( - K.map((k) => [k, rows.filter((r) => r.rank && r.rank <= k).length / scored]), -); -const mrr = rows.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / scored; +function metrics(set) { + const n = set.length || 1; + const recall = Object.fromEntries( + K.map((k) => [k, set.filter((r) => r.rank && r.rank <= k).length / n]), + ); + const mrr = set.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / n; + return { recall, mrr }; +} console.log( - `Feed: ${pages.length} pages | cases scored: ${scored}` + + `Feed: ${pages.length} pages | cases scored: ${rows.length}` + (broken.length ? ` | ${broken.length} BROKEN (expected url not in feed)` : "") + "\n", ); for (const r of rows) { const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; - console.log(`${tag} ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); + console.log(`${tag} [${r.kind.slice(0, 4)}] ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); } -console.log("\n--- retrieval quality ---"); -for (const k of K) console.log(`recall@${k}: ${(recall[k] * 100).toFixed(0)}%`); -console.log(`MRR: ${mrr.toFixed(3)}`); +const groups = [ + ["overall", rows], + ["command", rows.filter((r) => r.kind === "command")], + ["concept", rows.filter((r) => r.kind === "concept")], +]; +console.log("\n--- retrieval quality (recall@1 / @3 / @5 / @10 | MRR) ---"); +for (const [label, set] of groups) { + if (!set.length) continue; + const m = metrics(set); + const cells = K.map((k) => `${(m.recall[k] * 100).toFixed(0)}%`.padStart(4)).join(" / "); + console.log(`${label.padEnd(8)} (n=${String(set.length).padStart(2)}): ${cells} | ${m.mrr.toFixed(3)}`); +} if (broken.length) { console.log("\n--- BROKEN eval cases (fix ground truth) ---"); From cf2f90011207e6e510260bbf8775ce2bf0f18ed8 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 11:00:20 +0100 Subject: [PATCH 09/25] DOC-6809 Adopt balanced page-type weighting (drop command-boost overfit) Per the concept-case eval finding, switch page weighting from command-optimised to balanced: demote only release-notes/REST-API/references (x0.5); drop the /commands/* x1.5 boost and the blanket /operate/ x0.7 demotion. This trades command recall@5 86->73 for concept 46->62 and the best overall MRR (0.53), and stops burying legitimate /operate/ concept pages (persistence, replication). Command ranking should be lifted by better signal (vectors), not the boost. Eval + smoke green; SPEC decision marked resolved. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 26 +++++++++++------------- build/docs-mcp-server/node/README.md | 26 ++++++++++++------------ build/docs-mcp-server/node/src/search.ts | 24 ++++++++++++++-------- 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index c42f790dfd..fd78393864 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -202,15 +202,13 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. well-structured the corpus already is? - **Ranking quality (measured via the eval harness):** lexical BM25 with Porter stemming, stopword removal, title/summary/slug field boosts, and - page-type weighting (demote release-notes/REST-API/operator, modestly boost - `/commands/*`) gets **recall@5 86% / MRR 0.65** on 22 command-lookup cases — - up from a **59% / 0.42** un-stemmed, un-weighted baseline. Two caveats: (1) - the eval is command-heavy so the `/commands/*` boost partly flatters it — - **add concept/how-to eval cases** before concluding lexical is sufficient - generally; (2) the residual misses are pure semantic gaps ("remove a key" → - `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting - **weakened but did not eliminate** the §6 vector-search case — the eval now - lets that call be made on numbers. + balanced page-type weighting (demote release-notes/REST-API/references only) + gets, on the 35-case eval (22 command + 13 concept), **command recall@5 73% / + concept 62% / overall 69%, MRR 0.53** — up from a **59% / 0.42** un-stemmed, + un-weighted lexical baseline on the command set. Residual misses are pure + semantic gaps ("remove a key" → `flushdb` beats `del`) that only vector search + closes. Net: stemming+weighting **weakened but did not eliminate** the §6 + vector-search case — the eval now lets that call be made on numbers. - **Command boost is command-overfit (found after adding concept cases).** With 13 concept/how-to cases added, per-kind numbers diverge sharply: command recall@5 86% / MRR 0.65 vs **concept recall@5 46% / MRR 0.29**. The @@ -220,11 +218,11 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. notifications" → `expire`), and the blanket `/operate/` demotion drags down legitimate concept pages (persistence, replication). Ablation (neutralise the command boost, demote only REST-API/release-notes/references): concept @5 - 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Open decision:** command- - optimised (current) vs balanced weighting — a workload-mix call. A modest - command boost (×1.2) helps command none vs neutral, so the command signal - should really come from better lexical handling or vectors, not the thumb on - the scale. + 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Resolved: adopted the balanced + weighting** (no command boost, no blanket `/operate/` demotion). A modest + command boost (×1.2) helped command none vs neutral, so lifting command + ranking should come from better lexical handling or vectors, not a bigger + thumb on the scale — the next lever, measured against this eval. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 28b6c1cd75..b95e0f761a 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -75,25 +75,25 @@ Cases are tagged `command` (22) or `concept` (13, how-to / concept pages) so the runner reports recall per kind — because command and concept queries behave very differently. -**Current results (shipped config: Porter stemming + field boosts + page-type -weighting with `/commands/*` ×1.5, `/operate/` ×0.7, REST-API/release-notes ×0.5):** +**Current results (shipped config: Porter stemming + field boosts + *balanced* +page-type weighting — demote REST-API/release-notes/references ×0.5, no command +boost, no blanket `/operate/` demotion):** | group | recall@1 | @3 | @5 | @10 | MRR | |---|---|---|---|---|---| -| command (22) | 50% | 68% | 86% | 95% | 0.65 | -| concept (13) | 15% | 31% | 46% | 62% | 0.29 | -| overall (35) | 37% | 54% | 71% | 83% | 0.52 | +| command (22) | 41% | 64% | 73% | 95% | 0.57 | +| concept (13) | 31% | 54% | 62% | 77% | 0.45 | +| overall (35) | 37% | 60% | 69% | 89% | 0.53 | (Un-stemmed, un-weighted lexical baseline on the command set was 59%@5 / 0.42.) -**Finding:** command retrieval is good, but **concept/how-to retrieval is about -half as good** — and the `/commands/*` boost is the cause: it lifts command -queries but pushes command pages *above* the canonical concept page when both -compete (e.g. "configure persistence" → `bgrewriteaof`; "set up replication" → -`cluster-replicate`). An ablation neutralising the command boost moves concept -@5 46%→62% and MRR 0.29→0.45, at the cost of command @5 86%→73%. That trade-off -(command-optimised vs balanced) is a product call; the residual misses are pure -semantic gaps that motivate vector search (SPEC §6/§10). +**Why balanced:** an earlier command-optimised config (`/commands/*` ×1.5, +`/operate/` ×0.7) scored command @5 86% but only concept @5 46% — the boost +ranked command pages above the canonical concept page when both competed +("configure persistence" → `bgrewriteaof`). We chose the balanced weighting: +concept @5 46%→62% for command @5 86%→73% (SPEC §10). The residual misses are +pure semantic gaps that motivate vector search (SPEC §6/§10) — the right lever +for lifting both, rather than a bigger thumb on the scale. ## Known limitations (v0) diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index d5d913285d..c0ad9ae710 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -8,17 +8,23 @@ import { stem } from "./stem.js"; const K1 = 1.5; const B = 0.75; -// Page-type weighting applied to the final score. Command reference pages are -// the canonical answer for command-shaped queries; release-notes / REST-API / -// operator pages are secondary and were observed outranking primary docs -// (e.g. "remove a key" -> operate/.../remove-node). Multipliers, not filters. -// NOTE: the retrieval eval is command-heavy, so keep the command boost modest -// to avoid tuning to the eval — the demotions are the more general fix. +// Page-type weighting applied to the final score. Demote clearly-secondary +// reference material (release-notes / REST-API / other references) that was +// observed outranking primary docs. Multipliers, not filters. +// +// Deliberately balanced, NOT command-optimised: an earlier config boosted +// /commands/* (x1.5) and demoted all of /operate/ (x0.7), which lifted command +// queries but ranked command pages above the canonical concept page when both +// competed (persistence -> bgrewriteaof) and buried legitimate /operate/ +// concept pages. The eval showed that cost concept recall@5 ~16pts for ~13pts +// of command gain, so we chose the balanced weighting (SPEC §10). Lifting +// command ranking further should come from better signal (vectors), not a +// bigger thumb on the scale. function pageWeight(url: string): number { const u = url.toLowerCase(); - if (u.includes("/release-notes") || u.includes("/rest-api/")) return 0.5; - if (u.includes("/operate/")) return 0.7; - if (u.includes("/commands/")) return 1.5; + if (u.includes("/release-notes") || u.includes("/rest-api/") || u.includes("/references/")) { + return 0.5; + } return 1; } From 25c1d25b0f7c74847df56b14b9224ff64beaa3d0 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 23 Jul 2026 13:50:03 +0100 Subject: [PATCH 10/25] DOC-6809 Add measure-first vector-search experiment; hybrid beats lexical MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A no-Redis experiment (vector-eval/) to decide whether vector search is worth the hosted infra before building it: embed the corpus with bge-small (fastembed/ONNX), rank in numpy, and score vector + hybrid (RRF fused with the production lexical ranker, dumped via dump-lexical.mjs) against the same 35-case eval. Verdict: hybrid clearly wins — overall recall@5 69->86, MRR .53->.66; command @5 73->91 — so the RediSearch/RedisVL build is justified, and it should be a HYBRID ranker (vector alone only modestly beats lexical). Concept queries stay weak (vector ties lexical; hybrid lifts @5 62->77 but not top-1/3), most likely an artefact of the coarse page-level chunking used here. A first section-level attempt (21.5k chunks) stalled on this machine: embedding runs only ~6 chunks/s (unoptimised CPU ONNX), so it ran >1h and the CPU collapsed with no checkpoint written. The committed version is the slim, instrumented rebuild: one page-level chunk per page (~2.5k), batched embedding with progress logging, end-of-run cache. Section-level chunking over the feed's sections[] is the next experiment and needs proper batching/caching to be runnable here. Learned: hybrid (lexical+vector RRF) clearly beats either alone; vector alone only modest; concept stays weak on coarse page-level chunks Directive: build the hosted path as HYBRID not pure vector; use the feed sections[] for section-level chunking to attack the concept gap; don't read the ~6 chunks/s rate as a production latency signal (unoptimised local ONNX) Gaps: section-level chunking and real production embedding latency not yet measured; concept n=13 so those numbers are directional Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 14 +- .../node/test/eval/dump-lexical.mjs | 26 +++ build/docs-mcp-server/vector-eval/.gitignore | 5 + .../vector-eval/eval_vector.py | 177 ++++++++++++++++++ .../vector-eval/requirements.txt | 2 + 5 files changed, 220 insertions(+), 4 deletions(-) create mode 100644 build/docs-mcp-server/node/test/eval/dump-lexical.mjs create mode 100644 build/docs-mcp-server/vector-eval/.gitignore create mode 100644 build/docs-mcp-server/vector-eval/eval_vector.py create mode 100644 build/docs-mcp-server/vector-eval/requirements.txt diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index fd78393864..f78cff27a5 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -196,10 +196,16 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. ## 10. Open questions - **Search backend:** resolved for v1 — lexical (BM25 over NDJSON), no - datastore, runs in both stdio and hosted modes (see §6). Open: do we add - vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / - RedisVL, and is the retrieval gain worth the added infra given how - well-structured the corpus already is? + datastore, runs in both stdio and hosted modes (see §6). **v2 vector: measured + and justified.** A measure-first experiment (`vector-eval/`, no Redis: + bge-small embeddings + numpy cosine, scored on the same 35-case eval) showed + **hybrid lexical+vector (RRF) clearly beats lexical alone** — overall recall@5 + 69%→86%, MRR 0.53→0.66; command @5 73%→91%. Vector *alone* only modestly beats + lexical, so fusion is the win. → Build the hosted RediSearch/RedisVL path with + a **hybrid** ranker (not pure vector). Caveat: concept queries stay weak + (vector ties lexical at @5; hybrid lifts @5 62%→77% but not top-1/3) — likely + because the experiment used coarse *page-level* chunks; **section-level + chunking** (the feed's `sections[]`) is the next lever, being measured next. - **Ranking quality (measured via the eval harness):** lexical BM25 with Porter stemming, stopword removal, title/summary/slug field boosts, and balanced page-type weighting (demote release-notes/REST-API/references only) diff --git a/build/docs-mcp-server/node/test/eval/dump-lexical.mjs b/build/docs-mcp-server/node/test/eval/dump-lexical.mjs new file mode 100644 index 0000000000..a68eda18e4 --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/dump-lexical.mjs @@ -0,0 +1,26 @@ +// Dump the REAL lexical ranker's results for every eval case, so the Python +// vector experiment can compare against (and fuse with) the exact production +// ranking rather than a reimplementation. Reuses the built dist/. +// node test/eval/dump-lexical.mjs > test/eval/lexical.json +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { loadFeed } from "../../dist/feed.js"; +import { DocsIndex } from "../../dist/search.js"; +import { searchDocs } from "../../dist/tools/search-docs.js"; + +const TOPN = 50; +const norm = (u) => u.trim().toLowerCase().replace(/\/+$/, ""); +const feedSrc = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./docs.ndjson.gz", import.meta.url)); +const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", import.meta.url)), "utf8")); + +const index = new DocsIndex(await loadFeed(feedSrc)); + +const out = cases.map((c) => ({ + q: c.q, + kind: c.kind ?? "command", + expected: c.expected.map(norm), + lexical: searchDocs(index, { query: c.q, limit: TOPN }).results.map((r) => norm(r.url)), +})); + +process.stdout.write(JSON.stringify(out, null, 2) + "\n"); diff --git a/build/docs-mcp-server/vector-eval/.gitignore b/build/docs-mcp-server/vector-eval/.gitignore new file mode 100644 index 0000000000..e689b46390 --- /dev/null +++ b/build/docs-mcp-server/vector-eval/.gitignore @@ -0,0 +1,5 @@ +.venv/ +__pycache__/ +embeddings.npz +lexical.json +*.log diff --git a/build/docs-mcp-server/vector-eval/eval_vector.py b/build/docs-mcp-server/vector-eval/eval_vector.py new file mode 100644 index 0000000000..079d61cfdf --- /dev/null +++ b/build/docs-mcp-server/vector-eval/eval_vector.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +""" +Measure-first vector-search experiment for the docs MCP server (no Redis). + +Compares three rankers on the same 35-case eval used by the lexical harness: + - lexical : the production Node ranker's output (read from lexical.json) + - vector : bge-small-en-v1.5 embeddings, cosine, ranked in numpy + - hybrid : reciprocal-rank fusion of lexical + vector + +v2 (slim): ONE page-level chunk per page (title + summary + lead section text), +~2.5k chunks instead of 21.5k section chunks — fast enough to iterate. Embedding +is batched with live progress and the corpus cache is written at the end +(embeddings.npz) for instant re-runs. The model (bge-small-en-v1.5) is what we'd +run in production via RedisVL; loaded here through fastembed (ONNX, no torch). + +Usage: + pip install -r requirements.txt + node ../node/test/eval/dump-lexical.mjs > lexical.json + python eval_vector.py +""" +import gzip +import json +import os +import sys +import time + +import numpy as np + +HERE = os.path.dirname(os.path.abspath(__file__)) +FEED = os.path.join(HERE, "..", "node", "test", "eval", "docs.ndjson.gz") +LEXICAL = os.path.join(HERE, "lexical.json") +CACHE = os.path.join(HERE, "embeddings.npz") +MODEL = "BAAI/bge-small-en-v1.5" +BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: " +LEAD_CHARS = 1200 +KS = [1, 3, 5, 10] + + +def norm_url(u): + return u.strip().lower().rstrip("/") + + +def load_pages(): + pages = [] + with gzip.open(FEED, "rt", encoding="utf-8") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + o = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(o, dict) and o.get("url") and o.get("title"): + pages.append(o) + return pages + + +def build_chunks(pages): + """One page-level chunk per page: title + summary + lead section text.""" + texts, owners = [], [] + for p in pages: + title = p.get("title", "") or "" + summary = p.get("summary", "") or "" + secs = " ".join((s.get("text", "") or "") for s in (p.get("sections") or [])) + parts = [x for x in (title, summary, secs[:LEAD_CHARS]) if x] + texts.append(". ".join(parts).strip() or title or p["url"]) + owners.append(norm_url(p["url"])) + return texts, np.array(owners) + + +def _model(): + from fastembed import TextEmbedding + + return TextEmbedding(model_name=MODEL) + + +def embed(texts, is_query=False, label="chunks"): + payload = [BGE_QUERY_PREFIX + t for t in texts] if is_query else texts + model = _model() + vecs, t0, total = [], time.time(), len(payload) + for i, v in enumerate(model.embed(payload, batch_size=64)): + vecs.append(v) + n = i + 1 + if n % 256 == 0 or n == total: + rate = n / (time.time() - t0) + print(f" {n}/{total} {label} ({rate:.0f}/s)", flush=True) + arr = np.asarray(vecs, dtype=np.float32) + arr /= np.linalg.norm(arr, axis=1, keepdims=True) + 1e-12 + return arr + + +def get_corpus_embeddings(texts, owners): + if os.path.exists(CACHE): + d = np.load(CACHE, allow_pickle=True) + if len(d["owners"]) == len(owners): + print(f" (using cached embeddings: {len(owners)} chunks)") + return d["emb"], d["owners"] + emb = embed(texts, is_query=False) + np.savez(CACHE, emb=emb, owners=owners) + print(" cached embeddings.npz") + return emb, owners + + +def rank_pages(qvec, emb, owners, topn=50): + sims = emb @ qvec + best = {} + for i in np.argsort(-sims): + u = owners[i] + if u not in best: + best[u] = float(sims[i]) + if len(best) >= topn: + break + return [u for u, _ in sorted(best.items(), key=lambda kv: -kv[1])] + + +def rrf(lists, k=60, topn=50): + scores = {} + for lst in lists: + for rank, url in enumerate(lst): + scores[url] = scores.get(url, 0.0) + 1.0 / (k + rank + 1) + return [u for u, _ in sorted(scores.items(), key=lambda kv: -kv[1])][:topn] + + +def best_rank(ranking, expected): + for i, u in enumerate(ranking): + if u in expected: + return i + 1 + return None + + +def metrics(ranks): + n = len(ranks) or 1 + rec = {k: sum(1 for r in ranks if r and r <= k) / n for k in KS} + mrr = sum((1.0 / r) if r else 0.0 for r in ranks) / n + return rec, mrr + + +def main(): + if not os.path.exists(LEXICAL): + sys.exit("lexical.json missing — run: node ../node/test/eval/dump-lexical.mjs > lexical.json") + cases = json.load(open(LEXICAL)) + + print("Loading feed + building page-level chunks ...") + pages = load_pages() + texts, owners = build_chunks(pages) + print(f" {len(pages)} pages -> {len(texts)} chunks") + + print("Embedding corpus ...") + emb, owners = get_corpus_embeddings(texts, owners) + + print("Embedding queries ...") + qvecs = embed([c["q"] for c in cases], is_query=True, label="queries") + + systems, groups = ["lexical", "vector", "hybrid"], ["overall", "command", "concept"] + data = {s: {g: [] for g in groups} for s in systems} + for c, qv in zip(cases, qvecs): + expected = set(c["expected"]) + lex, vec = c["lexical"], rank_pages(qv, emb, owners) + ranks = {"lexical": best_rank(lex, expected), + "vector": best_rank(vec, expected), + "hybrid": best_rank(rrf([lex, vec]), expected)} + for s in systems: + data[s]["overall"].append(ranks[s]) + data[s][c["kind"]].append(ranks[s]) + + for g in groups: + n = len(data["lexical"][g]) + print(f"\n=== {g} (n={n}) === recall@1 / @3 / @5 / @10 | MRR") + for s in systems: + rec, mrr = metrics(data[s][g]) + cells = " / ".join(f"{rec[k]*100:3.0f}%" for k in KS) + print(f" {s:8} {cells} | {mrr:.3f}") + + +if __name__ == "__main__": + main() diff --git a/build/docs-mcp-server/vector-eval/requirements.txt b/build/docs-mcp-server/vector-eval/requirements.txt new file mode 100644 index 0000000000..7f7505c751 --- /dev/null +++ b/build/docs-mcp-server/vector-eval/requirements.txt @@ -0,0 +1,2 @@ +fastembed>=0.3 +numpy>=1.24 From a2766985c1806cb9037356e197019062e3e5d4d2 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 23 Jul 2026 14:39:02 +0100 Subject: [PATCH 11/25] DOC-6809 Section-level chunking for vector eval; refines the hybrid verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add section-level chunk mode (one chunk per feed section + a page anchor, ~15.3k chunks) with checkpoint/resume, alongside the page-level mode. Re-run on the same 35-case eval sharpens the vector picture: - Section-level chunking is the big lift, and it fixes concept: vector MRR .47->.64 (concept), .60->.72 (overall). The coarse page-level chunks were the concept bottleneck, as hypothesised. - Surprise: with strong section-level embeddings, equal-weight RRF hybrid now *dilutes* the top ranks. Pure VECTOR is best by MRR/@1-@3 (overall .72 vs hybrid .67; concept .64 vs .53); HYBRID is best by recall@5/@10 (concept @5 92% / @10 100%). Fusing a strong retriever with a weaker one pulls its confident #1 hits down. Refined decision (SPEC §6/§10): build section-level embeddings + a WEIGHTED fusion favouring vector (not equal-weight RRF), or ship pure vector; tune against this eval before hosting. Checkpoint/resume proved out — this run resumed cleanly from 5120/15300 after the prior session's process exited. Learned: section-level chunks fix vector's concept gap; equal-weight RRF dilutes a strong vector retriever's top ranks (precision vs recall trade-off) Directive: use section-level chunking + weighted fusion (favour vector) for the hosted build; don't equal-weight-RRF a strong vector with weak lexical Gaps: weighted/score fusion not yet measured; small eval (35 cases); single small model (bge-small); sections truncated ~1200 chars Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 24 ++-- build/docs-mcp-server/vector-eval/.gitignore | 2 +- build/docs-mcp-server/vector-eval/README.md | 46 ++++++++ .../vector-eval/eval_vector.py | 106 ++++++++++++------ 4 files changed, 135 insertions(+), 43 deletions(-) create mode 100644 build/docs-mcp-server/vector-eval/README.md diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index f78cff27a5..8505ecd715 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -198,14 +198,22 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. - **Search backend:** resolved for v1 — lexical (BM25 over NDJSON), no datastore, runs in both stdio and hosted modes (see §6). **v2 vector: measured and justified.** A measure-first experiment (`vector-eval/`, no Redis: - bge-small embeddings + numpy cosine, scored on the same 35-case eval) showed - **hybrid lexical+vector (RRF) clearly beats lexical alone** — overall recall@5 - 69%→86%, MRR 0.53→0.66; command @5 73%→91%. Vector *alone* only modestly beats - lexical, so fusion is the win. → Build the hosted RediSearch/RedisVL path with - a **hybrid** ranker (not pure vector). Caveat: concept queries stay weak - (vector ties lexical at @5; hybrid lifts @5 62%→77% but not top-1/3) — likely - because the experiment used coarse *page-level* chunks; **section-level - chunking** (the feed's `sections[]`) is the next lever, being measured next. + bge-small embeddings + numpy cosine, scored on the same 35-case eval) shows + vector/hybrid clearly beats lexical → build the hosted RediSearch/RedisVL path. + Two rounds: + - *Page-level chunks (coarse):* hybrid best (overall recall@5 69%→86%, MRR + 0.53→0.66); vector alone only modest; **concept stayed weak** (hybrid @5 77%). + - *Section-level chunks (feed `sections[]`):* the big lift, and it fixed + concept. **Vector alone becomes the strongest by MRR** (overall 0.72 vs + hybrid 0.67; concept 0.64 vs 0.47 page-level) and best @1/@3; **hybrid RRF is + best by recall@5/@10** (concept @5 92% / @10 100%; overall @5 91%). Vanilla + equal-weight RRF now *dilutes* the top ranks because it fuses the strong + vector retriever with the weaker lexical one. + - **Decision:** build **section-level embeddings** + a **weighted** fusion + (favour vector) rather than equal-weight RRF, to keep vector's top-1 + precision *and* hybrid's top-k recall. Test weighted RRF / score fusion + against this eval before finalising. Pure vector is a viable fallback if + fusion tuning isn't worth it. - **Ranking quality (measured via the eval harness):** lexical BM25 with Porter stemming, stopword removal, title/summary/slug field boosts, and balanced page-type weighting (demote release-notes/REST-API/references only) diff --git a/build/docs-mcp-server/vector-eval/.gitignore b/build/docs-mcp-server/vector-eval/.gitignore index e689b46390..b741d0feb6 100644 --- a/build/docs-mcp-server/vector-eval/.gitignore +++ b/build/docs-mcp-server/vector-eval/.gitignore @@ -1,5 +1,5 @@ .venv/ __pycache__/ -embeddings.npz +embeddings*.npz lexical.json *.log diff --git a/build/docs-mcp-server/vector-eval/README.md b/build/docs-mcp-server/vector-eval/README.md new file mode 100644 index 0000000000..f611fa844c --- /dev/null +++ b/build/docs-mcp-server/vector-eval/README.md @@ -0,0 +1,46 @@ +# vector-eval — measure-first vector search experiment + +Answers "does vector / hybrid retrieval beat the tuned lexical ranker, and is it +worth the hosted RediSearch infra?" **before** building any of it. No Redis: it +embeds the corpus with an open model (bge-small-en-v1.5 via fastembed/ONNX), +ranks in numpy, and scores against the **same 35-case eval** the lexical harness +uses (`../node/test/eval/cases.json`). + +## Run + +```bash +python3 -m venv .venv && .venv/bin/pip install -r requirements.txt +node ../node/test/eval/dump-lexical.mjs > lexical.json # real lexical rankings +.venv/bin/python eval_vector.py section # or: page +``` + +Embedding is CHECKPOINTED to `embeddings-.npz` every 1024 chunks (a +crash/kill resumes, doesn't restart). Everything except the scripts is +gitignored (venv, caches, `lexical.json`). Note: embedding runs slowly here +(~6–11 chunks/s on unoptimised CPU ONNX) — **not** a production latency signal. + +## Results (recall@5 / MRR) + +| group | | lexical | vector | hybrid (RRF) | +|---|---|---|---|---| +| overall | page-level | 69 / .53 | 74 / .60 | **86 / .66** | +| overall | section-level | 69 / .53 | **83 / .72** | 91 / .67 | +| command | section-level | 73 / .57 | 91 / **.76** | 91 / .76 | +| concept | section-level | 62 / .46 | 69 / **.64** | **92** / .53 | + +(section-level concept: hybrid @10 = 100%, vector @10 = 85%.) + +## Findings + +1. **Section-level chunking (feed `sections[]`) is the big win** — especially + for concept queries (vector MRR .47 → .64). Coarse page-level chunks were + what held vector back. +2. **With strong section-level embeddings, vanilla equal-weight RRF hurts the + top ranks**: pure vector is best by MRR / @1–@3, hybrid is best by recall@5/@10. + Fusing a strong retriever with a weaker one dilutes its confident top hits. +3. **Direction:** build section-level embeddings + a **weighted** fusion + (favour vector), or ship pure vector. Tune against this eval before hosting. + See `../SPEC.md` §6/§10. + +Limitations: small eval (35 cases, 13 concept — directional, not definitive); +sections truncated to ~1200 chars; single small model (bge-small). diff --git a/build/docs-mcp-server/vector-eval/eval_vector.py b/build/docs-mcp-server/vector-eval/eval_vector.py index 079d61cfdf..ee5b3ca39e 100644 --- a/build/docs-mcp-server/vector-eval/eval_vector.py +++ b/build/docs-mcp-server/vector-eval/eval_vector.py @@ -4,19 +4,25 @@ Compares three rankers on the same 35-case eval used by the lexical harness: - lexical : the production Node ranker's output (read from lexical.json) - - vector : bge-small-en-v1.5 embeddings, cosine, ranked in numpy + - vector : bge-small-en-v1.5 embeddings, best-chunk cosine, ranked in numpy - hybrid : reciprocal-rank fusion of lexical + vector -v2 (slim): ONE page-level chunk per page (title + summary + lead section text), -~2.5k chunks instead of 21.5k section chunks — fast enough to iterate. Embedding -is batched with live progress and the corpus cache is written at the end -(embeddings.npz) for instant re-runs. The model (bge-small-en-v1.5) is what we'd -run in production via RedisVL; loaded here through fastembed (ONNX, no torch). +Chunk mode (argv[1], default "section"): + - page : one chunk/page (title + summary + lead section text) ~2.5k chunks + - section : one chunk per section (page title + section title + text) + a + page anchor chunk ~15-18k chunks; better for concept queries. + +Embedding is batched with live progress and CHECKPOINTED every CKPT_EVERY chunks +to embeddings-.npz, so a crash/kill resumes instead of restarting (the +21.5k-chunk first attempt died after >1h with nothing saved). Model is +bge-small-en-v1.5 (what we'd run via RedisVL in production); loaded here through +fastembed (ONNX, no torch). Embedding is ~6 chunks/s on this CPU — an +unoptimised-local artefact, not a production latency signal. Usage: pip install -r requirements.txt node ../node/test/eval/dump-lexical.mjs > lexical.json - python eval_vector.py + python eval_vector.py section """ import gzip import json @@ -29,12 +35,17 @@ HERE = os.path.dirname(os.path.abspath(__file__)) FEED = os.path.join(HERE, "..", "node", "test", "eval", "docs.ndjson.gz") LEXICAL = os.path.join(HERE, "lexical.json") -CACHE = os.path.join(HERE, "embeddings.npz") MODEL = "BAAI/bge-small-en-v1.5" +DIM = 384 # bge-small-en-v1.5 BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: " LEAD_CHARS = 1200 +MAX_SECTIONS = 8 +CKPT_EVERY = 1024 KS = [1, 3, 5, 10] +MODE = (sys.argv[1] if len(sys.argv) > 1 else "section").lower() +CACHE = os.path.join(HERE, f"embeddings-{MODE}.npz") + def norm_url(u): return u.strip().lower().rstrip("/") @@ -56,49 +67,75 @@ def load_pages(): return pages -def build_chunks(pages): - """One page-level chunk per page: title + summary + lead section text.""" +def build_chunks(pages, mode): texts, owners = [], [] for p in pages: + url = norm_url(p["url"]) title = p.get("title", "") or "" summary = p.get("summary", "") or "" - secs = " ".join((s.get("text", "") or "") for s in (p.get("sections") or [])) - parts = [x for x in (title, summary, secs[:LEAD_CHARS]) if x] - texts.append(". ".join(parts).strip() or title or p["url"]) - owners.append(norm_url(p["url"])) + sections = p.get("sections") or [] + if mode == "page": + secs = " ".join((s.get("text", "") or "") for s in sections)[:LEAD_CHARS] + parts = [x for x in (title, summary, secs) if x] + texts.append(". ".join(parts).strip() or title or url) + owners.append(url) + else: # section + anchor = ". ".join(x for x in (title, summary) if x).strip() + if anchor: + texts.append(anchor) + owners.append(url) + n = 0 + for s in sections: + body = (s.get("text", "") or "").strip() + if len(body) < 20: + continue + st = (s.get("title", "") or "").strip() + texts.append(f"{title} — {st}. {body[:LEAD_CHARS]}".strip()) + owners.append(url) + n += 1 + if n >= MAX_SECTIONS: + break + if not anchor and n == 0: + texts.append(title or url) + owners.append(url) return texts, np.array(owners) def _model(): from fastembed import TextEmbedding - return TextEmbedding(model_name=MODEL) + return TextEmbedding(model_name=MODEL, threads=os.cpu_count()) -def embed(texts, is_query=False, label="chunks"): +def embed_batch(model, texts, is_query=False): payload = [BGE_QUERY_PREFIX + t for t in texts] if is_query else texts - model = _model() - vecs, t0, total = [], time.time(), len(payload) - for i, v in enumerate(model.embed(payload, batch_size=64)): - vecs.append(v) - n = i + 1 - if n % 256 == 0 or n == total: - rate = n / (time.time() - t0) - print(f" {n}/{total} {label} ({rate:.0f}/s)", flush=True) - arr = np.asarray(vecs, dtype=np.float32) + arr = np.asarray(list(model.embed(payload, batch_size=64)), dtype=np.float32) arr /= np.linalg.norm(arr, axis=1, keepdims=True) + 1e-12 return arr def get_corpus_embeddings(texts, owners): + total = len(texts) + emb = np.zeros((total, DIM), dtype=np.float32) + start = 0 if os.path.exists(CACHE): d = np.load(CACHE, allow_pickle=True) - if len(d["owners"]) == len(owners): - print(f" (using cached embeddings: {len(owners)} chunks)") - return d["emb"], d["owners"] - emb = embed(texts, is_query=False) - np.savez(CACHE, emb=emb, owners=owners) - print(" cached embeddings.npz") + if int(d["total"]) == total: + emb = d["emb"] + start = int(d["n_done"]) + if start >= total: + print(f" (full cache: {total} chunks)") + return emb, owners + print(f" resuming from {start}/{total}") + model = _model() + t0 = time.time() + for s in range(start, total, CKPT_EVERY): + e = min(s + CKPT_EVERY, total) + emb[s:e] = embed_batch(model, texts[s:e]) + rate = (e - start) / (time.time() - t0) + print(f" {e}/{total} chunks ({rate:.0f}/s)", flush=True) + np.savez(CACHE, emb=emb, owners=owners, total=total, n_done=e) + print(f" cached {os.path.basename(CACHE)}") return emb, owners @@ -141,16 +178,16 @@ def main(): sys.exit("lexical.json missing — run: node ../node/test/eval/dump-lexical.mjs > lexical.json") cases = json.load(open(LEXICAL)) - print("Loading feed + building page-level chunks ...") + print(f"Mode: {MODE}. Loading feed + building chunks ...") pages = load_pages() - texts, owners = build_chunks(pages) + texts, owners = build_chunks(pages, MODE) print(f" {len(pages)} pages -> {len(texts)} chunks") print("Embedding corpus ...") emb, owners = get_corpus_embeddings(texts, owners) print("Embedding queries ...") - qvecs = embed([c["q"] for c in cases], is_query=True, label="queries") + qvecs = embed_batch(_model(), [c["q"] for c in cases], is_query=True) systems, groups = ["lexical", "vector", "hybrid"], ["overall", "command", "concept"] data = {s: {g: [] for g in groups} for s in systems} @@ -164,6 +201,7 @@ def main(): data[s]["overall"].append(ranks[s]) data[s][c["kind"]].append(ranks[s]) + print(f"\n### chunk mode: {MODE} ###") for g in groups: n = len(data["lexical"][g]) print(f"\n=== {g} (n={n}) === recall@1 / @3 / @5 / @10 | MRR") From 0c3e7df85d59405d542b4aefd924c087c29ca0ab Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 23 Jul 2026 14:44:32 +0100 Subject: [PATCH 12/25] DOC-6809 Fusion sweep resolves hybrid weighting: vector-weighted RRF wins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With section embeddings cached, sweep fusion methods on the eval (no re-embedding): pure vector, equal RRF, and weighted RRF favouring vector at 2x/3x/5x. Result: weighted RRF (~2-3x vector) recovers the top-1 precision that equal-weight RRF lost AND keeps hybrid's top-k recall — overall MRR .73 (vs .72 pure vector, .69 equal RRF), command MRR .80, concept @5 92% / @10 100%. So the earlier "RRF dilutes the top ranks" was specifically *equal* weighting; weight vector above lexical and it's the best of both. n=35 so 2x vs 3x is noise. Resolves the open "which fusion" question (SPEC §6/§10): the hosted build is section-level embeddings + vector-weighted RRF. Learned: equal-weight RRF dilutes a strong retriever; weighting it ~2-3x above the weak one recovers top-1 precision while keeping top-k recall Directive: hosted build = section-level embeddings + vector-weighted RRF (not equal-weight); don't read 2x-vs-3x as significant at n=35 Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 12 ++-- build/docs-mcp-server/vector-eval/README.md | 19 ++++++- .../vector-eval/fusion_sweep.py | 56 +++++++++++++++++++ 3 files changed, 79 insertions(+), 8 deletions(-) create mode 100644 build/docs-mcp-server/vector-eval/fusion_sweep.py diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 8505ecd715..95c74d1032 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -209,11 +209,13 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. best by recall@5/@10** (concept @5 92% / @10 100%; overall @5 91%). Vanilla equal-weight RRF now *dilutes* the top ranks because it fuses the strong vector retriever with the weaker lexical one. - - **Decision:** build **section-level embeddings** + a **weighted** fusion - (favour vector) rather than equal-weight RRF, to keep vector's top-1 - precision *and* hybrid's top-k recall. Test weighted RRF / score fusion - against this eval before finalising. Pure vector is a viable fallback if - fusion tuning isn't worth it. + - **Decision (measured — fusion sweep):** build **section-level embeddings** + + **weighted RRF favouring vector ~2–3×**. The sweep (`fusion_sweep.py`) shows + vector-weighted RRF recovers the top-1 precision equal-weight RRF lost *and* + keeps the top-k recall: overall MRR **.73** (vs .72 pure vector, .69 equal + RRF), command MRR **.80**, concept @5 **92%** / @10 **100%**. So the dilution + was specifically *equal* weighting. (n=35 is small, so treat the 2× vs 3× + choice as noise — just weight vector above lexical.) - **Ranking quality (measured via the eval harness):** lexical BM25 with Porter stemming, stopword removal, title/summary/slug field boosts, and balanced page-type weighting (demote release-notes/REST-API/references only) diff --git a/build/docs-mcp-server/vector-eval/README.md b/build/docs-mcp-server/vector-eval/README.md index f611fa844c..858149f9c7 100644 --- a/build/docs-mcp-server/vector-eval/README.md +++ b/build/docs-mcp-server/vector-eval/README.md @@ -38,9 +38,22 @@ gitignored (venv, caches, `lexical.json`). Note: embedding runs slowly here 2. **With strong section-level embeddings, vanilla equal-weight RRF hurts the top ranks**: pure vector is best by MRR / @1–@3, hybrid is best by recall@5/@10. Fusing a strong retriever with a weaker one dilutes its confident top hits. -3. **Direction:** build section-level embeddings + a **weighted** fusion - (favour vector), or ship pure vector. Tune against this eval before hosting. - See `../SPEC.md` §6/§10. +3. **Direction:** build section-level embeddings + **weighted RRF favouring + vector ~2–3×** (`fusion_sweep.py`). Weighted RRF recovers the top-1 precision + that equal-weight RRF lost while keeping top-k recall — overall MRR .73 (vs + .72 pure vector, .69 equal RRF), concept @5 92% / @10 100%. The dilution was + *equal* weighting, not fusion per se. See `../SPEC.md` §6/§10. + +## Fusion sweep (section-level, cached embeddings) + +`python fusion_sweep.py section` — resolves which fusion to build: + +| system | overall MRR | command MRR | concept @5 / @10 | +|---|---|---|---| +| lexical | .53 | .57 | 62 / 77 | +| pure vector | .72 | .76 | 69 / 85 | +| equal RRF (1:1) | .69 | .76 | 92 / 100 | +| **weighted RRF (3:1)** | **.73** | **.80** | **92 / 100** | Limitations: small eval (35 cases, 13 concept — directional, not definitive); sections truncated to ~1200 chars; single small model (bge-small). diff --git a/build/docs-mcp-server/vector-eval/fusion_sweep.py b/build/docs-mcp-server/vector-eval/fusion_sweep.py new file mode 100644 index 0000000000..8bbff41e03 --- /dev/null +++ b/build/docs-mcp-server/vector-eval/fusion_sweep.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +""" +Fusion sweep: with section-level embeddings CACHED, compare lexical / pure +vector / equal-weight RRF / weighted RRF (favouring vector) on the eval, to +resolve which fusion to build. Reuses eval_vector's corpus embeddings (cache +hit — no re-embedding) and the dumped lexical rankings. + + python fusion_sweep.py section +""" +import json + +import eval_vector as ev + + +def wrrf(lists_weights, k=60, topn=50): + """Weighted reciprocal-rank fusion. lists_weights: [(ranked_urls, weight)].""" + scores = {} + for lst, w in lists_weights: + for r, u in enumerate(lst): + scores[u] = scores.get(u, 0.0) + w / (k + r + 1) + return [u for u, _ in sorted(scores.items(), key=lambda kv: -kv[1])][:topn] + + +cases = json.load(open(ev.LEXICAL)) +pages = ev.load_pages() +texts, owners = ev.build_chunks(pages, ev.MODE) +emb, owners = ev.get_corpus_embeddings(texts, owners) # cache hit +qvecs = ev.embed_batch(ev._model(), [c["q"] for c in cases], is_query=True) + +systems = { + "lexical": lambda lex, vec: lex, + "vector": lambda lex, vec: vec, + "rrf 1:1": lambda lex, vec: wrrf([(vec, 1), (lex, 1)]), + "wrrf v2": lambda lex, vec: wrrf([(vec, 2), (lex, 1)]), + "wrrf v3": lambda lex, vec: wrrf([(vec, 3), (lex, 1)]), + "wrrf v5": lambda lex, vec: wrrf([(vec, 5), (lex, 1)]), +} +groups = ["overall", "command", "concept"] +data = {s: {g: [] for g in groups} for s in systems} + +for c, q in zip(cases, qvecs): + exp = set(c["expected"]) + lex = c["lexical"] + vec = ev.rank_pages(q, emb, owners) + for s, fn in systems.items(): + r = ev.best_rank(fn(lex, vec), exp) + data[s]["overall"].append(r) + data[s][c["kind"]].append(r) + +for g in groups: + n = len(data["lexical"][g]) + print(f"\n=== {g} (n={n}) === recall@1 / @3 / @5 / @10 | MRR") + for s in systems: + rec, mrr = ev.metrics(data[s][g]) + cells = " / ".join(f"{rec[k] * 100:3.0f}%" for k in ev.KS) + print(f" {s:9} {cells} | {mrr:.3f}") From f54a8bba226a99e355f620814925e3af58e9ae1f Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 23 Jul 2026 15:09:20 +0100 Subject: [PATCH 13/25] DOC-6809 Paice/Lancaster stemmer bake-off vs Porter; keep Porter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a Paice/Lancaster stemmer (src/stem-paice.ts, standard Lancaster rule table) switchable via STEMMER=paice (default stays Porter, zero runtime impact when unused), and bake it off on the eval. Verdict: Porter wins or ties — Porter overall @5 69% / MRR .53 vs Paice 63% / .52. Paice edges @1 (aggressive conflation occasionally nabs the exact top hit; command @1 45% vs 41%) but is worse at @3-@10 and on concept, and over-stems (organization->org, maximum->maxim), which pulls in false matches and costs mid-rank recall. Kept Porter as the default analyzer. Small margins at n=35, so "Paice isn't better here", not "Paice is bad". And it matters least where we're headed: in the vector-weighted-RRF hybrid recipe lexical is the minority signal, so the stemmer mainly affects the pure-lexical (stdio/no-datastore) deployment — where Porter is the better default. Learned: Paice/Lancaster over-stems this technical corpus (organization->org); marginally better @1, worse mid-rank/concept than Porter; net keep Porter Directive: keep Porter as default; STEMMER=paice env flag left in for re-runs; stemmer choice is second-order given hybrid weights lexical low Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/README.md | 9 ++ build/docs-mcp-server/node/src/search.ts | 7 +- build/docs-mcp-server/node/src/stem-paice.ts | 98 ++++++++++++++++++++ 3 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 build/docs-mcp-server/node/src/stem-paice.ts diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index b95e0f761a..63deae733e 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -95,6 +95,15 @@ concept @5 46%→62% for command @5 86%→73% (SPEC §10). The residual misses a pure semantic gaps that motivate vector search (SPEC §6/§10) — the right lever for lifting both, rather than a bigger thumb on the scale. +**Stemmer:** Porter (`src/stem.ts`) is the default. A Paice/Lancaster stemmer +(`src/stem-paice.ts`) is available behind `STEMMER=paice` for comparison. Bake-off +on this eval: Porter wins or ties — Porter overall @5 69% / MRR 0.53 vs Paice +63% / 0.52; Paice edges @1 (its aggressive conflation occasionally nabs the exact +top hit) but is worse at @3–@10 and on concept, and over-stems (e.g. +`organization → org`). Kept Porter. (Small margins at n=35, and the stemmer only +really matters for the pure-lexical deployment — in the hybrid recipe lexical is +the minority signal.) + ## Known limitations (v0) - **Ranking is lexical (BM25 + Porter stemming + field/page-type weighting).** diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index c0ad9ae710..1dff08c72f 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -1,5 +1,10 @@ import type { Page } from "./types.js"; -import { stem } from "./stem.js"; +import { stem as stemPorter } from "./stem.js"; +import { stem as stemPaice } from "./stem-paice.js"; + +// Stemmer is switchable for the eval bake-off (STEMMER=paice|porter). Porter is +// the default/shipped analyzer. +const stem = (process.env.STEMMER ?? "porter").toLowerCase() === "paice" ? stemPaice : stemPorter; // Self-contained BM25 lexical index. No external search dependency: at // ~4,100 docs the whole index builds in-memory in well under a second, which diff --git a/build/docs-mcp-server/node/src/stem-paice.ts b/build/docs-mcp-server/node/src/stem-paice.ts new file mode 100644 index 0000000000..3958a075e7 --- /dev/null +++ b/build/docs-mcp-server/node/src/stem-paice.ts @@ -0,0 +1,98 @@ +// Paice/Lancaster stemmer — an iterative, rule-table-driven stemmer, more +// aggressive than Porter (heavier conflation, editable rules). Included to +// bake off against Porter (src/stem.ts) on the retrieval eval. +// +// Rules use Paice's compact notation, one per line: the leading letters are the +// ending REVERSED; optional `*` = apply only while the word is still intact; +// digits = characters to remove; trailing letters = characters to append; +// `>` = continue (re-scan), `.` = stop. Rule set is the widely-published +// Lancaster default (as used by NLTK's LancasterStemmer). + +const RAW_RULES = [ + "ai*2.", "a*1.", "bb1.", "city3s.", "ci2>", "cn1t>", "dd1.", "dei3y>", + "deec2ss.", "dee1.", "de2>", "dooh4>", "e1>", "feil1v.", "fi2>", "gni3>", + "gai3y.", "ga2>", "gg1.", "ht*2.", "hsiug5ct.", "hsi3>", "i*1.", "i1y>", + "ji1d.", "juf1s.", "ju1d.", "jo1d.", "jeh1r.", "jrev1t.", "jsim2t.", "jn1d.", + "j1s.", "lbaifi6.", "lbai4y.", "lba3>", "lbi3.", "lib2l>", "lc1.", "lufi4y.", + "luf3>", "lu2.", "lai3>", "lau3>", "la2>", "ll1.", "mui3.", "mu*2.", "msi3>", + "mm1.", "nois4j>", "noix4ct.", "noi3>", "nai3>", "na2>", "nee0.", "ne2>", + "nn1.", "pihs4>", "pp1.", "re2>", "rae0.", "ra2.", "ro2>", "ru2>", "rr1.", + "rt1>", "rei3y>", "sei3y>", "sis2.", "si2>", "ssen4>", "ss0.", "suo3>", + "su*2.", "s*1>", "s0.", "tacilp4y.", "ta2>", "tnem4>", "tne3>", "tna3>", + "tpir2b.", "tpro2b.", "tcud1.", "tpmus2.", "tpec2iv.", "tulo2v.", "tsis0.", + "tsi3>", "tt1.", "uqi3.", "ugo1.", "vis3j>", "vie0.", "vi2>", "ylb1>", + "yli3y>", "ylp0.", "yl2>", "ygo1.", "yhp1.", "ymo1.", "ypo1.", "yti3>", + "yte3>", "ytl2.", "yrtsi5.", "yra3>", "yro3>", "yfi3.", "ycn2t>", "yca3>", + "zi2>", "zy1s.", +]; + +interface Rule { + end: string; // actual ending (un-reversed) + intact: boolean; // apply only if the word is unmodified + remove: number; + append: string; + cont: boolean; // continue re-scanning after applying +} + +function parseRule(s: string): Rule { + let i = 0; + let rev = ""; + while (i < s.length && s[i] >= "a" && s[i] <= "z") rev += s[i++]; + const intact = s[i] === "*"; + if (intact) i++; + let num = ""; + while (i < s.length && s[i] >= "0" && s[i] <= "9") num += s[i++]; + let append = ""; + while (i < s.length && s[i] >= "a" && s[i] <= "z") append += s[i++]; + return { + end: rev.split("").reverse().join(""), + intact, + remove: parseInt(num || "0", 10), + append, + cont: s[i] === ">", + }; +} + +// Group rules by the word's last character (= last char of the ending) for +// fast lookup, preserving Paice's rule order within each group. +const RULES_BY_LAST: Map = (() => { + const m = new Map(); + for (const raw of RAW_RULES) { + const r = parseRule(raw); + const key = r.end[r.end.length - 1]; + (m.get(key) ?? m.set(key, []).get(key)!).push(r); + } + return m; +})(); + +function acceptable(stem: string): boolean { + if (stem.length === 0) return false; + if ("aeiou".includes(stem[0])) return stem.length >= 2; + // consonant start: need >= 3 chars and at least one vowel (y counts) + if (stem.length < 3) return false; + for (const c of stem) if ("aeiouy".includes(c)) return true; + return false; +} + +export function stem(word: string): string { + let w = word.toLowerCase(); + let intact = true; + for (let guard = 0; guard < 50; guard++) { + const rules = RULES_BY_LAST.get(w[w.length - 1]); + if (!rules) return w; + let applied = false; + for (const r of rules) { + if (!w.endsWith(r.end)) continue; + if (r.intact && !intact) continue; + const candidate = w.slice(0, w.length - r.remove); + if (!acceptable(candidate)) return w; // matched ending but unacceptable → stop + w = candidate + r.append; + intact = false; + if (!r.cont) return w; + applied = true; + break; // re-scan with the shortened word + } + if (!applied) return w; + } + return w; +} From 0001afee674c9a586a291be7251c8aac7ba208b5 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 23 Jul 2026 16:35:36 +0100 Subject: [PATCH 14/25] DOC-6809 Model bake-off: bge-base not worth it over bge-small; keep bge-small MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Embed the section-level corpus with bge-base-en-v1.5 (per-model cache) and compare at the chosen weighted-RRF 3:1 recipe. bge-base overall MRR .74 vs bge-small .73, command .82 vs .80, concept .61 vs .62 — recall@5 identical (91/91/92). The only real gain is command (already strong); concept (the weak spot) is flat within n=13 noise. bge-base costs ~2x the vector size + compute and embedded at half the speed (~3 vs ~6-11 chunks/s), so it isn't justified. Keep bge-small for the hosted build. Learned: a bigger embedding model (bge-base) gives only a rounding-error overall gain here, concentrated on already-strong command queries, nothing on concept — bge-small is good enough Directive: use bge-small for the hosted build; revisit a bigger model only if the eval grows and concept is still short Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/vector-eval/README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/build/docs-mcp-server/vector-eval/README.md b/build/docs-mcp-server/vector-eval/README.md index 858149f9c7..4661a73482 100644 --- a/build/docs-mcp-server/vector-eval/README.md +++ b/build/docs-mcp-server/vector-eval/README.md @@ -55,5 +55,14 @@ gitignored (venv, caches, `lexical.json`). Note: embedding runs slowly here | equal RRF (1:1) | .69 | .76 | 92 / 100 | | **weighted RRF (3:1)** | **.73** | **.80** | **92 / 100** | +## Model comparison (bge-small vs bge-base) + +`EMBED_MODEL=BAAI/bge-base-en-v1.5 python eval_vector.py section` (per-model +cache). At weighted RRF 3:1: bge-base overall MRR .74 vs bge-small .73, +command .82 vs .80, concept .61 vs .62 — **recall@5 identical** (91/91/92). +The only real gain is command (already strong); concept (the weak spot) is +flat. **Kept bge-small**: bge-base costs ~2× vector size + compute (and embeds +at half the speed here) for a rounding-error overall gain. + Limitations: small eval (35 cases, 13 concept — directional, not definitive); -sections truncated to ~1200 chars; single small model (bge-small). +sections truncated to ~1200 chars. From 85e1d0d177043f28350f325bfe6473f75fc70b87 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 24 Jul 2026 09:55:22 +0100 Subject: [PATCH 15/25] DOC-6809 Commit the EMBED_MODEL parametrization behind the bge-base bake-off The bge-base bake-off (0001afee6) committed only the README documenting the result; the eval_vector.py change that actually enabled running a second model was left uncommitted in the working tree. Restore it so the tree matches the documented state. Makes the embedding model selectable via EMBED_MODEL (default stays bge-small-en-v1.5), gives non-default models their own per-model cache file (default model keeps embeddings-{mode}.npz for back-compat), and detects the embedding dimension from the first batch instead of hard-coding DIM=384 (bge-base is 768-dim, so the old constant would have mis-shaped the array). Learned: a documented experiment can leave its enabling code uncommitted if only the write-up is staged; check the tree matches the claim Directive: default path is byte-for-byte unchanged; re-run other models with EMBED_MODEL=BAAI/bge-base-en-v1.5 Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../vector-eval/eval_vector.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/build/docs-mcp-server/vector-eval/eval_vector.py b/build/docs-mcp-server/vector-eval/eval_vector.py index ee5b3ca39e..b16599350f 100644 --- a/build/docs-mcp-server/vector-eval/eval_vector.py +++ b/build/docs-mcp-server/vector-eval/eval_vector.py @@ -35,8 +35,8 @@ HERE = os.path.dirname(os.path.abspath(__file__)) FEED = os.path.join(HERE, "..", "node", "test", "eval", "docs.ndjson.gz") LEXICAL = os.path.join(HERE, "lexical.json") -MODEL = "BAAI/bge-small-en-v1.5" -DIM = 384 # bge-small-en-v1.5 +DEFAULT_MODEL = "BAAI/bge-small-en-v1.5" +MODEL = os.environ.get("EMBED_MODEL", DEFAULT_MODEL) # e.g. BAAI/bge-base-en-v1.5 BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: " LEAD_CHARS = 1200 MAX_SECTIONS = 8 @@ -44,7 +44,13 @@ KS = [1, 3, 5, 10] MODE = (sys.argv[1] if len(sys.argv) > 1 else "section").lower() -CACHE = os.path.join(HERE, f"embeddings-{MODE}.npz") +# Default model keeps its original cache name (back-compat); other models get a +# per-model cache so runs don't clobber each other. +_SLUG = MODEL.split("/")[-1] +CACHE = os.path.join( + HERE, + f"embeddings-{MODE}.npz" if MODEL == DEFAULT_MODEL else f"embeddings-{MODE}-{_SLUG}.npz", +) def norm_url(u): @@ -116,7 +122,7 @@ def embed_batch(model, texts, is_query=False): def get_corpus_embeddings(texts, owners): total = len(texts) - emb = np.zeros((total, DIM), dtype=np.float32) + emb = None # allocated once we know the model's embedding dimension start = 0 if os.path.exists(CACHE): d = np.load(CACHE, allow_pickle=True) @@ -131,7 +137,10 @@ def get_corpus_embeddings(texts, owners): t0 = time.time() for s in range(start, total, CKPT_EVERY): e = min(s + CKPT_EVERY, total) - emb[s:e] = embed_batch(model, texts[s:e]) + batch = embed_batch(model, texts[s:e]) + if emb is None: + emb = np.zeros((total, batch.shape[1]), dtype=np.float32) + emb[s:e] = batch rate = (e - start) / (time.time() - t0) print(f" {e}/{total} chunks ({rate:.0f}/s)", flush=True) np.savez(CACHE, emb=emb, owners=owners, total=total, n_done=e) From 27084d07a26837af13f0de5cea2179e566ff3478 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 24 Jul 2026 10:16:33 +0100 Subject: [PATCH 16/25] DOC-6809 Step 1 hosted prototype: RediSearch FLAT KNN parity with offline eval MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First step of the hosted-phase prototype (redis-eval/, sibling to vector-eval/). Loads the cached bge-small section vectors into a Redis 8 FLAT/COSINE index and checks that KNN + app-layer weighted RRF reproduces the offline numpy ranking, holding embedding constant (same cached corpus + Python query vectors on both sides) so the only variable is Redis vs numpy. It does: command metrics are bit-identical (MRR .795), KNN p50 4.1ms at K=200. The lone concept-MRR delta (.638->.599) is a single query where redis-py/connect and ioredis/connect carry an exactly-equal cosine score (gap 0.00e+00) that numpy's stable argsort and Redis's internal order tie-break differently — parity to tie-breaking, not a ranking defect (diag_divergence.py proves it). That tie also exposed a corpus quirk, not a Redis one: client "connect" pages embed identically because their section anchor text is templated per-client, so the embedding can't separate redis-py from ioredis for a "Python" query — a chunking lever for later, not now. Learned: RediSearch FLAT reproduces numpy cosine ranking to within tie-breaking; equal-score chunks are the only divergence and don't move recall@k/MRR except at an exact tie Constraint: keep app-layer weighted RRF favouring vector ~2-3x — command parity depended on fusing the Redis vector ranking with the dumped lexical exactly as the offline recipe did Directive: store owner (page URL) as an unindexed hash field returned via RETURN, NOT in the index SCHEMA — redis-py rejects a non-indexed non-sortable field, and owner never needs indexing Gaps: parity used Python-embedded query vectors; Node/ONNX embedding parity is Step 2 and still unverified Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/redis-eval/README.md | 68 ++++++ .../redis-eval/diag_divergence.py | 44 ++++ build/docs-mcp-server/redis-eval/parity.py | 197 ++++++++++++++++++ .../redis-eval/requirements.txt | 4 + 4 files changed, 313 insertions(+) create mode 100644 build/docs-mcp-server/redis-eval/README.md create mode 100644 build/docs-mcp-server/redis-eval/diag_divergence.py create mode 100644 build/docs-mcp-server/redis-eval/parity.py create mode 100644 build/docs-mcp-server/redis-eval/requirements.txt diff --git a/build/docs-mcp-server/redis-eval/README.md b/build/docs-mcp-server/redis-eval/README.md new file mode 100644 index 0000000000..69e9d13b8f --- /dev/null +++ b/build/docs-mcp-server/redis-eval/README.md @@ -0,0 +1,68 @@ +# redis-eval — hosted-phase prototype (DOC-6809) + +Takes the recipe the offline `vector-eval/` experiment settled on (section-level +bge-small chunks + weighted RRF favouring vector ~2–3×) and proves it on a real +Redis 8 Query Engine backend, one step at a time. Each step isolates a single +source of variance so a divergence can only come from one place. + +Runs against a local **Redis 8** with the `search` module on `localhost:6379` +(verified: Redis 8.8.0, RediSearch 8.8). Reuses `../vector-eval/.venv` (adds +`redis-py`) and the cached `../vector-eval/embeddings-section.npz` — no +re-embedding. + +## Step 1 — Redis retrieval parity (`parity.py`) ✅ PASS + +**Question:** does RediSearch `FLAT`/`COSINE` KNN + app-layer weighted RRF +reproduce the offline numpy-cosine ranking? Embedding is held constant: BOTH the +numpy reference and the Redis path use the **same cached corpus vectors** and the +**same Python-embedded query vectors**. The only moving part is Redis vs numpy. + +Design: load all 15,300 section chunks into a HASH index with a single indexed +`VECTOR FLAT` field (FLOAT32, DIM 384, COSINE); `owner` (page URL) rides on the +hash and comes back via `RETURN` (unindexed). Per query: KNN over the full chunk +population → dedup to best (nearest) chunk per page → top-50 pages, exactly +mirroring numpy `rank_pages`. Fuse with the dumped lexical ranking via weighted +RRF (v2/v3) and score against the same 35-case eval. + +**Result (2529 pages → 15,300 chunks):** + +| | numpy (offline) | redis (this) | +|---|---|---| +| command MRR (v3) | 0.795 | **0.795** (identical) | +| command recall@1/@5 | 73% / 91% | **73% / 91%** (identical) | +| concept MRR (vector) | 0.638 | 0.599 | +| overall MRR (wrrf v3) | 0.731 | 0.712 | +| top-1 page match | — | 34/35 | +| top-50 exact order | — | 19/35 | +| KNN latency (K=200) | — | **p50 4.1 ms, max 29 ms** | + +**Verdict:** parity holds to tie-breaking. Command metrics are bit-identical. The +single concept-MRR delta traces to ONE query — *"connect to Redis from a Python +application"* — where two chunks (`redis-py/connect`, `ioredis/connect`) have an +**exactly equal** cosine score (gap `0.00e+00`); numpy's stable argsort and +Redis's internal order break the tie differently. Deep-tail order diverges on +more queries (hence top-50 exact only 19/35) but that rarely moves the first +expected hit (top-1 34/35), so recall@k / MRR are unaffected except at that one +tie. `diag_divergence.py` reproduces the tie evidence. + +Side note (not a Redis issue): those client "connect" pages embed identically +because their section anchor text is templated the same across clients — the +embedding can't distinguish redis-py from ioredis for a "Python" query. A +chunking/corpus tuning candidate for later. + +## Next + +- **Step 2 — Node ONNX embedding parity:** embed corpus + queries in Node + (fastembed-js / onnxruntime-node), confirm cosine ≈ 1.0 vs Python fastembed and + that re-running this eval on Node vectors lands on the same numbers. Guards the + "embed in-process in the Node server" decision. +- **Step 3 — Redis-native hybrid** vs this app-layer weighted RRF: does the + built-in hybrid query reproduce the ranking with vector-favoured weighting? +- **Step 4 (later):** wire the winning path into the MCP `search_docs` handler. + +## Run + +``` +../vector-eval/.venv/bin/python parity.py section # Step 1 +../vector-eval/.venv/bin/python diag_divergence.py section +``` diff --git a/build/docs-mcp-server/redis-eval/diag_divergence.py b/build/docs-mcp-server/redis-eval/diag_divergence.py new file mode 100644 index 0000000000..a2b4eacf21 --- /dev/null +++ b/build/docs-mcp-server/redis-eval/diag_divergence.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +"""Diagnostic: for any query whose Redis top-1 page != numpy top-1 page, print +the numpy top-3 chunk cosine scores. If the top-2 gap is ~1e-5 or less, the +divergence is FLOAT32 tie-break noise (summation order), not a ranking defect. +""" +import os +import sys +import json + +import numpy as np +import redis +from redis.commands.search.query import Query + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 +import parity # noqa: E402 + +cases = json.load(open(ev.LEXICAL, encoding="utf-8")) +pages = ev.load_pages() +texts, owners = ev.build_chunks(pages, ev.MODE) +emb, owners = ev.get_corpus_embeddings(texts, owners) +qvecs = ev.embed_batch(ev._model(), [c["q"] for c in cases], is_query=True) + +r = redis.Redis(host="localhost", port=6379, decode_responses=False) +parity.build_index(r, emb.shape[1]) +parity.load_chunks(r, emb, owners) + +for c, qv in zip(cases, qvecs): + np_pages = ev.rank_pages(qv, emb, owners) + rd_pages = parity.redis_knn_pages(r, qv, k=emb.shape[0]) + if np_pages[0] == rd_pages[0]: + continue + sims = emb @ qv + order = np.argsort(-sims)[:3] + print(f"\nQUERY ({c['kind']}): {c['q']}") + print(f" numpy top-1 page: {np_pages[0]}") + print(f" redis top-1 page: {rd_pages[0]}") + print(" numpy top-3 chunk cosine scores:") + for j in order: + print(f" {sims[j]:.8f} {owners[j]}") + print(f" top-1 vs top-2 gap: {sims[order[0]] - sims[order[1]]:.2e}") + +r.ft(parity.INDEX).dropindex(delete_documents=True) diff --git a/build/docs-mcp-server/redis-eval/parity.py b/build/docs-mcp-server/redis-eval/parity.py new file mode 100644 index 0000000000..f329c01129 --- /dev/null +++ b/build/docs-mcp-server/redis-eval/parity.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +""" +Step 1 of the hosted-phase prototype (DOC-6809): Redis retrieval parity. + +Question this isolates: does RediSearch FLAT/COSINE KNN + app-layer weighted RRF +reproduce the OFFLINE numpy-cosine eval ranking (overall MRR ~.73, concept @10 +100%)? To keep embedding out of the picture entirely (that's Step 2 — Node ONNX +parity), BOTH the numpy reference path and the Redis path use the SAME cached +bge-small corpus vectors (embeddings-section.npz) and the SAME Python-embedded +query vectors. The only moving part here is Redis vs numpy. + +What it does: + 1. Reuse eval_vector to build the section chunks + load cached corpus vectors. + 2. Load every chunk into a RediSearch HASH index (FLAT, COSINE, FLOAT32, 384). + 3. Embed the 35 eval queries once with Python fastembed (the reference vectors). + 4. Per query: Redis KNN over all chunks -> dedup to best-per-page -> top-50 pages. + Compare that page ranking against numpy rank_pages on the identical vector. + 5. Fuse each ranking with the dumped lexical ranking via weighted RRF and score + recall@k / MRR, side by side numpy-vector vs redis-vector. + 6. Report KNN latency at a realistic K (separate from the full-K parity query). + +Usage: + ../vector-eval/.venv/bin/python parity.py section +Requires: local Redis 8 with search (localhost:6379), cached embeddings-section.npz. +""" +import os +import sys +import time + +import numpy as np +import redis +from redis.commands.search.field import VectorField +from redis.commands.search.index_definition import IndexDefinition, IndexType +from redis.commands.search.query import Query + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 (MODE/CACHE resolve from argv[1], default "section") + +INDEX = "docs_parity" +PREFIX = "parity:chunk:" +REALISTIC_K = 200 # for the latency figure; parity uses full-K + + +def connect(): + r = redis.Redis(host="localhost", port=6379, decode_responses=False) + r.ping() + return r + + +def build_index(r, dim): + try: + r.ft(INDEX).dropindex(delete_documents=True) + except redis.ResponseError: + pass + # Only the vector is indexed; `owner` lives on the hash and is pulled back + # via RETURN at query time (RediSearch returns unindexed hash fields fine). + schema = ( + VectorField( + "vector", + "FLAT", + {"TYPE": "FLOAT32", "DIM": dim, "DISTANCE_METRIC": "COSINE"}, + ), + ) + r.ft(INDEX).create_index( + schema, + definition=IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH), + ) + + +def load_chunks(r, emb, owners): + pipe = r.pipeline(transaction=False) + for i, (vec, owner) in enumerate(zip(emb, owners)): + pipe.hset( + f"{PREFIX}{i}", + mapping={ + "vector": np.asarray(vec, dtype=np.float32).tobytes(), + "owner": owner, + }, + ) + if i % 2000 == 0: + pipe.execute() + pipe.execute() + + +def redis_knn_pages(r, qvec, k, topn=50): + """KNN over `k` chunks, dedup to best (nearest) page, return top-n page urls.""" + blob = np.asarray(qvec, dtype=np.float32).tobytes() + q = ( + Query(f"*=>[KNN {k} @vector $blob AS score]") + .sort_by("score") # COSINE distance, ascending = nearest first + .return_fields("owner", "score") + .paging(0, k) + .dialect(2) + ) + res = r.ft(INDEX).search(q, query_params={"blob": blob}) + best = {} + for doc in res.docs: + owner = doc.owner.decode() if isinstance(doc.owner, bytes) else doc.owner + if owner not in best: # docs are distance-sorted, so first = nearest + best[owner] = float(doc.score) + if len(best) >= topn: + break + return list(best.keys()) + + +def main(): + import json + + cases = json.load(open(ev.LEXICAL)) + print(f"Mode: {ev.MODE}. Building chunks + loading cached corpus vectors ...") + pages = ev.load_pages() + texts, owners = ev.build_chunks(pages, ev.MODE) + emb, owners = ev.get_corpus_embeddings(texts, owners) # cache hit, no re-embed + total, dim = emb.shape + print(f" {len(pages)} pages -> {total} chunks, dim {dim}") + + print("Connecting to Redis + (re)building FLAT index ...") + r = connect() + build_index(r, dim) + t0 = time.time() + load_chunks(r, emb, owners) + print(f" loaded {total} chunks in {time.time()-t0:.1f}s") + + print("Embedding 35 queries with Python fastembed (reference vectors) ...") + qvecs = ev.embed_batch(ev._model(), [c["q"] for c in cases], is_query=True) + + # --- Parity: Redis full-K KNN vs numpy rank_pages on identical vectors --- + print("\nParity check: Redis KNN page ranking vs numpy rank_pages ...") + identical, top1_match, latencies = 0, 0, [] + redis_rank_by_case = [] + for c, qv in zip(cases, qvecs): + np_pages = ev.rank_pages(qv, emb, owners) # numpy reference (all chunks) + rd_pages = redis_knn_pages(r, qv, k=total) # Redis, full-K = same population + redis_rank_by_case.append(rd_pages) + if np_pages[:50] == rd_pages[:50]: + identical += 1 + if np_pages and rd_pages and np_pages[0] == rd_pages[0]: + top1_match += 1 + # realistic-K latency (not the full-K parity query) + t = time.time() + redis_knn_pages(r, qv, k=REALISTIC_K) + latencies.append((time.time() - t) * 1000) + n = len(cases) + print(f" top-50 page order identical: {identical}/{n}") + print(f" top-1 page match: {top1_match}/{n}") + lat = sorted(latencies) + print(f" KNN K={REALISTIC_K} latency: p50 {lat[n//2]:.1f}ms max {lat[-1]:.1f}ms") + + # --- Metrics: numpy-vector vs redis-vector, each fused with lexical --- + def score(vec_ranker): + groups = ["overall", "command", "concept"] + systems = { + "vector": lambda lex, vec: vec, + "wrrf v2": lambda lex, vec: _wrrf([(vec, 2), (lex, 1)]), + "wrrf v3": lambda lex, vec: _wrrf([(vec, 3), (lex, 1)]), + } + data = {s: {g: [] for g in groups} for s in systems} + for c, qv in zip(cases, qvecs): + exp = set(c["expected"]) + lex = c["lexical"] + vec = vec_ranker(c, qv) + for s, fn in systems.items(): + rk = ev.best_rank(fn(lex, vec), exp) + data[s]["overall"].append(rk) + data[s][c["kind"]].append(rk) + return data + + def report(title, data): + print(f"\n### {title} ###") + for g in ["overall", "command", "concept"]: + m = len(data["vector"][g]) + print(f"\n=== {g} (n={m}) === recall@1 / @3 / @5 / @10 | MRR") + for s in data: + rec, mrr = ev.metrics(data[s][g]) + cells = " / ".join(f"{rec[k]*100:3.0f}%" for k in ev.KS) + print(f" {s:8} {cells} | {mrr:.3f}") + + np_idx = {id(c): ev.rank_pages(qv, emb, owners) for c, qv in zip(cases, qvecs)} + report("numpy vector (offline reference)", score(lambda c, qv: np_idx[id(c)])) + rd_idx = {id(c): rd for c, rd in zip(cases, redis_rank_by_case)} + report("redis vector (this prototype)", score(lambda c, qv: rd_idx[id(c)])) + + r.ft(INDEX).dropindex(delete_documents=True) + print("\nDropped index; done.") + + +def _wrrf(lists_weights, k=60, topn=50): + scores = {} + for lst, w in lists_weights: + for rank, u in enumerate(lst): + scores[u] = scores.get(u, 0.0) + w / (k + rank + 1) + return [u for u, _ in sorted(scores.items(), key=lambda kv: -kv[1])][:topn] + + +if __name__ == "__main__": + main() diff --git a/build/docs-mcp-server/redis-eval/requirements.txt b/build/docs-mcp-server/redis-eval/requirements.txt new file mode 100644 index 0000000000..6d85d73e8b --- /dev/null +++ b/build/docs-mcp-server/redis-eval/requirements.txt @@ -0,0 +1,4 @@ +# Reuses ../vector-eval/.venv (fastembed + numpy). Adds the Redis client. +# Local Redis 8 with the `search` module must be reachable on localhost:6379. +-r ../vector-eval/requirements.txt +redis>=8.0 From ee3e586a20e3ed3dfdca989626a64b6d3ab09681 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 24 Jul 2026 10:27:34 +0100 Subject: [PATCH 17/25] DOC-6809 Step 2 hosted prototype: Node fastembed-js parity with Python fastembed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Confirms that embedding queries in-process in the Node MCP server is safe. Node `fastembed` (v2.1.0, the Node port of the same Qdrant fastembed the Python eval uses) embeds bge-small identically to Python: cosine 1.00000 across all 545 compared texts (35 queries + a 510-chunk corpus sample), 0 below 0.999. So the cached Python corpus vectors and every offline number stay valid regardless of which language does the embedding. Input drift is eliminated by having Python export the exact strings both sides embed. The eval pure-vector metrics are identical; the only movement is wrrf-v3 command MRR .795->.789 on one query — the same RRF tie-break sensitivity noted in Step 1, where cosine-1.0 is not bit-identical so a couple of near-tied chunks reorder in the tail and fusion flips one fused rank while the vector-only metric doesn't move. Not an embedding discrepancy. Learned: fastembed-js reproduces Python fastembed bge-small vectors to cosine 1.0, so embed-in-process-in-Node is safe and cross-language corpus/query mixing is valid Directive: embed with plain embed() and prepend the BGE query prefix manually — do NOT use queryEmbed/passageEmbed, whose built-in prefix wording can differ from the eval's and silently break parity Constraint: fastembed-js downloads a ~128M model cache to node/local_cache/ (gitignored) — keep it out of git and out of the npm package Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/.gitignore | 1 + build/docs-mcp-server/node/package.json | 1 + .../docs-mcp-server/node/test/embed-dump.mjs | 55 ++++++++++ build/docs-mcp-server/redis-eval/.gitignore | 4 + build/docs-mcp-server/redis-eval/README.md | 46 +++++++- .../redis-eval/embed_parity.py | 100 ++++++++++++++++++ .../redis-eval/export_texts.py | 34 ++++++ 7 files changed, 236 insertions(+), 5 deletions(-) create mode 100644 build/docs-mcp-server/node/test/embed-dump.mjs create mode 100644 build/docs-mcp-server/redis-eval/.gitignore create mode 100644 build/docs-mcp-server/redis-eval/embed_parity.py create mode 100644 build/docs-mcp-server/redis-eval/export_texts.py diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore index d458242e86..144260fb0f 100644 --- a/build/docs-mcp-server/node/.gitignore +++ b/build/docs-mcp-server/node/.gitignore @@ -2,3 +2,4 @@ node_modules/ dist/ *.log test/eval/docs.ndjson* +local_cache/ diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json index 90f51e6444..434ef96d5a 100644 --- a/build/docs-mcp-server/node/package.json +++ b/build/docs-mcp-server/node/package.json @@ -26,6 +26,7 @@ }, "devDependencies": { "@types/node": "^20.0.0", + "fastembed": "^2.1.0", "tsx": "^4.0.0", "typescript": "^5.0.0" } diff --git a/build/docs-mcp-server/node/test/embed-dump.mjs b/build/docs-mcp-server/node/test/embed-dump.mjs new file mode 100644 index 0000000000..764d8f78b9 --- /dev/null +++ b/build/docs-mcp-server/node/test/embed-dump.mjs @@ -0,0 +1,55 @@ +// Step 2b: embed the exported parity strings with fastembed-js (bge-small, +// ONNX in Node) and dump the vectors for Python-side comparison. Uses plain +// embed() and prepends the SAME BGE query prefix as the Python eval, so the +// only variable under test is the Node ONNX embedding itself (tokenizer + +// model + pooling), not prefix wording or method choice. +// +// node test/embed-dump.mjs +import { FlagEmbedding, EmbeddingModel } from "fastembed"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const IN = path.join(HERE, "..", "..", "redis-eval", "parity_texts.json"); +const OUT = path.join(HERE, "..", "..", "redis-eval", "node_vectors.json"); + +function l2normalize(v) { + let n = 0; + for (const x of v) n += x * x; + n = Math.sqrt(n) + 1e-12; + return v.map((x) => x / n); +} + +async function embedAll(model, texts) { + const out = []; + // embed() yields batches of Float32-ish arrays. + for await (const batch of model.embed(texts, 64)) { + for (const v of batch) out.push(l2normalize(Array.from(v))); + } + return out; +} + +const { query_prefix, queries, corpus } = JSON.parse(fs.readFileSync(IN, "utf-8")); + +const model = await FlagEmbedding.init({ model: EmbeddingModel.BGESmallENV15 }); + +console.log(`embedding ${queries.length} queries + ${corpus.length} corpus chunks ...`); +const t0 = Date.now(); +const queryVecs = await embedAll(model, queries.map((q) => query_prefix + q)); +const corpusVecs = await embedAll(model, corpus); +const secs = (Date.now() - t0) / 1000; + +fs.writeFileSync( + OUT, + JSON.stringify({ + model: "fast-bge-small-en-v1.5", + dim: queryVecs[0].length, + queries: queryVecs, + corpus: corpusVecs, + }) +); +console.log( + `wrote ${OUT}: dim ${queryVecs[0].length}, ` + + `${(queries.length + corpus.length) / secs | 0} texts/s` +); diff --git a/build/docs-mcp-server/redis-eval/.gitignore b/build/docs-mcp-server/redis-eval/.gitignore new file mode 100644 index 0000000000..274f9886e8 --- /dev/null +++ b/build/docs-mcp-server/redis-eval/.gitignore @@ -0,0 +1,4 @@ +# Regenerable parity artifacts (export_texts.py / embed-dump.mjs) +parity_texts.json +node_vectors.json +__pycache__/ diff --git a/build/docs-mcp-server/redis-eval/README.md b/build/docs-mcp-server/redis-eval/README.md index 69e9d13b8f..99b7449cc2 100644 --- a/build/docs-mcp-server/redis-eval/README.md +++ b/build/docs-mcp-server/redis-eval/README.md @@ -50,12 +50,44 @@ because their section anchor text is templated the same across clients — the embedding can't distinguish redis-py from ioredis for a "Python" query. A chunking/corpus tuning candidate for later. +## Step 2 — Node ONNX embedding parity ✅ PASS + +**Question:** does embedding in Node (`fastembed` npm, ONNX via `onnxruntime-node`) +reproduce Python `fastembed` vectors closely enough to (a) embed queries +in-process in the Node MCP server and (b) keep the cached Python corpus vectors + +offline numbers valid? `fastembed` npm is the Node port of the same Qdrant +fastembed the Python side uses (same `@anush008/tokenizers`, same HF model files). + +Design: Python exports the exact query + 510-chunk corpus-sample strings +(`export_texts.py` → `parity_texts.json`); Node embeds them with plain `embed()` +prepending the same BGE query prefix (`node/test/embed-dump.mjs` → +`node_vectors.json`); Python re-embeds the identical strings and compares +(`embed_parity.py`). Using plain `embed()` (not `queryEmbed`/`passageEmbed`) keeps +the only variable the ONNX embedding itself, not prefix wording. + +**Result:** + +| check | result | +|---|---| +| cosine(node, python), 35 queries | min/p50/mean **1.00000** | +| cosine(node, python), 510 corpus chunks | min/p50/mean **1.00000** | +| texts below 0.999 cosine | **0 / 545** | +| eval, pure-vector MRR (Node vs Python queries) | identical (.717 / .763 / .638) | +| eval, wrrf-v3 command MRR | .795 → .789 | + +**Verdict:** the Node embeddings are identical to Python's (cosine 1.0), so +embedding in-process in the Node server is safe and the cached corpus vectors +remain valid. The lone wrrf-v3 delta (command MRR .795→.789, one query) is the +same tie-break sensitivity seen in Step 1: cosine-1.0 is not bit-identical, so a +couple of near-tied corpus chunks reorder in the tail and RRF — sensitive to +exact rank positions — flips one fused rank, even though the vector-only metric +is unchanged. Noise at n=22, not an embedding discrepancy. + +Node embedding ran ~4 texts/s here (unoptimised local ONNX, same caveat as the +Python side — not a production latency signal; the Step 1 KNN latency is). + ## Next -- **Step 2 — Node ONNX embedding parity:** embed corpus + queries in Node - (fastembed-js / onnxruntime-node), confirm cosine ≈ 1.0 vs Python fastembed and - that re-running this eval on Node vectors lands on the same numbers. Guards the - "embed in-process in the Node server" decision. - **Step 3 — Redis-native hybrid** vs this app-layer weighted RRF: does the built-in hybrid query reproduce the ranking with vector-favoured weighting? - **Step 4 (later):** wire the winning path into the MCP `search_docs` handler. @@ -63,6 +95,10 @@ chunking/corpus tuning candidate for later. ## Run ``` -../vector-eval/.venv/bin/python parity.py section # Step 1 +../vector-eval/.venv/bin/python parity.py section # Step 1 ../vector-eval/.venv/bin/python diag_divergence.py section + +../vector-eval/.venv/bin/python export_texts.py # Step 2 +(cd ../node && node test/embed-dump.mjs) # (downloads model 1st run) +../vector-eval/.venv/bin/python embed_parity.py section ``` diff --git a/build/docs-mcp-server/redis-eval/embed_parity.py b/build/docs-mcp-server/redis-eval/embed_parity.py new file mode 100644 index 0000000000..596e43abc8 --- /dev/null +++ b/build/docs-mcp-server/redis-eval/embed_parity.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Step 2c: compare Node fastembed-js vectors against Python fastembed on the +SAME strings, and confirm the eval is stable when queries are embedded in Node. + +Two checks: + 1. Per-text cosine(node_vec, python_vec) for the 35 queries + corpus sample. + Near-1.0 means the Node ONNX path reproduces Python embeddings (so the + cached Python corpus vectors and the offline numbers stay valid). + 2. Re-run the 35-case eval with Node-embedded QUERY vectors against the cached + Python corpus (the most likely near-term wiring: build offline in Python, + embed queries in Node at request time). Metrics should match Step 1. + +Run export_texts.py + node test/embed-dump.mjs first. + ../vector-eval/.venv/bin/python embed_parity.py section +""" +import json +import os +import sys + +import numpy as np + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 + + +def cosine(a, b): + a = np.asarray(a, dtype=np.float32) + b = np.asarray(b, dtype=np.float32) + return float(a @ b / ((np.linalg.norm(a) * np.linalg.norm(b)) + 1e-12)) + + +def main(): + texts = json.load(open(os.path.join(HERE, "parity_texts.json"), encoding="utf-8")) + node = json.load(open(os.path.join(HERE, "node_vectors.json"), encoding="utf-8")) + prefix = texts["query_prefix"] + + # Python embeddings of the identical strings. + py_q = ev.embed_batch(ev._model(), texts["queries"], is_query=True) + py_c = ev.embed_batch(ev._model(), texts["corpus"], is_query=False) + + q_cos = [cosine(n, p) for n, p in zip(node["queries"], py_q)] + c_cos = [cosine(n, p) for n, p in zip(node["corpus"], py_c)] + + def stat(name, xs): + xs = sorted(xs) + print(f" {name:14} min {xs[0]:.5f} p50 {xs[len(xs)//2]:.5f} " + f"mean {sum(xs)/len(xs):.5f} (n={len(xs)})") + + print(f"Node dim {node['dim']} vs Python dim {py_q.shape[1]}") + print("Per-text cosine(node, python):") + stat("queries", q_cos) + stat("corpus", c_cos) + below = sum(1 for x in q_cos + c_cos if x < 0.999) + print(f" texts below 0.999 cosine: {below}/{len(q_cos)+len(c_cos)}") + + # --- Eval stability: Node query vectors vs cached Python corpus --- + cases = json.load(open(ev.LEXICAL, encoding="utf-8")) + pages = ev.load_pages() + corpus_texts, owners = ev.build_chunks(pages, ev.MODE) + emb, owners = ev.get_corpus_embeddings(corpus_texts, owners) + node_q = np.asarray(node["queries"], dtype=np.float32) + + groups = ["overall", "command", "concept"] + + def run(qvecs, label): + systems = { + "vector": lambda lex, vec: vec, + "wrrf v3": lambda lex, vec: _wrrf([(vec, 3), (lex, 1)]), + } + data = {s: {g: [] for g in groups} for s in systems} + for c, qv in zip(cases, qvecs): + exp = set(c["expected"]) + vec = ev.rank_pages(qv, emb, owners) + for s, fn in systems.items(): + rk = ev.best_rank(fn(c["lexical"], vec), exp) + data[s]["overall"].append(rk) + data[s][c["kind"]].append(rk) + print(f"\n### {label} ###") + for g in groups: + print(f"=== {g} (n={len(data['vector'][g])}) === @1/@3/@5/@10 | MRR") + for s in data: + rec, mrr = ev.metrics(data[s][g]) + cells = " / ".join(f"{rec[k]*100:3.0f}%" for k in ev.KS) + print(f" {s:8} {cells} | {mrr:.3f}") + + run(py_q, "Python query vectors (baseline)") + run(node_q, "Node query vectors (this prototype)") + + +def _wrrf(lists_weights, k=60, topn=50): + scores = {} + for lst, w in lists_weights: + for rank, u in enumerate(lst): + scores[u] = scores.get(u, 0.0) + w / (k + rank + 1) + return [u for u, _ in sorted(scores.items(), key=lambda kv: -kv[1])][:topn] + + +if __name__ == "__main__": + main() diff --git a/build/docs-mcp-server/redis-eval/export_texts.py b/build/docs-mcp-server/redis-eval/export_texts.py new file mode 100644 index 0000000000..577729968e --- /dev/null +++ b/build/docs-mcp-server/redis-eval/export_texts.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +"""Step 2a: export the EXACT strings both embedders must see, so Node vs Python +embedding parity has zero input drift. Writes parity_texts.json: + { "query_prefix": "...", "queries": [...35 raw query strings...], + "corpus": [...sampled chunk texts...] } +The BGE query prefix is emitted here and applied identically on both sides. +""" +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 + +cases = json.load(open(ev.LEXICAL, encoding="utf-8")) +pages = ev.load_pages() +texts, owners = ev.build_chunks(pages, ev.MODE) + +# Spread a ~500-chunk sample across the corpus (deterministic stride). +stride = max(1, len(texts) // 500) +sample_idx = list(range(0, len(texts), stride)) +corpus_sample = [texts[i] for i in sample_idx] + +out = { + "query_prefix": ev.BGE_QUERY_PREFIX, + "queries": [c["q"] for c in cases], + "corpus": corpus_sample, + "corpus_idx": sample_idx, # so the comparator can align to cached vectors +} +path = os.path.join(HERE, "parity_texts.json") +with open(path, "w", encoding="utf-8") as f: + json.dump(out, f) +print(f"wrote {path}: {len(out['queries'])} queries, {len(corpus_sample)} corpus chunks") From 72b91c8a88876362349f5aaa091d2b074ce1b6f2 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 24 Jul 2026 10:38:02 +0100 Subject: [PATCH 18/25] DOC-6809 Step 3 hosted prototype: Redis-native FT.HYBRID vs app-layer weighted RRF MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the fusion fork: keep BM25 lexical + weighted-RRF fusion in the app layer, use Redis only for vector KNN. Redis 8.8 has native FT.HYBRID (8.4.4+), but it does not reproduce the validated recipe (vector ~3x lexical, overall MRR .73) for two independent reasons, and native_hybrid.py decomposes both on the 35-case eval. First, its COMBINE RRF exposes only CONSTANT/WINDOW — no per-retriever weights, so it is equal-weight RRF (the variant the fusion sweep already found dilutes the top ranks); COMBINE LINEAR can weight but fuses raw scores, a different algorithm, and lands at .501. Second and more decisive, its lexical side is Redis's own BM25 over a body-text index (MRR .117, 0% command recall@5) versus our Porter-stemmed, field-boosted Node ranker (.530): a command query like "append an entry to a stream" carries no xadd token, so plain body-text BM25 ranks streams tutorials above the XADD page, and only the Node ranker's title/slug boosts + page-type weighting fix that. app-wrrf over our lexical + Redis vector reproduces Step 1 at .731; the same fusion over Redis's own BM25 collapses to .431. A corollary worth remembering: RRF gives every input list fixed rank-reciprocal mass regardless of that list's quality, so it injects a bad retriever's distractors — which is why app-wrrf-over-Redis-BM25 (.431) scored below native LINEAR (.501) on the same signals; RRF only wins when both retrievers are good. This matches SPEC section 6 — lexical has to live app-side for the stdio no-datastore mode anyway. Learned: native FT.HYBRID can't express our recipe — RRF is equal-weight-only and its raw BM25 lexical is far weaker than our tuned Node ranker (0% command recall@5); the all-in-Redis showcase costs ~.23 MRR Constraint: lexical BM25 + weighted-RRF fusion stay app-side; Redis is the vector-KNN backend only — do not route search through FT.HYBRID Directive: RRF is only safe when every fused list is a good retriever (fixed rank-reciprocal mass injects a bad list's distractors); if a weak signal must be fused, prefer LINEAR score fusion with a low weight Directive: FT.HYBRID call gotchas are in redis-eval/README — VSIM @field $param, KNN/COMBINE counts are k/v-pair counts, no DIALECT, project with LOAD not RETURN Gaps: native BM25 was queried naively (OR of terms, body-only index); a Redis index mirroring the Node analyzer would narrow the lexical gap but not enable weighted fusion Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/redis-eval/README.md | 68 ++++++- .../redis-eval/native_hybrid.py | 191 ++++++++++++++++++ 2 files changed, 256 insertions(+), 3 deletions(-) create mode 100644 build/docs-mcp-server/redis-eval/native_hybrid.py diff --git a/build/docs-mcp-server/redis-eval/README.md b/build/docs-mcp-server/redis-eval/README.md index 99b7449cc2..d116f1a99f 100644 --- a/build/docs-mcp-server/redis-eval/README.md +++ b/build/docs-mcp-server/redis-eval/README.md @@ -86,11 +86,71 @@ is unchanged. Noise at n=22, not an embedding discrepancy. Node embedding ran ~4 texts/s here (unoptimised local ONNX, same caveat as the Python side — not a production latency signal; the Step 1 KNN latency is). +## Step 3 — Redis-native FT.HYBRID vs app-layer weighted RRF ✅ (resolves the fork) + +**Question:** does Redis's native `FT.HYBRID` (8.4.4+) reproduce our validated +app-layer weighted-RRF recipe (vector ~3× lexical, overall MRR .73)? + +Two structural facts, found in the command spec: +- `COMBINE RRF` exposes only `CONSTANT` + `WINDOW` — **no per-retriever weights**. + Native RRF is equal-weight, the variant the fusion sweep already found dilutes + the top ranks. +- `COMBINE LINEAR` takes `ALPHA`/`BETA` but fuses raw **scores** linearly — a + different algorithm from our rank-based weighted RRF. +- Native hybrid also uses Redis's **own** BM25 over an indexed TEXT field, not our + Porter-stemmed / field-boosted Node lexical (`lexical.json`). + +`native_hybrid.py` decomposes both axes (fusion + lexical) on the 35-case eval. +Working `FT.HYBRID` invocation (gotchas noted for reuse): +`FT.HYBRID SEARCH "" SCORER BM25 VSIM @vec $qv KNN 2 K 200 +[COMBINE RRF 2 CONSTANT 60 | COMBINE LINEAR 4 ALPHA a BETA b] LOAD 1 @owner +LIMIT 0 200 PARAMS 2 qv ` — `VSIM @field $param`; KNN/COMBINE counts are +**k/v-pair counts** not the k value; **no `DIALECT`**; project with `LOAD`, not +`RETURN`. + +**Results (overall MRR):** + +| system | overall | command | concept | +|---|---|---|---| +| redis bm25 only | 0.117 | 0.032 | 0.262 | +| our lexical only | 0.530 | 0.571 | 0.461 | +| native rrf (equal-weight) | 0.425 | 0.448 | 0.387 | +| native linear α.2/β.8 | 0.501 | 0.527 | 0.458 | +| app wrrf, Redis BM25 + vec | 0.431 | 0.396 | 0.491 | +| **app wrrf, our lexical + vec (recipe)** | **0.731** | **0.795** | **0.621** | + +**Verdict — resolves the "showcase vs control" fork toward app-layer fusion:** +native `FT.HYBRID` does **not** reproduce the recipe, for two independent reasons. +(1) Its RRF is equal-weight-only; LINEAR weights but underperforms weighted-RRF. +(2) More decisively, its lexical side (Redis raw BM25, MRR .117; **0% command +recall@5**) is far weaker than our Node ranker (.530) — command queries like +"append an entry to a stream" carry no `xadd` token, so a plain body-text BM25 +ranks streams *tutorials* above the *XADD command page*; our title/slug field +boosts + page-type weighting are what fix that, and a plain Redis TEXT index +doesn't carry that signal. A subtle corollary: RRF gives every list fixed +rank-reciprocal mass regardless of quality, so it injects a bad retriever's +distractors (hence app-wrrf-over-Redis-BM25 .431 < native-linear .501); RRF wins +only when both retrievers are good, which they are with our Node lexical (.731). + +**Architecture conclusion (matches SPEC §6):** use **Redis for vector KNN** +(proven in Step 1, ~4 ms), keep **lexical BM25 + weighted-RRF fusion in the app +(Node) layer** — where the lexical path already has to live for the stdio +no-datastore mode. `FT.HYBRID`'s all-in-Redis showcase costs ~0.23 MRR and can't +express the weighted fusion, so it's not the path. + +*Honest caveat:* native BM25's showing is depressed partly by a deliberately +naive OR-of-terms query and by indexing only the section body (no boosted +title/slug fields). A Redis index mirroring the Node analyzer would narrow the +lexical gap — but that is re-implementing our lexical ranker inside Redis, with +native RRF still unable to do the weighted fusion. The conclusion holds either +way. (The eval's vector side here is numpy `rank_pages`; Step 1 already showed +Redis KNN ≈ numpy to tie-breaking, so this isolates fusion + lexical cleanly.) + ## Next -- **Step 3 — Redis-native hybrid** vs this app-layer weighted RRF: does the - built-in hybrid query reproduce the ranking with vector-favoured weighting? -- **Step 4 (later):** wire the winning path into the MCP `search_docs` handler. +- **Step 4 — wire the winning path into the MCP `search_docs` handler:** Redis + KNN for vector + the existing Node BM25, fused with weighted RRF (vector ~3×). + This is the first change to the shipped server; scope with Andy first. ## Run @@ -101,4 +161,6 @@ Python side — not a production latency signal; the Step 1 KNN latency is). ../vector-eval/.venv/bin/python export_texts.py # Step 2 (cd ../node && node test/embed-dump.mjs) # (downloads model 1st run) ../vector-eval/.venv/bin/python embed_parity.py section + +../vector-eval/.venv/bin/python native_hybrid.py section # Step 3 ``` diff --git a/build/docs-mcp-server/redis-eval/native_hybrid.py b/build/docs-mcp-server/redis-eval/native_hybrid.py new file mode 100644 index 0000000000..7f92de28a2 --- /dev/null +++ b/build/docs-mcp-server/redis-eval/native_hybrid.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""Step 3: does Redis-native FT.HYBRID reproduce our validated app-layer +weighted-RRF recipe (vector ~3x lexical, overall MRR .73)? + +FT.HYBRID (Redis 8.4.4+) fuses a text search and a vector search server-side. +Two findings drive the design: + - COMBINE RRF exposes only CONSTANT + WINDOW — NO per-retriever weights. Native + RRF is therefore EQUAL-weight, the exact variant our fusion sweep found + dilutes the top ranks (.69 vs .73). + - COMBINE LINEAR takes ALPHA/BETA weights but fuses raw SCORES linearly, a + different algorithm from our rank-based weighted RRF. +Also, native hybrid uses Redis's OWN BM25 over an indexed TEXT field, not our +Porter-stemmed/boosted Node BM25 (lexical.json). So "native" differs on two axes: +fusion AND lexical signal. This harness decomposes both: + + native RRF Redis BM25 + Redis KNN, native equal-weight RRF + native LINEAR a/b Redis BM25 + Redis KNN, native weighted score fusion + app wRRF (redis) Redis BM25 + Redis KNN, OUR weighted-rank RRF [isolates fusion] + app wRRF (ours) lexical.json + Redis KNN, our recipe (= Step 1) [isolates lexical] + + ../vector-eval/.venv/bin/python native_hybrid.py section +""" +import json +import os +import re +import sys + +import numpy as np +import redis + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 + +INDEX = "docs_hybrid" +PREFIX = "hybrid:chunk:" +KNN_K = 200 +TOPN = 50 + + +def connect(): + r = redis.Redis(host="localhost", port=6379, decode_responses=False) + r.ping() + return r + + +def build_index(r, dim): + try: + r.execute_command("FT.DROPINDEX", INDEX, "DD") + except redis.ResponseError: + pass + r.execute_command( + "FT.CREATE", INDEX, "ON", "HASH", "PREFIX", "1", PREFIX, "SCHEMA", + "text", "TEXT", + "owner", "TAG", + "vec", "VECTOR", "FLAT", "6", + "TYPE", "FLOAT32", "DIM", dim, "DISTANCE_METRIC", "COSINE", + ) + + +def load_chunks(r, texts, emb, owners): + pipe = r.pipeline(transaction=False) + for i, (t, v, o) in enumerate(zip(texts, emb, owners)): + pipe.hset(f"{PREFIX}{i}", mapping={ + "text": t, "owner": o, + "vec": np.asarray(v, dtype=np.float32).tobytes(), + }) + if i % 2000 == 0: + pipe.execute() + pipe.execute() + + +def or_query(q): + """Natural-language query -> BM25-friendly OR of alphanumeric terms.""" + terms = re.findall(r"[a-z0-9]+", q.lower()) + return " | ".join(terms) if terms else "*" + + +def _dedup_pages(rows): + best = [] + seen = set() + for owner in rows: + if owner not in seen: + seen.add(owner) + best.append(owner) + if len(best) >= TOPN: + break + return best + + +def _owner(doc): + o = doc.get(b"owner") if isinstance(doc, dict) else None + return o.decode() if isinstance(o, bytes) else o + + +def native_hybrid_pages(r, q, blob, combine): + args = ["FT.HYBRID", INDEX, "SEARCH", or_query(q), "SCORER", "BM25", + "VSIM", "@vec", "$qv", "KNN", "2", "K", str(KNN_K)] + args += combine + args += ["LOAD", "1", "@owner", "LIMIT", "0", str(KNN_K), + "PARAMS", "2", "qv", blob] + res = r.execute_command(*args) + rows = res[b"results"] if isinstance(res, dict) else res.get("results", []) + return _dedup_pages([_owner(d) for d in rows]) + + +def redis_bm25_pages(r, q): + res = r.ft(INDEX).search(_bm25_query(or_query(q))) + return _dedup_pages([_owner_obj(d) for d in res.docs]) + + +def _bm25_query(text): + from redis.commands.search.query import Query + return Query(text).scorer("BM25").return_fields("owner").paging(0, KNN_K) + + +def _owner_obj(doc): + o = getattr(doc, "owner", None) + return o.decode() if isinstance(o, bytes) else o + + +def main(): + cases = json.load(open(ev.LEXICAL, encoding="utf-8")) + pages = ev.load_pages() + texts, owners = ev.build_chunks(pages, ev.MODE) + emb, owners = ev.get_corpus_embeddings(texts, owners) + dim = emb.shape[1] + print(f"{len(pages)} pages -> {len(texts)} chunks, dim {dim}") + + r = connect() + build_index(r, dim) + load_chunks(r, texts, emb, owners) + print("index built + loaded") + + qvecs = ev.embed_batch(ev._model(), [c["q"] for c in cases], is_query=True) + + systems = ["redis bm25 only", "our lexical only", + "native rrf", "native lin .2/.8", "native lin .1/.9", + "app wrrf (redis)", "app wrrf (ours)"] + groups = ["overall", "command", "concept"] + data = {s: {g: [] for g in groups} for s in systems} + + for c, qv in zip(cases, qvecs): + exp = set(c["expected"]) + blob = np.asarray(qv, dtype=np.float32).tobytes() + + rankings = { + "native rrf": native_hybrid_pages( + r, c["q"], blob, ["COMBINE", "RRF", "2", "CONSTANT", "60"]), + "native lin .2/.8": native_hybrid_pages( + r, c["q"], blob, + ["COMBINE", "LINEAR", "4", "ALPHA", "0.2", "BETA", "0.8"]), + "native lin .1/.9": native_hybrid_pages( + r, c["q"], blob, + ["COMBINE", "LINEAR", "4", "ALPHA", "0.1", "BETA", "0.9"]), + } + # app-layer fusions: same Redis KNN, two different lexical sources + redis_vec = ev.rank_pages(qv, emb, owners) # matches Step 1 vector side + redis_bm25 = redis_bm25_pages(r, c["q"]) + rankings["redis bm25 only"] = redis_bm25 + rankings["our lexical only"] = c["lexical"] + rankings["app wrrf (redis)"] = _wrrf([(redis_vec, 3), (redis_bm25, 1)]) + rankings["app wrrf (ours)"] = _wrrf([(redis_vec, 3), (c["lexical"], 1)]) + + for s in systems: + rk = ev.best_rank(rankings[s], exp) + data[s]["overall"].append(rk) + data[s][c["kind"]].append(rk) + + for g in groups: + n = len(data[systems[0]][g]) + print(f"\n=== {g} (n={n}) === recall@1 / @3 / @5 / @10 | MRR") + for s in systems: + rec, mrr = ev.metrics(data[s][g]) + cells = " / ".join(f"{rec[k]*100:3.0f}%" for k in ev.KS) + print(f" {s:18} {cells} | {mrr:.3f}") + + r.execute_command("FT.DROPINDEX", INDEX, "DD") + print("\ndropped index; done.") + + +def _wrrf(lists_weights, k=60, topn=TOPN): + scores = {} + for lst, w in lists_weights: + for rank, u in enumerate(lst): + scores[u] = scores.get(u, 0.0) + w / (k + rank + 1) + return [u for u, _ in sorted(scores.items(), key=lambda kv: -kv[1])][:topn] + + +if __name__ == "__main__": + main() From 0ccf6c3366d03d5ec606a330538fc55840471640 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 24 Jul 2026 10:55:07 +0100 Subject: [PATCH 19/25] DOC-6809 Step 4 hosted prototype: hybrid search_docs (Redis vector KNN + BM25) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First change to the shipped server: a hosted hybrid search mode behind a REDIS_URL feature flag, implementing the recipe the measure-first work (Steps 1-3) validated. When REDIS_URL is set, search_docs fuses the existing in-memory BM25 lexical ranker with vector KNN from Redis via weighted reciprocal-rank fusion (vector ~3x); the query is embedded in-process with fastembed-js. Without REDIS_URL the server stays lexical-only and datastore-free (the stdio mode), and that default path is deliberately untouched. Redis is the vector backend only — lexical and fusion stay app-side, per the Step 3 finding that native FT.HYBRID can't express the weighted recipe. New modules: embed.ts, chunk.ts (a faithful port of the Python build_chunks so Node corpus vectors match the offline ones), vector-store.ts, hybrid.ts; a build-time loader (scripts/load-index.mjs) and a through-the-tool eval (test/eval/run-hybrid.mjs). Verified live against local Redis 8.8: hybrid overall MRR .704 / command .783 / concept .570 (concept @10 100%), within tie-break noise of the offline .731 — the gap stacks the Step 1 KNN and Step 2 embedding tie-breaks, not a ranking difference. Lexical-only default unchanged (.525) and smoke passes. Net payoff: overall MRR .53 -> .70, concept @10 77% -> 100%. Two things bit and are worth flagging. Making search_docs async (the hybrid path must await embedding + Redis) silently broke every synchronous caller — run.mjs, dump-lexical.mjs, smoke.ts — which failed only at runtime, not at compile time, because they discarded the now-Promise return. And node-redis v6 renamed its schema enums (SCHEMA_FIELD_TYPE / SCHEMA_VECTOR_FIELD_ALGORITHM, not SchemaFieldTypes / VectorAlgorithms) — that one does fail at compile time. For iteration speed the loader seeded vectors from the cached Python dump rather than re-embedding 15k chunks in Node at ~4/s; Step 2 proved the two are cosine-1.0 identical, so it is equivalent, and the production --embed path is still there. Learned: hybrid lifts overall MRR .53->.70 and concept @10 77%->100% live through the real tool; the ~.03 gap under offline .73 is stacked FLOAT32 KNN + embedding tie-break noise, not a ranking regression Constraint: the no-REDIS_URL lexical-only path must stay datastore-free and behaviourally unchanged — it is the stdio deployment (SPEC §6); the feature flag is the only thing that turns on Redis Directive: making a tool handler async ripples to every caller — after such a change grep for sync callers (eval/dump/smoke scripts), they fail at runtime not compile time Directive: the loader --vectors seed path is a dev shortcut (valid because Step 2 proved Node==Python vectors); production must run load-index without it so corpus embedding goes through fastembed-js Gaps: no HTTP/SSE endpoint, rate-limiting or deploy yet (stopped at the vertical slice for review); FLAT index only (HNSW deferred); package-lock is gitignored so deps float on ^ranges Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/README.md | 34 +++++ build/docs-mcp-server/node/package.json | 7 +- .../node/scripts/load-index.mjs | 72 +++++++++++ build/docs-mcp-server/node/src/chunk.ts | 48 ++++++++ build/docs-mcp-server/node/src/embed.ts | 48 ++++++++ build/docs-mcp-server/node/src/hybrid.ts | 77 ++++++++++++ build/docs-mcp-server/node/src/index.ts | 25 +++- build/docs-mcp-server/node/src/search.ts | 29 +++++ build/docs-mcp-server/node/src/smoke.ts | 2 +- .../node/src/tools/search-docs.ts | 9 +- .../docs-mcp-server/node/src/vector-store.ts | 116 ++++++++++++++++++ .../node/test/eval/dump-lexical.mjs | 16 ++- .../node/test/eval/run-hybrid.mjs | 95 ++++++++++++++ build/docs-mcp-server/node/test/eval/run.mjs | 2 +- build/docs-mcp-server/redis-eval/.gitignore | 2 + build/docs-mcp-server/redis-eval/README.md | 21 +++- .../redis-eval/dump_vectors_bin.py | 36 ++++++ 17 files changed, 618 insertions(+), 21 deletions(-) create mode 100644 build/docs-mcp-server/node/scripts/load-index.mjs create mode 100644 build/docs-mcp-server/node/src/chunk.ts create mode 100644 build/docs-mcp-server/node/src/embed.ts create mode 100644 build/docs-mcp-server/node/src/hybrid.ts create mode 100644 build/docs-mcp-server/node/src/vector-store.ts create mode 100644 build/docs-mcp-server/node/test/eval/run-hybrid.mjs create mode 100644 build/docs-mcp-server/redis-eval/dump_vectors_bin.py diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 63deae733e..c0ee45d4bb 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -47,6 +47,40 @@ Add to a Claude Code / Cursor MCP config after `npm run build`: } ``` +## Hybrid mode (hosted) — DOC-6809 Step 4 prototype + +`search_docs` has two backends, chosen by whether `REDIS_URL` is set: + +- **Lexical-only (default):** in-memory BM25, no datastore — the stdio mode above. +- **Hybrid (hosted):** when `REDIS_URL` is set, fuses the same BM25 lexical ranker + with **vector KNN from Redis** using weighted reciprocal-rank fusion (vector + ~3× lexical). The query is embedded in-process with `fastembed-js` + (bge-small-en-v1.5). This is the recipe the measure-first work settled on; + Redis is the vector backend only — lexical + fusion stay in-process, because + native `FT.HYBRID` can't express the weighted recipe (see `../redis-eval/`). + +Load the vector index once (rebuild whenever `docs.ndjson` changes), then run: + +```bash +# 1. build the vector index in Redis (embeds all section chunks with fastembed-js) +REDIS_URL=redis://localhost:6379 npm run load-index + +# 2. start the server in hybrid mode +REDIS_URL=redis://localhost:6379 DOCS_NDJSON= npm run start +``` + +`npm run eval:hybrid` scores the hybrid path through the real `search_docs` tool. +**Measured (local Redis 8.8, same 35-case eval), hybrid vs lexical-only:** + +| group | lexical MRR | **hybrid MRR** | hybrid @5 / @10 | +|---|---|---|---| +| overall | 0.53 | **0.70** | 91% / 94% | +| command | 0.57 | **0.78** | 91% / 91% | +| concept | 0.45 | **0.57** | 92% / 100% | + +(Hybrid sits within tie-break noise of the offline recipe's .73; the small gap is +FLOAT32 KNN + embedding tie-breaks, not a ranking difference — see `../redis-eval/`.) + ## Tools | Tool | Inputs | Returns | diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json index 434ef96d5a..e740cf5730 100644 --- a/build/docs-mcp-server/node/package.json +++ b/build/docs-mcp-server/node/package.json @@ -12,7 +12,9 @@ "start": "tsx src/index.ts", "dev": "tsx watch src/index.ts", "smoke": "tsx src/smoke.ts", - "eval": "npm run build && node test/eval/run.mjs" + "eval": "npm run build && node test/eval/run.mjs", + "eval:hybrid": "npm run build && node test/eval/run-hybrid.mjs", + "load-index": "npm run build && node scripts/load-index.mjs" }, "keywords": [ "redis", @@ -22,11 +24,12 @@ "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "^1.0.0", + "fastembed": "^2.1.0", + "redis": "^6.1.0", "zod": "^3.22.0" }, "devDependencies": { "@types/node": "^20.0.0", - "fastembed": "^2.1.0", "tsx": "^4.0.0", "typescript": "^5.0.0" } diff --git a/build/docs-mcp-server/node/scripts/load-index.mjs b/build/docs-mcp-server/node/scripts/load-index.mjs new file mode 100644 index 0000000000..b320635e92 --- /dev/null +++ b/build/docs-mcp-server/node/scripts/load-index.mjs @@ -0,0 +1,72 @@ +// Build-time index loader for hybrid mode (DOC-6809 Step 4). Chunks the feed +// (section-level), embeds each chunk with fastembed-js, and loads the vectors +// into Redis as the docs_vec FLAT/COSINE index. Run once whenever docs.ndjson +// is regenerated (SPEC freshness model); the server only queries at runtime. +// +// REDIS_URL=redis://localhost:6379 npm run load-index +// REDIS_URL=... node scripts/load-index.mjs --vectors ../redis-eval/vecdump +// +// --vectors seeds precomputed vectors (meta.json/owners.json/vectors.f32) +// instead of embedding in Node. Step 2 proved fastembed-js == those Python +// vectors (cosine 1.0), so seeding is equivalent — used to avoid a ~1h local +// re-embed while iterating. Production runs without the flag. +import { readFile } from "node:fs/promises"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { resolve } from "node:path"; +import { loadFeed } from "../dist/feed.js"; +import { buildChunks } from "../dist/chunk.js"; +import { embedPassages } from "../dist/embed.js"; +import { VectorStore } from "../dist/vector-store.js"; + +const REDIS_URL = process.env.REDIS_URL ?? "redis://localhost:6379"; +const FEED = + process.env.DOCS_NDJSON ?? + fileURLToPath(new URL("../test/eval/docs.ndjson.gz", import.meta.url)); + +function arg(name) { + const i = process.argv.indexOf(name); + return i >= 0 ? process.argv[i + 1] : undefined; +} + +async function fromSeed(dir) { + // Resolve against cwd (intuitive for a CLI arg), not the script location. + const base = pathToFileURL(resolve(process.cwd(), dir) + "/"); + const meta = JSON.parse(await readFile(new URL("meta.json", base), "utf8")); + const owners = JSON.parse(await readFile(new URL("owners.json", base), "utf8")); + const buf = await readFile(new URL("vectors.f32", base)); + const floats = new Float32Array(buf.buffer, buf.byteOffset, buf.byteLength / 4); + const { n, dim } = meta; + const chunks = []; + for (let i = 0; i < n; i++) { + chunks.push({ owner: owners[i], vec: floats.subarray(i * dim, (i + 1) * dim) }); + } + console.error(`[load-index] seeded ${n} vectors (dim ${dim}) from ${dir}`); + return chunks; +} + +async function fromEmbed() { + const pages = await loadFeed(FEED); + const chunks = buildChunks(pages); + console.error(`[load-index] ${pages.length} pages -> ${chunks.length} chunks; embedding (fastembed-js) ...`); + const vecs = await embedPassages(chunks.map((c) => c.text)); + return chunks.map((c, i) => ({ owner: c.owner, vec: vecs[i] })); +} + +async function main() { + const seedDir = arg("--vectors"); + const chunks = seedDir ? await fromSeed(seedDir) : await fromEmbed(); + + const store = new VectorStore(REDIS_URL); + await store.connect(); + await store.dropIndex(); + await store.ensureIndex(chunks[0].vec.length); + const t0 = Date.now(); + await store.loadChunks(chunks); + console.error(`[load-index] loaded ${chunks.length} chunks into ${REDIS_URL} in ${((Date.now() - t0) / 1000).toFixed(1)}s`); + await store.close(); +} + +main().catch((e) => { + console.error("[load-index] fatal:", e); + process.exit(1); +}); diff --git a/build/docs-mcp-server/node/src/chunk.ts b/build/docs-mcp-server/node/src/chunk.ts new file mode 100644 index 0000000000..c5bd53f9e5 --- /dev/null +++ b/build/docs-mcp-server/node/src/chunk.ts @@ -0,0 +1,48 @@ +// Section-level chunking for the vector index. Faithful port of the Python +// vector-eval build_chunks() (mode "section"), so Node-embedded corpus vectors +// correspond 1:1 with the offline experiment's chunks: +// - one "anchor" chunk per page: ". <summary>" +// - up to MAX_SECTIONS section chunks: "<title> — <section title>. <body>" +// - body truncated to LEAD_CHARS; sections with <20 chars of body skipped +// - owner of every chunk is the page's normalized url +// Section-level chunking is what fixed the concept-query gap (DOC-6809 SPEC §10). +import type { Page } from "./types.js"; + +const LEAD_CHARS = 1200; +const MAX_SECTIONS = 8; + +export interface Chunk { + text: string; + owner: string; // normalized page url +} + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +export function buildChunks(pages: Page[]): Chunk[] { + const chunks: Chunk[] = []; + for (const p of pages) { + const owner = normalizeUrl(p.url); + const title = p.title ?? ""; + const summary = p.summary ?? ""; + const sections = p.sections ?? []; + + const anchor = [title, summary].filter(Boolean).join(". ").trim(); + if (anchor) chunks.push({ text: anchor, owner }); + + let n = 0; + for (const s of sections) { + const body = (s.text ?? "").trim(); + if (body.length < 20) continue; + const st = (s.title ?? "").trim(); + chunks.push({ + text: `${title} — ${st}. ${body.slice(0, LEAD_CHARS)}`.trim(), + owner, + }); + if (++n >= MAX_SECTIONS) break; + } + if (!anchor && n === 0) chunks.push({ text: title || owner, owner }); + } + return chunks; +} diff --git a/build/docs-mcp-server/node/src/embed.ts b/build/docs-mcp-server/node/src/embed.ts new file mode 100644 index 0000000000..0c81b95fb6 --- /dev/null +++ b/build/docs-mcp-server/node/src/embed.ts @@ -0,0 +1,48 @@ +// bge-small-en-v1.5 embedding via fastembed-js (ONNX, in-process). Proven in +// the DOC-6809 Step 2 experiment to reproduce Python fastembed vectors to +// cosine 1.0, so query-time embedding here is interchangeable with the offline +// corpus vectors. IMPORTANT: use plain embed() and prepend the BGE query prefix +// manually for queries — do NOT use fastembed's queryEmbed/passageEmbed, whose +// built-in prefix wording differs and silently breaks parity (Step 2 finding). +import { FlagEmbedding, EmbeddingModel } from "fastembed"; + +const QUERY_PREFIX = "Represent this sentence for searching relevant passages: "; + +let modelPromise: Promise<FlagEmbedding> | null = null; + +function model(): Promise<FlagEmbedding> { + return (modelPromise ??= FlagEmbedding.init({ + model: EmbeddingModel.BGESmallENV15, + })); +} + +function l2normalize(v: number[]): Float32Array { + let n = 0; + for (const x of v) n += x * x; + n = Math.sqrt(n) + 1e-12; + const out = new Float32Array(v.length); + for (let i = 0; i < v.length; i++) out[i] = v[i] / n; + return out; +} + +async function embedAll(texts: string[]): Promise<Float32Array[]> { + const m = await model(); + const out: Float32Array[] = []; + for await (const batch of m.embed(texts, 64)) { + for (const v of batch) out.push(l2normalize(Array.from(v))); + } + return out; +} + +/** Embed a search query (applies the BGE query prefix). Returns a unit vector. */ +export async function embedQuery(text: string): Promise<Float32Array> { + const [v] = await embedAll([QUERY_PREFIX + text]); + return v; +} + +/** Embed corpus passages (no prefix). Returns unit vectors, input order. */ +export async function embedPassages(texts: string[]): Promise<Float32Array[]> { + return embedAll(texts); +} + +export const EMBED_DIM = 384; diff --git a/build/docs-mcp-server/node/src/hybrid.ts b/build/docs-mcp-server/node/src/hybrid.ts new file mode 100644 index 0000000000..6172607518 --- /dev/null +++ b/build/docs-mcp-server/node/src/hybrid.ts @@ -0,0 +1,77 @@ +// Hybrid searcher (hosted mode). Fuses the existing app-side BM25 lexical ranker +// (DocsIndex) with Redis vector KNN using weighted reciprocal-rank fusion, +// favouring the vector signal ~3x. This is the recipe the DOC-6809 measure-first +// work settled on (SPEC §6/§10): section-level bge-small embeddings + weighted +// RRF. Query embedding happens in-process via fastembed-js (embed.ts). Lexical +// and fusion stay here in the app because native FT.HYBRID can't express the +// weighted recipe and its raw BM25 is much weaker than this ranker (Step 3). +import type { DocsIndex, SearchHit, SearchOptions } from "./search.js"; +import type { VectorStore } from "./vector-store.js"; +import { embedQuery } from "./embed.js"; + +const RRF_K = 60; +const DEFAULT_VECTOR_WEIGHT = 3; +const LEXICAL_POOL = 50; // lexical candidates fused +const VECTOR_POOL = 200; // vector chunks fetched (deduped to <=50 pages) + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +/** Weighted reciprocal-rank fusion. Returns url -> fused score. */ +function weightedRrf( + lists: Array<{ urls: string[]; weight: number }>, +): Map<string, number> { + const scores = new Map<string, number>(); + for (const { urls, weight } of lists) { + urls.forEach((url, rank) => { + scores.set(url, (scores.get(url) ?? 0) + weight / (RRF_K + rank + 1)); + }); + } + return scores; +} + +export class HybridSearcher { + constructor( + private readonly index: DocsIndex, + private readonly store: VectorStore, + private readonly vectorWeight = DEFAULT_VECTOR_WEIGHT, + ) {} + + async search(query: string, opts: SearchOptions = {}): Promise<SearchHit[]> { + const limit = opts.limit ?? 10; + + // Lexical side (already page-type filtered) + vector side, in parallel. + const lexHits = this.index.search(query, { limit: LEXICAL_POOL, pageType: opts.pageType }); + const qvec = await embedQuery(query); + const vecUrls = await this.store.knn(qvec, VECTOR_POOL, LEXICAL_POOL); + + const lexByUrl = new Map(lexHits.map((h) => [normalizeUrl(h.url), h])); + const lexUrls = lexHits.map((h) => normalizeUrl(h.url)); + + const fused = weightedRrf([ + { urls: vecUrls, weight: this.vectorWeight }, + { urls: lexUrls, weight: 1 }, + ]); + + const ranked = [...fused.entries()].sort((a, b) => b[1] - a[1]); + + const out: SearchHit[] = []; + for (const [url, score] of ranked) { + // Page-type filter: lexical hits are pre-filtered; a vector-only url must + // be checked against the page's type here. + const lex = lexByUrl.get(url); + let hit: SearchHit | undefined; + if (lex) { + hit = { ...lex, score: Number(score.toFixed(4)) }; + } else { + hit = this.index.hitForUrl(url, query, score); + } + if (!hit) continue; + if (opts.pageType && hit.page_type !== opts.pageType) continue; + out.push(hit); + if (out.length >= limit) break; + } + return out; + } +} diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index 21fbcd51ec..a1e2020af6 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -7,7 +7,9 @@ import { import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; import { loadFeed } from "./feed.js"; -import { DocsIndex } from "./search.js"; +import { DocsIndex, type Searcher } from "./search.js"; +import { HybridSearcher } from "./hybrid.js"; +import { VectorStore } from "./vector-store.js"; import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; import { getPage, GetPageInput } from "./tools/get-page.js"; import { toolResult, fail } from "./response.js"; @@ -16,6 +18,12 @@ import { toolResult, fail } from "./response.js"; const FEED_SOURCE = process.env.DOCS_NDJSON ?? "https://redis.io/docs/latest/docs.ndjson"; +// Hosted hybrid mode activates only when a Redis backend is configured. Without +// REDIS_URL the server stays lexical-only, which needs no datastore and runs +// client-side over stdio (SPEC §6). The vector index must be pre-loaded by +// `npm run load-index` against the same REDIS_URL. +const REDIS_URL = process.env.REDIS_URL; + const TOOLS = [ { name: "search_docs", @@ -69,6 +77,19 @@ async function main() { // Log to stderr — stdout is reserved for the MCP protocol. console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); + // search_docs backend: hybrid when Redis is configured, else lexical-only. + // get_page always uses the lexical index (feed lookup, no ranking). + let searcher: Searcher = index; + if (REDIS_URL) { + const store = new VectorStore(REDIS_URL); + await store.connect(); + await store.ensureIndex(); + searcher = new HybridSearcher(index, store); + console.error(`[redis-docs-mcp] hybrid mode: vector KNN via ${REDIS_URL}`); + } else { + console.error("[redis-docs-mcp] lexical-only mode (no REDIS_URL)"); + } + const server = new Server( { name: "redis-docs-mcp", version: "0.0.1" }, { capabilities: { tools: {} } }, @@ -81,7 +102,7 @@ async function main() { try { switch (name) { case "search_docs": - return toolResult(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + return toolResult(await searchDocs(searcher, SearchDocsInput.parse(args ?? {}))); case "get_page": return toolResult(getPage(index, GetPageInput.parse(args ?? {}))); default: diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index 1dff08c72f..19834f158c 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -104,6 +104,12 @@ export interface SearchHit { matching_section_ids: string[]; } +/** A ranking backend for search_docs. DocsIndex (lexical) is sync; the hosted + * HybridSearcher is async — the tool awaits either. */ +export interface Searcher { + search(query: string, opts?: SearchOptions): SearchHit[] | Promise<SearchHit[]>; +} + export class DocsIndex { readonly pages: Page[]; // The feed's `id` is the last URL path segment (e.g. "config", "acl") and is @@ -181,6 +187,29 @@ export class DocsIndex { return this.pages.filter((p) => normalizeUrl(p.url).endsWith(anchored)); } + /** + * Build a SearchHit for a page by url, for hybrid fusion — a page surfaced by + * vector KNN may not appear in the lexical results, so it has no hit yet. The + * caller supplies the fused score; matching sections are computed from the + * query with the same analyzer as search(). + */ + hitForUrl(url: string, query: string, score: number): SearchHit | undefined { + const p = this.getByUrl(url); + if (!p) return undefined; + const raw = [...new Set(tokenize(query))].filter((t) => !STOPWORDS.has(t)); + const base = raw.length ? raw : [...new Set(tokenize(query))]; + const qset = new Set(base.map(stem)); + return { + id: p.id, + title: p.title, + url: p.url, + summary: p.summary ?? "", + page_type: p.page_type ?? "content", + score: Number(score.toFixed(4)), + matching_section_ids: matchingSections(p, qset), + }; + } + search(query: string, opts: SearchOptions = {}): SearchHit[] { // Filter stopwords on RAW tokens (before stemming), then stem + dedupe. const raw = [...new Set(tokenize(query))]; diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index fedc6c450d..996d119220 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -26,7 +26,7 @@ const index = new DocsIndex(pages); console.log(`loaded ${index.size} pages from ${feed}\n`); // --- search_docs --- -const stream = searchDocs(index, { query: "append an entry to a stream" }); +const stream = await searchDocs(index, { query: "append an entry to a stream" }); check("search returns hits", stream.count > 0); check("search hits carry a url", Boolean(stream.results[0]?.url)); diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts index 08f1574feb..e17089a356 100644 --- a/build/docs-mcp-server/node/src/tools/search-docs.ts +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import type { DocsIndex } from "../search.js"; +import type { Searcher } from "../search.js"; export const SearchDocsInput = z.object({ query: z.string().min(1, "query is required"), @@ -8,9 +8,10 @@ export const SearchDocsInput = z.object({ }); export type SearchDocsInput = z.infer<typeof SearchDocsInput>; -/** Rank pages by relevance. Returns refs + summaries only — never full text. */ -export function searchDocs(index: DocsIndex, input: SearchDocsInput) { - const results = index.search(input.query, { +/** Rank pages by relevance. Returns refs + summaries only — never full text. + * The searcher may be lexical (sync) or hybrid (async), so this awaits. */ +export async function searchDocs(searcher: Searcher, input: SearchDocsInput) { + const results = await searcher.search(input.query, { limit: input.limit ?? 10, pageType: input.page_type, }); diff --git a/build/docs-mcp-server/node/src/vector-store.ts b/build/docs-mcp-server/node/src/vector-store.ts new file mode 100644 index 0000000000..6ceedbccd7 --- /dev/null +++ b/build/docs-mcp-server/node/src/vector-store.ts @@ -0,0 +1,116 @@ +// Redis vector-KNN backend for hybrid mode. Redis is used ONLY for vector +// search (DOC-6809 Step 3 verdict: keep BM25 lexical + weighted-RRF fusion +// app-side; native FT.HYBRID can't express the weighted recipe). Section chunks +// are stored as HASHes with a FLAT/COSINE vector field; `owner` (page url) is +// unindexed and pulled back via RETURN. FLAT = exact KNN, which reproduced the +// offline numpy ranking to tie-breaking in Step 1; swap to HNSW later if the +// corpus grows enough to need it. +import { + createClient, + SCHEMA_FIELD_TYPE, + SCHEMA_VECTOR_FIELD_ALGORITHM, + type RedisClientType, +} from "redis"; +import { EMBED_DIM } from "./embed.js"; + +const INDEX = "docs_vec"; +const PREFIX = "docvec:"; + +function vecBuffer(v: Float32Array): Buffer { + return Buffer.from(v.buffer, v.byteOffset, v.byteLength); +} + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +export class VectorStore { + private client: RedisClientType; + + constructor(url: string) { + this.client = createClient({ url }); + this.client.on("error", (e) => console.error("[redis-docs-mcp] redis:", e)); + } + + async connect(): Promise<void> { + if (!this.client.isOpen) await this.client.connect(); + } + + async close(): Promise<void> { + if (this.client.isOpen) await this.client.close(); + } + + /** Create the FLAT/COSINE index if absent (idempotent). */ + async ensureIndex(dim = EMBED_DIM): Promise<void> { + try { + await this.client.ft.create( + INDEX, + { + vec: { + type: SCHEMA_FIELD_TYPE.VECTOR, + ALGORITHM: SCHEMA_VECTOR_FIELD_ALGORITHM.FLAT, + TYPE: "FLOAT32", + DIM: dim, + DISTANCE_METRIC: "COSINE", + }, + }, + { ON: "HASH", PREFIX }, + ); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (!/index already exists/i.test(msg)) throw e; + } + } + + async dropIndex(): Promise<void> { + try { + await this.client.ft.dropIndex(INDEX, { DD: true }); + } catch { + // no index — nothing to drop + } + } + + /** Bulk-load chunk vectors. Each chunk becomes one HASH {owner, vec}. */ + async loadChunks(chunks: Array<{ owner: string; vec: Float32Array }>): Promise<void> { + const BATCH = 2000; + for (let i = 0; i < chunks.length; i += BATCH) { + const slice = chunks.slice(i, i + BATCH); + await Promise.all( + slice.map((c, j) => + this.client.hSet(`${PREFIX}${i + j}`, { + owner: c.owner, + vec: vecBuffer(c.vec), + }), + ), + ); + } + } + + /** + * KNN over `k` chunks, deduped to the nearest chunk per page. Returns page + * urls, nearest first (mirrors the offline rank_pages()). + */ + async knn(qvec: Float32Array, k = 200, topn = 50): Promise<string[]> { + const res = await this.client.ft.search( + INDEX, + `*=>[KNN ${k} @vec $BLOB AS score]`, + { + PARAMS: { BLOB: vecBuffer(qvec) }, + SORTBY: { BY: "score", DIRECTION: "ASC" }, // cosine distance: nearest = smallest + RETURN: ["owner"], + LIMIT: { from: 0, size: k }, + DIALECT: 2, + }, + ); + const seen = new Set<string>(); + const pages: string[] = []; + for (const doc of res.documents) { + const owner = normalizeUrl(String((doc.value as { owner?: string }).owner ?? "")); + if (!owner || seen.has(owner)) continue; + seen.add(owner); + pages.push(owner); + if (pages.length >= topn) break; + } + return pages; + } +} diff --git a/build/docs-mcp-server/node/test/eval/dump-lexical.mjs b/build/docs-mcp-server/node/test/eval/dump-lexical.mjs index a68eda18e4..678b603410 100644 --- a/build/docs-mcp-server/node/test/eval/dump-lexical.mjs +++ b/build/docs-mcp-server/node/test/eval/dump-lexical.mjs @@ -16,11 +16,15 @@ const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", im const index = new DocsIndex(await loadFeed(feedSrc)); -const out = cases.map((c) => ({ - q: c.q, - kind: c.kind ?? "command", - expected: c.expected.map(norm), - lexical: searchDocs(index, { query: c.q, limit: TOPN }).results.map((r) => norm(r.url)), -})); +const out = []; +for (const c of cases) { + const { results } = await searchDocs(index, { query: c.q, limit: TOPN }); + out.push({ + q: c.q, + kind: c.kind ?? "command", + expected: c.expected.map(norm), + lexical: results.map((r) => norm(r.url)), + }); +} process.stdout.write(JSON.stringify(out, null, 2) + "\n"); diff --git a/build/docs-mcp-server/node/test/eval/run-hybrid.mjs b/build/docs-mcp-server/node/test/eval/run-hybrid.mjs new file mode 100644 index 0000000000..861c5d5218 --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/run-hybrid.mjs @@ -0,0 +1,95 @@ +// Hybrid retrieval eval (DOC-6809 Step 4 acceptance). Runs the 35 cases through +// the REAL search_docs tool function backed by HybridSearcher — the exact path +// index.ts calls — so this exercises query embedding (fastembed-js) + Redis KNN +// + app-side weighted RRF end to end. Confirms the shipped hybrid mode +// reproduces the offline recipe (overall MRR ~.73). +// +// Prereq: load the vector index first, against the same REDIS_URL: +// REDIS_URL=redis://localhost:6379 node scripts/load-index.mjs --vectors ../redis-eval/vecdump +// REDIS_URL=redis://localhost:6379 node test/eval/run-hybrid.mjs +// +// Imports the BUILT server (dist/) — run `npm run build` first (eval:hybrid does). +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { loadFeed } from "../../dist/feed.js"; +import { DocsIndex } from "../../dist/search.js"; +import { HybridSearcher } from "../../dist/hybrid.js"; +import { VectorStore } from "../../dist/vector-store.js"; +import { searchDocs } from "../../dist/tools/search-docs.js"; + +const K = [1, 3, 5, 10]; +const LIMIT = 10; +const REDIS_URL = process.env.REDIS_URL ?? "redis://localhost:6379"; +const norm = (u) => u.trim().toLowerCase().replace(/\/+$/, ""); +const short = (u) => (u ? u.replace("https://redis.io/docs/latest", "") : "—"); + +const feedSrc = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./docs.ndjson.gz", import.meta.url)); +const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", import.meta.url)), "utf8")); + +const pages = await loadFeed(feedSrc); +const index = new DocsIndex(pages); +const feedUrls = new Set(pages.map((p) => norm(p.url))); + +const store = new VectorStore(REDIS_URL); +await store.connect(); +await store.ensureIndex(); +const hybrid = new HybridSearcher(index, store); + +const broken = []; +const rows = []; +for (const c of cases) { + const expected = c.expected.map(norm); + if (expected.every((u) => !feedUrls.has(u))) { + broken.push({ q: c.q, missing: expected }); + continue; + } + const { results } = await searchDocs(hybrid, { query: c.q, limit: LIMIT }); + const urls = results.map((r) => norm(r.url)); + let rank = null; + for (let i = 0; i < urls.length; i++) { + if (expected.includes(urls[i])) { + rank = i + 1; + break; + } + } + rows.push({ kind: c.kind ?? "command", q: c.q, rank, top: urls[0] }); +} +await store.close(); + +function metrics(set) { + const n = set.length || 1; + const recall = Object.fromEntries( + K.map((k) => [k, set.filter((r) => r.rank && r.rank <= k).length / n]), + ); + const mrr = set.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / n; + return { recall, mrr }; +} + +console.log( + `Hybrid via ${REDIS_URL} | ${pages.length} pages | cases scored: ${rows.length}` + + (broken.length ? ` | ${broken.length} BROKEN` : "") + + "\n", +); +for (const r of rows) { + const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; + console.log(`${tag} [${r.kind.slice(0, 4)}] ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); +} + +const groups = [ + ["overall", rows], + ["command", rows.filter((r) => r.kind === "command")], + ["concept", rows.filter((r) => r.kind === "concept")], +]; +console.log("\n--- hybrid retrieval quality (recall@1 / @3 / @5 / @10 | MRR) ---"); +for (const [label, set] of groups) { + if (!set.length) continue; + const m = metrics(set); + const cells = K.map((k) => `${(m.recall[k] * 100).toFixed(0)}%`.padStart(4)).join(" / "); + console.log(`${label.padEnd(8)} (n=${String(set.length).padStart(2)}): ${cells} | ${m.mrr.toFixed(3)}`); +} + +if (broken.length) { + console.log("\n--- BROKEN eval cases ---"); + for (const b of broken) console.log(` "${b.q}" -> ${b.missing.map(short).join(", ")}`); +} diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs index 6c8da79419..b59fbaf882 100644 --- a/build/docs-mcp-server/node/test/eval/run.mjs +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -35,7 +35,7 @@ for (const c of cases) { broken.push({ q: c.q, missing: expected }); continue; } - const results = searchDocs(index, { query: c.q, limit: LIMIT }).results.map((r) => norm(r.url)); + const results = (await searchDocs(index, { query: c.q, limit: LIMIT })).results.map((r) => norm(r.url)); let rank = null; for (let i = 0; i < results.length; i++) { if (expected.includes(results[i])) { diff --git a/build/docs-mcp-server/redis-eval/.gitignore b/build/docs-mcp-server/redis-eval/.gitignore index 274f9886e8..44746405a4 100644 --- a/build/docs-mcp-server/redis-eval/.gitignore +++ b/build/docs-mcp-server/redis-eval/.gitignore @@ -2,3 +2,5 @@ parity_texts.json node_vectors.json __pycache__/ +# Portable vector dump for the Step 4 loader seed (dump_vectors_bin.py) +vecdump/ diff --git a/build/docs-mcp-server/redis-eval/README.md b/build/docs-mcp-server/redis-eval/README.md index d116f1a99f..30436bd804 100644 --- a/build/docs-mcp-server/redis-eval/README.md +++ b/build/docs-mcp-server/redis-eval/README.md @@ -146,11 +146,22 @@ native RRF still unable to do the weighted fusion. The conclusion holds either way. (The eval's vector side here is numpy `rank_pages`; Step 1 already showed Redis KNN ≈ numpy to tie-breaking, so this isolates fusion + lexical cleanly.) -## Next - -- **Step 4 — wire the winning path into the MCP `search_docs` handler:** Redis - KNN for vector + the existing Node BM25, fused with weighted RRF (vector ~3×). - This is the first change to the shipped server; scope with Andy first. +## Step 4 — wired into the MCP server ✅ (vertical slice) + +Implemented in `../node/` behind a `REDIS_URL` feature flag: hybrid `search_docs` += app-side BM25 + Redis vector KNN, fused with weighted RRF (vector 3×), query +embedded in-process via fastembed-js. Loader `node/scripts/load-index.mjs` builds +the `docs_vec` FLAT/COSINE index (embeds with fastembed-js, or `--vectors +<dir>` to seed from `dump_vectors_bin.py` output — used here to skip a ~1h local +re-embed, valid because Step 2 proved Node ≡ Python vectors). + +Live eval through the real `search_docs` tool (`npm run eval:hybrid`): overall +MRR **.704**, command **.783**, concept **.570** (concept @10 **100%**) — within +tie-break noise of the offline .731 (stacks the Step 1 KNN + Step 2 embedding +tie-breaks). Lexical-only default is unchanged (overall .525) and smoke passes. +Payoff: hybrid lifts overall MRR **.53 → .70**, concept @10 **77% → 100%**. See +`../node/README.md` (Hybrid mode). Stopped here for review — no HTTP endpoint / +rate-limiting / deploy yet. ## Run diff --git a/build/docs-mcp-server/redis-eval/dump_vectors_bin.py b/build/docs-mcp-server/redis-eval/dump_vectors_bin.py new file mode 100644 index 0000000000..551903d3fe --- /dev/null +++ b/build/docs-mcp-server/redis-eval/dump_vectors_bin.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""Dump the cached bge-small section vectors to a portable binary the Node +build-index loader can seed from (--vectors), so the vertical-slice eval need +not re-embed 15k chunks in Node at ~4/s. Step 2 proved Node fastembed-js == these +Python vectors (cosine 1.0), so seeding is equivalent to the Node embed path. + +Writes to redis-eval/vecdump/: + meta.json {n, dim} + owners.json [url, ...] (aligned to rows) + vectors.f32 raw little-endian float32, n*dim + + ../vector-eval/.venv/bin/python dump_vectors_bin.py section +""" +import json +import os +import sys + +import numpy as np + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "vector-eval")) +import eval_vector as ev # noqa: E402 + +pages = ev.load_pages() +texts, owners = ev.build_chunks(pages, ev.MODE) +emb, owners = ev.get_corpus_embeddings(texts, owners) +emb = np.ascontiguousarray(emb, dtype="<f4") # little-endian float32 + +out = os.path.join(HERE, "vecdump") +os.makedirs(out, exist_ok=True) +emb.tofile(os.path.join(out, "vectors.f32")) +with open(os.path.join(out, "owners.json"), "w", encoding="utf-8") as f: + json.dump([str(o) for o in owners], f) +with open(os.path.join(out, "meta.json"), "w", encoding="utf-8") as f: + json.dump({"n": int(emb.shape[0]), "dim": int(emb.shape[1])}, f) +print(f"wrote {out}: n={emb.shape[0]} dim={emb.shape[1]}") From a8b273120bdd38dccef82cb14f810d5a3a54d293 Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 11:18:42 +0100 Subject: [PATCH 20/25] DOC-6809 Address Bugbot: empty-index guard + defer matchingSections past top-k MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two safe fixes from the Bugbot review of the hybrid work. (1) main() now throws if loadFeed returns zero pages, instead of advertising readiness over an empty index and then serving only empty results / not-found errors (Medium 3542934577). (2) search() scores into an interim array and defers the per-section matchingSections() re-analysis until after the top-k slice, so it runs for the ~10 returned pages rather than every positive-score page in the corpus (Low 3638621205). Verified the search refactor preserves ranking exactly: lexical eval still overall MRR .525 / command .571, hybrid still .704 / .783, smoke all green. Those eval numbers are the regression oracle for any change to the scoring loop. Learned: the 35-case eval (npm run eval / eval:hybrid) is the regression oracle for search() changes — lexical .525 / hybrid .704 must still hold after touching the scoring loop Directive: matchingSections is display-only metadata and must not influence ranking — compute it after sort+slice, never let it move results Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- build/docs-mcp-server/node/src/index.ts | 7 ++++++ build/docs-mcp-server/node/src/search.ts | 28 +++++++++++++----------- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index a1e2020af6..2e20b9cd12 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -73,6 +73,13 @@ const TOOLS = [ async function main() { const pages = await loadFeed(FEED_SOURCE); + // Refuse to start on an empty index (empty file, bad path, or no valid NDJSON + // lines) rather than advertising readiness and serving only empty results. + if (pages.length === 0) { + throw new Error( + `No documents loaded from ${FEED_SOURCE} — refusing to start with an empty index.`, + ); + } const index = new DocsIndex(pages); // Log to stderr — stdout is reserved for the MCP protocol. console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index 19834f158c..4b153292ce 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -225,7 +225,10 @@ export class DocsIndex { } const qset = new Set(qterms); - const hits: SearchHit[] = []; + // Score first; defer the expensive per-section matchingSections() until + // after the top-k slice, so we only re-analyze section text for the handful + // of pages we actually return, not every positive-score page in the corpus. + const scored: Array<{ page: Page; score: number }> = []; for (const d of this.docs) { if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; @@ -243,19 +246,18 @@ export class DocsIndex { if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; } if (score > 0) { - score *= pageWeight(d.page.url); - hits.push({ - id: d.page.id, - title: d.page.title, - url: d.page.url, - summary: d.page.summary ?? "", - page_type: d.page.page_type ?? "content", - score: Number(score.toFixed(4)), - matching_section_ids: matchingSections(d.page, qset), - }); + scored.push({ page: d.page, score: Number((score * pageWeight(d.page.url)).toFixed(4)) }); } } - hits.sort((a, b) => b.score - a.score); - return hits.slice(0, opts.limit ?? 10); + scored.sort((a, b) => b.score - a.score); + return scored.slice(0, opts.limit ?? 10).map(({ page, score }) => ({ + id: page.id, + title: page.title, + url: page.url, + summary: page.summary ?? "", + page_type: page.page_type ?? "content", + score, + matching_section_ids: matchingSections(page, qset), + })); } } From defc9662ae919e20712fbecf28dd10c1ae2dbb6c Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 11:36:43 +0100 Subject: [PATCH 21/25] DOC-6809 Address Bugbot: dep-free default path + single normalizeUrl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tidies the Step-4 hybrid module boundary in response to two Bugbot findings, both about the feature-flag boundary not being clean. (1) index.ts no longer statically imports HybridSearcher/VectorStore — it `await import()`s them inside the `if (REDIS_URL)` block, so the default lexical-only stdio mode no longer loads fastembed (native onnxruntime) or the redis client at startup. This restores the invariant the Step-4 note claimed but the implementation broke: the no-REDIS_URL path is genuinely dependency-free, and can't crash on a platform lacking the native ONNX binary when it isn't even doing vector search (Medium 3644667453). (2) The normalizeUrl helper, previously copy-pasted byte-identically into search.ts, chunk.ts, hybrid.ts and vector-store.ts, is now a single export in url.ts — the hybrid path matches pages across those modules by normalized url, so a divergent copy would silently break cross-module findability (Low 3644535220). Lexical .525 / hybrid .704 and smoke unchanged after the refactor. Learned: my own Step-4 static imports violated the dep-free-default-path constraint I'd documented — a feature flag gates *use* but a static import still loads the module graph; only dynamic import actually defers the cost Constraint: the no-REDIS_URL lexical path must not statically import hybrid.ts/vector-store.ts/embed.ts (they pull in native onnxruntime + redis) — load them via dynamic import inside the REDIS_URL branch only Constraint: normalizeUrl has ONE definition (url.ts); the chunker, vector store, hybrid fusion and lexical index all match pages by its output, so a second copy that drifts breaks findability silently Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- build/docs-mcp-server/node/src/chunk.ts | 5 +---- build/docs-mcp-server/node/src/hybrid.ts | 5 +---- build/docs-mcp-server/node/src/index.ts | 8 ++++++-- build/docs-mcp-server/node/src/search.ts | 5 +---- build/docs-mcp-server/node/src/url.ts | 8 ++++++++ build/docs-mcp-server/node/src/vector-store.ts | 5 +---- 6 files changed, 18 insertions(+), 18 deletions(-) create mode 100644 build/docs-mcp-server/node/src/url.ts diff --git a/build/docs-mcp-server/node/src/chunk.ts b/build/docs-mcp-server/node/src/chunk.ts index c5bd53f9e5..1c95be9ed0 100644 --- a/build/docs-mcp-server/node/src/chunk.ts +++ b/build/docs-mcp-server/node/src/chunk.ts @@ -7,6 +7,7 @@ // - owner of every chunk is the page's normalized url // Section-level chunking is what fixed the concept-query gap (DOC-6809 SPEC §10). import type { Page } from "./types.js"; +import { normalizeUrl } from "./url.js"; const LEAD_CHARS = 1200; const MAX_SECTIONS = 8; @@ -16,10 +17,6 @@ export interface Chunk { owner: string; // normalized page url } -function normalizeUrl(u: string): string { - return u.trim().toLowerCase().replace(/\/+$/, ""); -} - export function buildChunks(pages: Page[]): Chunk[] { const chunks: Chunk[] = []; for (const p of pages) { diff --git a/build/docs-mcp-server/node/src/hybrid.ts b/build/docs-mcp-server/node/src/hybrid.ts index 6172607518..3527264ab5 100644 --- a/build/docs-mcp-server/node/src/hybrid.ts +++ b/build/docs-mcp-server/node/src/hybrid.ts @@ -8,16 +8,13 @@ import type { DocsIndex, SearchHit, SearchOptions } from "./search.js"; import type { VectorStore } from "./vector-store.js"; import { embedQuery } from "./embed.js"; +import { normalizeUrl } from "./url.js"; const RRF_K = 60; const DEFAULT_VECTOR_WEIGHT = 3; const LEXICAL_POOL = 50; // lexical candidates fused const VECTOR_POOL = 200; // vector chunks fetched (deduped to <=50 pages) -function normalizeUrl(u: string): string { - return u.trim().toLowerCase().replace(/\/+$/, ""); -} - /** Weighted reciprocal-rank fusion. Returns url -> fused score. */ function weightedRrf( lists: Array<{ urls: string[]; weight: number }>, diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index 2e20b9cd12..adc02e2a30 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -8,8 +8,6 @@ import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" import { loadFeed } from "./feed.js"; import { DocsIndex, type Searcher } from "./search.js"; -import { HybridSearcher } from "./hybrid.js"; -import { VectorStore } from "./vector-store.js"; import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; import { getPage, GetPageInput } from "./tools/get-page.js"; import { toolResult, fail } from "./response.js"; @@ -88,6 +86,12 @@ async function main() { // get_page always uses the lexical index (feed lookup, no ranking). let searcher: Searcher = index; if (REDIS_URL) { + // Load the hybrid path lazily: it pulls in fastembed (native onnxruntime) + // and the redis client, which the default lexical-only stdio mode never + // needs and which would otherwise crash startup on platforms lacking the + // native ONNX binary. Keep the no-REDIS_URL path dependency-free. + const { VectorStore } = await import("./vector-store.js"); + const { HybridSearcher } = await import("./hybrid.js"); const store = new VectorStore(REDIS_URL); await store.connect(); await store.ensureIndex(); diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index 4b153292ce..2b1bb580e3 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -1,6 +1,7 @@ import type { Page } from "./types.js"; import { stem as stemPorter } from "./stem.js"; import { stem as stemPaice } from "./stem-paice.js"; +import { normalizeUrl } from "./url.js"; // Stemmer is switchable for the eval bake-off (STEMMER=paice|porter). Porter is // the default/shipped analyzer. @@ -60,10 +61,6 @@ function analyze(text: string): string[] { return tokenize(text).map(stem); } -function normalizeUrl(u: string): string { - return u.trim().toLowerCase().replace(/\/+$/, ""); -} - /** Everything worth matching against for a page: slug, title, summary, section text. */ function searchableText(p: Page): string { const parts: string[] = [p.id ?? "", p.title ?? "", p.summary ?? ""]; diff --git a/build/docs-mcp-server/node/src/url.ts b/build/docs-mcp-server/node/src/url.ts new file mode 100644 index 0000000000..04516dac8e --- /dev/null +++ b/build/docs-mcp-server/node/src/url.ts @@ -0,0 +1,8 @@ +// Canonical URL normalization, shared across the lexical index, chunker, vector +// store, and hybrid fusion. It MUST be the single definition: the hybrid path +// matches pages across modules by normalized url (chunk.ts indexes by it, +// vector-store.ts returns it, hybrid.ts fuses on it, search.ts looks up by it), +// so any divergence would silently break cross-module findability. +export function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} diff --git a/build/docs-mcp-server/node/src/vector-store.ts b/build/docs-mcp-server/node/src/vector-store.ts index 6ceedbccd7..f50716167a 100644 --- a/build/docs-mcp-server/node/src/vector-store.ts +++ b/build/docs-mcp-server/node/src/vector-store.ts @@ -12,6 +12,7 @@ import { type RedisClientType, } from "redis"; import { EMBED_DIM } from "./embed.js"; +import { normalizeUrl } from "./url.js"; const INDEX = "docs_vec"; const PREFIX = "docvec:"; @@ -20,10 +21,6 @@ function vecBuffer(v: Float32Array): Buffer { return Buffer.from(v.buffer, v.byteOffset, v.byteLength); } -function normalizeUrl(u: string): string { - return u.trim().toLowerCase().replace(/\/+$/, ""); -} - export class VectorStore { private client: RedisClientType; From dbd7987fe9c46aebf821c0d0e3271231fd297fb1 Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 13:02:50 +0100 Subject: [PATCH 22/25] DOC-6809 Address Bugbot: fix Float32Array alignment in load-index seed path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The --vectors seed path built a Float32Array view directly over the readFile Buffer (`new Float32Array(buf.buffer, buf.byteOffset, ...)`). A pooled Node Buffer's byteOffset isn't guaranteed 4-byte aligned, and an unaligned offset makes the typed-array view throw RangeError. It worked in every test only because our ~23MB vectors.f32 gets a dedicated allocation at offset 0 — not a contract. Now copies into a fresh 0-offset ArrayBuffer first (Low 3644788302). Verified: loader still seeds 15300 vectors and hybrid eval holds at MRR .704. Learned: a Float32Array view over a Node Buffer can throw on a non-4-aligned pooled byteOffset — large files hide it (offset 0); copy into a fresh ArrayBuffer before viewing, don't trust the direct view Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- build/docs-mcp-server/node/scripts/load-index.mjs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/build/docs-mcp-server/node/scripts/load-index.mjs b/build/docs-mcp-server/node/scripts/load-index.mjs index b320635e92..a36ee8a8ea 100644 --- a/build/docs-mcp-server/node/scripts/load-index.mjs +++ b/build/docs-mcp-server/node/scripts/load-index.mjs @@ -34,7 +34,11 @@ async function fromSeed(dir) { const meta = JSON.parse(await readFile(new URL("meta.json", base), "utf8")); const owners = JSON.parse(await readFile(new URL("owners.json", base), "utf8")); const buf = await readFile(new URL("vectors.f32", base)); - const floats = new Float32Array(buf.buffer, buf.byteOffset, buf.byteLength / 4); + // Copy into a fresh 0-offset ArrayBuffer before the Float32Array view: a + // pooled Node Buffer's byteOffset isn't guaranteed 4-byte aligned, and an + // unaligned offset makes `new Float32Array(buffer, offset)` throw RangeError. + const ab = buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength); + const floats = new Float32Array(ab); const { n, dim } = meta; const chunks = []; for (let i = 0; i < n; i++) { From 2bdf8549b0ef5a2d57f3eae45d1952e45ec36f39 Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 13:36:22 +0100 Subject: [PATCH 23/25] DOC-6809 Address Bugbot: retry-safe embedder init + loader empty guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two Low findings from Bugbot's round-4 re-scan, both in Step-4 code. (1) embed.ts memoized the FlagEmbedding.init() promise with `??=`, which permanently caches a *rejected* init — a transient first-run failure (model download timeout, fs perms) would poison the singleton and break hybrid mode until process restart. Now clears the cache on rejection so the next embedQuery retries (3645222048). (2) load-index.mjs accessed chunks[0].vec.length with no empty guard, throwing an opaque "Cannot read properties of undefined" on an empty/invalid feed; now it fails with a clear message, and fromEmbed() bails before loading native ONNX to embed nothing — which also removes an ugly libc++abi teardown crash on the empty path (3645222055). Verified: smoke green, seed path still loads 15300 vectors, hybrid MRR .704 unchanged, empty-feed now errors cleanly on both embed and seed paths. Learned: a promise memoized with `??=` caches rejections too — for lazy init of a fallible singleton (model load, DB connect) always clear the cache in .catch() so a transient failure doesn't permanently wedge the feature Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- .../docs-mcp-server/node/scripts/load-index.mjs | 10 ++++++++++ build/docs-mcp-server/node/src/embed.ts | 16 +++++++++++++--- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/build/docs-mcp-server/node/scripts/load-index.mjs b/build/docs-mcp-server/node/scripts/load-index.mjs index a36ee8a8ea..e9d70b8b2e 100644 --- a/build/docs-mcp-server/node/scripts/load-index.mjs +++ b/build/docs-mcp-server/node/scripts/load-index.mjs @@ -51,6 +51,9 @@ async function fromSeed(dir) { async function fromEmbed() { const pages = await loadFeed(FEED); const chunks = buildChunks(pages); + // Bail before loading the embedder: no point spinning up native ONNX to embed + // nothing (main() guards too, but this avoids the wasted model init on empty). + if (chunks.length === 0) return chunks; console.error(`[load-index] ${pages.length} pages -> ${chunks.length} chunks; embedding (fastembed-js) ...`); const vecs = await embedPassages(chunks.map((c) => c.text)); return chunks.map((c, i) => ({ owner: c.owner, vec: vecs[i] })); @@ -59,6 +62,13 @@ async function fromEmbed() { async function main() { const seedDir = arg("--vectors"); const chunks = seedDir ? await fromSeed(seedDir) : await fromEmbed(); + // Guard before chunks[0]: an empty/invalid feed or seed would otherwise throw + // an opaque "Cannot read properties of undefined" on the ensureIndex line. + if (chunks.length === 0) { + throw new Error( + `No chunks to load from ${seedDir ?? FEED} — aborting (empty or invalid source).`, + ); + } const store = new VectorStore(REDIS_URL); await store.connect(); diff --git a/build/docs-mcp-server/node/src/embed.ts b/build/docs-mcp-server/node/src/embed.ts index 0c81b95fb6..383afce6a1 100644 --- a/build/docs-mcp-server/node/src/embed.ts +++ b/build/docs-mcp-server/node/src/embed.ts @@ -11,9 +11,19 @@ const QUERY_PREFIX = "Represent this sentence for searching relevant passages: " let modelPromise: Promise<FlagEmbedding> | null = null; function model(): Promise<FlagEmbedding> { - return (modelPromise ??= FlagEmbedding.init({ - model: EmbeddingModel.BGESmallENV15, - })); + // Memoize the loaded model, but DON'T cache a rejected init: a transient + // failure (first-run download timeout, fs perms) must not poison the singleton + // and break hybrid mode until restart. Clear the cache on failure so the next + // call retries. + if (!modelPromise) { + modelPromise = FlagEmbedding.init({ model: EmbeddingModel.BGESmallENV15 }).catch( + (e) => { + modelPromise = null; + throw e; + }, + ); + } + return modelPromise; } function l2normalize(v: number[]): Float32Array { From 0df2793c34eec2e8b4c78f2af0fa268dd7e54d28 Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 13:47:10 +0100 Subject: [PATCH 24/25] DOC-6809 Harden hybrid mode from Codex review: resilience, lifecycle, validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independent Codex review of the Step-4 hybrid code surfaced a deeper class of issue than Bugbot's line-level drip — resource lifecycle, error suppression, resilience, input validation. Fixes the four highest-value findings; two production-grade items (atomic index swap on reload; FT.INFO schema/population verification at startup) are left as in-code TODOs, deferred as beyond the offline single-instance prototype. - Graceful degradation (hybrid.ts): a transient embedding or Redis KNN failure no longer fails search_docs — it logs and returns the already-computed lexical results. Hybrid is an enhancement over a working lexical base, so it should fall back to it, not collapse. - Redis lifecycle (index.ts): the store is now closed on SIGINT/SIGTERM and on connection close, and on startup failure — previously the open socket could keep the stdio process alive after the client disconnected. - dropIndex error scoping (vector-store.ts): only the "index doesn't exist" error is swallowed; auth/network/permission errors propagate instead of being silently ignored and letting the loader reuse a stale index. - Seed validation (load-index.mjs): the --vectors seed is checked for dim == EMBED_DIM, owners length == n, and byteLength == n*dim*4 before loading, so a truncated/mismatched dump fails loudly instead of "loading" and being silently misindexed by Redis. - EMBED_DIM moved to a dep-free constants.ts so vector-store and the loader's seed path can validate the dimension without importing embed.ts (which pulls in fastembed's native ONNX runtime) — same dep-free-default-path principle as the dynamic-import fix. Verified: smoke green, lexical MRR .525 and hybrid .704 both unchanged, validated seed still loads 15300 vectors. Learned: the independent Codex sweep caught a whole class Bugbot's per-line scan missed (lifecycle/resilience/validation) in one pass — worth running once on a large new subsystem rather than waiting out the bot's round-by-round drip Constraint: hybrid search MUST degrade to lexical-only when the vector path (embedding or Redis KNN) fails — never fail search_docs when lexical results exist Constraint: the Redis backend must be closed on shutdown (signals + connection close) so a live socket can't keep the stdio server alive after disconnect Directive: EMBED_DIM lives in constants.ts (dep-free) — import it there, not from embed.js, in any module that must stay off the fastembed/native-ONNX graph (vector-store, loader seed path) Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- .../node/scripts/load-index.mjs | 24 +++++++++++++- build/docs-mcp-server/node/src/constants.ts | 6 ++++ build/docs-mcp-server/node/src/embed.ts | 3 +- build/docs-mcp-server/node/src/hybrid.ts | 19 +++++++++-- build/docs-mcp-server/node/src/index.ts | 33 +++++++++++++++++-- .../docs-mcp-server/node/src/vector-store.ts | 10 ++++-- 6 files changed, 85 insertions(+), 10 deletions(-) create mode 100644 build/docs-mcp-server/node/src/constants.ts diff --git a/build/docs-mcp-server/node/scripts/load-index.mjs b/build/docs-mcp-server/node/scripts/load-index.mjs index e9d70b8b2e..459f46ae11 100644 --- a/build/docs-mcp-server/node/scripts/load-index.mjs +++ b/build/docs-mcp-server/node/scripts/load-index.mjs @@ -17,6 +17,7 @@ import { loadFeed } from "../dist/feed.js"; import { buildChunks } from "../dist/chunk.js"; import { embedPassages } from "../dist/embed.js"; import { VectorStore } from "../dist/vector-store.js"; +import { EMBED_DIM } from "../dist/constants.js"; const REDIS_URL = process.env.REDIS_URL ?? "redis://localhost:6379"; const FEED = @@ -34,12 +35,27 @@ async function fromSeed(dir) { const meta = JSON.parse(await readFile(new URL("meta.json", base), "utf8")); const owners = JSON.parse(await readFile(new URL("owners.json", base), "utf8")); const buf = await readFile(new URL("vectors.f32", base)); + // Validate the seed before trusting it: mismatched/truncated vectors would + // otherwise "load" but be silently rejected or misindexed by Redis while the + // loader reports success. + const { n, dim } = meta; + if (!Number.isInteger(n) || !Number.isInteger(dim) || n <= 0 || dim <= 0) { + throw new Error(`seed meta.json invalid: n=${n} dim=${dim}`); + } + if (dim !== EMBED_DIM) { + throw new Error(`seed dim ${dim} != expected EMBED_DIM ${EMBED_DIM}`); + } + if (!Array.isArray(owners) || owners.length !== n) { + throw new Error(`seed owners length ${owners?.length} != n ${n}`); + } + if (buf.byteLength !== n * dim * 4) { + throw new Error(`seed vectors.f32 is ${buf.byteLength} bytes, expected ${n * dim * 4} (n*dim*4)`); + } // Copy into a fresh 0-offset ArrayBuffer before the Float32Array view: a // pooled Node Buffer's byteOffset isn't guaranteed 4-byte aligned, and an // unaligned offset makes `new Float32Array(buffer, offset)` throw RangeError. const ab = buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength); const floats = new Float32Array(ab); - const { n, dim } = meta; const chunks = []; for (let i = 0; i < n; i++) { chunks.push({ owner: owners[i], vec: floats.subarray(i * dim, (i + 1) * dim) }); @@ -72,6 +88,12 @@ async function main() { const store = new VectorStore(REDIS_URL); await store.connect(); + // TODO (production hardening, deferred — Codex review): this drops the live + // index before loading, so a hosted server reloading under traffic would see + // an absent/partial corpus mid-load (and a partial index if loading fails). + // For zero-downtime reloads, build under a versioned index+prefix, verify it, + // then atomically switch an alias. Fine for the current offline, + // single-instance prototype. await store.dropIndex(); await store.ensureIndex(chunks[0].vec.length); const t0 = Date.now(); diff --git a/build/docs-mcp-server/node/src/constants.ts b/build/docs-mcp-server/node/src/constants.ts new file mode 100644 index 0000000000..0aa85d977b --- /dev/null +++ b/build/docs-mcp-server/node/src/constants.ts @@ -0,0 +1,6 @@ +// Dependency-free shared constants. Kept separate from embed.ts so modules that +// only need the dimension (vector-store, the loader's seed path) don't +// transitively import fastembed and load the native ONNX runtime. + +/** bge-small-en-v1.5 embedding dimension. */ +export const EMBED_DIM = 384; diff --git a/build/docs-mcp-server/node/src/embed.ts b/build/docs-mcp-server/node/src/embed.ts index 383afce6a1..6d280e0ae3 100644 --- a/build/docs-mcp-server/node/src/embed.ts +++ b/build/docs-mcp-server/node/src/embed.ts @@ -5,6 +5,7 @@ // manually for queries — do NOT use fastembed's queryEmbed/passageEmbed, whose // built-in prefix wording differs and silently breaks parity (Step 2 finding). import { FlagEmbedding, EmbeddingModel } from "fastembed"; +import { EMBED_DIM } from "./constants.js"; const QUERY_PREFIX = "Represent this sentence for searching relevant passages: "; @@ -55,4 +56,4 @@ export async function embedPassages(texts: string[]): Promise<Float32Array[]> { return embedAll(texts); } -export const EMBED_DIM = 384; +export { EMBED_DIM }; diff --git a/build/docs-mcp-server/node/src/hybrid.ts b/build/docs-mcp-server/node/src/hybrid.ts index 3527264ab5..325e3d9807 100644 --- a/build/docs-mcp-server/node/src/hybrid.ts +++ b/build/docs-mcp-server/node/src/hybrid.ts @@ -38,10 +38,23 @@ export class HybridSearcher { async search(query: string, opts: SearchOptions = {}): Promise<SearchHit[]> { const limit = opts.limit ?? 10; - // Lexical side (already page-type filtered) + vector side, in parallel. + // Lexical side (already page-type filtered) is computed first and always + // usable. The vector side (query embedding + Redis KNN) can fail transiently + // — if it does, degrade to lexical-only rather than failing the whole tool + // call, since hybrid is meant to be an enhancement over a working lexical base. const lexHits = this.index.search(query, { limit: LEXICAL_POOL, pageType: opts.pageType }); - const qvec = await embedQuery(query); - const vecUrls = await this.store.knn(qvec, VECTOR_POOL, LEXICAL_POOL); + let vecUrls: string[]; + try { + const qvec = await embedQuery(query); + vecUrls = await this.store.knn(qvec, VECTOR_POOL, LEXICAL_POOL); + } catch (e) { + console.error( + `[redis-docs-mcp] vector search failed, returning lexical-only: ${ + e instanceof Error ? e.message : String(e) + }`, + ); + return lexHits.slice(0, limit); + } const lexByUrl = new Map(lexHits.map((h) => [normalizeUrl(h.url), h])); const lexUrls = lexHits.map((h) => normalizeUrl(h.url)); diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index adc02e2a30..f7be55b925 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -85,6 +85,9 @@ async function main() { // search_docs backend: hybrid when Redis is configured, else lexical-only. // get_page always uses the lexical index (feed lookup, no ranking). let searcher: Searcher = index; + // Cleanup for the hybrid backend (no-op in lexical-only mode). Called on + // shutdown so an open Redis socket can't keep the stdio process alive. + let closeBackend: () => Promise<void> = async () => {}; if (REDIS_URL) { // Load the hybrid path lazily: it pulls in fastembed (native onnxruntime) // and the redis client, which the default lexical-only stdio mode never @@ -93,8 +96,20 @@ async function main() { const { VectorStore } = await import("./vector-store.js"); const { HybridSearcher } = await import("./hybrid.js"); const store = new VectorStore(REDIS_URL); - await store.connect(); - await store.ensureIndex(); + try { + await store.connect(); + // TODO (deferred — Codex review): ensureIndex creates a missing index but + // accepts any existing one without checking it. Verify via FT.INFO (dim, + // metric, prefix, non-zero doc count) so a misconfig can't advertise + // hybrid over an empty/mismatched index — or explicitly fall back to + // lexical. Operator runs load-index before serving today, so deferred. + await store.ensureIndex(); + } catch (e) { + // Don't leak the socket if startup fails partway through. + await store.close().catch(() => {}); + throw e; + } + closeBackend = () => store.close(); searcher = new HybridSearcher(index, store); console.error(`[redis-docs-mcp] hybrid mode: vector KNN via ${REDIS_URL}`); } else { @@ -124,6 +139,20 @@ async function main() { } }); + // Close the Redis backend on shutdown so a live socket can't keep the process + // alive after the client disconnects (stdio EOF fires the connection close). + let shuttingDown = false; + const shutdown = async (reason: string) => { + if (shuttingDown) return; + shuttingDown = true; + console.error(`[redis-docs-mcp] shutting down (${reason})`); + await closeBackend().catch(() => {}); + process.exit(0); + }; + server.onclose = () => void shutdown("connection closed"); + process.on("SIGINT", () => void shutdown("SIGINT")); + process.on("SIGTERM", () => void shutdown("SIGTERM")); + const transport = new StdioServerTransport(); await server.connect(transport); console.error("[redis-docs-mcp] ready on stdio"); diff --git a/build/docs-mcp-server/node/src/vector-store.ts b/build/docs-mcp-server/node/src/vector-store.ts index f50716167a..63d7056ccc 100644 --- a/build/docs-mcp-server/node/src/vector-store.ts +++ b/build/docs-mcp-server/node/src/vector-store.ts @@ -11,7 +11,7 @@ import { SCHEMA_VECTOR_FIELD_ALGORITHM, type RedisClientType, } from "redis"; -import { EMBED_DIM } from "./embed.js"; +import { EMBED_DIM } from "./constants.js"; import { normalizeUrl } from "./url.js"; const INDEX = "docs_vec"; @@ -62,8 +62,12 @@ export class VectorStore { async dropIndex(): Promise<void> { try { await this.client.ft.dropIndex(INDEX, { DD: true }); - } catch { - // no index — nothing to drop + } catch (e) { + // Swallow ONLY "index doesn't exist" (nothing to drop). Auth, permission, + // or network errors must propagate — otherwise a failed drop is silently + // ignored and the loader reuses/mixes into a stale index. + const msg = e instanceof Error ? e.message : String(e); + if (!/unknown index|no such index|not exist/i.test(msg)) throw e; } } From 753fd94136ee97db29c32ca5c19f087394381fc5 Mon Sep 17 00:00:00 2001 From: Andy Stark <andrew.stark@redis.com> Date: Fri, 24 Jul 2026 14:43:48 +0100 Subject: [PATCH 25/25] DOC-6809 assess-comments skill ledger --- .claude/state/assess-comments.coverage.md | 49 ++++++++++++++++++----- 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/.claude/state/assess-comments.coverage.md b/.claude/state/assess-comments.coverage.md index d4c3ffc3b1..9badb2c4ca 100644 --- a/.claude/state/assess-comments.coverage.md +++ b/.claude/state/assess-comments.coverage.md @@ -27,23 +27,23 @@ whether to commit the change. | Capability | Confidence | Real encounters | Last verified | Evidence | |---|---|---|---|---| -| Branch/PR identification + arg handling | 🟢 corroborated | 8 | 2026-07-02 | #3415, #3507, #3374, #3536, #3542, #3543, #3573, #2531 | -| Multi-source collection (inline + top-level + reviews) | 🟢 corroborated | 9 | 2026-07-02 | #3415, #3507, #3510, #3374, #3536, #3542, #3543, #3573, #2531 | -| GraphQL thread-resolution pull (`isResolved`/`isOutdated`) | 🟢 corroborated | 7 | 2026-07-02 | #3510 (12/12 resolved), #3374 (15/15), #3536 (2/2 open), #3542 (3 resolved/1 open), #3543 (1/1 open), #3573 (2/2 open), #2531 (r1: 11 open/2 res-outdated/1 open; r2: 11 of 14 outdated after fixes + APPROVED) | -| Source-role tagging (bugbot/security/history/summary/ci/human) | 🟢 corroborated | 8 | 2026-07-02 | #3415, #3507, #3374, #3536, #3542, #3543, #3573, #2531 | -| Open/resolved split | 🟢 corroborated | 7 | 2026-07-02 | #3510, #3374, #3536 (0 resolved/2 open), #3542 (3 resolved/1 open), #3543, #3573 (r1/r2 mixed), #2531 (11 open / 3 resolved) | -| Fix-quality spot-check (genuinely fixed vs silenced) | 🟢 corroborated | 5 | 2026-07-02 | #3510 (term removals landed), #3374 (`num_docs`, dropIndex landed), #3542 (xargs+guard, narrowed exclude, SHA pin landed), #3573 (relpath + `--add` dedup landed), #2531 (r1 found decimal thread 2619563419 reverted; r2 doc fixes landed + engineer ZdravkoDonev **APPROVED**) | +| Branch/PR identification + arg handling | 🟢 corroborated | 9 | 2026-07-24 | #3415, #3507, #3374, #3536, #3542, #3543, #3573, #2531, #3585 | +| Multi-source collection (inline + top-level + reviews) | 🟢 corroborated | 10 | 2026-07-24 | #3415, #3507, #3510, #3374, #3536, #3542, #3543, #3573, #2531, #3585 | +| GraphQL thread-resolution pull (`isResolved`/`isOutdated`) | 🟢 corroborated | 8 | 2026-07-24 | #3510 (12/12 resolved), #3374 (15/15), #3536 (2/2 open), #3542 (3 resolved/1 open), #3543 (1/1 open), #3573 (2/2 open), #2531 (r1: 11 open/2 res-outdated/1 open; r2: 11 of 14 outdated after fixes + APPROVED), #3585 (10 threads: 2 open / 8 resolved, of which 1 resolved+not-outdated) | +| Source-role tagging (bugbot/security/history/summary/ci/human) | 🟢 corroborated | 9 | 2026-07-24 | #3415, #3507, #3374, #3536, #3542, #3543, #3573, #2531, #3585 (bugbot + Jit security + CLA/Jira ci + **Redis Memory history bot** + Cursor summary) | +| Open/resolved split | 🟢 corroborated | 8 | 2026-07-24 | #3510, #3374, #3536 (0 resolved/2 open), #3542 (3 resolved/1 open), #3543, #3573 (r1/r2 mixed), #2531 (11 open / 3 resolved), #3585 (2 open / 8 resolved) | +| Fix-quality spot-check (genuinely fixed vs silenced) | 🟢 corroborated | 6 | 2026-07-24 | #3510 (term removals landed), #3374 (`num_docs`, dropIndex landed), #3542 (xargs+guard, narrowed exclude, SHA pin landed), #3573 (relpath + `--add` dedup landed), #2531 (r1 found decimal thread 2619563419 reverted; r2 doc fixes landed + engineer ZdravkoDonev **APPROVED**), #3585 (dup-id High genuinely fixed in get-page.ts via ambiguous()+candidates; EMBED_MODEL fix landed 85e1d0d17 — both real, not silenced; r2: the 2 prior-open findings confirmed genuinely fixed — empty-index guard added to index.ts main() [now resolved+not-outdated, Medium], matchingSections deferred past top-k with lexical/hybrid MRR .525/.704 unchanged) | | "Resolved ≠ fixed" flag — **legitimate deferral** variant | 🟡 seen once | 1 | 2026-06-23 | #3510 (TS.BGET:122 left pending eng) | | "Resolved ≠ fixed" flag — **still-broken / reverted** variant | 🟡 seen once | 1 | 2026-06-30 | #2531 (resolved+outdated thread 2619563419 said decimal default=`string`; a later rewrite reverted current code to `precise`, so the resolved fix is no longer in the code — engineer re-raised it as 3496835587). Regression flavour; see worked examples | -| Cross-tool **agreement** | 🟡 seen once | 1 | 2026-06-23 | #3374 (Claude + bugbot independently on `num_docs`) | +| Cross-tool **agreement** | 🟢 corroborated | 2 | 2026-07-24 | #3374 (Claude + bugbot independently on `num_docs`); #3585 (bugbot finding 3639447417 "EMBED_MODEL documented but never read" independently matched the orphaned diff the author had already identified & committed as 85e1d0d17 — both flagged the parametrization missing from committed code) | | **Contradiction** detection | 🟢 corroborated | 2 | 2026-07-02 | #3415 (approval vs open bugbot finding); #2531 (RDI engineer's repo ground truth contradicts the page's Debezium-docs claims on ≥4 points — version, decimal default, temporal pass-through, MariaDB connector — **and** engineer-vs-existing-doc on temporal normalization). *(#3507 was an off-branch manual demo — not counted.)* | | **Ping-pong loop** detection | ❓ untested | 0 | 2026-07-02 | still no true tool A↔B loop across #3536 (4 rounds), #3542 (r2 "empty-scope"), #3573 (r1 fixed point; r2 independent), or #2531. #3542/#3573 were churn not loops; #2531's nearest reopened-concern was the decimal regression (resolved Dec → reverted by a June rewrite → re-raised) — a regression across one rewrite, not a cycle | | **Subsystem churn** detection (repeated findings on one patched area) | 🟢 corroborated | 3 PRs | 2026-07-02 | 3 distinct PRs. #3536 — 3 instances (review-handling / churn-feature / cap↔report contract). #3542 — 2 instances on the extraction *fail-loud-on-empty* contract (r1 ARG_MAX silent-green → r2 sibling zero-files `exit 0`). #3573 — 2 instances on the `--add` virtual-merge mechanism (r1 dup-vs-disk → r2 dropped-under-collapse). Worked examples below | | Approval-over-open-finding cross-check | 🟢 corroborated | 6 | 2026-07-02 | #3415 (dwdougherty), #3374 (low-confidence over open HIGH), #3536 (high-confidence over 2 open Mediums: benign), #3542 (paoloredis "yep go ahead" 7 min after open Medium #3498159511; unacknowledged), #3573 (dwdougherty "Sure, why not?" APPROVED 13:41 over open findings; 2 bot findings landed 13:49 after), #2531 (run1 correct **negative** — no approval; run2 **positive** — ZdravkoDonev APPROVED 13:13 then bugbot finding 3499796857 landed 15:08, and he approved over 2-3 of his own still-open findings incl. the temporal one) | | Depth cap / prioritisation under load | 🟢 corroborated | 2 | 2026-07-02 | #3374 (19 candidate findings → 4 deep-verified); #2531 (r1: 14 threads → 5 deep-verified, 6 deferred). *(#3542/#3573 were under cap — not load tests)* | -| Mandatory deep-verify of resolved+not-outdated HIGH | 🟡 seen once | 1 | 2026-07-02 | #3542 #3467309496 (High "Grep failure skips link check", resolved + isOutdated:false) — deep-verified against current code: xargs+guard genuinely present, so legitimately fixed (not still-broken). First real firing of the rule | -| Bot calibration (fixed-vs-dismissed ratio) | 🟢 corroborated | 6 | 2026-07-02 | #3374 (bugbot mostly accepted); #3536 (5/5 valid); #3542 (3/3 valid — 2 fixed, 1 open); #3543 (1/1 valid; Jit 0); #3573 (4/4 valid; Jit 0); #2531 (r1 bugbot 0 findings; r2 bugbot 1/1 valid — caught the ledger duplicate-rows defect 3499796857; Jit 0) | -| Codex second-opinion availability gate | 🟢 corroborated | 6 | 2026-07-02 | #3415, #3374 (CLI on PATH; #3374 had a real Codex review), #3542, #3543, #3573, #2531 (codex on PATH) | +| Mandatory deep-verify of resolved+not-outdated HIGH | 🟢 corroborated | 2 | 2026-07-24 | #3542 #3467309496 (High "Grep failure skips link check", resolved + isOutdated:false) — xargs+guard genuinely present, legitimately fixed. #3585 3513924736 (High "Duplicate page IDs break get_page", resolved + isOutdated:false) — deep-verified get-page.ts: non-unique id now returns ambiguous()+candidates, search hits carry unique url; genuinely fixed *elsewhere* than the flagged line (hence not outdated). 2nd distinct PR | +| Bot calibration (fixed-vs-dismissed ratio) | 🟢 corroborated | 7 | 2026-07-24 | #3374 (bugbot mostly accepted); #3536 (5/5 valid); #3542 (3/3 valid — 2 fixed, 1 open); #3543 (1/1 valid; Jit 0); #3573 (4/4 valid; Jit 0); #2531 (r1 bugbot 0 findings; r2 bugbot 1/1 valid — caught ledger dup-rows 3499796857; Jit 0); #3585 (bugbot 13/13 valid across ≥5 rounds — r1 8 fixed incl. all 3 High; then empty-index + matchingSections fixed; r2 normalizeUrl dup 3644535220 + eager native import 3644667453 fixed; r3 Float32Array misaligned-buffer 3644788302 Low, valid, open; Jit 0; consistently high trust) | +| Codex second-opinion availability gate | 🟢 corroborated | 7 | 2026-07-24 | #3415, #3374 (CLI on PATH; #3374 had a real Codex review), #3542, #3543, #3573, #2531, #3585 (codex on PATH) | | Ledger self-integrity after `main` merge (no duplicate rows) | 🟡 seen twice | 2 | 2026-07-02 | #2531 r2 — bugbot 3499796857 caught the shared ledger gaining duplicate rows when `main` (carrying a #3573-era ledger) merged in and git kept both blocks. **2026-07-02**: merging `main` again produced a real conflict as #3542/#3573 edited the same rows — union-merged per capability. Recurring shared-file hazard; see worked examples + step-11 refinement | ## Worked examples library @@ -71,6 +71,35 @@ pushed, and bugbot's next re-scan came back **clean — no comments**. A real lo would have spawned another round; this settled. So the "not a loop" judgement is borne out by what happened next: assess → fix → re-scan reached a fixed point. +**Near-miss (NOT a loop) — #3585, 2026-07-24.** Bugbot Low #3638621205 +(`matchingSections` eager compute) landed in `search.ts` — a file the author had +just edited in Step 4 (added `hitForUrl`, which also calls `matchingSections`). +Superficially loop-shaped (finding in a freshly-edited file). But round-1 search.ts +findings were all *correctness* (dup-ids, suffix match, version) and were resolved; +this is a *new, independent perf* observation, not the same concern reopened or an +A↔B cycle. Normal iteration → fresh cluster, not ping-pong. Also NOT churn: distinct +concern class (perf vs correctness), not repeated patching of one under-specified area. + +*Round-2 re-confirmation (2026-07-24):* fixing the 2 open findings triggered a +re-scan that surfaced 2 **new, independent** findings in the Step-4 code +(normalizeUrl dup across 4 files 3644535220; eager native import 3644667453) — +again the fix→rescan→new-findings-in-changed-files pattern, NOT a loop (no reopened +concern, no A↔B). The prior findings stayed resolved. Consistent with the near-miss +rule; still no true ping-pong on this PR. + +*Round-3 (2026-07-24):* fixing those 2 triggered a 3rd re-scan → 1 new independent +finding (Float32Array misaligned-buffer in load-index.mjs 3644788302). **New +distinction worth naming: DRIP-FEED, not churn.** 3 consecutive rounds of findings +in the *same newly-added feature* (the ~7-file Step-4 hybrid code), but each on a +**different file** (index.ts → hybrid.ts+3 → load-index.mjs), each an independent +well-understood nit (import strategy / DRY / buffer alignment), each fix clean & +non-reopening. That is NOT churn (churn = repeated patches to the *same +under-specified area*, each exposing the next adjacent gap in that spot) and NOT +ping-pong (no reopened concern, no A↔B). It's Bugbot draining a backlog from a large +new diff, ~1-2 findings/scan. Right move is NOT redesign but a **proactive +self-review sweep** of the remaining new files to get ahead of the drip. See +step-11 refinement suggestion. + **Near-miss (NOT a loop) — #3542, 2026-07-01.** Round-1 bugbot High #3467309496 (ARG_MAX / `|| true` silent-green) was fixed (commit `21f079e1d`: xargs + zero-URL `exit 1` guard). Round-2 re-scan raised Medium #3498159511 ("empty scope