From 6ac97a387a4b4e992d207906d735d8a10e8394af Mon Sep 17 00:00:00 2001 From: ChronicallyJD Date: Wed, 12 Aug 2026 16:57:10 -0600 Subject: [PATCH] docs: reconcile the #426 prune ceiling to the measured ~91% (was ~93%) The disposition cited a ~93% prune plateau (line 10) and a ~91% prune ceiling (line 101) -- internally inconsistent, since a plateau cannot exceed the ceiling. Verified by re-measuring the %google% zero-match 1024-row-block fraction on the ClickBench data: 91.01% on the full 11.1M sample, 93.45% on the first 1M. Both are correct for their scale; the 1M figure is a prefix and over-counts because the matches cluster, so ~91% is the truer ceiling. Use it consistently and note why the prefix over-read -- the same prefix-unrepresentativeness that #595 hit. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_017N82wDmsawqSWoWkmxtHmW --- design/ISSUE_426_SUBSTRING_FILTER.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/design/ISSUE_426_SUBSTRING_FILTER.md b/design/ISSUE_426_SUBSTRING_FILTER.md index aa2ddf8..905c719 100644 --- a/design/ISSUE_426_SUBSTRING_FILTER.md +++ b/design/ISSUE_426_SUBSTRING_FILTER.md @@ -7,8 +7,11 @@ merged in #584) and `design/ISSUE_452_PHASE2_DECODE_GATING.md`. ## DISPOSITION (2026-08-11): feasible, measured, NOT built The design below is sound and the step-1 measurement was taken on real ClickBench -`hits` URLs (1,000,000 rows via duckdb). Result: reaching the ~93% prune plateau -costs **~8 bytes/row** on the filtered string column (65536 bits per 1024-row +`hits` URLs (1,000,000 rows via duckdb; re-checked on the full 11.1M sample). +Result: reaching the ~91% prune ceiling (the initial 1M-row probe read ~93%, but +that is a prefix, and the `%google%` matches cluster, so the full sample is the +truer ~91% -- a prefix over-counts skippable vectors) costs **~8 bytes/row** on +the filtered string column (65536 bits per 1024-row vector; real URLs carry ~7,251 distinct trigrams/vector, so a smaller filter saturates and prunes nothing). On URL that is ~40% overhead on the column, ~6% of the whole hits table, and it buys only q21/q22 (2.2x/1.8x) because q24 -- the big