docs: fix confidence scoring accuracy and default threshold#132
Merged
Conversation
rajkumarsakthivel
added a commit
that referenced
this pull request
Jul 13, 2026
* feat: expose dashboard format controls * docs: fix confidence scoring accuracy and default threshold (#132) * docs: add confidence scoring explanation page * docs: fix confidence scoring doc accuracy and default threshold value --------- Co-authored-by: ahfoysal <ahfoysal30@gmail.com> * fix: full-project review findings + retrieval precision (26% fewer tokens served) (#123) * docs: token-savings program design spec * fix: resolve 2026-07-03 full-project review findings Config writers (data loss): - string-aware JSONC comment stripping; URLs in values no longer corrupt configs - parse failures skip the file with a warning instead of overwriting user MCP configs - non-dict server sections handled; uninstall matches CCE markers only, never bare "cce" - uninstall strips only the CCE block from shared git hooks Memory (index desync): - bootstrap without sqlite-vec no longer stamps CURRENT_VERSION; vec tables created on later connect - turn re-compression uses DELETE+INSERT so FTS/vec delete triggers fire - compression queue dead-letters after 5 attempts; savings recorded only on success - rollup enqueue dedup via -1 sentinel; PII scrub applied to prompt/tool-payload writes - non-numeric started_at no longer 500s Retrieval/storage (ranking + consistency): - FTS-only chunks get worst-case vector distance instead of perfect 0.0 - FTS queries tokenized per-term (OR-joined, stopwords dropped) instead of one phrase - chunks_vec created with distance_metric=cosine; legacy L2 tables detected and rebuilt - vector/FTS/graph stores roll back on mid-batch failure; FTS re-ingest no longer duplicates Indexer (correctness + secrets): - tree-sitter byte offsets slice bytes, not str; non-ASCII files chunk correctly - single-file target path (watcher) now honors is_secret_file and .cceignore - project_dir resolved once; symlinked roots no longer crash targeted reindex - deleted-but-tracked targets prune index + manifest - unquoted credential assignments redacted; *.env suffix files skipped - Stripe/GitHub vendor regexes fixed (capturing groups leaked full key values) Dashboard (XSS): - all API-sourced strings escaped (sessions, decisions, file paths, chart labels) - inline onclick handlers replaced with data attributes + listeners - state.json written atomically 1040 tests pass (~60 new regression tests, all confirmed failing pre-fix). * docs: Phase 1 retrieval-precision implementation plan * feat(retrieval): persist chunk modified_ts through vector store * feat(indexer): stamp chunks with source file mtime for recency scoring * fix(indexer): stamp git commit chunks with commit-time modified_ts * feat(retrieval): collapse overlapping same-file chunks before packing * feat(retrieval): marginal-utility stop with top-1 guarantee, config-driven * docs: correct omitted-results note design in Phase 1 plan (truthful dropped-count signal) * fix(retrieval): truthful dropped-results signal for the omitted-results note * feat(benchmarks): A/B mode for retrieval precision; tune Phase 1 defaults from evidence Adds --ab flag to run_benchmark.py: indexes the repo once, then runs each query twice (baseline retrieve() vs tuned with threshold+marginal_ratio) and reports per-query token deltas plus aggregate reduction%. Evidence from 6 A/B runs on this repo shows marginal_ratio=0.75 achieves 26.1% token-served reduction with hit rate unchanged (4/8→4/8), clearing the ≥25% gate. confidence_threshold had no effect (all scores 0.70–0.95). Updates retrieval_marginal_ratio default: 0.5 → 0.75. Results committed as benchmarks/results/cce-phase1-ab.{json,md}. * docs(benchmarks): disclose cliff shape and sample-size limits in Phase 1 A/B evidence * fix: point omitted-results note at marginal_ratio; document new retrieval config * fix(indexer): repopulate index after cosine-metric migration The cosine rebuild in VectorStore._ensure_tables wipes on-disk chunks but left manifest.json claiming every file was still indexed, so an incremental reindex (the default for cce index and the watcher) skipped every unchanged file and left context_search returning nothing until cce index --full. VectorStore now flags metric_rebuilt; run_indexing clears the manifest when set, forcing the scan to re-ingest. Mirrors the existing dim-migration guard. Regression test confirms an incremental reindex repopulates post-migration. * fix: atomic write in /api/format; _state_path and _default_top_k in test helper --------- Co-authored-by: zerone0x <39543393+zerone0x@users.noreply.github.com> Co-authored-by: ahfoysal <ahfoysal30@gmail.com> Co-authored-by: Fazle Elahee <felahee@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the inaccuracies found in #126 and merges the content cleanly.
Changes
docs/wiki/Confidence-Scoring.md(new, from #126 with corrections):/2normalization step applied beforeConfidenceScorermax(0, 1 - kd/5)) and the_MAX_KEYWORD_DISTANCE = 5capconfidence_thresholdis compared against the blended score, not raw confidence0.5to0.2(matchesconfig.py)docs/wiki/Configuration.md(existing file):confidence_thresholddefault from0.5to0.2in both the example block and the Retrieval Tuning descriptionCloses #126