fix(extract): hollow, unparseable and omitting chunks count as incomplete so the shrink guard stays armed (#3105) - #3114
Conversation
…lete (Graphify-Labs#3105) The Graphify-Labs#479 shrink guard is bypassed (force=True) on a run classified as complete, and _extraction_incomplete tracked hard failures only: a crashed pass, a chunk that raised. A chunk that comes back hollow after every retry, or as invalid JSON, or that simply omits some of its files does not raise - it returns fewer nodes and counts as a SUCCEEDED chunk. So two consecutive --update runs on an unchanged repo: the first had 3 raised chunks and the guard refused; the second had 0 raised and 6 hollow, read as complete, and wrote 111 nodes over a 570-node graph without a word. With an LLM backend that is the normal way an extraction silently produces a fraction of the graph, so it now arms the guard exactly like a crashed chunk does: a non-empty uncovered_files list (files the model omitted, which is where invalid-JSON chunks land) or any partial/hollow file marks the run incomplete, with a stderr line saying so. A complete run keeps force=True as before; --allow-partial still overrides; a refused write still leaves the manifest unstamped so the next run retries.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Treats hollow, unparseable, or file-omitting semantic chunks as an incomplete extraction: dispatch_command now sets _extraction_incomplete when fresh reports uncovered_files or partial-marked source files, so the #479 shrink guard stays armed (force=False) instead of silently overwriting a larger graph with a fraction of it (#3105). Previously these cases returned fewer nodes without raising and counted as SUCCEEDED, bypassing the guard; --allow-partial still forces the write, and an ordinary fully-covered run keeps its force-write path. Adds test_hollow_chunks_arm_shrink_guard covering the omit, hollow, allow-partial override, clean-run, and manifest-not-stamped-on-refusal cases.
No blocking issues surfaced.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 256 functions depend on the 67 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 122 callees - new:
_stale_graph_sources()— 7 callers, 6 callees - new:
_run_hook_guard()— 4 callers, 7 callees - new:
test_poisoned_manifest_is_healed()— 0 callers, 6 callees
Verification — 256 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 209 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
· 4 more finding(s) on lines outside this diff (see the check run).
Closes #3105.
The problem
A full build writes with
force=True, bypassing the #479 shrink guard, unless the run was classified as incomplete._extraction_incompletetracked hard failures only: a crashed extractor pass, a chunk that raised. A chunk that comes back hollow after every retry, or as invalid JSON, or that simply omits some of its files does not raise — it returns fewer nodes and counts as a succeeded chunk. So the reporter's two consecutive--updateruns on an unchanged repo went:Run 2 even printed
10/24 dispatched file(s) produced no nodes and are absent from the graphand still counted as complete. With an LLM backend this is the normal way an extraction silently produces a fraction of the graph — the case the guard exists for, and the one case where it was disabled.The change
After the semantic pass, a non-empty
uncovered_fileslist (files the model omitted — which is also where an invalid-JSON chunk's files land) or any partial/hollow file now sets_extraction_incomplete = True, with one stderr line saying how many files produced no nodes and how many came back truncated or hollow, and that--allow-partialoverrides. From there the existing machinery does the rest:force=False, the shrink guard compares against the existing graph, a refusal exits non-zero and leaves the manifest unstamped so the next run retries those files.A complete run — every dispatched file covered, nothing partial — keeps
force=Trueexactly as before, so a legitimately shrinking full build (dedup, deleted code) is unaffected.Tests
tests/test_hollow_chunks_arm_shrink_guard.py— 5 tests through the real CLI with a stubbed semantic pass: an omitting chunk arms the guard (force=False); a hollow chunk (files marked partial, nothing raised — the reporter's run 2) arms it;--allow-partialstill overrides; a fully covered run keepsforce=Truewith no warning; and a refused write exits non-zero without stamping the manifest. With the fix reverted, the two "arms the guard" tests fail.test_incomplete_build_guard,test_extract_cliandtest_extract_code_only_cliare unchanged (60 passed); the full suite matches thev8baseline.