quality-report: no threshold watches editor_truncated_rate

[bbertucc]

/v1/quality gained editor_truncated_rate in #151: the share of documents where a correction round's response hit the model's output ceiling, so the round was discarded and the document shipped as it entered that round, with <!-- @editor-truncated --> in it. Nothing reads it. .github/workflows/quality-report.yml has thresholds for unresolved_rate, links_dropped_rate, lint_error_rate, mean_rounds and per-rule share, so a rise in truncated rounds is the one quality regression in the tally that files no issue.

It was left out of #151 on purpose. Every other threshold in that workflow is a number someone chose against observed rates, and the file says so; picking one for a metric with no history means picking a number that is either so loose it never fires or so tight it files an issue in its first week. A month of real values is the input this needs.

What to do once there are numbers:

• Add EDITOR_TRUNCATED_RATE_MAX alongside the other *_MAX inputs and into the THRESHOLDS object.
• Add a finding branch. Suggested severity 2, between dropped links and unresolved: like a dropped link this is content loss — corrections the editor actually made were thrown away — but unlike a dropped link the source content is still present, so it does not outrank it.
• Title has to be stable in the sense the comment above sev insists on: no rate and no threshold in it. Something like "Quality regression: correction rounds are being discarded at the output ceiling".
• The body already gets metric/value/threshold, which is where the numbers belong.

Before choosing the number, note that this rate is not independent of document length: the editor's output length follows the length of the document it is rewriting, not the number of issues in it, so a week of unusually long inputs moves this rate without anything being wrong with the pipeline. Whatever threshold is chosen should be loose enough to survive that, or the finding should say so in its body.

[github-actions]

Possibly a duplicate of #143 — left open for a human.

#143 needs a code change in runEditor (src/pipeline/review.ts:527 still rethrows on TruncatedResponseError) so a truncated editor response costs the round rather than the whole document. #159 needs a new EDITOR_TRUNCATED_RATE_MAX input, THRESHOLDS entry and severity-2 finding branch in .github/workflows/quality-report.yml so the resulting rate is watched in CI. Fixing #143 makes truncation survivable but files no issue when it rises; fixing #159 adds a threshold for a metric that only exists because of #143's fix. Same subsystem and a direct causal chain (#159 is the slice #151 deliberatel

What closing this would lose: The only request anywhere in the tracker to actually watch editor_truncated_rate. Every other quality regression in the /v1/quality tally has a threshold; closing #159 leaves discarded correction rounds as the one metric that can climb silently, plus the specific guidance it carries: severity 2 (content loss, but source content still present, so below dropped links), a rate-free stable finding title, and the warning that the rate co-varies with document length so a week of long inputs moves it without a pipeline defect.

Not stale: I grepped the checkout and quality-report.yml (lines 225-284) has thresholds only for unresolved/links_dropped/lint_error, with no truncation entry. But note the metric itself is absent at this sha — editor_truncated_rate and <!-- @editor-truncated --> match nowhere in the tree, and review.ts still rethrows, so #151's containment work is not on main here. #159 is therefore blocked twice over: on #151 landing, and on its own stated precondition of ~a month of observed values before a number can be chosen. Consider labelling it blocked/needs-data rather than leaving it in the acti

Not closed because the verdict is related, not duplicate; confidence is medium, and only high can close; no refutation session ran.

---

Posted by the issue-triage workflow (run). It closes duplicates only when two independent sessions agree; this one did not qualify, so the call is yours. Add no-auto-close if this issue should never be closed automatically.

[bbertucc]

Round 8 (a1ead8e, 4 chunks / 100 pages) puts editor_truncated_rate at 3 of 4 documents, up from 2 of 4 in round 7 and 1 of 4 in round 6. Nothing watches it, so nothing said anything.

The details, for whatever threshold this ends up with:

• ceiling 32,000 tokens; the truncated responses were p50 80,396 characters
• all 3 fell back to the sectioned rewrite and 13/13 sections came back — 0 rounds lost entirely, so the fallback (The copy editor cannot finish a 25-page round under a 32,000-token ceiling: 2 of 4 documents delivered with their corrections unapplied #165) is doing its job
• the largest document split into 5 sections at a 33,574-token budget on 140,003 characters
• copy_editor cost $5.86 of the round's $17.83 (32.9% of spend) across 17 calls, against 4 documents — the section retries are most of that call count

The reason a threshold is worth having rather than a fix: at 100 pages per document this is now the normal path, not the exception. A whole-body rewrite of a 140,000-character document does not fit in a 32,000-token response and never will, so editor_truncated_rate on documents this size is really measuring "how often did the editor take the expensive route", and a threshold that fires at, say, 50% would have fired on the last three rounds in a row. That is either the signal you want or evidence the ceiling is the wrong place to put the alarm — see #185's still-open item 1, which is the same fact from the cost side.

Round-over-round: 1/4 (round 6) → 2/4 (round 7) → 3/4 (round 8). The corpus sample changed between rounds 7 and 8, so that is not a clean trend; the documents got bigger, which is enough to explain it.