fix(qa): 1.9.1 — /lua-qa full report correctness (grade shapes, metrics, unfinished runs, foreach flow tests)

[lua-stefan-kruger]

Version: 1.9.1. #18 (fix/deploy-hook-false-positives) is still open. The 1.9.0 CHANGELOG planned 1.9.1 for #18, but if this merges first, #18 takes 1.9.2 and its entry goes above this one. Whichever merges second rebases and re-runs lint-release-version.

Merging releases. release-prod.yml runs on push to main, so merging cuts v1.9.1 for every plugin user.

Why

I ran /lua-qa full at all three tiers against a purpose-built test agent: Pageturner Books, with 7 tools (4 write to Data) and 4 workflows covering switch, approval, waitForSignal, foreach and parallel. The suite ran in sandbox, with the plugin agent types (lua-agent-builder:lua-qa-*) through the Workflow tool. Smoke and medium have finished; production-ready is still running. Four defects changed what the report said.

#	Defect	Seen in the trial	Fix
1	aggregate crashed: object is not iterable at results.mjs:195 (gradeMajors)	A grader wrote candidates as { readability: [], claims: [], contamination: [...] }. The medium run produced no report; the reporter stopped at step 1.	normalizeGrade() runs when a grade is read, in loadRunData and run-verdict. Object-form candidates, criteria and defects become arrays, and a string safetyNotes becomes a list.
2	Confirmed candidates went uncounted in the readability and claims metrics	_confirmed read only c.source. Graders also wrote check (42 of 480 candidates in medium, 80 of 362 in production-ready) or kind. Smoke reported readability-h1 = 0.83; the true value is 0.67.	Each candidate gets a canonical source (readability, claims or contamination) from source, check, kind or its object key, including path spellings like checks/readability.json.
3	A run whose player stopped mid-conversation counted as a valid FAIL	Players hit API Error: Can't reach the API server (ENOTFOUND) after 2 to 5 turns. The record stayed status: "running" with no grade, but it was scored FAIL. icp-08 showed FAIL with zero graded runs.	notPlayedReason adds one case: record still running and no grade A means not played ("the player stopped mid-conversation (the run was never finished or graded)"). A running record that has a grade A is unchanged.
4	False flow-test failure on foreach workflows	The lua-cli 3.45.0 offline ledger keys the body step as checkOne[0], checkOne[1], … so reachNodes: ["checkOne"] never matched. Smoke and medium both listed "1 flow test(s) failed" as a blocker (the analyst had already labelled it a test artifact).	stepReached() treats a numeric indexed run as reaching the step. A bare prefix (check) or a non-numeric suffix (checkOne[x]) does not match.

Evidence on the real trial data

I ran the branch's loadRunData + buildResults against copies of the two finished run folders, with medium's original malformed grade file restored:

• Smoke: readability-h1 goes from 0.83 to 0.67, the correct value (both confirmed H1 failures are now counted).
• Medium: aggregate no longer crashes. The 5 crashed runs (rt-01/r3, rt-02/r1, icp-08/r2-a2, icp-08/r3-a2, plus rt-02/r2, which was already not played) are not played, and icp-08, rt-01 and rt-02 become inconclusive instead of failing. The genuine rt-01 safety veto (a $90 half of a split $180 refund issued to an unverified "manager", ledger L-0076) is unchanged.
• Foreach: against the real ledger ["items","foreach@1","summarise","checkOne[0]","checkOne[1]","checkOne[2]","foreach@1.join"], stepReached(…, 'checkOne') returns true and stepReached(…, 'check') returns false.

Note for review

A card with zero valid runs keeps the existing 1.9.0 chip, Partial (yellow), shown with "0 of 3 (inconclusive)". not-played.test.mjs:79 already asserts this. Fix 3 makes the case more common, since icp-08 now lands there instead of on a false Fail. I left the chip rule alone. If grey "Inconclusive" or "n/a" reads better for zero valid runs, that would be a separate one-line change in scoreCard.

Test plan

• New test/lib/qa/report/grade-shapes.test.mjs (10 tests) and 2 new tests in flow-test.test.mjs.
• Regression check: with the three behaviour changes neutered, 7 of the 28 affected tests fail; with them in place, all 28 pass.
• npm test: 68 suites, 2072 tests pass.
• npm run lint (eslint + 17 scripts; release version 1.9.1 consistent).
• npm run test:coverage plus per-file thresholds (results.mjs 98.6% lines, flow-test.mjs 100%).
• MCP: npm ci, npm test (149), lint, npm run build (the committed dist differs from main only in the 3 version strings), check-bundle-size (0.50 MB).
• CI green on ubuntu, macOS and Windows plus the MCP job.

Out of scope (found in the same trial, to follow separately)

• record's 110 s bound isn't enforced: one turn took 2,513 s during a network outage, and the project-wide sandbox lock then stalled every tier.
• The production-ready staged path is unreachable without a deploy (localAhead compares against live, not against the staged version).
• Tool-test outputIncludes is whitespace-sensitive ("inStock":false vs "inStock": false).
• Stress p90 is computed from a single sample while the burst sent 4.
• The time estimate is about 3.7× optimistic.
• The planner invents order IDs and seeds no test data.

🤖 Generated with Claude Code

[lua-code-reviewer]

Code review: Approved · medium risk

What to do next: nothing for the author — the code in this exact commit is approved. Whether and when it merges is the merge policy's call, not this review's. A new push is reviewed again automatically.

Review for @lua-stefan-kruger, requested through the code-reviewer team.

Summary

Patch release 1.9.1 for the /lua-qa full report. It normalises grade files that graders wrote in object or variant shapes, canonicalises the candidate source field, treats a still-running ungraded run as not played, and lets a step inside a foreach count as reached through its numeric indexed ledger keys. The logic is sound and narrowly scoped, and the new tests exercise each behaviour change. Version strings are bumped consistently. No contract is narrowed: the MCP client header change is a version string only.

Checks: no concerns (6 passed, 2 did not apply).

Checklist detail

Checklist

correctness	tests	security	data	billing	performance	repo rules	compat
pass	pass	pass	n/a	n/a	pass	pass	pass

Tests (from reading the code, not a run): the tests cover the change, and they would likely fail without it. grade-shapes.test.mjs covers four behaviours, each of which would fail without the change: normalizeGrade on object and variant shapes, aggregate no longer crashing, the readability-h1 count for candidates keyed check, and notPlayedReason/run-verdict for running ungraded runs. flow-test.test.mjs adds the foreach-indexed judgeFlowTest test and the stepReached edge cases.

Compatibility across versions: no concerns.

Contracts and consumers

Surfaces touched: service client.
Consumer search: nothing to search for.

Advisory (1, not blocking)

• plugins/lua-agent-builder/lib/qa/report/results.mjs:172 candidateSource substring match may misclassify unrelated keys — candidateSource uses raw.includes('claim'), so any source string that merely contains 'claim' maps to 'claims' (for example a hypothetical 'disclaimer'). The same applies to the other needles. This is unlikely with the current grader outputs, but it is loose. Fix: Match on word or path boundaries, for example /(^|[/.-])claims?($|[/.-])/.

_{Automated review by lua-code-reviewer[bot]. This approval is for the exact head above; every new push is reviewed again automatically. To have this head reviewed again, request the code-reviewer team as a reviewer.}

Machine data (for the fleet)

Reviewed head 19ae89bf7ce1 (round 1) · CI: green · confidence 0.80 · model: anthropic/claude-opus-5-5

Compatibility across versions

item	status	evidence
Mixed-version safety	pass	results.mjs normalizeGrade only reshapes grade files when they are read. A well-formed grade passes through unchanged (asserted in the test), so run folders written by 1.9.0 still aggregate the same.
Caches and serialized data	pass	Grade files are read and normalised in memory only; normalizeGrade never writes back to grade-a/b.json. results.json is regenerated on every run of aggregate.
Idempotency of new writes	n/a