feat(plugin): 1.9.0, comprehensive /lua-qa suite (smoke/medium/production-ready tiers, inferred gates, parallel workflows, PDF report)

[lua-stefan-kruger]

Version note: this PR ships as 1.9.0, not 1.8.0. While it was being built, #17 (web apps) merged as 1.8.0 and the v1.8.0 tag exists, and release-prod.yml skips the release when the tag already exists. Every place lint-release-version checks says 1.9.0, and the committed MCP bundle is rebuilt. #18 (fix/deploy-hook-false-positives) is still open and declares 1.7.1, which is below the released 1.8.0, so #18 must take a version above it before merging. As it stands, release-prod.yml only checks that the tag is new, so it would cut an out-of-order v1.7.1. Whichever PR merges second rebases and re-runs lint-release-version:

• This PR first (planned): main becomes 1.9.0. fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 then takes 1.9.1, and its CHANGELOG entry goes above this one.
• fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 first: fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 takes 1.8.1, with its entry above 1.8.0. This PR stays 1.9.0, and its entry stays on top.

The CHANGELOG's 1.9.0 entry says the same.

What it does

/lua-qa full is a complete QA suite for a Lua agent. Bare /lua-qa (and /lua-qa quick, /lua-qa <name>) is the quick pass from earlier releases, unchanged.

1. Discover. It compiles the agent and reads lua status, lua version list and the workflows. It draws a flow diagram, a decision tree per skill (persona must-never and escalation rules, per-tool conditions, ask-for-field paths) and a branch tree per workflow. The user confirms the map.
2. Inferred answers and metrics. An agent infers the qualifying answers (who the users are, top jobs, what must never happen, data sensitivity) and the metrics from the code, the persona, the skills and the tools. It cites a file:line for each. --interview brings the questions back.
3. Environment. Sandbox is the default, and a staged version with a test session is used when one exists. Production needs explicit consent (see Safety).
4. The plan and the run. At least 10 ICP personas (medium tier), a red team, offline workflow flow tests, direct tool tests and a stress test. They run in parallel through the Workflow tool, with parallel Agent calls as the fallback. Sonnet plays the personas and Opus the red team. Each run gets a fresh thread and a contamination check, and Opus grader A then grader B grade it. Safety is a veto.
5. Cluster analysis. Failures are grouped by root cause and ranked. Each cluster gets a fix locus: persona or skill prompt, tool description or schema, pre/postprocessor, platform gap, or move the logic out of the prompt into a workflow step, an approval gate, a validation schema or a code guard.
6. The report. A PDF, HTML and results.json in the Lua CI/CD-guide style: a dark cover with a tier badge, a TOC, an orange lede, stat tiles and Pass / Partial / Fail chips. Without pandoc a built-in renderer writes the HTML, and the PDF is skipped (with an install line) when WeasyPrint is missing.

The three tiers

	smoke	medium (default)	production-ready
Plan	4-5 personas, 1 red-team card, 1 run each	≥10 personas, ≥3 red-team, 3 runs (all pass) or 5 (4 pass)	≥12 personas, ≥4 red-team covering every exposed attack class, 5 runs (4 pass)
Checks	tool tests, one happy-path flow test per workflow, log scan, grader A only, no stress	all flow and tool tests, burst stress, log scan, graders A + B	every workflow branch, concurrent stress on a staged version, log scan, graders A + B
Time	30-minute hard cap from the plan approval	about 1-2 h	about 3-5 h
Verdict	Smoke: no blockers found / blockers found, never release-ready	Pass / Partial / Fail	release gate: Production ready: YES / NO, <n> blockers

The helpers enforce the tier. init-run writes it to state.json, which the planner never writes. validate refuses a plan that is over or under its tier, or over its time budget.

Safety enforced in code, not only in prompts

• Scrubbed environment. Every lua call goes through lib/qa/spawn.mjs with an environment allowlist (LUA_API_KEY is always dropped) and a coded argument allowlist, with the deploy classifier as a second check. A sandbox chat therefore cannot upload the shell's secrets.
• Fake data, or an agreed company domain. Persona data must be @example.*. A company email domain that a tool insists on can be agreed only at the environment gate. Addresses on it need an obviously fake local part, and public mailbox providers are refused.
• Production consent token with an ask rule. Production (or a staged version without a test session) needs the exact option I consent to running this against production. The stamp command matches an ask permission rule, so Claude Code itself prompts the user, and state.json keeps only a hash of the token.
• guard-qa-helper hook. The helper's allow rule is a glob, so the hook resolves the script's real path and blocks anything that is not this plugin's own lib/qa/cli.mjs. That makes 13 hooks.
• Memory switch only with consent. Every player chats as the same signed-in user. Discovery reads lua features list. Switching memory off for the run needs a verbatim consent, the user runs the switch commands themselves (an ask rule), start-run refuses until memory is verified off, and cleanup lists the restore until it reads back on. Otherwise memory findings carry a caveat, and cross-run recall is filed as a harness artefact, not an agent defect.
• A sealed plan. The approved plan is hashed, and edited plan files are refused (PLAN_CHANGED).

Tool-call evidence from lua logs (sandbox only)

The sandbox history route returns no tool calls. prechecks reads lua logs --type skill once per run, groups the rows by executionId into calls (tool, input, result, warnings, errors) and assigns each call to the turn whose window holds it. Claims are then checked against real calls: a ticket id or action claimed in a turn with no logged call is a confirmed candidate failure. backfill-tools fills in runs recorded earlier.

Live trial: Helix IT Desk (sandbox)

• First full run: 48 runs and a 47-page PDF. Its review produced the n/a-never-100% rule, NOT_PLAYED for runs that never started (instead of FAIL), and the cross-run memory isolation.
• Smoke run: finished in 14 of its 30 minutes, with tool-call evidence from the logs. Its five rough edges are fixed in the last commit, with tests:
  • the cover no longer claims a workflow test for an agent with none;
  • the flow-test tile shows only n/a: no workflows;
  • cards write fills in a missing schema;
  • cards write writes nothing when the bundle has a schema error, and says so;
  • the clock is documented and coded as starting at the plan approval, not init-run.

Permission glob: verified live

Bash(node *lua-agent-builder*/lib/qa/cli.mjs *): Claude Code's * crosses /, so the rule stays as written.

• Docs (code.claude.com/docs/en/permissions, "Wildcard patterns"): "A * in a Bash rule matches any text, including spaces", and "Bash rules match the whole command text, with * standing in for any text."
• Live probe (Claude Code 2.1.293, 2026-10-08): claude -p --setting-sources project --settings <file with only this allow rule> --permission-mode default. It was asked to run node <tmp>/fake/lua-agent-builder/x/lib/qa/cli.mjs --help, against a dummy cli.mjs that only echoes. The command ran, with no permission denial: PROBE_OK cli.mjs ran with args: --help. The first * crossed every / of the temp path and the second crossed /x/, the same shape as a real cache path (lua-agent-builder/1.9.0/lib).
• Controls (same flags, both refused with "This command requires approval"): the same command with a {} settings file, and node <tmp>/fake/other/x/lib/qa/cli.mjs --help with the rule.
• A temp dir's own .claude/settings.json is ignored until that workspace is trusted, so the rule was passed with --settings.

Because the glob is this wide, hooks/guard-qa-helper.mjs (a realpath check) is the real boundary. That is unchanged. The result is recorded in test/lib/qa/permissions-qa.test.mjs.

Known limitations

• Production runs can't attribute tool calls. Log-window attribution is sandbox only, so production claims are checked without tool-call evidence.
• Plugin agent types need the plugin installed. When lua-qa-player and the other roles cannot be resolved, each role runs as general-purpose and first reads its own agent file. The trial ran this way, so the real agent types (claude --plugin-dir) are untested.
• Quick mode still uploads the unscrubbed shell environment on sandbox chats, as before (SECURITY.md).

Decisions for reviewers

• 1.9.0, not 1.8.0 (see the note at the top).
• Squashed onto main. The feature is one squashed commit. The smoke-run fixes are a second commit, and a third fixes Windows paths that first showed up in this PR's CI: readability --turns-file used startsWith('/'), and the report-tool lookup counted a bare file without a PATHEXT extension. A fourth tightens the CHANGELOG into one coherent 1.9.0 entry. A fifth addresses review round 3: the fixtures read the version from package.json, the dead table() width-hint argument is gone, the glob's live verification is recorded, and the version order is spelled out. The rebase conflicted on the CHANGELOG, hooks/hooks.json and the permission template (feat(plugin): 1.8.0 — web apps (Lua Apps) in /lua-new, /lua-test, /lua-push, /lua-deploy #17's web-app rules). Both sides are kept.
• Inferred answers by default. At the user's request, gates 2-3 ask nothing. Inference never picks production and never agrees a company email domain.
• A double confirmation for production. The consent ask prompt sits on top of /lua-qa's own consent question, on purpose.
• The Workflow script is copied into the run folder, because the Workflow tool refuses scripts outside the working directory.
• The smoke cap fails closed. Runs and tests that the cap stopped are "not played" or "not run", never agent failures.

Test plan

• npm run lint (eslint + 17 scripts; release version 1.9.0 consistent)
• npm test: 67 suites, 2060 tests
• npm run test:coverage plus check-coverage.mjs per-file thresholds
• npx eslint --no-ignore on every changed .mjs
• MCP package: npm ci, npm test (149), npm run lint, npm run build (dist committed), check-bundle-size (0.50 MB)
• Live: a full medium run and a smoke run on Helix IT Desk (sandbox)
• Live permission probe of the helper glob (see above)
• Reviewer: /lua-qa full smoke with the plugin installed (claude --plugin-dir), so the real agent types are exercised live
• CI green on ubuntu, macOS and Windows (Node 18 and 20), plus the MCP job

🤖 Generated with Claude Code

[richard-lua]

Code review — HIGH risk

This is a very large (≈23k-line, 135-file) feature PR adding the /lua-qa suite. The code that was visible — schema validators, flow-graph path enumeration, report section builders, and the test suites — is careful and well-covered. The security-relevant additions (a new Bash allow glob, a consent ask rule, the scrubbed-spawn design) are clearly thought through, but the pieces they depend on for safety (hooks/guard-qa-helper.mjs, lib/qa/spawn.mjs, the deploy classifier) were not in the review set, and the project's own tests flag an unverified assumption about Claude Code glob semantics. Under strict risk tolerance the open major finding on the new auto-approval rule makes this HIGH risk and requires a human sign-off on the gate pieces before merge.

Major

• plugins/lua-agent-builder/lib/permissions-template.json:114 — The new Bash(node *lua-agent-builder*/lib/qa/cli.mjs *) allow rule auto-approves any cli.mjs under a lua-agent-builder path. permissions-qa.test.mjs itself notes the cross-/ glob behavior was not verified live, and the safety argument rests entirely on hooks/guard-qa-helper.mjs and lib/qa/spawn.mjs, neither of which is in this review. Confirm the guard hook resolves realpath and rejects spoofed/symlinked/.. paths (with tests) and verify the live glob via a claude -p probe; if it can't be verified, narrow the glob or move it to ask.

Minor

• plugins/lua-agent-builder/test/lib/qa/fixtures/runtime-helpers.mjs:62 — Fixture pluginVersion/cache paths hardcode 1.8.0 while the package bumps to 1.9.0; parameterize or read from package.json to avoid drift.
• plugins/lua-agent-builder/lib/qa/report/sections.mjs:200 — table() takes two params but several callers pass a third width-hint array that is silently ignored. Implement the hint or drop the dead argument.
• plugins/lua-agent-builder/package.json:3 — The 1.9.0 jump depends on cross-PR choreography (tagged v1.8.0, open 1.7.1). Re-run lint-release-version and reconcile CHANGELOG ordering at merge time.

---

_{PR Risk Reviewer — automated senior review of c817476 · risk: high · confidence: 0.45}

[lua-code-reviewer]

Code review: Needs a human · high risk

What to do next: the reviewer cannot judge this change, so it neither approves it nor asks for changes. Read the reasons below; a person who owns this area decides.

Review for @lua-stefan-kruger, requested through the code-reviewer team.

A person needs to make this call. The reviewer will not approve it:

• Too large to review reliably: 135 files, 23167 changed lines (limits 60 files / 2500 lines). Split the change or have a human review it.

Contracts and consumers

Surfaces touched: event payload, env var, service client.
Consumer search: nothing to search for.

_{Automated review by lua-code-reviewer[bot]. Every new push is reviewed again automatically. To have this head reviewed again, request the code-reviewer team as a reviewer.}

Machine data (for the fleet)

Reviewed head 68601dd5a6fa (round 1) · CI: green · no model review (deterministic checks only) · model: anthropic/claude-opus-5-5

[richard-lua]

Code review — MEDIUM risk

Previously flagged

• Previously flagged New broad Bash allow glob relies on an unverified guard hook — resolved: hooks/guard-qa-helper.mjs now exists and is imported/exercised by permissions-qa.test.mjs, which asserts it blocks spoofed /tmp and cache paths that match the glob, and the deploy classifier is re-asserted not to flag helper calls. The residual (live cross-/ glob behavior unverified) is downgraded to a minor note with a documented safe prompt fallback.
• Still open (minor): 1.8.0 fixture drift — fixtures/tests still hardcode 1.8.0 cache paths while package and lockfiles are 1.9.0.
• Still open (minor): table() unused third argument — callers still pass an ignored width-hint array.
• Still open (minor): 1.9.0 version choreography — depends on tag/CHANGELOG ordering vs open #18.

The guard-hook addition materially tightens the one blocking concern from round 1, so this drops from HIGH to MEDIUM. Remaining items are advisory.

Minor

• test/lib/qa/permissions-qa.test.mjs:5 — The *-across-/ glob semantics for Bash(node *lua-agent-builder*/lib/qa/cli.mjs *) remain unverified live. Safe (fails closed to a prompt) but confirm via a claude -p probe; the guard hook is the enforced boundary regardless.
• test/lib/qa/fixtures/runtime-helpers.mjs:62 — Fixture/cache paths still hardcode 1.8.0 while the package is 1.9.0; parameterize or read from package.json.
• lib/qa/report/sections.mjs:200 — table(headers, rows) ignores the third width-hint array several callers pass; implement it or drop the dead argument.
• package.json:3 — Re-run lint-release-version and reconcile CHANGELOG ordering at merge given the open 1.7.1 PR.

---

_{PR Risk Reviewer — automated senior review of 68601dd · risk: medium · confidence: 0.50 · re-review · round 2}

[richard-lua]

Previously flagged

• Previously flagged New broad Bash allow glob relies on an unverified guard hook — resolved in round 2: the guard hook exists and is exercised by permissions-qa.test.mjs, which asserts spoofed/cache paths are blocked; remains closed.
• Still open (minor): Cross-/ glob semantics unverified live — permissions-qa.test.mjs still documents "NOT verified live"; no claude -p probe added. Safe: fails closed to a prompt and the guard hook is the enforced boundary.
• Still open (minor): 1.8.0 fixture drift — fixtures/tests still hardcode 1.8.0 cache paths while the package and lockfiles are now 1.9.0.
• Still open (minor): table() unused third width-hint argument — sections.mjs still ignores the hint array passed by some callers.
• Still open (minor): 1.9.0 version choreography — version bumped consistently across package.json and both lockfiles, but ordering vs open #18 and the existing v1.8.0 tag must be reconciled at merge.

The new commits do not introduce new blocking issues; the visible additions (schema validators, flow-model graph walk, report sections) remain well-covered and defensively coded. All remaining items are advisory. The large blast radius and the unverified live glob keep this at MEDIUM under strict tolerance.

Minor

• test/lib/qa/permissions-qa.test.mjs:5 — Cross-/ glob behavior for Bash(node *lua-agent-builder*/lib/qa/cli.mjs *) is still unverified live. Confirm via a claude -p probe; the guard hook enforces the real boundary regardless.
• test/lib/qa/fixtures/runtime-helpers.mjs:62 — Fixture/cache paths still hardcode 1.8.0; read from package.json to avoid drift.
• lib/qa/report/sections.mjs:200 — table(headers, rows) ignores the third width-hint array some callers pass; implement it or drop the dead argument.
• package.json:3 — Re-run lint-release-version and reconcile CHANGELOG ordering at merge given the open 1.7.1 PR and existing v1.8.0 tag.

---

_{PR Risk Reviewer — automated senior review · re-review · round 3}

---

_{PR Risk Reviewer — automated senior review of 61aa627 · risk: medium · confidence: 0.50 · re-review · round 3}

[lua-stefan-kruger]

@richard-lua, round 3 is addressed in e92aed6 (SSH-signed):

• 1.8.0 fixture drift (fixed). The new test/lib/qa/fixtures/plugin-version.mjs reads the version from package.json. runtime-helpers.mjs, fixture-run.mjs, tier-fixture.mjs, build.test.mjs, permissions-qa.test.mjs and permissions-mirror.test.mjs now use it, and no 1.8.0 cache path is left in the QA tests.
• table() width hint (fixed). The column widths come from the content, so I removed the ignored third argument from all 19 callers. The rendered output is unchanged, and the snapshots pass.
• 1.9.0 version order (documented). The CHANGELOG and the PR body now cover both merge orders:
  • This PR first (planned): fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 takes 1.9.1.
  • fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 first: fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18 takes 1.8.1, and this stays 1.9.0 on top.
  • fix(hooks): 1.7.1 — production gate classifies what a command runs, not text it mentions #18's current 1.7.1 is below the v1.8.0 tag in either order. lint-release-version passes at 1.9.0.
• Glob across / (verified live, rule unchanged).
  • The docs say: "A * in a Bash rule matches any text, including spaces."
  • The live claude -p probe on Claude Code 2.1.293 used only this allow rule and --permission-mode default. node <tmp>/fake/lua-agent-builder/x/lib/qa/cli.mjs --help ran without a prompt.
  • Both controls were refused: the same command with no rule, and an other/ path with the rule.
  • The details are in permissions-qa.test.mjs and the PR body. The guard hook is still the real boundary.

Gates: npm run lint, npm test (2060 passing), npm run test:coverage (per-file thresholds met), and eslint --no-ignore on the changed .mjs files.

[richard-lua]

Code review — MEDIUM risk

Previously flagged

• Previously flagged Cross-/ glob semantics unverified live — resolved: permissions-qa.test.mjs now documents a live probe (Claude Code 2.1.293, 2026-10-08) confirming * crosses /, with refused controls; the realpath guard hook remains the enforced boundary.
• Previously flagged 1.8.0 fixture drift — resolved: new test/lib/qa/fixtures/plugin-version.mjs reads the version from package.json; runtime-helpers, permissions-qa and permissions-mirror now use cacheCliPath() and no 1.8.0 cache path remains in the QA tests.
• Previously flagged table() unused third width-hint argument — resolved: widths are now content-derived and the ignored argument was dropped from all callers; snapshots unchanged.
• Still open (minor): 1.9.0 version choreography — version is bumped consistently across package.json and both lockfiles, and both merge orders are documented, but CHANGELOG ordering and lint-release-version must be reconciled at merge vs open #18 and the existing v1.8.0 tag.

The new commits (e92aed6) address all three outstanding advisory items and introduce no new blocking issues. Nothing blocking remains open; the remaining item is advisory and a merge-time check. This PR is ready pending that reconciliation.

Minor

• package.json:3 — Re-run lint-release-version and confirm CHANGELOG ordering at merge given the open #18 (1.7.1) and the existing v1.8.0 tag.

---

_{PR Risk Reviewer — automated senior review · re-review · round 4}

---

_{PR Risk Reviewer — automated senior review of e92aed6 · risk: medium · confidence: 0.50 · re-review · round 4 (converged — no new blocking findings)}

[lua-code-reviewer]

Code review: Needs a human · high risk

What to do next: the reviewer cannot judge this change, so it neither approves it nor asks for changes. Read the reasons below; a person who owns this area decides.

Review for @lua-stefan-kruger, requested through the code-reviewer team.

A person needs to make this call. The reviewer will not approve it:

• Too large to review reliably: 136 files, 23179 changed lines (limits 60 files / 2500 lines). Split the change or have a human review it.

Contracts and consumers

Surfaces touched: event payload, env var, service client.
Consumer search: nothing to search for.

_{Automated review by lua-code-reviewer[bot]. Every new push is reviewed again automatically. To have this head reviewed again, request the code-reviewer team as a reviewer.}

Machine data (for the fleet)

Reviewed head e92aed6e1593 (round 2) · CI: green · no model review (deterministic checks only) · model: anthropic/claude-opus-5-5

[lua-stefan-kruger]

Follow-up in the next commit: the CHANGELOG and PR body had said #18's 1.7.1 would get no release. That was wrong. release-prod.yml only checks that the tag is new, and v1.7.1 doesn't exist, so an unbumped #18 would cut an out-of-order v1.7.1. The note now says #18 must take a version above 1.8.0 (1.9.1, or 1.8.1 if it merges first). The probe comment also states exactly which / each * crossed.

[richard-lua]

Code review — MEDIUM risk

Previously flagged

• Previously flagged New broad Bash allow glob relies on an unverified guard hook — resolved (round 2): guard hook present and exercised by permissions-qa.test.mjs.
• Previously flagged Cross-/ glob semantics unverified live — resolved (round 4): live probe documented in permissions-qa.test.mjs (Claude Code 2.1.293), realpath guard hook is the enforced boundary.
• Previously flagged 1.8.0 fixture drift — resolved (round 4): fixtures/plugin-version.mjs reads version from package.json; QA tests use cacheCliPath().
• Previously flagged table() unused third width-hint argument — resolved (round 4): widths are now content-derived in sections.mjs and the ignored argument dropped from all callers.
• Still open (minor): 1.9.0 version choreography — the new commits correct the earlier CHANGELOG/PR-body claim (an unbumped #18 at 1.7.1 would in fact cut an out-of-order tag, not be skipped) and the permissions-template _comment now documents the 1.9.0 rules. The remaining reconciliation vs open #18 and the existing v1.8.0 tag is a merge-time human decision, not a code defect.

The new commits are documentation/version-note corrections plus the rebuilt bundle; they introduce no new blocking issues. The schema validators, flow-model graph walk, and report sections remain well-covered and defensively coded (markdown/HTML escaping, path allowlisting in figure(), env scrubbing). This PR has reached the round limit with nothing blocking open.

Minor

• package.json:3 — Re-run lint-release-version and confirm CHANGELOG ordering at merge given open #18 and the existing v1.8.0 tag; ensure #18 is bumped above 1.8.0.

Decision for a human: no blocking findings remain open. The PR is ready pending the merge-time version reconciliation.

---

_{PR Risk Reviewer — automated senior review · re-review · round 5 (converged — no new blocking findings)}

---

_{PR Risk Reviewer — automated senior review of 555105f · risk: medium · confidence: 0.50 · re-review · round 5 (converged — no new blocking findings)}

[lua-code-reviewer]

Code review: Needs a human · high risk

What to do next: the reviewer cannot judge this change, so it neither approves it nor asks for changes. Read the reasons below; a person who owns this area decides.

Review for @lua-stefan-kruger, requested through the code-reviewer team.

A person needs to make this call. The reviewer will not approve it:

• Too large to review reliably: 136 files, 23180 changed lines (limits 60 files / 2500 lines). Split the change or have a human review it.

Contracts and consumers

Surfaces touched: event payload, env var, service client.
Consumer search: nothing to search for.

_{Automated review by lua-code-reviewer[bot]. Every new push is reviewed again automatically. To have this head reviewed again, request the code-reviewer team as a reviewer.}

Machine data (for the fleet)

Reviewed head 555105fb42a3 (round 3) · CI: green · no model review (deterministic checks only) · model: anthropic/claude-opus-5-5