Skip to content

feat(software-factory): target:measure proposes a target's excludes and resources - #859

Open
blove wants to merge 14 commits into
blove/targets-initfrom
blove/targets-measure
Open

blove wants to merge 14 commits into
blove/targets-initfrom
blove/targets-measure

Conversation

@blove

@blove blove commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Framework-gaps item 5, PR 2 of 2. Plan: docs/superpowers/plans/2026-09-25-targets-generated-from-a-package.md, Tasks 12-19.

Stacked on #858. The base is blove/targets-init. After #858 squash-merges, this branch needs git rebase --onto origin/main blove/targets-init, and the PR needs retargeting to main.

What it adds

pnpm --filter @b4-example/software-factory-controller target:measure <id> [--pin] [--runs N] [--allow-decrease] [--targets-dir] [--write]

The command runs each test file of a target alone, in the target's image, with the network denied, exactly as the verifier would run it. It then proposes the target's excludes and resources as a diff plus a committed targets/<id>/measurement.md. It writes only with --write.

  • Verdicts.
    • pass
    • fail
    • hang
    • killed (exit 137 with no report)
    • writes (the file changed the workspace, or left a workspace the verifier's inspection refuses)
    • flaky
  • Flaky files are never excluded. Every non-pass runs once more in a fresh container. If the two runs disagree, the file is flaky: listed, but never proposed for exclusion.
  • Fresh session after side effects. A new session starts after any file that wrote, hung or was killed. Listing runs in a session of its own.
  • Capture omissions are named. An ENOENT on a path that exists at the pin is labelled a capture omission.
  • Resources.
    • Measured in fresh whole-suite containers.
    • The proposal is the larger of the existing and the measured value, unless --allow-decrease.
    • The verifier deadline always covers each session running one command to its timeout, so a hung candidate is rejected rather than blocked.
    • The proposal is confirmed by a suite run at exactly the proposed values. That run must pass the way the verifier grades it: a valid report, at least one passing test and no failures.
  • Promotion set. When a build fails at the promotion check, the set is learned from the failed build and proposed as a Dockerfile diff.
  • measurement.md is deterministic. Test output is sanitised, UUIDs and temp names are masked, and each entry shows its test counts. A reviewer can see when a mostly-passing file is dropped.
  • Unmeasured targets can be measured. target:measure loads a target with placeholder resources. Those targets stay hidden from the drafter until measured.

Proof

  • Docker lane (devkit). measure on the generated devkit target proposes exactly the nine committed excludes and the committed test command, and names each exclude's cause: the uncaptured templates/ directory. Proposed resources are 512/2/60000/180000, beside the committed 768/2/60000/240000. The lane takes about 85 s locally.

  • cli by hand, in a scratch catalog. The generated cli target covers 169 files and took about 23 minutes on a warm image.

    Verdict Files
    pass alone 114
    proposed exclude 55
    flaky 0
    • All eight files the hand-written target runs pass.
    • 29 of the excludes reach @b4run/testing's build output by relative path, which needs a capture edited by hand.
    • The first run found a real defect, fixed here. A test that chmods the built CLI made the verifier's inspection refuse the workspace. That is now the file's writes verdict instead of a stop.
    • With the deadline fix, cli's proposal would be 1280/2/420000/960000.

Review

Every task had a spec review and a quality review, then the branch had a final review. The "As landed" notes record every fix.

Verification (local, Node 24)

  • Unit tests: controller 1036 passed, server 21 passed, drafter 17 passed.
  • test:sandbox: 13 files passed and 1 skipped (opt-in); 32 tests passed and 5 skipped, in 299 s.
  • pnpm lint, node scripts/check-docs.mjs and the factory typechecks pass.

Examples only, so there is no changeset.

🤖 Generated with Claude Code

blove and others added 14 commits September 25, 2026 19:40
…ed; a measure capture role

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sources from suite samples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a whole

measureSuite over an injected session: each listed file alone, a fresh
container after a file that wrote, hung or was killed; each non-pass once
more in a fresh container (flaky when they disagree); the suite sampled in
fresh containers and confirmed at the proposed resources. The listing drops
vitest's [project] prefix and an empty one stops; session.ts has the pure
helpers the Docker session reads vitest's output with (relative paths, the
per-file status exactly).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs, keep measurement.md deterministic

Test output reaches report.md, measurement.md and the printed diff only
through sanitize(): escape sequences and C0/C1 controls (but newline and
tab) stripped, `<` escaped. Paths in a reason are JSON-quoted; listed file
names holding a control are refused; error lines in the record are code
spans. Exit 0 with the file not passed is a MeasureError. proposeResources
refuses non-positive or non-finite samples; settleResources ignores a
placeholder prior. The committed record carries no limits, no second run's
evidence (new secondRun field, report only) and sorts by file.
captureOmissions normalises paths and ignores the workspace root. A
MeasureError in the first per-file pass carries the files measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…haped like the verifier's

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…resources and promotion set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cludes on a generated target

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… grades a pass

A suite sample or the confirmation must pass gradeVitestReport, not merely
exit 0: a missing report or an all-skipped suite is refused by name. An
exclude withExcludes cannot write is a MeasureError naming the file (with
the partial report), and is noted when listed. The listing runs in its own
session; a build killed or timed out names the limits; SIGTERM tears the
sessions down; writeProposal's refusals are neutral; measurement.md no
longer claims every file passed when flaky files are listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-landed note

The README and target-init's header now describe the two-command flow
(init, measure, review the diff and measurement.md, commit) instead of
calling target:measure a follow-up; the placeholder-resource refusal stays
as the fail-safe for a target committed without measuring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fier's inspection refuses as writes

The hand measurement of the generated cli target stopped after 16 files on
"Executable workspace file: packages/cli/dist/index.js": check-command.test.ts
chmods the built CLI 0755, and the inspection refuses an executable file, as
the verifier's tamper check does on every verification. A refused snapshot
after a file's run now makes the file writes (reason names the refusal) and
the next file gets a fresh session. A refusal before any test ran, or after
the suite, stops the measurement; any other snapshot failure is a MeasureError
that keeps the files measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eout per session; stable, counted measurement.md

The verifier runs its visible and independent sessions under one deadline, and a
hung suite runs one command to its timeout in each. The proposed deadline could
sit below one command timeout (the generated cli target: 300000 beside 420000),
so a hung candidate went inconclusive instead of rejected. verifierDeadlineMs is
now at least twice (commandTimeoutMs + the slowest session), re-applied after
settleResources (coveringDeadline).

measurement.md masks UUIDs and mkdtemp suffixes, drops code-frame lines and caps
each error line at 240 characters, so a re-measurement that agrees rewrites it
byte for byte; each entry shows its test counts, and the record opens with a
count of every verdict (D15). A round-trip test loads a written proposal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… measure lane at 15 minutes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd the final review

--with-dev-builds does not reach the 29 files that import @b4run/testing by
relative path; run-typegen and verify-command are capture omissions the detector
missed (follow-up); the cli deadline recomputed as 960000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
b4-run Ignored Ignored Preview Sep 26, 2026 5:33am UTC

Request Review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: this PR received an intelligent (AI) code review. See the review comments on this PR.

@blove
blove deployed to vercel-preview September 26, 2026 05:33 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
vercel-preview — 6cefa501 Deployed Sep 26, 2026 by blove via vercel-native #2433
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant