Verify local environment during setup | **Head:** issue-313-setup-verification

[heyoub]

Draft pull request

Title: Verify local environment during setup

Repository: CodeForPhilly/benefit-decision-toolkit

Base: main

Head: issue-313-setup-verification

Draft: yes

---

Summary

This change makes the developer setup lifecycle verify that a Devbox-managed local environment or GitHub Codespace is not merely installed, but actually usable end to end.

bin/setup now runs the Java test suites and then delegates to the existing local E2E lifecycle so setup verifies the complete stack with:

• Library API Maven tests
• Builder API Maven tests
• Bruno requests against the local Library API
• Playwright Chromium tests against the local frontend and emulators
• Python-based local readiness and socket checks
• transactional handling of emulator-data
• verified service shutdown and conservative cleanup

The implementation is intentionally limited to:

• bin/setup
• bin/run-e2e-tests

No product code, workflow file, lockfile, or permanent documentation file is changed.

Closes #313.

Problem

Issue #313 asks setup to run E2E tests, Java Maven tests, and Bruno tests so a new Devbox-managed local environment or Codespace proves that it is configured and started correctly.

Before this change, setup could finish after dependency installation and Maven packaging without proving the full local graph. In particular:

• Builder API packaging used -DskipTests, so its 40 tests were not part of setup verification. Library API packaging already ran its 14 tests.
• Bruno and Playwright were not run as part of the setup lifecycle.
• The local E2E runner assumed a narrower Devbox/process-compose state than a fresh isolated worktree actually has.
• Cleanup could not distinguish partial-startup and partial-restoration states strongly enough to preserve recoverable emulator state conservatively before further mutation.

A setup command that only installs dependencies can still leave a contributor with a broken service graph, missing browser runtime, failing API behavior, or E2E fixtures that cannot be loaded. This PR changes Devbox-managed setup from an installation-only path into a local environment verification path.

Root cause and state model

The key orchestration bug was treating the absence of a process-compose registration as an error.

In a fresh or fully stopped worktree, all of the following may legitimately be absent:

• a registered process-compose instance
• an active process-compose control listener
• .env files before the Codespaces bootstrap pre-step
• emulator-data
• an existing feature-worktree service graph

That is different from a registered graph, an occupied application port, malformed process-compose data, or an indeterminate socket state.

This PR codifies the distinction explicitly:

Observed state	Meaning	Behavior
No process-compose registration and every managed application port is free	Valid fresh or stopped worktree	Continue
Valid process-compose registration with its control port listening	Existing local graph	Refuse during setup, or stop deliberately inside the E2E runner
One or more managed application ports occupied without a known active project graph	Unknown ownership/conflict	Refuse before touching emulator data
Registered control port exists but is not listening, while managed ports are free	Stopped registration	Continue after confirming all ports are closed
Invalid process-compose port data	Ownership cannot be determined safely	Fail closed
Socket availability cannot be determined reliably	State is indeterminate	Fail closed

The managed application ports are:

5173  builder frontend
8080  Firestore emulator
8081  Builder API
8083  Library API
9099  Auth emulator
9199  Storage emulator

The ordering is deliberate: port ownership and shutdown are resolved before emulator-data is moved or replaced. The runner never mutates emulator state while another or unknown local process may still own the graph.

Implementation

bin/setup

Setup now performs a preflight when running inside Devbox:

1. Requires Python for deterministic local socket checks.
2. Checks every managed application port.
3. Reads the process-compose port as optional state.
4. Refuses an active project graph or occupied managed port.
5. Fails closed on malformed process-compose data or indeterminate socket results.

The Java builds now run their tests:

cd library-api
mvn clean package

cd builder-api
mvn clean package

The previous Builder API -DskipTests behavior is removed. Library API packaging remains unchanged because it already runs its test suite.

After environment files are prepared, Devbox setup runs:

bin/run-e2e-tests

This makes the normal setup lifecycle prove the complete local development environment instead of reporting success after dependency installation alone.

bin/run-e2e-tests

The E2E runner now treats local state as a transaction.

Preconditions

It verifies:

• devbox, python3, node, npm, and bru are available
• the checked-in emulator fixture directory exists
• root emulator-data is either absent or a real directory
• symbolic links, regular files, sockets, pipes, and device nodes are refused
• the repository-local Playwright executable exists after npm ci

Browser installation

The runner uses the repository-local Playwright version and installs only Chromium:

./node_modules/.bin/playwright install --with-deps chromium

This avoids an ambient npx version and keeps the browser runtime tied to the lockfile. On Linux, the first run may invoke the system package manager through sudo; Codespaces supports that noninteractively.

Service ownership

Before fixture mutation, it:

1. Checks every managed application port.
2. Reads process-compose registration as optional state.
3. Stops a known active project graph deliberately.
4. Waits for the control and application ports to close.
5. Refuses any remaining or unknown listener.

Emulator-data transaction

When original emulator-data exists, it is moved into a recovery directory under logs/ and the move is verified before fixture installation.

The runner then:

1. Installs the E2E emulator fixture.
2. Starts the temporary Devbox graph.
3. Waits for Firestore, Storage, Auth, and the Library API.
4. Runs the Bruno collection against the local Library API.
5. Waits for the frontend.
6. Runs the Chromium Playwright suite.

The cleanup trap preserves the original test exit status while independently tracking cleanup failure.

When the cleanup trap runs, it will not remove or restore emulator data until it can prove the service graph is stopped. It verifies:

• any temporary graph has stopped
• all managed ports are closed
• E2E fixture data was removed
• an expected backup is still present
• the restore destination is absent
• the original data was moved back successfully
• no backup payload remains afterward

If restoration cannot be proven safe, cleanup fails loudly, reports the surviving backup/current paths, and tells the operator not to rerun until recovery is resolved. It does not paper over an unsafe state with || true.

On successful cleanup, the same transaction restores either:

• the original emulator-data directory, or
• the original absence of emulator data

No shell trap can recover from SIGKILL or abrupt container destruction. In those cases, any completed backup remains the recovery source and the runner reports recovery paths whenever it still has an opportunity to do so.

Codespaces lifecycle

The validated Codespaces entrypoint is the repository's existing devcontainer lifecycle command:

bin/ensure-root-env-file && devbox run setup

The pre-step matters because devbox.json loads the root file through env_from before Devbox invokes bin/setup. A completely absent .env therefore prevents the standalone devbox run setup command from entering the setup script.

This PR does not change that configuration contract. It verifies the complete Codespaces lifecycle already defined by .devcontainer/devcontainer.json.

Validation

The exact lifecycle command completed successfully from an isolated feature worktree:

bin/ensure-root-env-file && devbox run setup

Exit status: 0
Runtime:     365 seconds (6m 5s)

Java

Library Maven:       14/14 passed
Builder Maven:       40/40 passed
Failures:             0
Errors:               0
Skipped:              0

Library API classes:

• DynamicEndpointPatternTest: 4
• ModelDiscoveryTest: 4
• OpenAPISchemaPatternTest: 4
• SituationTypeValidationTest: 2

Builder API classes:

• DmnParserTest: 4
• FormDataTransformerTest: 13
• InputSchemaServiceTest: 18
• LibraryApiServiceTest: 5

Bruno

Requests:    36/36 passed
Assertions:  51/51 passed
Runtime:     604 ms

All requests targeted the local Library API. No cloud credentials were requested.

Playwright

Browser: Chromium
Tests:   8/8 passed
Runtime: approximately 1 minute

The suite covered:

• database setup
• screener creation
• adding and configuring a benefit
• creating and previewing a form
• publishing a screener
• authenticated landing-page smoke coverage

The repository-local Playwright 1.57 installation downloaded:

• Chromium 143.0.7499.4, build 1200
• Chromium headless shell, build 1200
• FFmpeg, build 1011

State-machine and shell validation

Disposable state checks: 54 passed
bash syntax checks:       passed
git diff checks:          passed

The disposable checks covered fresh/stopped state, active registration, malformed registration, occupied ports, partial startup, cleanup behavior, backup restoration, and fail-closed socket outcomes without touching the real service graph.

CI coverage note

The current path-filtered workflows do not execute bin/setup or bin/run-e2e-tests for this two-script diff. The results above come from the exact Codespaces lifecycle command and the disposable state harness. Expanding workflow triggers is intentionally outside this PR's scope.

Temporary feature graph

The validation started a temporary graph, proved readiness, ran the tests, and stopped it successfully.

After cleanup:

process-compose registration: absent
5173:  closed
8080:  closed
8081:  closed
8083:  closed
9099:  closed
9199:  closed
emulator-data: absent (matching baseline)
recovery backups: none
E2E fixture data: removed

The separate validation harness then removed the validation-created Playwright cache to return the Codespace to its pre-run baseline. Browser-cache removal is not a cleanup responsibility of bin/run-e2e-tests.

Original developer graph preservation

The original main worktree's service graph was stopped gracefully before isolated validation and restarted afterward.

The original emulator-data digest immediately after graceful stop and after graph restoration matched exactly:

6328281d5ace97ea5432b9f25cb81ab140dd001dfafd6a21d6886ec6d1f25a08

The match covered 22 filesystem entries.

The original worktree remained at the same commit with only its known pre-existing devbox.lock metadata drift. The feature worktree returned to a clean state after commit, and the branch contains one commit changing only the two approved scripts.

Network boundary observed during validation

Directly evidenced external package/browser hosts were:

• repo.maven.apache.org
• deb.debian.org
• cdn.playwright.dev

Runtime application traffic used only localhost. Captured output showed no staging/production application requests and no Google Cloud or Firebase cloud service requests.

Relationship to other work

PR #460

This change builds on #460, which updated the E2E suite for current UI behavior and introduced the local bin/run-e2e-tests path. This PR hardens that runner and wires it into setup; it does not duplicate the UI selector or test-maintenance work from #460.

Issue #443

PR #460 introduced the local E2E runner and was intended to resolve #443. This PR builds on that implementation by hardening the runner and integrating it into setup.

This PR intentionally includes no closing keyword for #443. Maintainers can decide separately whether PR #460 already satisfied that issue.

Known limitation before ready-for-review

The underlying first-time Playwright dependency transaction was verified separately in this canonical Debian 13 Codespace. The repository-local Playwright 1.57 command installed the missing Chromium dependency set, added 92 Debian packages, downloaded Chromium, and launched it successfully.

The later successful full lifecycle run occurred after those operating-system packages were already present. During that run, Playwright reported:

0 upgraded
0 newly installed
0 removed
6 not upgraded

Therefore the combined evidence proves:

• first-time ./node_modules/.bin/playwright install --with-deps chromium succeeds on the canonical Debian 13 Codespace image
• the setup orchestration
• Maven, Bruno, and Chromium execution
• local graph startup/readiness
• fixture installation and cleanup
• emulator-data restoration
• use of the repository-local Playwright installation

The remaining unverified case is the entire setup lifecycle from a brand-new Codespace with those OS dependencies initially absent.

Before this draft is marked ready for review, rebuild or open a pristine Codespace and rerun:

bin/ensure-root-env-file && devbox run setup

Reviewer mental model

The safety invariant is:

Setup and E2E verification may own the managed local graph and emulator data only after ownership is known and every conflicting port is closed. When cleanup can run, it must restore prior data or prior absence; if safe restoration cannot be proven, it must preserve recoverable state and fail closed.

The implementation is intentionally conservative because emulator state is developer data. A false refusal is recoverable; mutating data while ownership is unknown is not.

Checklist

• Java tests run during setup
• Bruno runs against the local Library API
• Chromium Playwright suite runs locally
• Fresh/stopped process-compose registration is accepted
• Active or unknown port conflicts are refused before emulator mutation
• Invalid and indeterminate states fail closed
• Emulator fixtures are handled transactionally
• Cleanup restores prior data or prior absence
• Temporary feature graph leaves no managed listeners
• Original developer emulator data was preserved exactly
• Diff is limited to bin/setup and bin/run-e2e-tests
• First-time Playwright --with-deps chromium installed missing Debian 13 packages and launched Chromium
• Fresh Codespace full setup lifecycle verified with Playwright OS dependencies initially absent