test: end-to-end rollback harness against a real Fly.io app

[teckedd-code2save]

End-to-end rollback test harness against a real Fly.io app

Summary

The rollback path (medic, src/adapters/fly/runner.ts) exists for Fly.io but has never been tested against a real Fly app deployed by a previous Convoy apply. Scripted rollback mode uses canned data. Documented in PENDINGS.md (Critical, item 3) — "until then, the rollback is an educated guess."

What to do

• Build a test harness that:
  1. Deploys a real Fly app via Convoy
  2. Injects a failure (e.g., bad release / crash)
  3. Runs the rollback path end-to-end
  4. Verifies the app is restored to the previous known-good release
• Can be gated behind an env flag / opt-in so CI stays hermetic.

Why it matters

Rollback is the last line of defense for a deploy agent. An untested rollback can fail exactly when a deploy goes wrong — the highest-stakes moment in the product.

Acceptance criteria

• Harness deploys a real app, injects failure, and verifies recovery
• Rollback restores the previous release (image + config verified)
• Test results documented (e.g., in docs/ or the test README)
• Runnable on demand; clear instructions in repo docs

[teckedd-code2save]

Built — end-to-end rollback harness against a real Fly.io app

scripts/rollback-harness.sh + scripts/rollback-harness/bad.Dockerfile, documented in docs/rollback-harness.md and wired into the README's Verification section. PENDINGS Critical #3 marked fixed (f3cae86, f414e61).

Acceptance criteria mapping

• Harness deploys a real app, injects failure, and verifies recovery — Phase 1 ships demo-app/ to Fly via Convoy's real pipeline (convoy ship --platform fly → canary deploy → promote live-verify → observe). Phase 2 injects a broken release: an HTTP-500-on-every-request image (bad.Dockerfile), deployed as a new release. Phase 4 verifies recovery.
• Rollback restores the previous release (image + config verified) — Phase 3 rolls back via convoy rollback <app> -y (the real flyRollbackPreview + flyRollback path in src/adapters/fly/runner.ts). Phase 4 asserts three independent signals: current release image ref == known-good image ref, release status complete, /health 200 + /orders serving data.
• Test results documented — docs/rollback-harness.md: phases, pass criteria, troubleshooting, scope notes.
• Runnable on demand; clear instructions in repo docs — one command: scripts/rollback-harness.sh (needs fly auth login first — the preflight gate, which also keeps CI hermetic).

Why a bad image, not an env flag

Fly secrets are app-level, not per-release — rolling back the image does not unset them. A broken image is the only failure mode a rollback can actually reverse, so it's the honest test (DEMO_HEALTH_FAIL-style injection would stay broken after rollback and fail for the wrong reason).

What was verified in this environment

• npm run typecheck ✅, npm test (58/58) ✅
• Harness syntax + helper logic (health polling, release-image parsing) against fixtures ✅
• Failure-injection image built and run in Docker: boots, logs BAD_RELEASE_READY, answers /health with HTTP 500 ✅
• Full convoy ship flag surface exercised up to the auth boundary (clean preflight blocker, exit 2) ✅
• Preflight gate: without a fly session the harness refuses with flyctl not authenticated. Run: fly auth login ✅

The live end-to-end run is a single command but needs an interactive fly auth login (browser flow), which a scheduled environment can't do — so the harness is built to be run on demand, exactly as the issue requests: fly auth login && scripts/rollback-harness.sh. Expected runtime ~8–15 min, ending in ROLLBACK HARNESS PASSED — <app> with the good/bad/restored release evidence printed.