Δ1 — Bake Piccolissimo solve AMI (runner_image: pkgimage + julia 1.12.6 + CairoMakie)

[jack-champagne]

C0. Add a solve-runner image recipe/component to the existing runner_image Image Builder that bakes the Piccolissimo pkgimage (from the rel-* release the GCP julia-builder produces) + julia 1.12.6 into /opt/julia-depot, plus CairoMakie (the #24 recheck found it absent from the bundle). Point the S/M/L launch templates at it — replacing today's install-Piccolo-at-boot user-data.

Acceptance (TDD — write the boot-smoke first): an instance launched from the baked AMI runs BundleLoader.load() + a 2-level X-gate solve to FINISHED{completed} with no network install and no CairoMakie import error, and julia --version == 1.12.6.

⛔ Gate: blocked by the wk-3 freeze — run-dir contract + SchemaPackage v0 (amicode #15 + the #28–31 stack). RemoteExecutor/the S3 layout is the run-dir contract, so do not start until that lands.

📄 Refs: Spec armonissima/specs/spec-20260628-amicode-cloud-solve-service.md (Δ1) · phased plan §5a (plan-20260603-124231-amicode-phased-build) · aws-infra/INFRASTRUCTURE.md → The Analysis Pipeline (reuse step_function/ec2_launch_template/dynamodb/runner_image).

Evolve, not build — a working v0 (the aws-infra Analysis Pipeline) already does size→S/M/L EC2→julia→S3→email→terminate. Validate on staging (mirrors prod), then two-step promote.

🏗 Implementation repo: aws-infra (Terraform / lambda / Image Builder). Infra-side reference: harmoniqs/aws-infra#124.

[jack-champagne]

Implementation PR (infra-only — this delta is entirely aws-infra Terraform/Image Builder, no amicode code): harmoniqs/aws-infra#131 → staging (infra ref: #124).

Status: PR up + reviewed (the review reproduced the full bake green locally on the exact pkgimage + julia 1.12.6 — solve + plot_pulse + version assert all pass), review fixes applied (pinned CairoMakie/Makie, hardened the artifact bucket), terraform validate clean.

Remaining: merge #131 to staging → CI builds the bucket+pipeline → stage the pkgimage → trigger → on-AMI smoke + the definitive cold-start number. No amicode change lands here; the amicode-side cloud work is #32/#33/#34 (gated on Δ2/Δ4 + the wk-3 freeze).

[jack-champagne]

Δ1 shipped and verified on staging. Published AMI: ami-014ed5bb03b52b34d (harmoniqs-solve-runner-v2-2026-07-01T09-49-12.238Z).

What happened

The original implementation PR (aws-infra#131) had two real bugs that only surfaced once actual AWS builds ran against it (terraform plan/validate can't catch these — Image Builder's component schema check and Julia's own type resolution are both apply-time/runtime):

1. Two command lines in the component YAML were unquoted plain scalars containing "FATAL: — the colon-space made YAML parse them as nested maps, which Image Builder's CreateComponent rejected. Fixed in aws-infra#138.
2. Pkg.add([PackageSpec(...), PackageSpec(...), "JLD2", "TOML"]) mixed PackageSpec objects with plain strings, which Julia infers as Vector{Any} — no Pkg.add method matches that. Confirmed from the real build's CloudWatch log. Fixing this in place on generation "v1" (aws-infra#139) hit a structural wall: AWS won't let Terraform delete a recipe/component that any built image — even a FAILED one — still references (ResourceDependencyException), so it broke every subsequent staging apply until reverted (aws-infra#140).
3. Both fixes landed together as a new generation, "v2" (aws-infra#141), per this module's own documented convention: roll forward, never mutate a generation once a build has run against it. Generation "v1" stays in recipe_generations, frozen and orphaned, for the historical record.

Full sequence: #131 (broken) → #137 (revert) → #138 (YAML fix, relanded) → #139 (Pkg.add fix, but broke staging) → #140 (revert #139) → #141 (both fixes as v2, this is what's live).

Verification (on the published AMI, not just the build-time smoke)

Launched a fresh r8i.xlarge from the AMI and ran the acceptance script via SSM (not the build instance):

• julia --version → julia version 1.12.6 ✓
• Bundle loads source-free: package-image bundle loaded root="Piccolissimo" modules=475 — no network install ✓
• 2-level transmon X-gate SmoothPulseProblem solves, plot_pulse PNG saved via CairoMakie (no import error) ✓
• /tmp/FINISHED → status = "completed", exit_code = 0 ✓

Cold start: instance launch → FINISHED{completed} = 11m0s (boot + SSM-ready: ~3m30s; the Julia smoke script itself — bundle load + solve + plot, no precompile/install —: ~7m9s). This replaces the previous at-boot juliaup + Pkg.add + precompile flow.

Cost: build compute (m5.xlarge) across the two failed attempts + the successful one ≈ $0.11 one-time; the verification launch (r8i.xlarge, ~12 min) ≈ $0.05. Recurring: weekly rebuild (~24 min/week, Ubuntu-patch cadence) ≈ $0.14/month + AMI EBS snapshot storage, low single-digit \$/month — in line with the module README's estimate.

Maintenance

Added terraform/modules/solve_runner_image/README.md (mirrors runner_image's) covering the two independent update paths (weekly Ubuntu-patch cron vs. deliberate generation bump for Julia/pkgimage/content changes), plus the "any build freezes the generation, even a failed one" constraint learned the hard way above — so the next fix rolls forward correctly on the first try.

Not done here (unchanged from the original PR's scope)

Repointing the S/M/L launch templates at this AMI is still a separate follow-up delta, per aws-infra#131's original scope note.

[jack-champagne]

Launch-template repoint done — Δ1's full scope is now live on staging.

Follow-up to the AMI-bake comment above: the "point S/M/L launch templates at it, replacing the at-boot install" part of this issue landed in harmoniqs/aws-infra#142, merged to staging.

What shipped

• All three launch templates (small/medium/large) now boot from the baked solve-runner AMI (ami-014ed5bb03b52b34d) instead of running juliaup + Pkg.add("Piccolo") at boot.
• The analysis pipeline's SendCommand step now stages a small wrapper script before running a submitted solve file.

A real bug this caught before it shipped

A plain terraform plan/apply would have looked completely clean here but shipped a broken pipeline. The baked bundle loads Piccolo/Piccolissimo as private, unregistered Julia modules (see components/smoke.jl) — a bare using Piccolo, which every real submitted script has (see examples/*.jl), throws ArgumentError: Package Piccolo not found no matter what JULIA_DEPOT_PATH/PCL_BUNDLE/--project env is set. Caught this by actually launching a real instance and running the exact SendCommand invocation against it before merging (see scripts/smoke-tests/ in aws-infra — a manual, scripted check for exactly this class of apply-time/runtime bug, same family as the YAML-quoting and Pkg.add issues from #131-141).

Fix: the wrapper binds the loaded bundle modules into Main, does the equivalent of using .Piccolo (relative import, which does resolve), comments out the submitted script's now-redundant using <bundled package> line, then runs it. Verified end-to-end against staging with a real SmoothPulseProblem solve (IPOPT converges, exits 0) — not just a clean plan.

Verified live on staging

last-green recorded, all 3 launch templates confirmed pointing at the new AMI, step function definition confirmed matching what was tested.

Known follow-ups (not blocking)

• zip isn't baked into the AMI (only unzip is) — installed at boot for now via apt-get; fold into the component's AptInstallDeps on the next generation bump.
• examples/example1.jl references UnitarySmoothPulseProblem, which doesn't exist in the bundled Piccolo version (renamed to SmoothPulseProblem at some point) — pre-existing example staleness, unrelated to this change, would fail the same way under the old at-boot flow too.

Both halves of C0 (bake + repoint) are done and verified on staging.

[jack-champagne]

Verified done — see comment above. Both the AMI bake and the launch-template repoint are live on staging, confirmed with a real end-to-end solve through the actual pipeline invocation.