misc/scratch: derive AMIs by architecture and add instance-type override

[antiguru]

Motivation

Enumerating one config file per instance size meant the AMI was copy-pasted across every config and the instance type was baked into each filename.
That is how a recently added config drifted to name the wrong instance type internally.

Description

Centralize the two Ubuntu 26.04 AMIs in a single AMIS_BY_ARCH map and derive the AMI from the instance type's architecture, queried from EC2 via describe_instance_types (which also validates the type).
Add --instance-type and --size-gb overrides to create, go, and claude; --instance-type works without a preset by synthesizing a default config, so arbitrary sizes no longer need a checked-in file.
MachineDesc.ami becomes optional and the per-config ami field is dropped, leaving a single place to bump on the next release.
All configs move to Ubuntu 26.04 and the now-redundant r6gd/r8gd presets are removed.

While here, the rest of the scratch tooling is modernized:

• Provisioned tool versions are bumped in lockstep across provision.bash and ci/builder/Dockerfile so scratch matches CI: Node 24.16 and Compose 5.0.1. Docker stays on the 28 line until moby#51845 is fixed. kind/kubectl stay pinned at 0.29.0/1.24.3 — a Nightly run confirmed newer versions break Cloudtest (minio fails to come up on the kind cluster). The obsolete ec2-instance-connect OpenSSL-3 workaround is dropped — Ubuntu 26.04 already ships it fixed.
• bin/scratch list and the instance picker render a compact, borderless table (instance id first, relative expiry, no private IP) that fits a default terminal; CSV output is unchanged.
• Deprecated datetime.utcnow() (which silently produced tz-offset scratch-delete-after epochs) is replaced with timezone-aware UTC, get_event_loop().run_until_complete becomes asyncio.run, and mutable default arguments are removed.

🤖 Generated with Claude Code

[def-]

provision.bash bumps Docker (29.5.3)

https://materializeinc.slack.com/archives/C01LKF361MZ/p1773310371789569

I'd recommend not upgrading to Docker 29 until this issue is resolved (edited)

I did just notice that we already were on Docker 29.1.3 though, ouch. At least for me locally it was pretty unusable having to kill containers many times.

Node (24.16.0 LTS), kind (0.32.0), and kubectl (1.36.1, kept within one minor of kind's cluster),

It's nice to have the same versions in bin/ci-builder too, so if you want to bump them also do it in ci/builder/Dockerfile please.

[antiguru]

Thanks!

• Docker: agreed — rolled back from 29 to the latest 28 line (28.5.2) given moby#51845. ci-builder is unpinned (distro docker.io), so nothing to change there.
• Node / kind / kubectl: bumped in ci/builder/Dockerfile too, in lockstep with provision.bash — Node 24.16, kind 0.32.0, kubectl 1.36.1 (with fresh sha256 pins), Compose 5.0.1.

I removed your # TODO(def-) Upgrading kind/kubectl seems to cause Cloudtest failures since we're now upgrading. That TODO is from Feb, so it may be stale — but Cloudtest in this run is the real test. If it goes red, the fix is pinning kind/kubectl back in both files.

[def-]

I removed your # TODO(def-) Upgrading kind/kubectl seems to cause Cloudtest failures since we're now upgrading. That TODO is from Feb, so it may be stale — but Cloudtest in this run is the real test. If it goes red, the fix is pinning kind/kubectl back in both files.

It's only in Nightly though.

bin/scratch list and the instance picker render a compact, borderless table (instance id first, relative expiry, no private IP) that fits a default terminal; CSV output is unchanged.

Why only change it for ls and not create/destroy/(others I might have missed) too? Seems inconsistent to have multiple table styles now.

[antiguru]

It's only in Nightly though.

Good point — PR CI won't exercise it. We'll kick off a Nightly against this branch to validate the kind/kubectl bump before merge; if Cloudtest goes red there, we pin kind/kubectl back to 0.29.0/1.24.3 in both files (Node/Compose can still move).

Why only change it for ls and not create/destroy/...

It's not just ls — print_instances is shared, so create, destroy, go, claude, and the instance picker all already render the new borderless style. The PR wording was misleading, sorry. The one genuinely-different table was the machine-config chooser in pick_machine, which was still bordered. I've now routed both through a shared render_table helper, so there's a single style across the CLI.

[def-]

This doesn't look particularly borderless to me:

$ bin/scratch create
+----+----------------------+---------------+-------------------------------------------------------------------------+
| #  | Config               | Instance Type | Name                                                                    |
+----+----------------------+---------------+-------------------------------------------------------------------------+
| 1  | big-dev-box          | r5a.4xlarge   | one-off load testing                                                    |
| 2  | ci                   | c7a.2xlarge   | c7a.2xlarge as used by the CI                                           |
| 3  | ci-aarch64           | c8g.2xlarge   | c8g.2xlarge as used by the CI (aarch64)                                 |
| 4  | ci-aarch64-large     | c8g.12xlarge  | c8g.12xlarge as used by CI, when a large box is needed (aarch64-large)  |
| 5  | ci-aarch64-medium    | c8g.4xlarge   | c8g.4xlarge as used by CI, when a medium box is needed (aarch64-medium) |
| 6  | ci-x86_64-large      | m7a.8xlarge   | m7a.8xlarge as used by CI                                               |
| 7  | dev-box              | t3.2xlarge    | one-off testing                                                         |
| 8  | persist-benchmarking | m6i.8xlarge   | persist benchmarking                                                    |
| 9  | prod-like-arm        | r6g.4xlarge   | production-like ARM                                                     |
| 10 | remote-dev           | c7a.4xlarge   | vs code remote dev                                                      |
| 11 | small                | t3.small      | small                                                                   |
+----+----------------------+---------------+-------------------------------------------------------------------------+
Select a machine config [#]:

Edit: Didn't see the new push, nevermind.

[def-]

Ubuntu 26.04

We're also not using Ubuntu 26.04 anywhere else, but probably doesn't matter much: #36053

No nightly yet: https://buildkite.com/materialize/nightly/builds?branch=antiguru%3Ascratch-modernize I'm guessing cloudtest will fail the same way.

[antiguru]

Nightly confirmed your TODO is still live — the kind/kubectl bump breaks Cloudtest:

ERROR at setup of test_envd_on_failing_node
materialize.ui.UIError: kubectl wait never returned 'condition met':
  kubectl wait --for condition=Available=True deployment.apps/minio-deployment --timeout 60s --context kind-mzcloud
  → timed out waiting for the condition on deployments/minio-deployment

minio never comes up on the kind v0.32 / k8s 1.36 cluster (nightly 16717). upstream/main has no relevant fix since, so rebasing wouldn't help.

Reverted kind/kubectl back to 0.29.0/1.24.3 in both files and restored your TODO. Kept Node 24.16 + Compose 5.0.1, and Docker on the 28 line.

[def-]

Thanks!