Relay scp downloads through S3 to bypass the SSM throughput ceiling

[joehoyle]

Problem

Downloading files with altis-cli stack scp tops out at ~1.3MB/s of file data no matter how fast the connection is. The bottleneck is the SSM agent itself: its shell output pump reads at most 1024 bytes from the PTY and then time.Sleep(time.Millisecond) on every iteration (shell.go writePump, StreamDataPayloadSize = 1024), a hard ~1-2MB/s per-session ceiling. It's compiled in — no agent config, session document parameter, or client-side change can lift it, and the port-forwarding plugins have the identical loop. Base64 framing eats a further 25% of that, so a 1GB database dump takes around 13 minutes. This is a well-known SSM limitation (aws/amazon-ssm-agent#227).

Proposed solution

Stop sending the bulk bytes through the session at all. The remote already has the aws cli and credentials for the stack's uploads bucket, so for downloads the CLI now:

1. Copies the file to s3://$S3_UPLOADS_BUCKET/tmp/altis-cli/<random>/<name> — with --acl private, since the uploads bucket is publicly readable by default.
2. Presigns a 5-minute GET URL and downloads it over plain HTTPS at full bandwidth.
3. Deletes the staged object (best-effort, in a finally).

The SSM session becomes a control channel only. Commands run through a new exec() helper that brackets output in unique sentinel tokens so output and exit codes can be captured reliably; the tokens are written with adjacent-quote splitting ("__ALTIS_""x_B__") so the PTY echoing the typed command back can never false-match them. The aws s3 cp progress line is parsed from the PTY echo and rendered in the existing progress bar UI, so both legs (remote→S3, S3→local) report live progress and speed.

Trade-offs and alternatives considered:

• Fixing the existing stream (windowed sends, stty -echo) only helps uploads — downloads are pinned at the agent's output ceiling, which we can't change. Uploads are unchanged here and still use the old path (presigning a PUT needs the SDK; the aws cli can only presign GETs).
• Port forwarding sessions hit the same hardcoded pacing in the agent, and would need backend support for non-shell session documents.
• If the S3 path is unavailable for any reason (no aws binary, missing IAM permissions, empty S3_UPLOADS_BUCKET), the command falls back to the existing base64-over-PTY transfer automatically, with the reason shown under -v — so no environment gets worse, slow is the floor.

Notes for reviewers / ops:

• --acl private requires s3:PutObjectAcl on the instance role; without it the cp fails and the CLI silently falls back to the slow path.
• A killed transfer (Ctrl-C between cp and rm) can leave a private object under tmp/altis-cli/. A lifecycle rule on that prefix would be the proper backstop; deliberately not handled CLI-side.
• Also fixes a pre-existing display bug: transfer speed was reported in base64-encoded bytes/s, overstating real throughput by 33%.

🤖 Generated with Claude Code