feat(aws-test-infra): add s3-stage, s3-download, s3-cleanup subcommands#175
Open
anvesh-loft wants to merge 1 commit into
Open
feat(aws-test-infra): add s3-stage, s3-download, s3-cleanup subcommands#175anvesh-loft wants to merge 1 commit into
anvesh-loft wants to merge 1 commit into
Conversation
loft-bot
reviewed
Jul 13, 2026
loft-bot
reviewed
Jul 13, 2026
Add the S3 half of the AWS test-infra lifecycle to the shared Go binary so the vcluster-pro conformance workflows can drop their duplicated S3 bucket-setup + presigned-URL bash (including the boto3 PUT-presign shim). Additive: provision and cleanup are unchanged. Subcommands: - s3-stage: ensure+tag a run-scoped bucket, upload files, emit presigned URLs as a JSON map (bucket, presigned-urls outputs). GET and PUT URLs are signed natively via aws-sdk-go-v2 (no boto3). URLs are registered with ::add-mask::, and the step warns if -expires-in outlives the STS session. - s3-download: fetch an object to a local path; a missing object is a no-op and a failed copy does not leave a truncated file behind. - s3-cleanup: empty+delete a bucket with a region-scoped, paginated, tag-based sweep fallback; best-effort by default, -strict to fail loudly. The s3-stage upload/presign list parsing lives in src/parse-list.sh (CRLF-safe) with a bats suite, wired into the Makefile and the test workflow. Go unit tests use fake S3 + presigner clients and cover bucket create/idempotency, tagging, region+pagination sweeps, DeleteObjects batching past 1000 keys, output formats + URL masking, and the download error paths. Closes ENGQA-1271 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anvesh-loft
force-pushed
the
engqa-1271-s3
branch
from
July 13, 2026 21:32
3a567e1 to
8fc50d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the S3 half of the AWS test-infra lifecycle to the
aws-test-infraGo binary: three new subcommands so callers can drop the duplicated S3 bucket-setup + presigned-URL bash currently inlined in the vcluster-pro conformance workflows.Additive and backward-compatible —
provisionandcleanupare untouched (same flags, logic, outputs). Existing consumers (selinux, prerelease, conformance) are unaffected.New subcommands
s3-stagebucket,presigned-urlsoutputs)s3-downloads3-cleanupcleanupdesign (best-effort by default,-strictto fail loudly)Both GET and PUT presigned URLs are generated natively via
aws-sdk-go-v2, which lets the caller drop the inlineboto3shim the workflows used for PUT URLs (aws-cli v2presignis GET-only).Design notes
S3API/S3Presignerinterfaces (likeEC2API), onerunXxxper subcommand with its ownflag.FlagSet, and theinstance-id-by-role-style JSON-map output for presigned URLs.s3-stagereadsuploads/presignas newline lists (same shape asingress-rules).run-id;us-east-1correctly omitsLocationConstraint.Testing
make test-aws-test-infra(go test -race) — new fake-S3 + fake-presigner suite (16 cases: bucket ensure/create/tag, us-east-1 constraint, stage upload+presign, download no-op-when-missing + writes-file, direct + tag-sweep cleanup, output emission) plus all pre-existing provision/cleanup tests still green.make check-docsclean (auto-doc tables regenerated).shellcheckclean on the composite run block; YAML validated.Rollout
vcluster-promigrates the conformance workflow's two staging steps (+ optionally the results download and S3 teardown) onto these subcommands and deletes the duplicated bash.curl -X PUT).