Skip to content

refactor(ci3): move logs and test caching to a server API - #25427

Open
ludamad wants to merge 19 commits into
nextfrom
ad/ci3-client-server
Open

ludamad wants to merge 19 commits into
nextfrom
ad/ci3-client-server

Conversation

@ludamad

@ludamad ludamad commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Move CI logs, test caching and run metadata behind one API.
Keep shared artifact uploads under AWS IAM and downloads on the trusted HTTPS cache; the dashboard password cannot publish builds.
Use environment-only setup with a local file server, system Python and tar/zstd.
Keep AWS OIDC on runners and instance roles on EC2; no new bastion authentication layer.

@ludamad
ludamad requested a review from charlielye as a code owner September 8, 2026 21:18
@ludamad
ludamad force-pushed the ad/ci3-client-server branch 2 times, most recently from 773bc53 to b0ed9fa Compare September 9, 2026 02:48
@ludamad ludamad changed the title refactor(ci3): logs, test cache and artifacts through a server API feat(ci3): server API, client scripts and reference servers Sep 9, 2026
@ludamad
ludamad force-pushed the ad/ci3-client-server branch from b0ed9fa to f0fd5c6 Compare September 9, 2026 03:30
@ludamad
ludamad added this pull request to stack #25431 September 9, 2026 04:14
@ludamad
ludamad force-pushed the ad/ci3-client-server branch 4 times, most recently from 96a21be to 0fcc6fc Compare September 9, 2026 22:37
@ludamad ludamad changed the title feat(ci3): server API, client scripts and reference servers refactor(ci3): logs, test cache and artifacts through a server API Sep 9, 2026
@ludamad
ludamad force-pushed the ad/ci3-client-server branch from 0fcc6fc to effb313 Compare September 9, 2026 22:40
@ludamad
ludamad force-pushed the ad/ci3-client-server branch 3 times, most recently from a64ded2 to b2176a9 Compare September 10, 2026 01:49
@ludamad ludamad added the ci-full Run all master checks. label Sep 10, 2026
@ludamad
ludamad force-pushed the ad/ci3-client-server branch 2 times, most recently from 8825f88 to 8f47016 Compare September 10, 2026 02:46
ci3 no longer knows about redis, the S3 log bucket or the dashboard. Everything it stores goes
through the HTTP API in ci3/CI3_SERVER_API.md via `ci3_client <command>`, which reads one file,
~/.ci3/config.json, written by ci3_setup:

- local mode (the default): ci3_server, the file-backed reference server, on this machine.
- aztec mode (CI3_PASSWORD in the environment, or CI): the labs CI dashboard at
  ci.aztec-labs.com, where CI logs live, behind its basic auth. The labs patch in
  labs-patches/0014 (aztec-node#140) makes the dashboard (rkapp) serve the API.

The entry points (bootstrap.sh, ci.sh, the CI launcher, Post-Actions) run ci3_setup before ci3
loads. An existing config is honoured; a non-interactive run whose environment asks for the
aztec server while the config says otherwise stops with an error, a terminal is asked.

Rewired: cache_log, denoise, run_test_cmd, filter_cached_test_cmd, exec_test (per-test
timings), log_ci_run, the Slack notifiers, ci.sh log/test-timings, cache_upload/download/
exists/ls (artifact reads fall back to the public build cache; ci3 never writes to S3 itself),
bench_compare, docker_isolate, the bb benchmark breakdown uploads. source_redis and
source_cache are gone. The release job downloads the packed release with aws directly.

Also fixed on the way, found in CI: the bb benchmark scripts sourced source_redis, and
make labs-use-local ran labs' bootstrap with the foundation's ci3 root exported.
@ludamad
ludamad force-pushed the ad/ci3-client-server branch from 8f47016 to 06443b5 Compare September 10, 2026 03:34
@ludamad ludamad removed the ci-full Run all master checks. label Sep 10, 2026
@ludamad ludamad added the ci-full Run all master checks. label Sep 11, 2026
@ludamad ludamad changed the title refactor(ci3): logs, test cache and artifacts through a server API refactor(ci3): move logs and test caching to a server API Sep 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full Run all master checks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant