Skip to content

Base image build process - #7270

Open
nccatoni wants to merge 17 commits into
mainfrom
nccatoni/base-image-build
Open

Base image build process#7270
nccatoni wants to merge 17 commits into
mainfrom
nccatoni/base-image-build

Conversation

@nccatoni

@nccatoni nccatoni commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Content-Addressed Base Images via GitLab CI

Summary

Automate weblog base image publishing with content-addressed tags, replacing the manual label-driven workflow (build-*-base-images PR labels + manual version bumps + human push coordination).

What Changed

  • New build_base_images GitLab CI job runs on every push: derives dependencies from COPY instructions in each *.base.Dockerfile, computes a 12-char content hash, and pushes datadog/system-tests:<name>.base-<hash> if the tag doesn't exist. Idempotent — never overwrites.
  • Lock file (utils/build/docker/base-images.lock.json) maps stable aliases (system_tests_base_nodejs_express4) → immutable tags. Consumer Dockerfiles use FROM <alias>, resolved via BuildKit --build-context at build time.
  • GitHub Actions no longer builds base images — polls Docker Hub (wait_for_base_image.py) for the locked tag instead.
  • Removed: per-language build scripts, build-*-base-images labels, _build_weblog_base_images workflow input, manual -vN tag management.

New Workflow

  1. Edit a base Dockerfile or its dependencies → push
  2. GitLab CI auto-publishes the new content-addressed tag
  3. Run python utils/scripts/build_base_images.py --update-lock + python utils/scripts/update_mirror_images.py
  4. Commit the updated lock/mirror files

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/test_the_test/test_build_base_images.py                           @DataDog/system-tests-core
utils/build/docker/base-images.lock.json                                @DataDog/system-tests-core
utils/scripts/base_image.py                                             @DataDog/system-tests-core
utils/scripts/build_base_images.py                                      @DataDog/system-tests-core
utils/scripts/wait_for_base_image.py                                    @DataDog/system-tests-core
.github/workflows/ci.yml                                                @DataDog/system-tests-core
.github/workflows/compute-workflow-parameters.yml                       @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
.github/workflows/system-tests.yml                                      @DataDog/system-tests-core
.gitignore                                                              @DataDog/system-tests-core
.gitlab-ci.yml                                                          @DataDog/system-tests-core
docs/CI/github-actions.md                                               @DataDog/system-tests-core
docs/edit/update-docker-images.md                                       @DataDog/system-tests-core
docs/understand/weblogs/weblog-metadata.md                              @DataDog/system-tests-core
mirror_images.lock.yaml                                                 @DataDog/system-tests-core
mirror_images.yaml                                                      @DataDog/system-tests-core
tests/test_the_test/test_ci_orchestrator.py                             @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/_context/weblog_metadata.py                                       @DataDog/system-tests-core
utils/build/build.sh                                                    @DataDog/system-tests-core
utils/build/docker/nodejs/docker-bake.hcl                               @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4-typescript.Dockerfile                @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4-typescript.base.Dockerfile           @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4.Dockerfile                           @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express4.base.Dockerfile                      @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express5.Dockerfile                           @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/express5.base.Dockerfile                      @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/fastify.Dockerfile                            @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/fastify.base.Dockerfile                       @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/nextjs.Dockerfile                             @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/nextjs.base.Dockerfile                        @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/uds-express4.Dockerfile                       @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.0-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.0.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.1-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.1.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.2-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.2.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.3-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.3.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.4-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-7.4.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.0-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.0.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.1-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.1.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.2-zts.Dockerfile                    @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod-8.2.Dockerfile                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/apache-mod.base.Dockerfile                       @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/docker-bake.hcl                                  @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.0.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.1.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.2.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.3.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-7.4.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.0.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.1.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.2.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm-8.5.Dockerfile                           @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm.base.Dockerfile                          @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/php-fpm/build.sh                                 @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/weblog_metadata.yml                              @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/python/django-poc.Dockerfile                         @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/django-poc.base.Dockerfile                    @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/django-py3.13.Dockerfile                      @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/django-py3.13.base.Dockerfile                 @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/docker-bake.hcl                               @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/fastapi.Dockerfile                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/fastapi.base.Dockerfile                       @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask-poc.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask-poc.base.Dockerfile                     @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/python3.12.Dockerfile                         @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/python3.12.base.Dockerfile                    @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/tornado.Dockerfile                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/tornado.base.Dockerfile                       @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uds-flask.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uwsgi-poc.Dockerfile                          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/uwsgi-poc.base.Dockerfile                     @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python_otel/flask-poc-otel.Dockerfile                @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/ci/gitlab/main.yml                                                @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core
utils/scripts/compute-workflow-parameters.py                            @DataDog/system-tests-core
utils/scripts/get_pr_merged_labels.sh                                   @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core
utils/scripts/update_mirror_images.py                                   @DataDog/system-tests-core
utils/build/build_nodejs_base_images.sh                                 @DataDog/system-tests-core
utils/build/build_php_base_images.sh                                    @DataDog/system-tests-core
utils/build/build_python_base_images.sh                                 @DataDog/system-tests-core
utils/scripts/build-base-image.py                                       @DataDog/system-tests-core

@nccatoni
nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from e2c8ca6 to 611f535 Compare July 6, 2026 16:25
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 6, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 3 Pipeline jobs failed

Testing the test | System Tests (nodejs, prod) / End-to-end #2 / fastify 2   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. 2 failed tests. Assertion Error: Should have at least one stats bucket in tests/stats/test_stats.py:682 and tests/stats/test_stats.py:726.

Testing the test | System Tests (nodejs, prod) / End-to-end #2 / express4 2   View in Datadog   GitHub Actions

See error 2 failed tests. Assertion Error: Should have at least one stats bucket in tests/stats/test_stats.py:682 and tests/stats/test_stats.py:726

Testing the test | all-jobs-are-green   View in Datadog   GitHub Actions

See error Check run failed for System Tests (nodejs, prod) including multiple failed end-to-end tests.
📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: nccatoni/base-image-build

Testing the test | System Tests (nodejs, prod) / End-to-end #2 / fastify 2
Commit: e71b9cb0d9c799559aaed67f5fa4294bfce40858
Error (code / test):
2 failed tests. Assertion Error: Should have at least one stats bucket in tests/stats/test_stats.py:682 and tests/stats/test_stats.py:726.
CI job: https://github.com/DataDog/system-tests/actions/runs/31610011644/job/94161076306

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f1605e2 | Docs | Datadog PR Page | Give us feedback!

@nccatoni
nccatoni force-pushed the nccatoni/base-image-build branch 2 times, most recently from dc70509 to b8f1ceb Compare July 7, 2026 11:52
@nccatoni
nccatoni force-pushed the nccatoni/base-image-build branch from d3cc9d8 to 1902e53 Compare July 30, 2026 10:14
@nccatoni
nccatoni force-pushed the nccatoni/base-image-build branch from 1902e53 to 578a92d Compare July 30, 2026 12:06
@nccatoni
nccatoni force-pushed the nccatoni/base-image-build branch from f90836f to 5d90e00 Compare July 30, 2026 13:44
@nccatoni
nccatoni marked this pull request as ready for review July 31, 2026 13:34
@nccatoni
nccatoni requested review from a team as code owners July 31, 2026 13:34
@nccatoni
nccatoni requested review from Kyle-Verhoog, christophe-papazian and dubloom and removed request for a team July 31, 2026 13:34

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5ac604fe36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/understand/weblogs/weblog-metadata.md Outdated
Comment thread utils/scripts/wait_for_base_image.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant