Skip to content

fix(ci): wire GitLab CI to publish the system-tests image #4079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MilanGarnier wants to merge 1 commit into from
+70 −80
Draft

fix(ci): wire GitLab CI to publish the system-tests image #4079

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 70 additions & 80 deletions .gitlab/generate-package.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1604,8 +1604,10 @@ function appsec_image_from_tag_mapping(string $tag): string
paths:
- packages/datadog-setup.php

# Runs on every non-default branch so system tests can be run against any (non-default) in-progress branch.
"publish docker image for system tests (token)":
# System-tests image publishing. Runs on every non-default branch so system tests can be
# run against any in-progress branch (DataDog/system-tests LIBRARY_TARGET_BRANCH).

"check open pr for system tests":
stage: release
image: registry.ddbuild.io/images/dd-octo-sts-ci-base:2025.06-1
tags: [ "arch:amd64" ]
Expand All @@ -1616,29 +1618,44 @@ function appsec_image_from_tag_mapping(string $tag): string
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: never
- when: always
allow_failure: true
variables:
GIT_STRATEGY: none
script:
- dd-octo-sts token --scope DataDog/dd-trace-php --policy gitlab-ci-publish-packages > github_token_system_tests.txt
script: |
set -e
# octo-sts token: pull_requests:read to gate the S3 upload on an open PR.
GITHUB_TOKEN=$(dd-octo-sts token --scope DataDog/dd-trace-php --policy gitlab-ci-publish-packages)

# Only publish for branches with an open PR (system tests pull the image per-PR),
# checked before anything is uploaded so throwaway branches never publish artifacts.
HTTP=$(curl -s -o /tmp/pulls.json -w "%{http_code}" -H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" \
"https://api.github.com/repos/DataDog/dd-trace-php/pulls?head=DataDog:${CI_COMMIT_BRANCH}&state=open")
if [ "${HTTP}" != "200" ]; then
echo "ERROR: failed to check for an open PR on branch ${CI_COMMIT_BRANCH} (HTTP ${HTTP}):"; cat /tmp/pulls.json; exit 1
fi
PR_COUNT=$(jq 'length' /tmp/pulls.json)
if [ "${PR_COUNT}" -eq 0 ]; then
echo "No open PR for branch ${CI_COMMIT_BRANCH}; skipping system-tests image publish."
echo "HAS_OPEN_PR=false" > pr_check.env
else
echo "Found open PR(s) on branch ${CI_COMMIT_BRANCH}"
echo "HAS_OPEN_PR=true" > pr_check.env
fi
artifacts:
paths:
- github_token_system_tests.txt
expire_in: 1 hour
when: on_success
reports:
dotenv: pr_check.env

"publish docker image for system tests":
"publish docker image for system tests (upload)":
stage: release
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/docker:29.4.0-noble
tags: [ "docker-in-docker:amd64" ]
resource_group: "publish-system-tests-image-${CI_COMMIT_REF_SLUG}"
interruptible: true
image: registry.ddbuild.io/images/mirror/amazon/aws-cli:2.17.32
tags: [ "arch:amd64" ]
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: never
- if: $HAS_OPEN_PR != "true"

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move PR gating out of the upload job rules

GitLab evaluates rules before any upstream job runs, so the dotenv report emitted by check open pr for system tests cannot populate HAS_OPEN_PR here (see the GitLab rules docs: https://docs.gitlab.com/ci/jobs/job_rules/). In a normal branch pipeline this variable is unset while the rule is evaluated, so $HAS_OPEN_PR != "true" matches and excludes the upload job even when the PR check would later write HAS_OPEN_PR=true; as a result the system-tests artifacts/image never get published for open PR branches.

Useful? React with 👍 / 👎.

when: never
- when: always
needs:
- job: "publish docker image for system tests (token)"
- job: "check open pr for system tests"
artifacts: true
- job: "datadog-setup.php"
artifacts: true
Expand All @@ -1648,74 +1665,47 @@ function appsec_image_from_tag_mapping(string $tag): string
artifacts: true
variables:
GIT_STRATEGY: none
allow_failure: true
script: |
set -e
IMAGE="ghcr.io/datadog/dd-trace-php/dd-library-php:${CI_COMMIT_REF_SLUG}"
VERSIONS_URL="https://api.github.com/orgs/DataDog/packages/container/dd-trace-php%2Fdd-library-php/versions"
GITHUB_TOKEN=$(<github_token_system_tests.txt)

apt-get update -qq && apt-get install -y -qq curl jq > /dev/null

STATUS=$(curl -s -o /tmp/pulls.json -w "%{http_code}" -H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" \
"https://api.github.com/repos/DataDog/dd-trace-php/pulls?head=DataDog:${CI_COMMIT_BRANCH}&state=open")
if [ "${STATUS}" != "200" ]; then
echo "ERROR: failed to check for an open PR on branch ${CI_COMMIT_BRANCH} (HTTP ${STATUS}):"; cat /tmp/pulls.json
exit 1
fi
PR_COUNT=$(jq 'length' /tmp/pulls.json)
if [ "${PR_COUNT}" -eq 0 ]; then
echo "No open PR for branch ${CI_COMMIT_BRANCH}; skipping system-tests image publish."
exit 3
fi
echo "Found open PR(s) on branch ${CI_COMMIT_BRANCH}, continuing"

docker login ghcr.io -u DataDog --password-stdin < github_token_system_tests.txt
echo "Docker login to ghcr.io succeeded"

gh_api() { curl -s -o /tmp/resp.json -w "%{http_code}" -X "$1" \
-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" "$2"; }

# delete any pre-existing image with the same tag so pushing with the same tag doesn't pile up untagged versions.
echo "== Checking token and existing image tagged ${CI_COMMIT_REF_SLUG} =="
STATUS=$(gh_api GET "$VERSIONS_URL")
if [ "$STATUS" != "200" ]; then
echo "WARNING: token check failed (HTTP ${STATUS}), skipping pre-delete:"; cat /tmp/resp.json
else
echo "Token OK (HTTP 200): listed package versions for dd-trace-php/dd-library-php"
VERSION_ID=$(jq -r --arg TAG "${CI_COMMIT_REF_SLUG}" \
'.[] | select(.metadata.container.tags[]? == $TAG) | .id' /tmp/resp.json | head -n1)
if [ -z "$VERSION_ID" ]; then
echo "No pre-existing image tagged ${CI_COMMIT_REF_SLUG} found, nothing to delete"
else
STATUS=$(gh_api DELETE "${VERSIONS_URL}/${VERSION_ID}")
if [ "$STATUS" = "204" ]; then
echo "Deleted existing image version ${VERSION_ID} (HTTP 204)"
else
echo "WARNING: failed to delete existing image version ${VERSION_ID} (HTTP ${STATUS}), continuing anyway:"
cat /tmp/resp.json
fi
fi
# Normalise the arch tarball names so the GitHub Actions side needs only the branch slug.
DEST="s3://dd-trace-php-builds/ci/${CI_COMMIT_REF_SLUG}"

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Serialize uploads to the per-branch S3 prefix

When two pipelines for the same branch overlap, both upload to the same ci/${CI_COMMIT_REF_SLUG} keys while the old per-slug resource_group was removed. The GitHub workflow later downloads from that shared prefix by slug, so a slower older pipeline can overwrite some or all of a newer pipeline's artifacts before either dispatch downloads them, causing the branch tag to be pushed with stale or mixed package contents; keep the upload/dispatch serialized per slug or use a unique prefix per pipeline/commit.

Useful? React with 👍 / 👎.

aws s3 cp packages/dd-library-php-*-x86_64-linux-gnu.tar.gz "${DEST}/dd-library-php-x86_64-linux-gnu.tar.gz"
aws s3 cp packages/dd-library-php-*-aarch64-linux-gnu.tar.gz "${DEST}/dd-library-php-aarch64-linux-gnu.tar.gz"
aws s3 cp packages/datadog-setup.php "${DEST}/datadog-setup.php"
echo "Uploaded system-tests artifacts to ${DEST}/"

"publish docker image for system tests (dispatch)":
stage: release
image: registry.ddbuild.io/images/dd-octo-sts-ci-base:2025.06-1
tags: [ "arch:amd64" ]
id_tokens:
DDOCTOSTS_ID_TOKEN:
aud: dd-octo-sts
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: never
- when: always
needs:
- job: "publish docker image for system tests (upload)"
Comment on lines +1687 to +1689

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate the dispatch job when upload is omitted

In the intended no-open-PR path, the upload job is configured to be omitted, but this dispatch job is still added for every non-default branch and has a hard needs edge to that omitted job. GitLab documents that needs pointing at a job excluded by rules can make pipeline creation fail unless the need is optional (https://docs.gitlab.com/ci/yaml/), so branches without an open PR can fail the package child pipeline instead of cleanly skipping system-tests image publishing.

Useful? React with 👍 / 👎.

variables:
GIT_STRATEGY: none
script: |
set -e
# octo-sts token: actions:write to dispatch the workflow.
GITHUB_TOKEN=$(dd-octo-sts token --scope DataDog/dd-trace-php --policy gitlab-ci-publish-packages)

# GHCR write must come from GitHub Actions' GITHUB_TOKEN, so trigger the workflow that
# pulls the artifacts back down from S3 and pushes the image.
PAYLOAD=$(jq -n --arg ref "${CI_DEFAULT_BRANCH}" --arg slug "${CI_COMMIT_REF_SLUG}" \
'{ref: $ref, inputs: {ref_slug: $slug}}')
HTTP=$(curl -s -o /tmp/dispatch.json -w "%{http_code}" -X POST \
-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "Accept: application/vnd.github+json" \
"https://api.github.com/repos/DataDog/dd-trace-php/actions/workflows/publish-system-tests-image.yml/dispatches" \
-d "${PAYLOAD}")
if [ "${HTTP}" != "204" ]; then
echo "ERROR: workflow_dispatch failed (HTTP ${HTTP}):"; cat /tmp/dispatch.json; exit 1
fi

echo "== Building and pushing ${IMAGE} (linux/amd64, linux/arm64) =="
# stage each arch's tarball under its own dir so the Dockerfile can pick the right one via buildx's TARGETARCH arg.
mkdir -p packages/amd64 packages/arm64
cp packages/dd-library-php-*-x86_64-linux-gnu.tar.gz packages/amd64/
cp packages/dd-library-php-*-aarch64-linux-gnu.tar.gz packages/arm64/
cp packages/datadog-setup.php packages/amd64/
cp packages/datadog-setup.php packages/arm64/

printf 'FROM scratch\nARG TARGETARCH\nCOPY packages/${TARGETARCH}/dd-library-php-*-linux-gnu.tar.gz /\nCOPY packages/${TARGETARCH}/datadog-setup.php /\n' \
> Dockerfile.system-tests
BUILDER="system-tests-builder-${CI_JOB_ID}"
docker buildx create --name "$BUILDER" --driver docker-container
docker buildx build --builder "$BUILDER" --platform linux/amd64,linux/arm64 \
-f Dockerfile.system-tests -t "$IMAGE" --push .
echo "Pushed $IMAGE"
after_script:
- rm -f github_token_system_tests.txt
- docker buildx rm "system-tests-builder-${CI_JOB_ID}" || true
echo "Dispatched publish-system-tests-image.yml (tag ${CI_COMMIT_REF_SLUG}) on ${CI_DEFAULT_BRANCH}"

"bundle for reliability env":
stage: shared-pipeline
Expand Down
Loading