Skip to content

Fix DDOT install for Agent >= 7.78.0#414

Draft
BaptisteFoy wants to merge 6 commits intomainfrom
baptiste.foy/FA/fix-ddot-install
Draft

Fix DDOT install for Agent >= 7.78.0#414
BaptisteFoy wants to merge 6 commits intomainfrom
baptiste.foy/FA/fix-ddot-install

Conversation

@BaptisteFoy
Copy link
Copy Markdown
Contributor

@BaptisteFoy BaptisteFoy commented Mar 27, 2026

Bug

For Agent >= 7.78.0, DDOT is installed by the Agent's own post-install script rather than the install script. The previous fix (#406) handled this by unsetting `DD_OTELCOLLECTOR_ENABLED`, which caused two regressions:

  1. `DD_OTELCOLLECTOR_ENABLED` was no longer present in the environment when running `apt-get`/`yum`/`zypper install datadog-agent`, so the Agent's post-install script couldn't see it and wouldn't install DDOT
  2. `update_ddot` and `manage_otel_config` were also skipped (they gate on `DD_OTELCOLLECTOR_ENABLED`), so the Datadog Agent config was not updated for DDOT

Why not pass DD_OTELCOLLECTOR_ENABLED to the package manager?

An intermediate approach passed `DD_OTELCOLLECTOR_ENABLED` to the `apt-get`/`yum`/`zypper` install command so the Agent's post-install script could pick it up. This failed because the Agent post-install runs before the install script creates `datadog.yaml`, causing:

failed to install extensions: extension ddot: could not install extension: failed to run hook (postInstallExtension): run failed: failed to write otel-config.yaml: failed to read datadog.yaml: open /etc/datadog-agent/datadog.yaml: no such file or directory

Fix

For Agent >= 7.78.0, instead of relying on the Agent post-install script:

  • Do not pass `DD_OTELCOLLECTOR_ENABLED` to the package manager (avoids triggering the broken post-install)
  • After `datadog.yaml` is created by the install script, call `datadog-installer extension install "oci://${ddot_oci_registry}/agent-package:${ddot_oci_tag}" "ddot"` directly — this handles both DDOT installation and configuration
  • Skip the `datadog-agent-ddot` deb/rpm package install and repo setup (no longer needed)

For Agent < 7.78.0, behavior is unchanged: the `datadog-agent-ddot` package is installed from the dedicated apt/yum/zypper repo, and `update_ddot`/`manage_otel_config` run as before.

Test plan

  • With `DD_OTELCOLLECTOR_ENABLED` set and Agent < 7.78.0: ddot repo setup and package install run as before, `update_ddot`/`manage_otel_config` still run
  • With `DD_OTELCOLLECTOR_ENABLED` set and Agent >= 7.78.0: ddot repo/package install are skipped, `datadog-installer extension install` is called after `datadog.yaml` is created
  • Without `DD_OTELCOLLECTOR_ENABLED`: no behavior change

@BaptisteFoy BaptisteFoy force-pushed the baptiste.foy/FA/fix-ddot-install branch from 4ecf902 to 0c3987a Compare March 30, 2026 13:17
@BaptisteFoy BaptisteFoy marked this pull request as ready for review March 30, 2026 13:20
@BaptisteFoy BaptisteFoy requested review from a team as code owners March 30, 2026 13:20
@BaptisteFoy BaptisteFoy requested review from raymondeah and removed request for a team March 30, 2026 13:20
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c3987a7e7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

# yum has a default retry of 10 https://github.com/Distrotech/yum/blob/f4e54aeed297158c563828aa3ebb93d0c8ce7e38/docs/yum.conf.5#L364-L366
$sudo_cmd bash -c "SYSTEMD_OFFLINE=${SYSTEMD_OFFLINE:-0} DATADOG_TRACE_ID=${DATADOG_TRACE_ID} DATADOG_PARENT_ID=${DATADOG_PARENT_ID} yum -y --disablerepo='*' --enablerepo='datadog' install $dnf_flag ${packages[*]}" 2> >($sudo_cmd tee /tmp/ddog_install_error_msg >&2) || $sudo_cmd bash -c "SYSTEMD_OFFLINE=${SYSTEMD_OFFLINE:-0} yum -y install $dnf_flag ${packages[*]}" 2> >($sudo_cmd tee /tmp/ddog_install_error_msg >&2)
if [ -n "$DD_OTELCOLLECTOR_ENABLED" ]; then
$sudo_cmd bash -c "SYSTEMD_OFFLINE=${SYSTEMD_OFFLINE:-0} DATADOG_TRACE_ID=${DATADOG_TRACE_ID} DATADOG_PARENT_ID=${DATADOG_PARENT_ID} DD_INSTALLER_REGISTRY_URL=${DD_INSTALLER_REGISTRY_URL} yum -y --disablerepo='*' --enablerepo='datadog' install $dnf_flag ${packages[*]}" 2> >($sudo_cmd tee /tmp/ddog_install_error_msg >&2) || $sudo_cmd bash -c "SYSTEMD_OFFLINE=${SYSTEMD_OFFLINE:-0} DD_INSTALLER_REGISTRY_URL=${DD_INSTALLER_REGISTRY_URL} yum -y install $dnf_flag ${packages[*]}" 2> >($sudo_cmd tee /tmp/ddog_install_error_msg >&2)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need to set DD_INSTALLER_REGISTRY_URL_AGENT_PACKAGE on the install commands? We're using DD_INSTALLER_REGISTRY_URL_AGENT_PACKAGE for ddot_oci_registry but this env var isn't passed to the install commands. So if DD_INSTALLER_REGISTRY_URL_AGENT_PACKAGE is set without DD_INSTALLER_REGISTRY_URL, we get the following error in upgrades/downgrades:

could not download image using install.datadoghq.com/agent-package:7.78.0-rc.2-1: GET https://install.datadoghq.com/v2/agent-package/manifests/7.78.0-rc.2-1: unexpected status code 404 Not Found: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>v2/agent-package/manifests/7.78.0-rc.2-1</Key><RequestId>EKNAE187CXD0RGH9</RequestId><HostId>zKo74kJNoNUgVjJfMUt7hjw+WGUoiyZ4lbB2E1x04XKbAhoiLxEKmqDos7a+qyZYXffbCInF2bI=</HostId></Error>
could not download image using install.datadoghq.com/agent-package:7.78.0-rc.2-1: GET https://gcr.io/v2/datadoghq/agent-package/manifests/7.78.0-rc.2-1: MANIFEST_UNKNOWN: Failed to fetch "7.78.0-rc.2-1"
failed to restore extensions: could not install extensions: could not download package: could not download package: could not download image from any registry: could not download image using install.datadoghq.com/agent-package:7.78.0-rc.2-1: GET https://install.datadoghq.com/v2/agent-package/manifests/7.78.0-rc.2-1: unexpected status code 404 Not Found: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>v2/agent-package/manifests/7.78.0-rc.2-1</Key><RequestId>EKNAE187CXD0RGH9</RequestId><HostId>zKo74kJNoNUgVjJfMUt7hjw+WGUoiyZ4lbB2E1x04XKbAhoiLxEKmqDos7a+qyZYXffbCInF2bI=</HostId></Error>; could not download image using install.datadoghq.com/agent-package:7.78.0-rc.2-1: GET https://gcr.io/v2/datadoghq/agent-package/manifests/7.78.0-rc.2-1: MANIFEST_UNKNOWN: Failed to fetch "7.78.0-rc.2-1"
W: --force-yes is deprecated, use one of the options starting with --allow instead.

else
echo -e " \033[33mInstalling package(s): ${packages[*]}\n\033[0m"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some testing and found that the rpm -> oci upgrade was failing

Suggested change
# When upgrading to 7.78+, DDOT is managed via OCI extension install.
# Remove the legacy datadog-agent-ddot RPM first to avoid strict version
# dependency conflicts that prevent yum from upgrading datadog-agent.
if [ -n "$ddot_use_installer" ] && rpm -q datadog-agent-ddot &>/dev/null; then
echo -e " \033[33mRemoving legacy datadog-agent-ddot RPM package before upgrade...\n\033[0m"
$sudo_cmd yum -y remove datadog-agent-ddot
fi

@BaptisteFoy BaptisteFoy marked this pull request as draft April 1, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants