perf: parallelize network calls and skip redundant operations#409
perf: parallelize network calls and skip redundant operations#409
Conversation
- Parallelize 6 network health-check curls in exit telemetry (was sequential with 5s timeout each, up to 30s total → now ~5s max) - Send logs and trace telemetry payloads in parallel (was sequential with 10s timeout + 5 retries each → now runs concurrently) - Download APT GPG keys in parallel and import in a single gpg call (was 5 sequential download+import cycles) - Download SUSE11 RPM GPG keys in parallel before importing - Skip apt-get update + dependency install when curl/gnupg are already present (common on most systems, saves a full apt-get update cycle) - Remove redundant HEAD request in _install_installer_script before the actual download (curl -f already fails on HTTP errors) These changes reduce install script execution time by ~15-40s per run, with the biggest gains from parallelizing exit telemetry network checks.
|
Approved pending @codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8c446864a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c2ca15f29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7db9a26fd2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed87cc1ccd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb86683fb6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b8168d963
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
install_script.sh.template
Outdated
| # Check if all dependencies are already installed to skip apt-get update + install | ||
| all_deps_installed=true | ||
| for dep in "${APT_DEPS[@]}"; do | ||
| if ! dpkg -s "$dep" >/dev/null 2>&1; then |
There was a problem hiding this comment.
The new all_deps_installed fast path uses dpkg -s as a boolean installed check, but dpkg -s also returns success for packages in non-installed states like deinstall ok config-files. In that state curl/gnupg binaries may be absent, yet this branch skips the dependency install loop and later fails when the script invokes curl/gpg for key setup. This introduces install failures on hosts with residual package metadata; the skip condition should verify the package status is actually installed (or validate command availability) before bypassing apt-get install.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9add5c9b5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| exit 1 | ||
| fi | ||
| # Import all keys in a single gpg call | ||
| $sudo_cmd cat "${APT_GPG_KEYS[@]/#//tmp/}" | $sudo_cmd gpg --import --batch --no-default-keyring --keyring "$apt_usr_share_keyring" |
There was a problem hiding this comment.
Importing all downloaded key files in one gpg --import call can silently accept partial failures: if one key file is malformed (for example, curl -sSL saved an HTTP error page) but another file is valid, gpg can still exit successfully, so the script continues without all expected signing keys. Before this change each file was imported separately, so a bad key aborted immediately. This regression can surface later as package signature failures (especially for version-pinned installs that need non-current keys) while the installer reports key setup as successful.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Investigation: batched gpg --import behavior with mixed valid/invalid keys
The concern: combining all downloaded key files into a single cat ... | gpg --import call could silently accept partial failures if one key file is malformed.
Local test (gpg 2.x on macOS):
Created a valid GPG key (good.gpg) and a garbage file (bad.gpg containing "not a real key"), then tested three scenarios:
$ cat bad.gpg good.gpg | gpg --import --batch
gpg: no valid OpenPGP data found.
Exit code: 2
$ cat good.gpg bad.gpg | gpg --import --batch
gpg: [don't know]: invalid packet (ctb=6e)
gpg: read_block: read error: Invalid packet
gpg: import from '[stdin]' failed: Invalid keyring
Exit code: 2
$ cat bad.gpg | gpg --import --batch
gpg: no valid OpenPGP data found.
Exit code: 2
In all three cases, gpg returned exit code 2 and imported zero keys — it did not silently import the valid key while skipping the bad one.
What the man page says:
RETURN VALUE: The program returns 0 if there are no severe errors, 1 if at least a signature was bad, and other error codes for fatal errors.
The man page doesn't specifically document per-stream --import behavior with mixed data, but classifies non-zero/non-one codes as
"fatal errors."
GnuPG maintainer (Werner Koch) on gnupg.org (T769): states that gpg "imports the parts of the key which are clean and ignores the garbled stuff" — suggesting that in some cases with partially malformed OpenPGP data (e.g., a valid key with a corrupted subpacket), gpg may import what it can. However, when a file is entirely non-OpenPGP data (which is the failure mode here — curl saving an HTTP error page), gpg treats this as a fatal error and rejects the entire stream.
Conclusion: The realistic failure scenario (curl saves an HTML error page instead of a key) causes gpg to hard-fail with exit 2, which is already caught by set -e. The batched import is not a regression for this case. The per-key loop on main would have failed on the bad key too — the only behavioral difference is that the loop would have already imported the preceding good keys before failing, while the batched version imports none. Both paths abort the script.
gh-worker-dd-devflow-36fce6
bot
added
mergequeue-status: waiting
and removed
mergequeue-status: error
labels
gh-worker-dd-devflow-36fce6
bot
added
mergequeue-status: done
and removed
mergequeue-status: waiting
labels
2 participants
Summary
Speeds up the install script by parallelizing network operations and skipping redundant work:
curl -I --max-time 5calls were running sequentially (up to 30s total), now run concurrently (~5s max)&+wait, imported in a singlegpg --importcallrpm --importapt-get updateif deps already present — checkdpkg -sfirst, skip entire update+install loop ifcurl/gnupgalready installed_install_installer_script—curl -falready fails on HTTP errors, no need for a separate availability check before downloadingMotivation
https://datadoghq.atlassian.net/browse/ACIX-1390
The
new-e2e-fleet-configCI job runs this install script ~10 times per platform (install + uninstall per test). These savings compound to 2.5-6.5 minutes per platform depending on network conditions. For production customers, install time is reduced by ~15-40s per run.Test plan
bash -nsyntax check (done locally)datadog-agentrepo test(e2e): Test install script from speedup-install-script branch datadog-agent#47627🤖 Generated with Claude Code