feat: add telegraf:<version>-distroless variant

[victor-gama]

Add a distroless Telegraf variant (1.37, 1.38, 1.39)

This adds an opt-in distroless variant for each supported Telegraf version, producing telegraf:{1.37,1.38,1.39}-distroless. Each image is built FROM scratch and ships only the static telegraf binary plus the handful of files it needs at runtime (no shell, no libc, no package manager, no OS userland), and runs non-root (uid 65532). The change is purely additive: the existing default and alpine images are untouched.

What to review

File	What changed
telegraf/{ver}/distroless/Dockerfile	New two-stage build (alpine fetch → scratch). The heart of the PR.
telegraf/{ver}/distroless/circle-test.sh	Per-variant smoke tests, sourced by the root loop below.
telegraf/{ver}/distroless/TESTING.md + local-test.sh	Manual runbook + one-shot script for the two runtime checks CI doesn't cover.
telegraf/manifest.json	Registers distroless for 1.37/1.38/1.39 on amd64 + arm64v8.
circle-test.sh (root)	Re-enables the shared "run every subdirectory circle-test.sh" loop (commented out before this branch). The only circle-test.sh files at depth ≥2 in the repo are the three added by this PR, so the loop runs exactly the new distroless smoke tests; it does not execute or alter any existing image's tests.

These files are near-duplicates across the three versions; review one set, then diff-check the other two. The three Dockerfiles are byte-identical except the TELEGRAF_VERSION default (the two ARG lines); TESTING.md and local-test.sh are byte-identical across all three (each derives its tag from its own path). Only circle-test.sh differs further, and only in the hardcoded tag and version-string literals. So review the 1.39 set in full, then confirm 1.37/1.38 differ only by version.

Each tag maps to a Telegraf release:

Tag	Telegraf
1.37-distroless	1.37.3
1.38-distroless	1.38.4
1.39-distroless	1.39.0

How the Dockerfile works

It's a two-stage build:

• An alpine fetch stage downloads the release, GPG-verifies it against InfluxData's signing key (24C975CBA61A024EE1B631787C3D57159FC2F927, the same key the alpine sibling uses), unpacks it, and assembles /rootfs as the exact final image tree (binary + config + CA bundle + tzdata + nsswitch.conf + passwd/group + home/tmp).
• The final scratch stage brings that tree over with a single plain COPY (no ADD --unpack, no COPY --parents, no other BuildKit-only feature), so it builds natively on the classic builder, docker/BuildKit, and podman build/buildah.
• Integrity is the GPG signature check in the fetch stage; the build fails loudly if verification fails. Nothing from alpine reaches the final image except the files explicitly staged under /rootfs.
• The runtime base is scratch, so there are no OS packages to track or bump.

What the final image contains: the complete inventory (everything a distro base would add is deliberately omitted)

Path	Why it's needed
/usr/bin/telegraf	the GPG-verified static (CGO_ENABLED=0) binary
/etc/telegraf/{telegraf.conf,telegraf.d}	default config + drop-in dir
/etc/ssl/certs/ca-certificates.crt	TLS trust roots for outbound HTTPS (e.g. influxdb_v2)
/usr/share/zoneinfo	IANA tz database: time.LoadLocation, json_timezone, cron schedules
/etc/nsswitch.conf	hosts: files dns so Go's resolver checks /etc/hosts before DNS
/etc/passwd, /etc/group	deterministic non-root identity nonroot:65532
/home/nonroot (owned 65532)	$HOME for uid 65532
/tmp (owned 65532)	os.TempDir() default; scratch ships neither

Not a drop-in replacement

This is a hardening-focused variant, not a swap-in for the default/alpine images. Because there is no shell and no setcap/setpriv, it does not support the ICMP ping input, privileged ports (below 1024), or shell-out plugins like exec, snmp, and sensors. It targets network-listener and push workloads. The Dockerfile header notes the image ships no shell or OS userland, the root cause of these limitations; the specific unsupported plugins/ports are documented only here.

Why scratch, and why a tag not a digest

Most of the CVEs we hit come from the parent image's userland (distro packages, the shell, coreutils) rather than from Telegraf itself. See this Debian CVE, which had no fix on any stable release as of June 26, 2026, and which forces us to keep swapping between the Debian and Alpine images to stay ahead of scanners. Building FROM scratch drops that whole layer, so CVE exposure comes down to the Telegraf binary plus a maintained CA bundle and tzdata. The image also runs non-root, so it satisfies Kubernetes runAsNonRoot and a read-only root filesystem out of the box.

Two integrity/versioning choices changed in this revision, per @srebhan's feedback:

• FROM scratch, not a distroless base image. Rather than depend on gcr.io/distroless/static, we assemble the rootfs ourselves in the alpine stage. That gives us a complete inventory of the image (see the table above) with nothing we don't use, and one fewer external base to track.
• Pull by tag/version, not by pinned digest. The release is selected by the TELEGRAF_VERSION build arg and GPG-verified; the alpine fetch base is the alpine:3.23 tag. Neither is pinned to a sha256 digest. This matches how InfluxData publishes to Docker Hub: a new Telegraf release is picked up by bumping the version and rebuilding, with no per-release digest to hand-edit. (The earlier revision sha256-pinned both the tarball via ADD --checksum= and a gcr.io/distroless/static base, each of which would need a manual digest bump on every release.) Integrity is still enforced, now by the GPG signature instead of a committed hash.

The tradeoff we accept: we own refreshing the CA bundle and tzdata. They're sourced from the alpine:3.23 fetch base at build time. Because that's a patch-rolling tag, a plain rebuild picks up alpine's latest CA/tzdata patches within the 3.23 series, so a periodic rebuild keeps the data fresh; that's the maintenance cost of scratch versus consuming a maintained base. Note that alpine:3.23 is minor-pinned: once 3.23 reaches EOL a plain rebuild no longer refreshes the data and the fetch base must be bumped to the next series, an explicit part of that accepted cost. One caveat for scanning: a scratch image exposes no OS-package database, so scanners report zero OS packages (a cleaner report, but the copied CA bundle and tzdata aren't scanner-visible).

Testing

I built and ran all three locally on linux/arm64 with Podman (they also build with Docker), in both cases with a plain build (no BuildKit container, no --privileged). The fetch-stage GPG check described above gates every build.

Reproduce locally (from any version's directory):

cd telegraf/1.39/distroless          # or 1.37 / 1.38
./local-test.sh                      # builds the image if needed, then runs both manual checks
# CONTAINER_CLI=podman ./local-test.sh   # use Podman   |   REBUILD=1 ./local-test.sh   # force a rebuild

To build/inspect the image on its own: docker build -t telegraf:1.39-distroless . (plain podman build also works). The re-enabled root circle-test.sh loop runs the smoke tests in CI; the full runbook is in each telegraf/{ver}/distroless/TESTING.md.

Smoke tests (now run in CI via the re-enabled root loop), in telegraf/{ver}/distroless/circle-test.sh:

• --version reports the expected version
• the image runs as non-root 65532
• it is genuinely shell-less (a /bin/sh entrypoint override fails)
• telegraf --test exits 0 on the default config

Manual runtime checks: the two capabilities the smoke tests deliberately don't cover (TESTING.md + local-test.sh)

The smoke tests are hermetic and by design don't exercise two things the shell-less image relies on, so each version ships a manual runbook (TESTING.md) and a one-shot script (local-test.sh, which derives its tag from its own directory so it stays byte-identical across versions). These are not wired into CI; automating them is a separate follow-up pending discussion with the maintainers.

Check	How	Pass criterion
Outbound TLS validates a real public cert	inputs.http_response against https://dl.influxdata.com (insecure_skip_verify=false)	metric shows result_code=0i
tzdata present, named zones resolve	inputs.file + json_timezone, UTC vs America/Los_Angeles	timestamps differ by exactly 28800000000000 ns (8h)

Results from local-test.sh (verified on all three versions):

• 1.37 (1.37.3), 1.38 (1.38.4), and 1.39 (1.39.0): both checks pass, clean exit 0, on the Podman path I ran. The same script picked the right tag per version and built each one's Dockerfile natively.
• TESTING.md's Troubleshooting section documents the negative case: pointing the TLS check at https://self-signed.badssl.com yields result_code=3i (connection_failed), confirming the CA bundle genuinely rejects an untrusted cert, so a passing result_code=0i run actually means something. (local-test.sh itself only runs the positive check.)
• shellcheck is clean, and re-runs are idempotent (local-test.sh reuses the existing image unless REBUILD=1).

[srebhan]

Thanks a lot for your work @victor-gama!

I don't think we need the staging approach as we can utilize the docker commands to get, check and unpack the released Telegraf binary.
Furthermore, I would really love to learn on how the circle-test.sh approach works. Do we need to execute this manually or is this something auto-executed?

Please also test at least the following:

• use a plugin to connect to an https endpoint for verifying TLS works
• use a plugin to parse times with timezone information to make sure the timezone-info is available

[srebhan]

Why not

Suggested change

	# Verified binary + default config + empty drop-in dir; users mount their own over these.
	COPY --from=fetch /out/usr/bin/telegraf /usr/bin/telegraf
	COPY --from=fetch /out/etc/telegraf/telegraf.conf /etc/telegraf/telegraf.conf
	COPY --from=fetch /out/etc/telegraf/telegraf.d /etc/telegraf/telegraf.d
	# Inject telegraf
	ADD --checksum=${TELEGRAF_CHECKSUM} --unpack=true https://dl.influxdata.com/telegraf/releases/telegraf-${TELEGRAF_VERSION}_${ARCH}.tar.gz /

[victor-gama]

Updated the code, I tried this method but ran into some issues, I couldn't find a way to only unpack the binaries and the files that I needed.

This process unpacks the whole .tar in the root directory, which also has the binary with the version on it.

My approach in the current PR is a bit more generic by only copying the binary and naming it as telegraf, that way we don't need to change the entry point across Dockerfiles.

Let me know what you think or if you would like me to take this approach instead.

[victor-gama]

Regarding the testing, I updated the circle-test.sh script in the root directory to also run the circle-test.sh file on each distroless sub directory, now this is automated on the pipeline 🥳

As for testing with different plugins I added instructions on how I tested this locally and also added a script that automates the same steps if anyone wants to validate this locally for the plugins you mentioned.

The Script/TESTING.md files are duplicated across directories for simplicity as I assume when you have a new version it will be copy/pasted, Let me know if you would prefer I unify them somewhere else.

[victor-gama]

Testing TL;DR - does TLS and timezones actually work in the distroless image?

The existing circle-test.sh smoke tests don't touch two things I wanted to be sure about: whether the image can make outbound TLS calls (i.e. does it actually have the system CA bundle) and whether named timezones work (does tzdata ship with it). I wrote a little runbook + script for each distroless variant to check both by hand. It's not wired into CI yet; that's a separate follow-up once I've talked it through with the maintainers.

What I added (same bytes in all three versions):

• telegraf/{1.37,1.38,1.39}/distroless/TESTING.md, the step-by-step so anyone can reproduce it
• telegraf/{1.37,1.38,1.39}/distroless/local-test.sh, a one-shot script. It figures out its own tag from the directory it lives in, so I can copy it around without editing anything.

The two checks:

What I'm checking	How	What "pass" looks like
Can it make a real HTTPS call and validate the cert?	inputs.http_response hitting https://dl.influxdata.com with insecure_skip_verify=false	metric comes back withresult_code=0i
Does tzdata exist / do named zones resolve?	inputs.file + json_timezone, comparing UTC to America/Los_Angeles	timestamps differ by exactly 28800000000000 ns (that's 8 hours)

Results (ran these against images I built locally):

• ✅ 1.37 (Telegraf 1.37.3) and 1.39 (1.39.0), both checks pass with a clean exit 0. The same script correctly picked the right tag per version and built each one's Dockerfile.
• ✅ Sanity check that it's not lying to me: I pointed the TLS check at https://self-signed.badssl.com and got result_code=3i (connection_failed). That means the cert validation
  genuinely fails on an untrusted cert, so the passing case actually means something.
• ✅ shellcheck is clean, both the plain-docker and CONTAINER_CLI=podman paths work, and re-running is safe (it reuses the existing image instead of rebuilding).

One gotcha with podman: the distroless Dockerfile uses ADD --checksum --unpack, which is a BuildKit-only feature, and plain podman build/buildah just can't do it.

The script notices when you're on Podman and builds through a moby/buildkit container instead (the same trick docker buildx uses under the hood), then loads the image in. For what it's worth, I ran all of this with podman + moby because that's my setup, but the docker path should work just as well.

[victor-gama]

I am not very familiar with CircleCI but this should allow your CI/CD to auto run the smoke tests I added on each /distroless directory (circle-test.sh), it was already built in but commented out 9 years ago on this commit

I can revert this if this could break your CI pipeline just let me know.

[victor-gama]

ah! it was commented because it was actually borked.
It should be fixed now.

CircleCI output can be found here

[victor-gama]

My distroless variant is circle-test.sh is being tested as part of the CI/CD