Operator monitors DCA and CLC health by adel121 · Pull Request #3348 · DataDog/datadog-operator

adel121 · 2026-08-11T14:34:39Z

What does this PR do?

Adds a ComponentHealth controller (beta, off by default) that watches the Datadog
Cluster Agent and Cluster Check Runner pods and reports Kubernetes-level health issues
derived from their pod status:

component_oomkilled — a container was OOMKilled
component_crash_looping — CrashLoopBackOff, or restarts past a threshold (default 5)
component_unschedulable — pod Pending + scheduler reported Unschedulable
image_pull_failure — ImagePullBackOff / ErrImagePull

Detection is per-pod but reporting is per-component: the same issue type across several
pods is a single issue instance keyed by (namespace, component, issue_type). Issues are
logged and exposed as Prometheus metrics (..._issues_detected_total counter,
..._issues_active gauge). Node Agent pods are intentionally excluded. Gated behind
--componentHealthEnabled / DD_COMPONENT_HEALTH_ENABLED.

Motivation

Cluster-level components have no platform-level health surface today: the operator mirrors
Deployment replica counts but never sees pod restarts, OOMKills, scheduling failures, or
image-pull problems. This is the operator-side collection layer from the CONTP-1865 brief,
a foundation for surfacing DCA/CLC health (a backend HealthReport emitter follows behind
the same interface).

Additional Notes

Reporting per-component (not per-pod) gives issues stable identity across pod churn, so a
rolling restart stays one active issue instead of emit/resolve churn. Metric labels are
limited to component + issue_type to bound cardinality.
Enabling the flag makes the operator retain DCA/CLC pod status in its cache (the existing
minimalist pod-cache transform is extended only when the flag is on). Node-agent-only
deployments are unaffected.
With rapidly cycling pods, crash_looping can briefly flap (emit→resolve→emit) until
restarts cross the threshold, since CrashLoopBackOff is only intermittently present
between restarts. This shows up as extra ..._detected_total increments; the
..._active gauge reflects true current state.

Minimum Agent Versions

No minimum agent versions required — the controller reads Kubernetes pod status only.

Agent: N/A
Cluster Agent: N/A

Describe your test plan

Unit:

go build ./...
go test ./pkg/componenthealth/... ./internal/controller/ -run 'ComponentHealth|DetectPodIssues|ManagedComponent'

Manual (kind) — no image build or credentials needed. Run the operator locally with
only this controller, then feed it component-labeled pods that deliberately fail:

kind create cluster --name contp-health
go run ./cmd/main.go --datadogAgentEnabled=false --componentHealthEnabled=true \
  --enable-leader-election=false --metrics-addr=:8085

Apply pods labeled agent.datadoghq.com/component: cluster-agent / cluster-checks-runner
that crash-loop (sh -c 'sleep 2; exit 1'), OOM (tail /dev/zero, memory: 16Mi), go
unschedulable (memory: 10000Gi request), and pull a bad image — plus one component: agent
pod to confirm exclusion. Check the log and curl -s localhost:8085/metrics | grep component_health.

Verified on kind (k8s v1.33.1):

Scenario	Result
CLC unschedulable / bad image, DCA OOMKill / crash loop	✅ detected with expected type + severity
Node agent crash loop	✅ ignored (no series, no logs)
3 DCA pods crash-looping	✅ one aggregated series, value `3`
Delete 1 of 3 crash pods	✅ gauge `3 → 2`, no resolve
Delete last unschedulable pod	✅ resolve emitted, gauge `→ 0`

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

datadog-datadog-prod-us1-2 · 2026-08-11T14:43:27Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 4 Pipeline jobs failed

DataDog/datadog-operator | build

validation | build

🔧 Fix in code. This looks caused by changes in this PR.
File is not properly formatted at pkg/config/config.go:85:1 due to 'gci' lint error.

pull request linter | Check Milestone

See error
Missing milestone or `qa/skip-qa` label.

View all 4 failed jobs.
📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: adelhajhassan/poc-monitor-clc-and-dca

validation | build
Commit: ee1ee78c3ecf02fffc189218a5d4dee2560ffd39
Error (code / quality):
File is not properly formatted at pkg/config/config.go:85:1 due to 'gci' lint error.
CI job: https://github.com/DataDog/datadog-operator/actions/runs/31606240693/job/94145799013

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ee1ee78 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee1ee78c3e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-12T14:25:45Z

+}
+
+func (r *ComponentHealthReconciler) setActiveGauge(key componentIssueKey, affectedPods int) {
+	metrics.ComponentHealthIssuesActive.WithLabelValues(key.component, key.issueType).Set(float64(affectedPods))


Sum active gauge across watched namespaces

With multi-namespace or cluster-wide watches, reported is keyed by namespace but the Prometheus series is not, so this setter writes only the affected-pod count for the single namespace being reconciled. If two namespaces both have cluster-agent/component_crash_looping, the later reconcile overwrites the earlier count, and resolving one namespace can set the shared gauge to 0 while the other namespace is still affected; either include namespace in the labels or set the gauge from the aggregate across all matching keys.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-12T14:25:45Z

+				// waiting reasons like ImagePullBackOff), scheduling conditions, phase, and
+				// termination context. Retaining these fields is what increases cache memory,
+				// which is why this is gated behind ComponentHealthEnabled.
+				if opts.ComponentHealthEnabled {


Keep full pod status limited to component pods

When ComponentHealthEnabled is combined with UntaintControllerWaitForCSIDriver, the earlier branch leaves podLabel nil so CSI node-server pods can be cached, which means this unconditional copy retains full phase/conditions/container statuses for every Pod in the agent and CSI namespaces. In namespaces such as kube-system that can include many unrelated pods and defeats the memory-saving transform; gate the full status copy on the DCA/CLC component label and leave only the untaint fields for the broader cache.

Useful? React with 👍 / 👎.

datadog-datadog-prod-us1-2 · 2026-08-12T14:31:06Z

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

pkg/config/config.go failed gci because the new ComponentHealthEnabled field was not aligned with the surrounding WatchOptions fields. Aligned the declaration without changing behavior.

Commit fix to this PR

View in Datadog | Reviewed commit ee1ee78 · Any feedback? Reach out in #deveng-pr-agent

github-actions Bot added the team/container-platform label Aug 11, 2026

adel121 force-pushed the adelhajhassan/poc-monitor-clc-and-dca branch from 6577d05 to ed9a81e Compare August 11, 2026 14:41

Operator monitors DCA and CLC health

2871f6a

adel121 force-pushed the adelhajhassan/poc-monitor-clc-and-dca branch from ed9a81e to 2871f6a Compare August 11, 2026 15:04

Merge branch 'main' into adelhajhassan/poc-monitor-clc-and-dca

ee1ee78

adel121 added this to the v1.31.0 milestone Aug 12, 2026

adel121 marked this pull request as ready for review August 12, 2026 14:21

adel121 requested a review from a team August 12, 2026 14:21

adel121 added the enhancement New feature or request label Aug 12, 2026

chatgpt-codex-connector Bot reviewed Aug 12, 2026

View reviewed changes

Navigation Menu

Uh oh!

Operator monitors DCA and CLC health - #3348

Operator monitors DCA and CLC health#3348
adel121 wants to merge 2 commits into
mainfrom
adelhajhassan/poc-monitor-clc-and-dca

adel121 commented Aug 11, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 11, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 12, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 12, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 12, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

adel121 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 11, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 12, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bits has a CI fix ready

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

adel121 commented Aug 11, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Aug 11, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

datadog-datadog-prod-us1-2 Bot commented Aug 12, 2026 •

edited by datadog-prod-us1-4 Bot

Loading