Add an experimental prepared blue-green Agent rollout

[AliDatadog]

Why

A DaemonSet update can delete a working Agent before Kubernetes pulls and starts its replacement. A slow or failed pull can leave a node without an Agent.

Native surge cannot complete this handoff when the replacement must wait for the old Agent to release host resources. This experiment keeps rollout control in the Operator.

Design

The feature uses this opt-in annotation:

experimental.agent.datadoghq.com/node-agent-rollout-mode: prepared-blue-green-v1

The Operator maintains two OnDelete DaemonSets. One slot serves. The other slot prepares the next revision. A one-time conventional rollout installs the lock wrapper before the Operator allows overlap.

A small init container copies s6-setlock from the core Agent image into a shared Pod volume. Each Agent container then waits on two locks:

• a host lock prevents both generations from running the same component;
• a Pod-local lock tells the startup probe when that component starts.

The Operator observes a Running target with no restarts twice. It then removes the source on a bounded set of nodes. Each target component starts as soon as its matching source component exits. The target must become Available before the Operator starts another batch.

The Operator uses the existing maxUnavailable value as a per-profile budget. Capacity fallback can use the same budget to remove a source that blocks target scheduling. This fallback is delete-first.

For hostNetwork: true, the Operator removes declared ports that would block scheduling. Processes still bind the node ports after lock handoff. Green uses a different core command port to prevent mixed-generation control traffic.

Trade-offs

• Preparation proves that images, init containers, and lock wrappers started. It does not prove Agent health.
• Prepared mode uses startup probes as lock signals. Readiness and liveness check the Agent after handoff. The Operator can extend liveness failure windows to preserve startup time. A failure visible only to the replaced startup handler can be missed.
• A failed target consumes the budget and stops new handoffs. Automatic rollback is not included.
• Capacity fallback has the same pull and startup downtime as the current rollout.
• Locks do not transfer queues, caches, assignments, or process state.
• The earlier custom-gate PoC used an incompatible lock type. Schema 3 rejects that PoC instead of allowing unsafe overlap.

Validation

• go test ./... -count=1 passed.
• make lint passed with 0 issues.
• Focused tests cover migration, new nodes, capacity fallback, controller restart, concurrent updates, and opt-out.

No Agent code change is required.

Related PR

• Experimental deployment and recovery runbook: https://github.com/DataDog/k8s-datadog-agent-ops/pull/9477

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 80.25%
• Overall Coverage: 51.79%

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6b02ba7 | Docs | View more details | Give us feedback!}