feat(evergreen-tracks): daily 06:00 ET cron for standard/trailing, gate kept as break-glass

[sfreudenthaler]

What

standard/trailing promotion is now end-to-end unattended: a daily cron at 06:00 ET runs plan → apply with no human in the loop. Per #36705 this is Option B, and it's surgical — the plan/apply structure, drift check, isolation test, and docs from #36652 all stay.

Manual dispatch is retained as the break-glass path: plan → gate → apply, where gate still waits on the evergreen-tracks-apply required-reviewer environment. Use it to review a plan before it lands, or to move tags off-cycle.

latest is untouched — the release pipeline moves it on every GA cut.

How the two paths diverge

	trigger	gate	applies
Daily	schedule 10:00 UTC	skipped (if: github.event_name == 'workflow_dispatch')	unattended
Break-glass	workflow_dispatch	waits for a dotDevelopers reviewer	after approval

apply accepts a skipped gate but not a failed or rejected one — hence explicit result checks rather than a bare always().

Why unattended daily promotion is safe

Moving a floating tag deploys nothing. Verified against dotCMS/infrastructure-as-code at HEAD:

• Every customer manifest pins an immutable <version>@sha256:<digest>; zero floating-tag references exist anywhere in the repo.
• The in-cluster reconciler is the only thing that rewrites those pins, and it resolves the track tag only at run time, inside its biweekly on-parity Wednesday window (timeZone: America/New_York). A tag moved on a Tuesday has no effect until that window.
• No continuous watcher: no Argo CD Image Updater, no Keel/Flux, no imagePullPolicy: Always on customer pods.
• The reconciler keeps no cross-run state and its NO-DOWNGRADE gate is a pure local compare, so intervening tag moves are irrelevant — it simply jumps to wherever the tag points now.
• Reconciler is armed on all four prod clusters (dotcms/infrastructure-as-code#8086, dotcms/infrastructure-as-code#8112) and has been rolling envs successfully.

Also: daily promotion ≠ daily tag movement. The planner is forward-only and age-gated (14d / 28d), so a track moves only on the day a release actually crosses its threshold.

Bug this would have shipped without

A schedule event carries no inputs, so github.event.inputs.* is empty on the cron path — #36652 had removed the fallbacks when it dropped the old cron. Restored in both jobs; without them the nightly run would have invoked the engine with an empty --repo and empty day thresholds.

Verification

• All six plan/gate result combinations resolve correctly — cron applies; rejected, cancelled, or failed-plan paths do not.
• End-to-end break-glass dispatch against dotcms/dotcms-test (run 31037723308): gate waited → approved → plan unchanged; applying → tag moved. Plan correctly excluded latest.
• 66 unit tests green. Both workflow files parse.

Docs

README cadence section rewritten (daily cron + break-glass + the safety rationale); RUNBOOK intro and quick-reference updated — routine promotion no longer needs an operator, so the taint/hold controls are framed as the intervention levers.

Closes: #36705

🤖 Generated with Claude Code

[alwaysmeticulous]

🤖 No test run has been triggered as your Meticulous project has been deactivated (since you haven't viewed any test results in a while). Click here to reactivate.

_{Last updated for commit 4f584fe. This comment will update as new commits are pushed.}

[claude]

Claude finished @sfreudenthaler's task in 6m 35s —— View job

---

PR Review — evergreen-tracks daily cron

• Read the workflow YAML changes
• Read README/RUNBOOK doc changes
• Analyze cron/gate/apply logic and Slack notifications
• Post review

I traced the workflow against the engine (cli.py, executor.py) and the notify-slack action interface. The cron/gate/apply wiring is sound: the if: >- always() && plan==success && (gate==success||skipped) correctly admits the scheduled skip while rejecting a failed/rejected gate, the || 'dotcms/dotcms' / || '14' / || '28' fallbacks correctly cover the empty-input schedule path, and the ^(standard|trailing) (\(held\) )?-> regex matches both the advance line (cli.py:83) and the held-reconcile line (cli.py:75) — the sed 's/ (sha256:.*//' strips only the digest, leaving (held) intact. Defaults (14/28), cron→06:00 ET, and lock semantics all line up with the docs.

One non-blocking finding:

New Issues

• 🟡 Medium: .github/workflows/cicd_evergreen-tracks-promote.yml:243-256 — The single failure payload fires on both plan failure and apply failure, but its body asserts "Tags are applied one at a time, so one track may have moved before the failure — check the run log for which actually applied." That's true for an apply-phase failure, but on a plan-phase failure (registry/API hiccup, uv error, docker-login failure — all plausible) nothing has been applied yet: apply is skipped and no point_tag ran. So a plan failure pages #dot-releases claiming a track may have moved when none could have. This is the same class of "the alert asserts the opposite of reality" issue you already corrected in 434251d for the half-move case — worth splitting the message, e.g. gate the "one track may have moved" line on needs.apply.result == 'failure' and keep a simpler "promotion errored; no tags moved" for the plan-failure branch. Fix this →

Notes (not blocking)

• Drift on the cron path → 🚨 FAILED alert. If a hold/taint marker changes in the seconds between plan and apply (or apply queues behind the release-pipeline lock long enough to cross a date/threshold boundary), the drift check exits 1 → apply.result == 'failure' → a 🚨 alert. On the cron this is benign (next morning re-plans), but it reads as an incident. Very rare, so fine to leave — just flagging that a benign drift and a real failure are indistinguishable in the channel.
• Semgrep mutable-tag finding (raised in your comment): accepting @v4/@v5/@v3 floating majors on trusted first-party/astral-sh/docker actions for auto-patching is a reasonable tradeoff and consistent with the rest of the repo. No objection.

Everything else — the six plan/gate combinations, the skipped-gate accept, the rejection-stays-silent path, the moved/summary extraction, and continue-on-error on both notify steps — checks out. The end-to-end verification runs you documented cover the paths that can be exercised pre-merge.

[sfreudenthaler]

Added: #dot-releases notifications (commit 4f584fe)

Wired up using the repo's existing core-cicd/notification/notify-slack action and the channel ID already in use for #dot-releases (CE1TBQU00) — no new pattern invented.

Designed to stay quiet, since a releases channel shouldn't get a daily "nothing to do" ping:

Outcome	Posts
plan or apply failed	🚨 yes — notes the tags did not move (unchanged, not half-moved) and that envs are unaffected until the next reconciler window
A track advanced	🌲 yes — with what moved where, and a reminder that envs pick it up at their next window, not immediately
Nothing to do (most days)	nothing

Two details worth calling out:

• A rejected break-glass approval never pages the channel. Declining leaves apply skipped, not failed, and the failure condition keys on == 'failure' — so a deliberate "no" is silent by construction.
• Both notify steps are continue-on-error, so Slack being down can never fail a promotion.

Verified the summary extraction against both-moved / one-moved / no-moves / held-track-only plans. YAML parses, 66 tests still green.

[sfreudenthaler]

⚠️ Slack smoke test found a real gap — one Slack action needed before this lands

Dispatched the full path against dotcms/dotcms-test (run 31056700820). Job routing was exactly right — Notify failure skipped, Notify moves fired, with a correctly rendered payload:

🌲 Evergreen tracks advanced on dotcms/dotcms-test — standard -> 26.03.12-01; trailing -> 26.03.12-01. Environments on these tracks pick this up at their next reconciler maintenance window, not immediately.

But the message never reached the channel. The Slack API returned:

Error: An API error occurred: not_in_channel

The bot behind SLACK_BOT_TOKEN is cicd (U0164AW6YKC) — it posts fine to #log-docker, #log-sdk-libs, etc., but it is not a member of #dot-releases. The release announcements already in that channel come from a different app (dotBot), which makes the channel look wired up when it isn't for this token.

Fix — one command in Slack, no code change:

/invite @cicd

run in #dot-releases. I'll re-run the smoke test afterwards to confirm.

This is a pre-existing bug, not one this PR introduces

cicd_comp_changelog-site-publish-phase.yml defaults to the same channel (CE1TBQU00) with the same token, and its notice fires only on failure/skip — so changelog-publish failures have almost certainly been silently swallowed the same way. Same /invite fixes both. Worth a look from whoever owns that workflow.

Why the job still reported success

Both notify steps are continue-on-error: true, which is deliberate — Slack being down must never fail a tag promotion — but it does mean a misconfigured channel fails quietly. That tradeoff is why this smoke test was worth running rather than trusting the green check.

[sfreudenthaler]

✅ Slack verified after the @cicd invite

Re-ran the full path (run 31057056392) — no not_in_channel, and the message landed in #dot-releases at 19:40 EDT:

🌲 Evergreen tracks advanced on dotcms/dotcms-test — standard -> 26.03.12-01; trailing -> 26.03.12-01. Environments on these tracks pick this up at their next reconciler maintenance window, not immediately.

Routing confirmed correct on a real run: Notify failure skipped, Notify moves fired.

Everything on this PR is now verified end-to-end — cron wiring, all six plan/gate outcome combinations, break-glass approval flow, drift check, and Slack delivery. Ready for review.

Two notes for after merge:

• The first scheduled run is the one path that can't be exercised pre-merge (scheduled triggers only fire from the default branch). It'll run 10:00 UTC / 06:00 ET the morning after merge; the gate job should show as skipped and apply should proceed unattended.
• The pre-existing #dot-releases gap in cicd_comp_changelog-site-publish-phase.yml is now also fixed by the same invite — those failure notices had been silently dropped and should work from here on.

[sfreudenthaler]

Both Medium findings confirmed and fixed (commit 434251d)

Checked each against the engine rather than taking them on faith — both were real.

🟡 "not half-moved" was a false guarantee. Confirmed: cmd_promote applies moves in a per-track loop (cli.py:81-84) and point_tag runs imagetools create with check=True (executor.py:27-30), so standard succeeding then trailing failing leaves a genuinely half-moved registry — the alert asserted the opposite of reality, which is the worst thing an incident message can do. Reworded to point responders at the run log for which tracks actually landed. Kept the "environments unaffected until the next reconciler window" line, which holds either way.

🟡 Held-track reconciles were silently dropped. Confirmed: the engine logs standard (held) -> reconcile to standard_hold (…) (cli.py:75) and then really does re-point that tag — but the anchored grep ^(standard|trailing) -> skips it. Demonstrated the miss, then fixed with one optional group: ^(standard|trailing) (\(held\) )?-> . Answering the reviewer's open question: yes, this should announce — an unattended cron physically moving a tag with no notice is exactly the silent-change case this notification exists to prevent.

Headline generalized *Evergreen tracks advanced* → *Evergreen track tags moved*, since a held reconcile is a correction, not an advance.

Verified across five plan shapes:

Plan	moved	Posts
both tracks advanced	true	✅
no track moves needed (most days)	false	quiet
held reconcile only — previously silent	true	✅ now fires
held reconcile + advance	true	✅ both listed
held but already consistent (no tag touched)	false	quiet — correct

That last row matters: when a held track already matches its marker the engine logs held at …, skipping promotion and calls no point_tag, so staying quiet is right — the fix doesn't over-trigger.

66 tests green; YAML parses. The DST note the reviewer flagged is already documented inline and in the README.

[sfreudenthaler]

semgrep doens't like the mutable tag but it's acceptable here because we want to pull the latest action of that major. minors and patch updates in rapidly devloping tech are worth the security boost we get from auto-patching on a floating major tag.

[sfreudenthaler]

🟡 Gate-failure alert: won't fix — the suggested condition would be harmful (commit 131bad7)

The finding's own "what to verify" was the right question, so I tested it rather than reasoned about it. Dispatched a run and rejected the deployment (run 31065188863):

job	result
plan	success
gate	failure ← a rejection, not an infra error
apply	skipped
notify	success — both Slack steps skipped, nothing posted

So the reviewer's stated assumption is wrong in the decisive way: GitHub reports a reviewer rejection as gate.result == 'failure' — the identical value a gate infrastructure failure produces. The two states are indistinguishable from needs.gate.result.

Adding || needs.gate.result == 'failure' would therefore page #dot-releases on every deliberate rejection, converting a correct, intentional silence into recurring noise. Telling them apart would mean calling the deployment-review API (/actions/runs/{id}/approvals) and parsing whether a human recorded a rejection — a real chunk of complexity for the residual case.

And that residual case is thin:

• It cannot occur on the unattended cron — gate is dispatch-only, so it never runs there. The cron is the entire reason this alert exists.
• On the manual path an operator just clicked Run workflow, and a gate failure leaves a red run (run_conclusion: failure) that GitHub already notifies them about.

No logic change. Added a comment recording the verified behavior and the reasoning, so the next reader doesn't "fix" it back into a noise source.

Bonus: this also empirically confirms a claim this PR has been making without proof — a deliberate "no" never pages the channel. Verified: both notify steps skipped, and no message appeared in #dot-releases.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request