Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 44 additions & 20 deletions .github/actions/core-cicd/evergreen-tracks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,26 +8,50 @@ tags plus `<version>_tainted` / `<track>_hold` markers).

- **`latest`** moves automatically on every GA release cut (the release pipeline calls
`promote --tracks latest --apply`).
- **`standard` / `trailing`** move **only when an operator manually dispatches** the
`evergreen-tracks-promote` GitHub Action. There is no cron — a human running the action
at a maintenance-window tag update is the cadence gate, so tracks never re-point
off-window. Automatic (e.g. daily) promotion stays off until the customer-experience
team green-lights it. When run, each track lands on the newest GA older than its age
threshold (`--standard-days` 14, `--trailing-days` 28).

The dispatch runs in three jobs: `plan` prints the intended moves (dry-run), `gate`
waits on the `evergreen-tracks-apply` environment's required-reviewer gate, and `apply`
runs after approval — nothing moves until a human reviews the plan and approves.
(One-time repo setup: Settings > Environments > `evergreen-tracks-apply` > Required
reviewers.) The dispatch is scoped to `--tracks standard,trailing` — it never moves
`latest` (the release pipeline owns that). `apply` re-derives its plan from live
registry state at run time and **fails if it no longer matches the approved plan**
(e.g. a hold/taint changed, or a release aged past a threshold during a long approval),
so it can never move tags nobody reviewed — just re-dispatch to review the new plan.
Because the plan excludes `latest`, an unattended `latest` move by the release pipeline
mid-approval doesn't trip the drift check. Only `apply` takes the shared
registry-mutation lock, so a pending approval never blocks the release from moving
`latest`.
- **`standard` / `trailing`** advance on a **daily cron at 06:00 ET**, unattended, via the
`evergreen-tracks-promote` GitHub Action (`10:00 UTC` — GitHub cron has no DST, so it's
05:00 ET in winter). Each track lands on the newest GA older than its age threshold
(`--standard-days` 14, `--trailing-days` 28). Daily promotion does not mean daily tag
movement: the planner is forward-only and age-gated, so a track moves only on the day a
release actually crosses its threshold.

The same workflow can be **dispatched manually as the break-glass path** — to review a
plan before it lands, or to move tags off-cycle. On that path a `gate` job waits on the
`evergreen-tracks-apply` environment's required-reviewer rule before `apply` runs. The
scheduled path skips `gate` entirely, which is what makes it unattended. (One-time repo
setup: Settings > Environments > `evergreen-tracks-apply` > Required reviewers.)

Both paths are scoped to `--tracks standard,trailing` — this workflow never moves
`latest` (the release pipeline owns that). `apply` re-derives its plan from live registry
state and **fails if it no longer matches what `plan` produced** (e.g. a hold/taint
changed), so it can never move tags nobody planned; the next morning's run re-plans.
Because the plan excludes `latest`, a `latest` move by the release pipeline never trips
the drift check. Only `apply` takes the shared registry-mutation lock, so a pending
approval on the manual path never blocks the release from moving `latest`.

### Notifications

The workflow posts to **#dot-releases**, deliberately only when there's something to say:

| Outcome | Posts |
|---|---|
| `plan` or `apply` failed | 🚨 yes — tags apply one at a time, so it points at the run log for which actually landed |
| A tag moved — a track advanced, or a held track was reconciled back to its hold marker | 🌲 yes — with what moved where |
| Nothing to do (most days) | nothing |

A rejected break-glass approval leaves `apply` *skipped* rather than failed, so declining a
plan never pages the channel. Notification failures are `continue-on-error` — Slack being
down never fails a promotion.

### Why unattended daily promotion is safe

Moving a floating tag deploys nothing. Every customer manifest in
`dotCMS/infrastructure-as-code` pins an immutable `<version>@sha256:<digest>`, and the
in-cluster evergreen-tracks reconciler is the only thing that rewrites those pins — it
resolves the track tag at run time, inside its biweekly on-parity Wednesday maintenance
window. A tag moved on a Tuesday has no effect until that window. Nothing in the clusters
watches the tags continuously (no Argo CD Image Updater / Keel, no floating-tag references,
no `imagePullPolicy: Always` on customer pods).

## Operator procedures

Expand Down
12 changes: 9 additions & 3 deletions .github/actions/core-cicd/evergreen-tracks/RUNBOOK.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
# Evergreen Tracks — Operator Runbook

Procedures for the three exception operations: **taint a release**, **hold a track**, and
**hold a single environment**. For the routine "advance standard/trailing" operation see
[README.md](README.md).
**hold a single environment**. Routine promotion needs no operator — `standard`/`trailing`
advance on a daily 06:00 ET cron and `latest` moves on every GA cut; see
[README.md](README.md) for that and for the manual break-glass dispatch.

Because promotion is unattended, these exception controls are how you intervene: **taint**
before a bad release can be picked up, **hold** to freeze or reverse a track, and the IaC
labels to park one environment.

Two systems are involved. Know which one you're touching:

Expand Down Expand Up @@ -187,7 +192,8 @@ Plus the Argo CD UI (Application = the **customer**, not the env) for Synced / H

| I need to… | Where | Action |
|---|---|---|
| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | dispatch → approve gate |
| Advance standard/trailing fleet-wide | `evergreen-tracks-promote` | automatic — daily 06:00 ET cron |
| Advance them off-cycle / review a plan first | `evergreen-tracks-promote` | dispatch → approve gate (break-glass) |
| Stop any track landing on a bad release | `evergreen-tracks-admin` | `taint` |
| Freeze a track / pull it off a bad release | `evergreen-tracks-admin` | `hold` |
| Resume a frozen track | `evergreen-tracks-admin` | `release-hold` |
Expand Down
Loading
Loading