Skip to content

fix(chart): topology-gate single-replica PDBs so single-node installs stay drainable (#560) - #710

Open
shujaatTracebloc wants to merge 3 commits into
developfrom
feature/560
Open

fix(chart): topology-gate single-replica PDBs so single-node installs stay drainable (#560)#710
shujaatTracebloc wants to merge 3 commits into
developfrom
feature/560

Conversation

@shujaatTracebloc

@shujaatTracebloc shujaatTracebloc commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem

client/templates/jobs-manager-pdb.yaml and client/templates/mysql-pdb.yaml rendered minAvailable: 1 unconditionally (gated only on podDisruptionBudget.*.create, default true). For a single-replica workload, minAvailable: 1 means the eviction API permits zero voluntary disruptions (currentHealthy 1 − 1 = 0 < 1), so kubectl drain/cordon blocks forever. On a single-node install this wedges all node maintenance (manual k3s upgrade, node drain), and the pod has nowhere to reschedule anyway.

Fix — topology-gated tri-state create

The chart already distinguishes topologies via hostPath.enabled (true = single-node/bare-metal installer; false = multi-node CSI: AKS/EKS/OpenShift) — the "storage/topology mode" the issue calls for, and the same signal SINGLE_NODE already defaults from.

podDisruptionBudget.{mysql,jobsManager}.create becomes tri-state:

  • null (default) = AUTO: render on multi-node CSI (hostPath.enabled=false), skip on single-node/bare-metal (hostPath.enabled=true)
  • true = force render regardless of topology
  • false = never render

Rationale for gating (vs maxUnavailable: 1 or a blanket off)

On multi-node CSI the PDB is a feature: it blocks a surprise autoscaler scale-down / node upgrade from silently evicting the only mysql/jobs-manager pod, forcing a deliberate operator drain of the single-replica stateful workload. Switching to maxUnavailable: 1 there would make it a no-op, and would also require rewriting the existing minAvailable-asserting unittest. On single-node the same PDB has no upside (nowhere to reschedule) and only wedges drains — so it is skipped. Topology-gating preserves the multi-node protection, keeps single-node drainable, and adds no existing-test rewrites. Default topology is CSI (hostPath.enabled=false), so default behavior is unchanged.

Changes

  • client/templates/jobs-manager-pdb.yaml, client/templates/mysql-pdb.yaml — topology-gated guard; minAvailable: 1 retained where it renders.
  • client/values.yamlcreate: truecreate: null with tri-state docs.
  • client/values.schema.jsoncreate type boolean["boolean","null"], default null.
  • client/Chart.yamlversion/appVersion 1.9.391.9.40 (chart-version-guard, lockstep).
  • client/tests/priority_class_pdb_test.yamladded 7 cases (single-node skip, CSI render, force-on/off overrides); all existing tests kept.

Verification

  • helm lint --strict — green on aks/bm/eks/oc profiles.
  • helm template — renders on all 4; PDB docs: aks/eks/oc = 2, bm = 0.
  • helm unittest — 7 new PDB tests pass; the local 5 failed / 5 errored are a pre-existing helm v4.2.0 schema-message artifact (identical on clean develop); CI pins helm-unittest 0.5.2.

Closes #560

🤖 Generated with Claude Code


Note

Medium Risk
Changes cluster maintenance behavior for PDBs on single-node vs multi-node topologies; mis-detection could drop drain protection on CSI fleets or reintroduce stuck drains on single-node installs.

Overview
Fixes #560: unconditional minAvailable: 1 PDBs on single-replica mysql and jobs-manager made kubectl drain hang forever on one-node clusters where the pod cannot reschedule.

podDisruptionBudget.{mysql,jobsManager}.create is now tri-state (null / true / false), default null (AUTO). In AUTO mode, PDBs render only on multi-node CSI (hostPath.enabled=false, typical AKS/EKS/OpenShift) and are omitted on single-node installs (hostPath.enabled=true or explicit env.SINGLE_NODE="true", matching jobs-manager’s SINGLE_NODE resolution). true / false still force render or disable regardless of topology. Where they render, minAvailable: 1 is unchanged.

Defaults move from create: true to create: null in values.yaml; the JSON schema allows boolean | null. Chart version bumps to 1.9.43. Helm unit tests cover AUTO skip/render, SINGLE_NODE edge cases, and explicit overrides.

Reviewed by Cursor Bugbot for commit 3e7c699. Bugbot is set up for automated code reviews on this repo. Configure here.

… drainable (client#560)

jobs-manager-pdb.yaml and mysql-pdb.yaml rendered minAvailable:1
unconditionally. For a 1-replica workload the eviction API then permits
zero voluntary disruptions, so kubectl drain/cordon blocks forever. On a
single-node install that wedges all node maintenance and the pod has
nowhere to reschedule anyway.

Make podDisruptionBudget.{mysql,jobsManager}.create tri-state:
  null (default) = AUTO: render on multi-node CSI (hostPath.enabled=false),
                   skip on single-node/bare-metal (hostPath.enabled=true)
  true  = force render
  false = never render

Preserves the protective PDB on multi-node CSI (deliberate operator drain
of a single-replica stateful DB) while keeping single-node installs
drainable. Bumps Chart.yaml 1.9.39 -> 1.9.40 (chart-version-guard) and
adds helm-unittest coverage for both topologies and the overrides.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@shujaatTracebloc shujaatTracebloc self-assigned this Aug 13, 2026
@shujaatTracebloc
shujaatTracebloc requested review from LukasWodka and saadqbal and removed request for saadqbal August 13, 2026 13:57
@shujaatTracebloc
shujaatTracebloc marked this pull request as ready for review August 14, 2026 10:37
Comment thread client/templates/jobs-manager-pdb.yaml Outdated
…h (client#560)

Bugbot (Medium): the AUTO-mode PDB skip keyed only on hostPath.enabled, so a
node-local single-node install (env.SINGLE_NODE=true, hostPath.enabled=false,
local-path CSI k3d, AGENTS=0) still rendered minAvailable:1 — undrainable on a
single node, blocking kubectl drain / k3s upgrade. Gate now mirrors
jobs-manager-deployment's SINGLE_NODE resolution (explicit env.SINGLE_NODE wins,
else hostPath.enabled), so both single-node topologies skip the PDB while
multi-node CSI still renders it. Added node-local + SINGLE_NODE=false regression
tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3e7c699. Configure here.

{{- $env := default dict .Values.env -}}
{{- $hostPathMode := (default dict .Values.hostPath).enabled | default false -}}
{{- $isSingleNode := ternary (eq (toString (index $env "SINGLE_NODE")) "true") $hostPathMode (hasKey $env "SINGLE_NODE") -}}
{{- $create := ternary $pdb.create (not $isSingleNode) (kindIs "bool" $pdb.create) -}}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PDB gate misuses SINGLE_NODE flag

Medium Severity

AUTO mode treats env.SINGLE_NODE=true as single-node topology and skips the minAvailable:1 PDBs. That flag only gates GPU-to-CPU fallback and is documented for multi-node NFS/CSI clusters with hostPath.enabled=false. Those clusters lose eviction protection on auto-upgrade even though pods can reschedule.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3e7c699. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PDBs minAvailable: 1 on single-replica workloads make the single node undrainable

2 participants