Adds FIFOPriorityQueue to Volcano

cmd/controller-manager/main.go

@@ -43,6 +43,7 @@ import (
 	_ "volcano.sh/volcano/pkg/controllers/jobflow"
 	_ "volcano.sh/volcano/pkg/controllers/jobtemplate"
 	_ "volcano.sh/volcano/pkg/controllers/podgroup"
+	_ "volcano.sh/volcano/pkg/controllers/podstartwatchdog"
 	_ "volcano.sh/volcano/pkg/controllers/queue"
 	_ "volcano.sh/volcano/pkg/controllers/sharding"
 	commonutil "volcano.sh/volcano/pkg/util"

farai/docs/changes/README.md

@@ -0,0 +1,21 @@
+# Changes from vanilla Volcano
+
+This fork (cut from upstream v1.15.0) is vanilla Volcano plus the
+changes documented here — one file per changed component, each with the
+motivation, the files touched, and how to use the feature:
+
+- [fifopriority.md](fifopriority.md) — in-tree scheduler plugin: FIFO-within-
+  PriorityClass ordering and preemption, driven by the submission-time stamp
+- [podstartwatchdog.md](podstartwatchdog.md) — controller in
+  `vc-controller-manager`: marks bound-but-unstarted pods Failed past the
+  `volcano.sh/pod-start-timeout` deadline (job label, copied onto pods); the
+  job's own `PodFailed → RestartJob` policy and `maxRetry` do the rest
+- [job-controller.md](job-controller.md) — eviction-triggered restarts don't
+  consume the retry budget; PodGroups stamped with the job's true submission
+  time (`volcano.sh/submitted-at`)
+- [helm-chart.md](helm-chart.md) — chart knobs for the podstartwatchdog
+
+Consumer context (the multi-tenant platform whose admission webhook defaults the
+preemptable annotation, eviction/restart policies, and the pod-start-timeout
+label onto tenant jobs, plus the design rationale for all of it) lives in the
+`flamingo_tenants` repo under `docs/agent/scheduler/`.

farai/docs/changes/fifopriority.md

@@ -0,0 +1,73 @@
+# fifopriority — FIFO-within-priority scheduler plugin
+
+## Motivation
+
+Stock Volcano orders and preempts by PriorityClass alone, so on a busy cluster a
+large job can be leapfrogged indefinitely by newer small jobs of the same class —
+starvation the `priority` plugin cannot express. fifopriority orders jobs by
+PriorityClass first and **submission time** second, and drives *both* allocation
+order and preempt-victim selection from that one comparator (a pending job may
+evict strictly-lower-class work, or equal-class work submitted in a strictly later
+second). The clock is the `volcano.sh/submitted-at` PodGroup label (epoch ms — the
+owning VCJob's immutable creationTimestamp, stamped by the job controller onto
+every (re)created PodGroup, so it survives RestartJob and cannot be forged), with
+PodGroup creationTimestamp as the fallback; comparison is at second granularity so
+same-second submissions never evict each other.
+
+Beyond ordering, the plugin reinstates the **queue-Open check at enqueue** (jobs
+submitted to Closed/Closing or missing queues do not enqueue and stay podless).
+Stock proportion only provides that check inside its enqueue-stage capacity vote,
+which deployments running open enqueue gates (`enableJobEnqueued: false`) must
+disable — losing the state check as collateral. State only; capacity is never
+gated at enqueue.
+
+## Changed component / files
+
+Scheduler (`vc-scheduler`):
+
+- `pkg/scheduler/plugins/fifopriority/fifopriority.go` — new plugin (comparator,
+  task/sub-job order, Preemptable + UnifiedEvictable victim rules, queue-Open
+  enqueue gate, starving test; stock `priority` parity everywhere else)
+- `pkg/scheduler/plugins/fifopriority/fifopriority_test.go` — unit tests
+- `pkg/scheduler/plugins/factory.go` — registration (config name `fifopriority`,
+  used in the scheduler config exactly where `priority` was)
+
+## Usage
+
+Name `fifopriority` in the scheduler configuration where `priority` would go
+(tier 1), with gang's and drf's victim vetoes off so it alone picks preempt
+victims:
+
+```yaml
+actions: "enqueue, allocate, reclaim, preempt, backfill"
+tiers:
+- plugins:
+  - name: fifopriority
+  - name: gang
+    enablePreemptable: false
+  - name: conformance
+- plugins:
+  - name: drf
+    enablePreemptable: false
+  - name: predicates
+  - name: proportion
+    enableJobEnqueued: false
+  - name: nodeorder
+  - name: binpack
+```
+
+`enableJobEnqueued: false` on proportion is part of the design, not an
+optimization: it disables proportion's enqueue-stage capacity vote so gates stay
+open. Do not rely on omitting it — that happens to behave the same only because a
+tier-1 Permit short-circuits later tiers' enqueue votes, and silently stops
+working if fifopriority moves out of tier 1.
+
+Nothing else to configure: PodGroups created by the job controller carry the
+`volcano.sh/submitted-at` stamp automatically (see job-controller.md); podgroups
+without one fall back to their own creation time. Two operational notes: eviction
+candidates must carry `volcano.sh/preemptable: "true"` on their pods (upstream
+drops unmarked candidates before any plugin runs), and a scheduler image WITHOUT
+this plugin does not crash on this config — an unknown plugin name is only an
+error log (`Failed to get plugin`) and tier 1 silently degrades to
+conformance-only preemption — so deployments should assert the running image and
+scan fresh scheduler logs after applying the config.

farai/docs/changes/helm-chart.md

@@ -0,0 +1,30 @@
+# Helm chart — podstartwatchdog knob
+
+## Motivation
+
+The chart has no generic extra-args passthrough for the controllers deployment;
+per-flag `custom.*` values are its native pattern (qps, worker threads, …). The
+podstartwatchdog's poll-cadence flag gets the same treatment so deployments can
+tune it through values instead of patching the deployment. Unset (the default)
+omits the arg and the compiled default (1m) applies. There is deliberately no
+timeout knob: deadlines come exclusively from the per-job
+`volcano.sh/pod-start-timeout` label, copied onto the job's pods (see
+podstartwatchdog.md).
+
+## Changed component / files
+
+Helm chart (`installer/helm/chart/volcano/`):
+
+- `values.yaml` — `custom.controller_pod_start_interval` (default `~`)
+- `templates/controllers.yaml` — conditional `--pod-start-interval` arg on the
+  controllers container
+
+## Usage
+
+Install (or upgrade) from the in-tree chart with the knob in values:
+
+```sh
+helm upgrade --install volcano installer/helm/chart/volcano \
+  --namespace volcano-system --create-namespace \
+  --set custom.controller_pod_start_interval=30s
+```

farai/docs/changes/job-controller.md

@@ -0,0 +1,89 @@
+# Job controller — retry accounting + PodGroup submission stamp
+
+Two changes to `vc-controller-manager`'s job controller.
+
+## 1. Evictions don't consume the retry budget
+
+### Motivation
+
+`RetryCount` counts every trip through a restart, whatever caused it, and a job
+whose count reaches `maxRetry` is terminally `Failed`. With a
+`PodEvicted → RestartJob` policy (whole-gang teardown on eviction), scheduler
+preemption/reclaim — which works by *deleting* pods, surfacing to the controller
+as `PodEvicted` — would bill every disruption against the budget, letting repeated
+preemption permanently kill an innocent job. Eviction means disrupted, not failed:
+eviction-triggered restarts are now free, while `PodFailed`-triggered and
+directly-issued restarts count exactly as before. (Requested upstream in
+volcano-sh/volcano#2560; a fix was invited but never landed.)
+
+### Changed files
+
+- `pkg/controllers/job/state/factory.go` — `state.Action` gains the triggering
+  `Event`; new `Action.CountsTowardRetry()` (false only for `PodEvicted`)
+- `pkg/controllers/job/job_controller_util.go` — `GetStateAction` forwards the
+  event from the delayed action
+- `pkg/controllers/job/state/running.go`, `state/pending.go` — the four
+  `RetryCount++` sites on `RestartJob`/`RestartTask|Pod|Partition` gated on
+  `CountsTowardRetry()` (the `ResumeJob` increments in aborting/aborted states are
+  untouched — resume is never eviction-triggered)
+- `pkg/controllers/job/state/retry_test.go` — unit tests (evicted vs failed vs
+  direct, both states, both action kinds)
+
+### Usage
+
+Nothing to configure — evictions never consume the budget. To get whole-gang
+teardown-and-requeue on eviction at no retry cost, give the job an eviction
+policy:
+
+```yaml
+spec:
+  maxRetry: 5              # bounds FAILURES only (CRD schema default: 3)
+  policies:
+  - {event: PodFailed,  action: RestartJob}
+  - {event: PodEvicted, action: RestartJob}
+```
+
+`PodFailed`-triggered and directly-issued restarts (`vcctl`, Commands) count
+exactly as before.
+
+## 2. PodGroups carry the job's true submission time
+
+### Motivation
+
+FIFO scheduling needs a restart-proof submission clock, but the scheduler only
+sees PodGroups, and a PodGroup's `creationTimestamp` resets on every `RestartJob`
+recreation — a restarted job would silently go to the back of its class's line.
+The owning VCJob's `metadata.creationTimestamp` is the true submission time:
+API-server-managed and immutable, so it also cannot be forged (previously an
+out-of-tree admission webhook stamped a label, which a user could edit
+post-create and launder onto the next recreated PodGroup). The fifopriority
+plugin orders by this stamp.
+
+### Changed files
+
+- `pkg/util/util.go` — `SubmittedAtPodGroupLabel = "volcano.sh/submitted-at"`
+  (shared by controller and scheduler plugin)
+- `pkg/controllers/job/job_controller_actions.go` — `podGroupLabels` stamps the
+  epoch-ms creation time onto every created PodGroup (force-set after the job
+  label copy, never inherited)
+- `pkg/controllers/job/podgroup_stamp_test.go` — unit tests (forgery override,
+  second-aligned epoch-ms format)
+
+### Usage
+
+Automatic — every PodGroup the job controller (re)creates carries the label.
+Inspect with:
+
+```sh
+kubectl get podgroup <name> -o jsonpath='{.metadata.labels.volcano\.sh/submitted-at}'
+```
+
+The value is the owning VCJob's `metadata.creationTimestamp` as epoch
+milliseconds; it cannot be set or forged from the job side (job labels are copied
+to the PodGroup, but this key is force-set after the copy). The stamp is written
+only at PodGroup creation — the controller does not re-reconcile it afterwards.
+Editing it directly on the PodGroup requires PodGroup write access, which tenants
+must not have: RBAC is the boundary there, not this controller. Two consequences:
+PodGroups that predate this feature stay unstamped (they order by their own
+creation time — the pre-feature status quo) until their next recreation stamps
+them, and the scheduler plugin's fallback covers them meanwhile.

farai/docs/changes/podstartwatchdog.md

@@ -0,0 +1,129 @@
+# podstartwatchdog — bound pods must start
+
+## Motivation
+
+A placed pod that cannot start — unpullable image, unmountable volume,
+container-create config error — never *fails* on its own: it sits Pending on its
+node, charging its full resource request against the cluster forever, and no
+`PodFailed` lifecycle policy ever fires. The stock `PodPending`-timeout policy
+cannot be used instead because its clock starts at pod creation — on a
+pass-through-enqueue cluster pods legitimately queue for hours, so it would kill
+innocent queued jobs. The watchdog anchors on **binding**: a pod counts as stuck
+only when bound (`spec.nodeName` set — it already holds capacity) and still
+Pending, whatever the cause. Stuck past its deadline → the pod is marked
+**Failed** (`status.phase`, reason `PodStartTimeout`) with a Warning Event on the
+pod.
+
+That is the watchdog's whole job. Everything downstream belongs to the owning
+job's own lifecycle policies, exactly as if a container had died: `PodFailed →
+RestartJob` (the platform default) tears down and requeues the gang, **consuming
+one retry per lap** — so transient causes (a registry blip, a slow cold pull that
+finishes next time) self-heal on the next attempt, while persistent causes burn
+`maxRetry` laps and park the job in terminal `Failed`: bounded, inspectable
+(`kubectl describe vcjob` shows the restart history and final state), and
+resubmittable.
+
+**Pod-scoped and stateless by design.** The watchdog never resolves, verifies, or
+trusts a pod→job mapping — no `volcano.sh/job-name` lookup, no ownerReference
+check, no job phase gate. Attribution of a failed pod to its job happens where it
+always has: in the job controller's own event handlers, with their ownership and
+version safeguards. There is no targeting decision to spoof — a hand-made pod
+carrying the label only ever condemns itself. Statelessness comes from two
+anchors that live on the pod:
+
+- **The clock**: the API server's binding subresource sets the
+  `PodScheduled=True` condition in the same write that sets `spec.nodeName`, so
+  its `lastTransitionTime` is the bind moment; pod phase never regresses, so
+  bound-and-Pending now means continuously unstarted since then. (A bound pod
+  without the condition can only be one created with `nodeName` preset — it never
+  queued, so its creationTimestamp is its bind time.) Controller restarts forget
+  nothing.
+- **The deadline**: the `volcano.sh/pod-start-timeout` label **on the pod**, a Go
+  duration — the only deadline source. The job controller copies it from the
+  owning VCJob's label onto every pod it creates (the platform's admission
+  webhook defaults it onto every job), so it is stamped by a trusted component;
+  tenants cannot edit controller-created pods. A pod without the label, or with
+  an unparseable or non-positive value (`"0"` is an explicit opt-out), is not
+  policed. Editing the job's label reaches the next pod incarnation (e.g. the
+  next restart lap).
+
+Marking a pod Failed from outside the kubelet is a one-way door the stack honors
+end to end: the API server does not restrict phase transitions in status updates;
+the kubelet begins teardown when it observes an API-terminal pod
+(`pod_workers.go`: "Pod is in a terminal phase (success/failed), begin teardown" —
+the same contract PodGC relies on); and the Volcano job controller classifies the
+transition as an ordinary `PodFailed` event. A racing kubelet status sync can at
+worst delay the mark by one interval — the watchdog re-asserts.
+
+**Cause-agnostic cuts both ways**: time spent in *legitimate* initialization — a
+cold pull of a huge image, a long-running init container (the pod stays Pending
+until init containers finish) — counts against the deadline too, because from the
+outside it is indistinguishable from an init that will never finish. The retry
+laps soften this (a slow pull that completes within a later lap survives), but
+jobs whose normal startup can approach the deadline should still set a larger
+label value, and the deployment default must be sized against worst-case
+cold-start time.
+
+## Changed component / files
+
+Controller manager (`vc-controller-manager`):
+
+- `pkg/controllers/podstartwatchdog/podstartwatchdog.go` — the whole controller:
+  informer plumbing, status-subresource Failed patch, and pod event, plus the
+  pure decision logic (bound+Pending predicate, bind-time clock anchor,
+  pod-label deadline) at the bottom of the file; flag `--pod-start-interval`
+  (poll cadence, default 1m); registered as `podstartwatchdog-controller`,
+  enabled by default
+- `pkg/controllers/podstartwatchdog/podstartwatchdog_test.go` — unit tests for
+  the decision logic
+- `pkg/controllers/job/job_controller_util.go` — the job controller copies the
+  `volcano.sh/pod-start-timeout` label from the job onto every pod it creates
+- `pkg/util/util.go` — `PodStartTimeoutLabel` (`volcano.sh/pod-start-timeout`)
+- `cmd/controller-manager/main.go` — package import for registration
+- `installer/helm/chart/volcano/templates/controllers.yaml`,
+  `installer/volcano-development.yaml` — controllers RBAC gains `patch` on
+  `pods/status` (the Failed mark)
+
+## Usage
+
+Runs by default in `vc-controller-manager` (disable with
+`--controllers=*,-podstartwatchdog-controller`). The deadline is set per job, via
+a label on the VCJob (a Go duration; `"0"` is an explicit opt-out), and rides
+onto the pods automatically:
+
+```yaml
+metadata:
+  labels:
+    volcano.sh/pod-start-timeout: "2h"
+```
+
+A deployment ensures coverage by defaulting the label at admission (the platform
+webhook injects `30m` onto jobs that don't set one). For the full
+restart-then-fail semantics, jobs need the restart policy and a retry budget
+(both platform webhook defaults):
+
+```yaml
+spec:
+  maxRetry: 5
+  policies:
+  - {event: PodFailed,  action: RestartJob}
+  - {event: PodEvicted, action: RestartJob}
+```
+
+The only chart knob is the poll cadence:
+
+```yaml
+custom:
+  controller_pod_start_interval: "30s"  # default 1m
+```
+
+On trigger the pod gets a Warning Event (reason `PodStartTimeout`) and is marked
+Failed; the job's `PodFailed → RestartJob` policy restarts the gang at the cost
+of one retry, and `maxRetry` exhaustion fails the job terminally. Jobs with
+legitimately slow startup (large cold image pulls, long init containers) should
+carry a label larger than their worst-case initialization time — the watchdog
+cannot tell slow from stuck, though a slow start that completes within a later
+retry lap survives on its own. Pair with `restartPolicy: Never` on task templates
+so mid-run container failures route through the same `PodFailed → RestartJob`
+path, and do not use `PodPending` lifecycle policies for the same purpose — their
+clock starts at pod creation, not binding, so they fire on healthy queued jobs.