🤖 fix: return an empty namespaced LIST where no Coder backend serves the namespace

docs/how-to/troubleshooting.md

@@ -166,6 +166,8 @@ This section is about `ServiceUnavailable` errors from the aggregated API server
 - `all` mode: no eligible `CoderControlPlane` exists yet. A control plane is eligible when its operator access is enabled and ready, its status has an operator token reference and a URL, and its name does not contain a `.` character.
 - Standalone mode (`--app=aggregated-apiserver`): set all three flags `--coder-url`, `--coder-session-token`, and `--coder-namespace`.
 
+A `list` in one namespace that contains no `CoderControlPlane` at all does not return this error. It returns an empty list. See [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend).
+
 The logs show which provider configuration the server used.
 
 ## Aggregated reads return `multiple eligible CoderControlPlane ...`

docs/reference/aggregated-api-behavior.md

@@ -10,6 +10,7 @@ The aggregated API server serves `coderworkspaces`, `codertemplates` and `codert
 | [Delete preconditions](#delete-preconditions) | The server checks `uid` and `resourceVersion`. A mismatch returns `409` and does not change Coder. |
 | [Workspace `resourceVersion`](#workspace-resourceversion) | An opaque fingerprint. Compare it only for equality. Workspace activity alone can cause `409`. |
 | [Watch](#watch) | Shows only writes made through this server. No replay, no initial events. |
+| [Namespaces without a Coder backend](#namespaces-without-a-coder-backend) | A `list` in one namespace returns an empty list. Other requests return an error. Thus such a namespace can be deleted. |
 | [Server-side apply](#server-side-apply) | Create-on-update works. The server does not keep field ownership. |
 | [Server-side dry-run](#server-side-dry-run) | Not supported for writes to `coderworkspaces` and `codertemplates`. `kubectl diff` and `--dry-run=server` return `400` and do not change Coder. A promotion with `dryRun=All` is a read-only preview. Start and stop accept dry-run too. |
 | [Template versions](#template-versions) | Read-only: `get` and `list`, no watch. Reads never download template source. |
@@ -137,6 +138,28 @@ The server rejects these requests:
 | `resourceVersionMatch` set | Rejected |
 | `sendInitialEvents=false` without a matching option | `422 Invalid` (rejected upstream) |
 
+## Namespaces without a Coder backend
+
+A namespace has no Coder backend in these cases:
+
+- `all` mode: the namespace contains no `CoderControlPlane` at all.
+- Standalone mode: the namespace is not the one that `--coder-namespace` names.
+
+In such a namespace, the server answers requests as follows:
+
+| Request | Result |
+| --- | --- |
+| `list` of `coderworkspaces`, `codertemplates`, or `codertemplateversions` in that namespace | `200` with an empty list. The server does not call Coder. |
+| `get`, `create`, `update`, `delete`, and the subresources | An error: `503` in `all` mode, `400` in standalone mode |
+| `watch` | Works as described in [Watch](#watch). It shows no events, because no write can succeed in that namespace. |
+| `list` in all namespaces (`-A`) | Unchanged. In `all` mode, it returns `503` when no `CoderControlPlane` is eligible in any namespace. |
+
+A namespace that contains a `CoderControlPlane` that is not eligible is not in this group. For example, operator access is not ready, or the name contains a `.` character. For the eligibility rules, see [Aggregated reads return `ServiceUnavailable`](../how-to/troubleshooting.md#aggregated-reads-return-serviceunavailable). In such a namespace, every request returns `503`, `list` included.
+
+The namespace controller lists every resource type in a namespace before it removes the namespace. An error from a list keeps the namespace in `Terminating` forever, so the empty list lets the deletion finish. If the namespace still contains a `CoderControlPlane`, the namespace controller deletes it in the same pass. The lists return `503` until the control plane is gone, and then they return empty lists.
+
+A short outage of a control plane does not empty a list. For example, the operator sets `operatorAccessReady=false` after a Postgres error. During that time, a list in its namespace returns `503`, so a client that caches a list, such as an informer, does not see objects as deleted.
+
 ## Server-side apply
 
 `kubectl apply --server-side` can create a resource that does not exist yet. For a missing workspace or template, the update path creates the object instead (`forceAllowCreate=true`).

hack/e2e-workspace-lifecycle.sh

@@ -575,19 +575,23 @@ wait_until "template test e2e-ns-delete to report CoderUnavailable" tt_unavailab
 T0=$SECONDS
 k delete namespace "$NS" --wait=false >/dev/null || fail "cannot delete namespace $NS"
 gone_obj() { ! k -n "$NS" get "$1" "$2" -o name >/dev/null 2>"$WORK/obj.err" && grep -q NotFound "$WORK/obj.err"; }
-ns_released() { # the test and the control plane are gone, and no namespace content or finalizer remains
+NS_SEEN=""
+ns_gone() { # the namespace disappears (#209), and with it the test and the control plane
   tt_state e2e-ns-delete || true # logs the ControlPlaneGone release while the object still exists
-  gone_obj codertemplatetest e2e-ns-delete && gone_obj codercontrolplane coder || return 1
-  k get namespace "$NS" -o json >"$WORK/ns.json" 2>"$WORK/ns.err" || { grep -q NotFound "$WORK/ns.err"; return; }
-  jq -e '[.status.conditions[]? | select(.type == "NamespaceContentRemaining" or .type == "NamespaceFinalizersRemaining")] |
-    length == 2 and all(.status == "False")' "$WORK/ns.json" >/dev/null
+  if k get namespace "$NS" -o json >"$WORK/ns.json" 2>"$WORK/ns.err"; then
+    # Diagnostics from the first poll on, so a run shows NamespaceDeletionContentFailure (the aggregated
+    # LIST errors while the control plane still exists): the phase and every True condition, on change.
+    local seen
+    seen=$(jq -r '[.status.phase // "unknown"] + [.status.conditions[]? | select(.status == "True") |
+      "\(.type): \(.message)"] | join("; ")' "$WORK/ns.json") || seen="unparsable namespace JSON"
+    [[ $seen == "$NS_SEEN" ]] || log "namespace $NS still exists: $seen"
+    NS_SEEN=$seen
+    return 1
+  fi
+  grep -q NotFound "$WORK/ns.err" && gone_obj codertemplatetest e2e-ns-delete && gone_obj codercontrolplane coder
 }
-TIMEOUT=$NS_DELETE_TIMEOUT wait_until "test, control plane, and content of namespace $NS deleted (ControlPlaneGone release)" ns_released
+TIMEOUT=$NS_DELETE_TIMEOUT wait_until "test, control plane, and namespace $NS deleted (ControlPlaneGone release)" ns_gone
 NS_DELETE_SECONDS=$((SECONDS - T0))
-# Known issue #209: a namespaced aggregated LIST without an eligible control plane answers 503, so the namespace
-# stays Terminating. Once #209 is fixed, this check becomes "the namespace disappears".
-[[ ! -s $WORK/ns.json ]] || log "namespace $NS is $(jq -r '.status.phase' "$WORK/ns.json"), known issue #209: $(jq -r '[.status.conditions[]? |
-  select(.type == "NamespaceDeletionContentFailure" and .status == "True") | .message] | join("; ")' "$WORK/ns.json")"
-log "namespace $NS: test, control plane, and content deleted in ${NS_DELETE_SECONDS}s"
+log "namespace $NS deleted in ${NS_DELETE_SECONDS}s, after its test and control plane"
 CASES+=("case: $CURRENT = passed") && CURRENT="" && RESULT=PASS
 log "PASS: workspace lifecycle"

hack/e2e-workspace-lifecycle_test.sh

@@ -93,7 +93,9 @@ case "${pos[0]}:${pos[1]:-}" in
       tt_json Failed AgentStartError '[{"type":"WorkspaceDeleted","status":"True","reason":"Deleted"}]'
     else tt_json Succeeded Succeeded '[{"type":"Ready","status":"True","reason":"Succeeded"},{"type":"WorkspaceDeleted","status":"True","reason":"Deleted"}]'
     fi ;;
-  get:namespace) # content gone, but the namespace stays Terminating (the aggregated API LIST answers 503)
+  get:namespace) # Terminating on the first read, then gone; ns-stays-terminating: Terminating forever (#209)
+    n=$(($(cat "$S/ns-reads" 2>/dev/null || echo 0) + 1)) && echo "$n" >"$S/ns-reads"
+    [[ $SCENARIO == ns-stays-terminating || $n -lt 2 ]] || err NotFound 'namespaces "coder" not found'
     jq -n '{status: {phase: "Terminating", conditions: [{type: "NamespaceDeletionContentFailure", status: "True", message: "no eligible CoderControlPlane"},
       {type: "NamespaceContentRemaining", status: "False"}, {type: "NamespaceFinalizersRemaining", status: "False"}]}}' ;;
   delete:namespace) touch "$S/ns-deleted"; echo 'namespace "coder" deleted' ;;
@@ -381,9 +383,9 @@ check "psql only counts rows: operator and tester api_keys, the first pass test'
 check "owner patch names the tester; the namespace test waits 120 s for its agent" eval 'grep -qF "\"ownerUserID\":\"user-tester\"" "$S/calls.log" &&
   jq -e ".spec.parameters == [{name: \"startup_delay\", value: \"120\"}] and .spec.template == \"coder.e2e-agent\"" "$T/work/tt-e2e-ns-delete.json" >/dev/null &&
   jq -e ".spec.version.active and (.spec | has(\"parameters\") | not)" "$T/work/tt-e2e-pass-1.json" >/dev/null'
-check "namespace deleted only after CoderUnavailable; the release went through ControlPlaneGone" eval '[[ $(grep -n "e2e-ns-delete: Running CoderUnavailable" "$T/out" | cut -d: -f1) -lt $(grep -n "content deleted in" "$T/out" | cut -d: -f1) ]] &&
+check "namespace deleted only after CoderUnavailable; the release went through ControlPlaneGone; the namespace disappeared" eval '[[ $(grep -n "e2e-ns-delete: Running CoderUnavailable" "$T/out" | cut -d: -f1) -lt $(grep -n "namespace coder deleted in" "$T/out" | cut -d: -f1) ]] &&
   out_has "e2e-ns-delete: Failed ControlPlaneGone deleted=Unknown/ControlPlaneGone" && grep -q "replicas.:0" "$S/calls.log" &&
-  out_has "namespace coder is Terminating, known issue #209: no eligible CoderControlPlane"'
+  out_has "namespace coder still exists: Terminating; NamespaceDeletionContentFailure: no eligible CoderControlPlane" && [[ $(<"$S/ns-reads") == 2 ]]'
 check "tester: password user in the default organization; receipt records its username and id" eval 'jq -e ".login_type == \"password\" and .organization_ids == [\"org-1\"]" "$S/tester-body.json" >/dev/null &&
   grep -qx "tester=e2e-tester/user-tester" "$T/work/receipt.txt"'
 # shellcheck disable=SC2034 # pw is used by the eval'd check below
@@ -586,7 +588,8 @@ ws-rows-2|expected exactly one workspaces row named ktt-e2e-pass-1 (deleted rows
 coder-rolls|setting templateTests.ownerUserID rolled deploy/coder|$TESTS
 keys-after-fails|cannot count the api_keys rows of the tester after the tests|$PHASED
 coder-stays-up|timed out after 3s waiting for: deploy/coder without pods|$PHASED,kubectl create,kubectl patch
-ns-delete-stuck|timed out after 2s waiting for: test, control plane, and content of namespace coder deleted|$PHASED,kubectl create,kubectl patch,kubectl delete
+ns-delete-stuck|timed out after 2s waiting for: test, control plane, and namespace coder deleted|$PHASED,kubectl create,kubectl patch,kubectl delete
+ns-stays-terminating|timed out after 2s waiting for: test, control plane, and namespace coder deleted|$PHASED,kubectl create,kubectl patch,kubectl delete
 fail-ignored|template test e2e-agent-fail ended Succeeded Succeeded deleted=True/Deleted, want Failed AgentStartError|$TESTS,kubectl create
 badparam-created|template test e2e-bad-param: expected 0 workspaces rows named ktt-e2e-bad-param (deleted rows included), found 1|$TESTS,kubectl create,kubectl create
 restart-failed|template test e2e-restart failed: Failed AgentStartError|$TESTS,kubectl create,kubectl create,kubectl create,kubectl delete

internal/aggregated/coder/controlplane_provider.go

@@ -73,14 +73,21 @@ func (p *ControlPlaneClientProvider) ClientForNamespace(ctx context.Context, nam
 		return nil, fmt.Errorf("assertion failed: secret reader must not be nil")
 	}
 
-	eligible, err := p.findEligibleControlPlanes(ctx, namespace)
+	eligible, listed, err := p.findEligibleControlPlanes(ctx, namespace)
 	if err != nil {
 		return nil, err
 	}
 
 	switch len(eligible) {
 	case 0:
-		return nil, apierrors.NewServiceUnavailable(noEligibleControlPlaneMessage(namespace))
+		unavailable := apierrors.NewServiceUnavailable(noEligibleControlPlaneMessage(namespace))
+		// Only a namespace without any CoderControlPlane is "not served". A control plane that exists
+		// but is not eligible can be in a short outage (operator access not ready after a Postgres
+		// error), so a namespaced LIST must keep the 503 instead of reporting every object as gone.
+		if namespace == "" || listed > 0 {
+			return nil, unavailable
+		}
+		return nil, newNamespaceNotServedError(unavailable)
 	case 1:
 		// handled below
 	default:
@@ -189,7 +196,7 @@ func (p *ControlPlaneClientProvider) ClientForNamespace(ctx context.Context, nam
 
 // DefaultNamespace resolves the namespace for all-namespaces LIST requests.
 func (p *ControlPlaneClientProvider) DefaultNamespace(ctx context.Context) (string, error) {
-	eligible, err := p.findEligibleControlPlanes(ctx, "")
+	eligible, _, err := p.findEligibleControlPlanes(ctx, "")
 	if err != nil {
 		return "", err
 	}
@@ -217,7 +224,7 @@ func (p *ControlPlaneClientProvider) EligibleNamespaces(ctx context.Context) ([]
 		return nil, fmt.Errorf("assertion failed: context must not be nil")
 	}
 
-	eligible, err := p.findEligibleControlPlanes(ctx, "")
+	eligible, _, err := p.findEligibleControlPlanes(ctx, "")
 	if err != nil {
 		return nil, err
 	}
@@ -247,34 +254,36 @@ func (p *ControlPlaneClientProvider) EligibleNamespaces(ctx context.Context) ([]
 	return namespaces, nil
 }
 
+// findEligibleControlPlanes returns the eligible CoderControlPlanes in namespace (all namespaces when
+// empty) and the number of CoderControlPlanes listed there, eligible or not.
 func (p *ControlPlaneClientProvider) findEligibleControlPlanes(
 	ctx context.Context,
 	namespace string,
-) ([]coderv1alpha1.CoderControlPlane, error) {
+) (eligible []coderv1alpha1.CoderControlPlane, listed int, err error) {
 	if p == nil {
-		return nil, fmt.Errorf("assertion failed: control plane client provider must not be nil")
+		return nil, 0, fmt.Errorf("assertion failed: control plane client provider must not be nil")
 	}
 	if ctx == nil {
-		return nil, fmt.Errorf("assertion failed: context must not be nil")
+		return nil, 0, fmt.Errorf("assertion failed: context must not be nil")
 	}
 	if p.cpReader == nil {
-		return nil, fmt.Errorf("assertion failed: control plane reader must not be nil")
+		return nil, 0, fmt.Errorf("assertion failed: control plane reader must not be nil")
 	}
 
 	controlPlaneList := &coderv1alpha1.CoderControlPlaneList{}
 	listOptions := make([]client.ListOption, 0, 1)
 	if namespace != "" {
 		listOptions = append(listOptions, client.InNamespace(namespace))
 	}
-	if err := p.cpReader.List(ctx, controlPlaneList, listOptions...); err != nil {
+	if err = p.cpReader.List(ctx, controlPlaneList, listOptions...); err != nil {
 		if namespace == "" {
-			return nil, fmt.Errorf("list CoderControlPlane resources across all namespaces: %w", err)
+			return nil, 0, fmt.Errorf("list CoderControlPlane resources across all namespaces: %w", err)
 		}
 
-		return nil, fmt.Errorf("list CoderControlPlane resources in namespace %q: %w", namespace, err)
+		return nil, 0, fmt.Errorf("list CoderControlPlane resources in namespace %q: %w", namespace, err)
 	}
 
-	eligible := make([]coderv1alpha1.CoderControlPlane, 0, 1)
+	eligible = make([]coderv1alpha1.CoderControlPlane, 0, 1)
 	for i := range controlPlaneList.Items {
 		controlPlane := controlPlaneList.Items[i]
 		if strings.Contains(controlPlane.Name, ".") {
@@ -301,7 +310,7 @@ func (p *ControlPlaneClientProvider) findEligibleControlPlanes(
 		eligible = append(eligible, controlPlane)
 	}
 
-	return eligible, nil
+	return eligible, len(controlPlaneList.Items), nil
 }
 
 func noEligibleControlPlaneMessage(namespace string) string {