From b015cf27b63c8b83dbf0304e89d8dbc666e64174 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Sat, 3 Oct 2026 17:36:52 +0000 Subject: [PATCH 1/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20update=20the=20Termi?= =?UTF-8?q?nating=20namespace=20step=20after=20the=20#209=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The troubleshooting step still said that a namespace without an eligible control plane never finishes deleting. #214 fixed that for a namespace without any control plane. The step now links the reference section and names the two cases that still answer the list with 503: a control plane that is not eligible, until the namespace controller deletes it, and a standalone server without Coder credentials (#215). Refs #209 Refs #215 Signed-off-by: Thomas Kosiewski --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ Change-Id: I6392df4a7a310671ef5b0bb06970e09ba16de436 --- docs/how-to/troubleshooting.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index 877c4572..32fd77a9 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -280,10 +280,15 @@ CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwis ## A namespace stays `Terminating` 1. Tests in the namespace wait for their cleanup. List them with `kubectl get codertemplatetests -n `, then see [A `CoderTemplateTest` does not finish deleting](#a-codertemplatetest-does-not-finish-deleting). -2. With the aggregated API server installed, a namespace without an eligible `CoderControlPlane` never finishes deleting ([#209](https://github.com/coder/coder-k8s/issues/209)). Its condition `NamespaceDeletionContentFailure` is `True` with the message `no eligible CoderControlPlane instances found in namespace ""`. The aggregated API answers the namespace controller's LIST with `503`. Read the conditions: +2. With the aggregated API server installed, the namespace controller lists `coderworkspaces`, `codertemplates`, and `codertemplateversions` before it removes the namespace. A namespace without any `CoderControlPlane` gets empty lists and finishes deleting. See [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend). Two cases still answer these lists with `503`, and the namespace waits: + + - **A control plane that is not eligible.** The namespace contains a `CoderControlPlane`, but its operator access is not ready, for example. The namespace controller deletes the control plane in the same pass. The lists return `503` until the control plane is gone, and then the deletion finishes. + - **A standalone server without Coder credentials** ([#215](https://github.com/coder/coder-k8s/issues/215)). A server in standalone mode (`--app=aggregated-apiserver`) without its Coder URL or session token answers `503` in every namespace. Set the flags, as described in [Aggregated reads return `ServiceUnavailable`](#aggregated-reads-return-serviceunavailable). + + Read the namespace conditions: ```bash kubectl get namespace -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}' ``` - If `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` are `False`, nothing is left in the namespace, and only #209 holds it. + In both cases, `NamespaceDeletionContentFailure` is `True`, and its message contains the `503` error of the failed list. If `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` are `False`, only these lists hold the namespace. From 9adbb8ac9ec4e4135fee920fc9f00d7cd8808d6a Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Sat, 3 Oct 2026 17:49:10 +0000 Subject: [PATCH 2/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20name=20the=20namespa?= =?UTF-8?q?ce=20cases=20by=20app=20mode=20and=20token=20state?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found four gaps in the rewritten step. The namespace controller lists only coderworkspaces and codertemplates, because codertemplateversions has no delete verb. The empty list applies to namespaces without a Coder backend in each app mode, as the reference defines them, and the pinned standalone namespace returns what Coder holds. A control plane whose operator token Secret lacks the key or holds an empty value also returns 503. A standalone server answers 503 only without both its URL and token, and does not start when only one is missing. Refs #209 Refs #215 Signed-off-by: Thomas Kosiewski --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ Change-Id: Iedda13df230ae69d834f9db07216015f9fb37b26 --- docs/how-to/troubleshooting.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index 32fd77a9..b80db624 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -280,10 +280,12 @@ CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwis ## A namespace stays `Terminating` 1. Tests in the namespace wait for their cleanup. List them with `kubectl get codertemplatetests -n `, then see [A `CoderTemplateTest` does not finish deleting](#a-codertemplatetest-does-not-finish-deleting). -2. With the aggregated API server installed, the namespace controller lists `coderworkspaces`, `codertemplates`, and `codertemplateversions` before it removes the namespace. A namespace without any `CoderControlPlane` gets empty lists and finishes deleting. See [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend). Two cases still answer these lists with `503`, and the namespace waits: +2. With the aggregated API server installed, the namespace controller lists `coderworkspaces` and `codertemplates` before it removes the namespace. A namespace without a Coder backend gets empty lists and finishes deleting. [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend) defines these namespaces for `all` mode and standalone mode. These cases still fail the lists, and the namespace waits: - - **A control plane that is not eligible.** The namespace contains a `CoderControlPlane`, but its operator access is not ready, for example. The namespace controller deletes the control plane in the same pass. The lists return `503` until the control plane is gone, and then the deletion finishes. - - **A standalone server without Coder credentials** ([#215](https://github.com/coder/coder-k8s/issues/215)). A server in standalone mode (`--app=aggregated-apiserver`) without its Coder URL or session token answers `503` in every namespace. Set the flags, as described in [Aggregated reads return `ServiceUnavailable`](#aggregated-reads-return-serviceunavailable). + - **A control plane that the server cannot use** (`all` mode). The namespace contains a `CoderControlPlane` that is not eligible, or whose operator token Secret lacks the key or holds an empty value. The lists return `503`. The namespace controller deletes the control plane in the same pass, and then the deletion finishes. + - **A standalone server without Coder credentials** ([#215](https://github.com/coder/coder-k8s/issues/215)). A server in standalone mode (`--app=aggregated-apiserver`) without both its Coder URL and its session token answers `503` in every namespace. If only one of them is missing, the server does not start. Set the flags, as described in [Aggregated reads return `ServiceUnavailable`](#aggregated-reads-return-serviceunavailable). + + In standalone mode, the namespace that `--coder-namespace` names has a Coder backend. Its lists return what Coder holds, as content of that namespace. Read the namespace conditions: From cfc8e162ac52fbd046f6c4347cae9f7a24cb8b01 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Sat, 3 Oct 2026 17:58:04 +0000 Subject: [PATCH 3/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20simplify=20the=20Ter?= =?UTF-8?q?minating=20namespace=20step?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each review round added cases to step 2, and each case drew new findings. The step now tells the reader to read the NamespaceDeletionContentFailure message, links the reference and the ServiceUnavailable entry for what each error means, and points to the control plane's finalizers and the controller logs. It no longer lists cases, and it drops the standalone --coder-namespace note. Refs #209 Refs #215 Signed-off-by: Thomas Kosiewski --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ Change-Id: I6ec365ecd630d27daa03923e36f5963ab739da6f --- docs/how-to/troubleshooting.md | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index b80db624..40a82e09 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -280,17 +280,9 @@ CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwis ## A namespace stays `Terminating` 1. Tests in the namespace wait for their cleanup. List them with `kubectl get codertemplatetests -n `, then see [A `CoderTemplateTest` does not finish deleting](#a-codertemplatetest-does-not-finish-deleting). -2. With the aggregated API server installed, the namespace controller lists `coderworkspaces` and `codertemplates` before it removes the namespace. A namespace without a Coder backend gets empty lists and finishes deleting. [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend) defines these namespaces for `all` mode and standalone mode. These cases still fail the lists, and the namespace waits: - - - **A control plane that the server cannot use** (`all` mode). The namespace contains a `CoderControlPlane` that is not eligible, or whose operator token Secret lacks the key or holds an empty value. The lists return `503`. The namespace controller deletes the control plane in the same pass, and then the deletion finishes. - - **A standalone server without Coder credentials** ([#215](https://github.com/coder/coder-k8s/issues/215)). A server in standalone mode (`--app=aggregated-apiserver`) without both its Coder URL and its session token answers `503` in every namespace. If only one of them is missing, the server does not start. Set the flags, as described in [Aggregated reads return `ServiceUnavailable`](#aggregated-reads-return-serviceunavailable). - - In standalone mode, the namespace that `--coder-namespace` names has a Coder backend. Its lists return what Coder holds, as content of that namespace. - - Read the namespace conditions: +2. If the namespace condition `NamespaceDeletionContentFailure` is `True`, read its message. It contains the error that the aggregated API returned when the namespace controller listed its resources, for example a `503` or a `400`. For what each error means, see [Namespaces without a Coder backend](../reference/aggregated-api-behavior.md#namespaces-without-a-coder-backend) and [Aggregated reads return `ServiceUnavailable`](#aggregated-reads-return-serviceunavailable). If a `CoderControlPlane` is still in the namespace, check its finalizers, for example `coder.com/workspace-rbac-cleanup`, and the controller logs. ```bash kubectl get namespace -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}' + kubectl get codercontrolplanes -n -o jsonpath='{range .items[*]}{.metadata.name}: {.metadata.finalizers}{"\n"}{end}' ``` - - In both cases, `NamespaceDeletionContentFailure` is `True`, and its message contains the `503` error of the failed list. If `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` are `False`, only these lists hold the namespace.