From ccc3b4a41fdf63f99256918e96339d0a5834496f Mon Sep 17 00:00:00 2001
From: Thomas Kosiewski <tk@coder.com>
Date: Sat, 3 Oct 2026 15:31:03 +0000
Subject: [PATCH 1/4] =?UTF-8?q?=F0=9F=A4=96=20docs:=20complete=20the=20Cod?=
 =?UTF-8?q?erTemplateTest=20how-to?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Expand the CoderTemplateTest how-to with a "Read the result" section
that lists every status.reason the controller sets (waiting, failure,
and WorkspaceDeleted condition reasons) with what to do, a ResourceQuota
example to cap test count, and a nightly CronJob with the RBAC it needs.
The existing notes on the tester, retain, TTL, keys, builds, and TLS
stay as they were.

Add troubleshooting entries for a test that stays Pending, a test that
does not finish deleting (finalizer), and a namespace stuck Terminating
because of the aggregated API LIST bug (#209). Add a sample manifest
(validated with a server-side dry run against the CRD in envtest) and
list CoderTemplateTest among the operator kinds in the README.

Refs #152

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

Change-Id: I0af2782b45a458869b2a1840611682a600cf06a3
---
 .cspell.json                                  |   1 +
 README.md                                     |   2 +-
 .../coder_v1alpha1_codertemplatetest.yaml     |  26 ++++
 docs/how-to/deploy-controller.md              |   5 +-
 docs/how-to/test-templates.md                 | 147 ++++++++++++++++++
 docs/how-to/troubleshooting.md                |  50 ++++++
 6 files changed, 229 insertions(+), 2 deletions(-)
 create mode 100644 config/samples/coder_v1alpha1_codertemplatetest.yaml

diff --git a/.cspell.json b/.cspell.json
index 993f3b694..e1c202ec4 100644
--- a/.cspell.json
+++ b/.cspell.json
@@ -19,6 +19,7 @@
     "rolebinding",
     "subresource",
     "subresources",
+    "finalizers",
     "configmap",
     "healthcheck",
     "ipairs",
diff --git a/README.md b/README.md
index 9da1622bd..067428cbe 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ Use native Kubernetes APIs to run and manage [Coder](https://coder.com).
 
 | Component | Manages | API group |
 | --- | --- | --- |
-| Operator | `CoderControlPlane`, `CoderProvisioner`, `CoderWorkspaceProxy` (custom resources) | `coder.com/v1alpha1` |
+| Operator | `CoderControlPlane`, `CoderProvisioner`, `CoderTemplateTest`, `CoderWorkspaceProxy` (custom resources) | `coder.com/v1alpha1` |
 | Aggregated API server | `CoderWorkspace`, `CoderTemplate`, `CoderTemplateVersion` (served from a live Coder instance) | `aggregation.coder.com/v1alpha1` |
 | MCP server | Tools that inspect and operate these resources over HTTP | — |
 
diff --git a/config/samples/coder_v1alpha1_codertemplatetest.yaml b/config/samples/coder_v1alpha1_codertemplatetest.yaml
new file mode 100644
index 000000000..b23378e9b
--- /dev/null
+++ b/config/samples/coder_v1alpha1_codertemplatetest.yaml
@@ -0,0 +1,26 @@
+# Tests one Coder template version: the controller creates a workspace from
+# it, waits until every agent is ready, records the result, and deletes the
+# workspace. See docs/how-to/test-templates.md.
+#
+# Prerequisites:
+# - CoderControlPlane codercontrolplane-sample with status.operatorAccessReady
+#   true and spec.templateTests.ownerUserID set to the tester's Coder user ID.
+# - The template below exists in Coder.
+#
+# The spec is immutable. To test again, delete this object or use a new name.
+apiVersion: coder.com/v1alpha1
+kind: CoderTemplateTest
+metadata:
+  name: codertemplatetest-sample
+  namespace: coder
+spec:
+  controlPlaneRef:
+    name: codercontrolplane-sample
+  # <organization>.<template>
+  template: "coder.docker"
+  version:
+    # Exactly one of active, name, or id.
+    active: true
+  timeoutSeconds: 900
+  # Optional: delete the test one hour after it finished.
+  # ttlSecondsAfterFinished: 3600
diff --git a/docs/how-to/deploy-controller.md b/docs/how-to/deploy-controller.md
index 1dd2ca449..aa3cc78b7 100644
--- a/docs/how-to/deploy-controller.md
+++ b/docs/how-to/deploy-controller.md
@@ -1,6 +1,6 @@
 # Deploy the controller
 
-Run `coder-k8s` as an operator only (`--app=controller`). The operator reconciles `CoderControlPlane`, `CoderProvisioner`, and `CoderWorkspaceProxy` resources.
+Run `coder-k8s` as an operator only (`--app=controller`). The operator reconciles `CoderControlPlane`, `CoderProvisioner`, `CoderWorkspaceProxy`, and `CoderTemplateTest` resources.
 
 Run the commands from a clone of this repository.
 
@@ -24,6 +24,9 @@ kubectl -n coder-system patch deployment/coder-k8s --type=json \
 !!! tip "Pin the image"
     The manifest uses `ghcr.io/coder/coder-k8s:latest`. To pin a version, change the tag before you apply the manifest.
 
+!!! warning "Upgrades: apply the CRDs and RBAC first"
+    Apply `config/crd/bases/` and `config/rbac/` from the new version before or together with the new image. The operator watches every kind it reconciles. If a kind's CRD is missing, for example `CoderTemplateTest` after an upgrade from an earlier version, the manager cannot start.
+
 ## 3. Verify
 
 ```bash
diff --git a/docs/how-to/test-templates.md b/docs/how-to/test-templates.md
index 12e7eda37..b0e394c93 100644
--- a/docs/how-to/test-templates.md
+++ b/docs/how-to/test-templates.md
@@ -61,6 +61,153 @@ The spec is immutable. To test again, create a new object.
 
 The operator's own RBAC comes with the install. People who run tests need `create`, `get`, `list`, `watch`, and `delete` on `codertemplatetests.coder.com` in their namespace. They never need access to Coder or to the operator token.
 
+## 4. Read the result
+
+`status.phase` is `Pending` until the controller may have sent a create request, then `Running`, and finally `Succeeded` or `Failed`. `status.reason` and `status.message` say what the controller waits for or why the test failed. `status.templateVersionName` shows which version ran.
+
+### Waiting reasons
+
+While the test waits, the condition `Reconciling` is `True` with the same reason. Every wait ends at `spec.timeoutSeconds`: the test then fails as `DeadlineExceeded`, and the message names the last wait.
+
+| Reason | Meaning | What to do |
+| --- | --- | --- |
+| `Initializing` | The controller started the test. | Nothing. |
+| `ControlPlaneNotReady` | The `CoderControlPlane` does not exist, is being deleted, has no usable `status.url`, or its operator token Secret has no token yet. | Check `spec.controlPlaneRef` and the control plane status. |
+| `OperatorAccessNotReady` | `status.operatorAccessReady` of the control plane is not `true`. | See [Deploy the controller](deploy-controller.md). |
+| `OwnerNotConfigured` | The control plane has no `spec.templateTests.ownerUserID`. | Set it, as in step 1. |
+| `OwnerNotEligible` | The tester does not exist, is not `active`, has a person's login type, has a role that is not allowed, or is not a member of the template's organization. | Fix the tester as described in step 1. The message names the problem. |
+| `TemplateNotFound` | The organization or the template in `spec.template` does not exist. | Check `spec.template` (`<organization>.<template>`). |
+| `TemplateVersionNotFound` | The version in `spec.version` does not exist. | Check `spec.version`, or push the version. |
+| `TemplateVersionImporting` | The version is still importing. | Wait. |
+| `CreatingWorkspace`, `CreateRetrying` | The controller sends the create request, or sends it again after a request that had no effect. | Wait. |
+| `ConfirmingCreate` | The answer to the create request was lost. The controller reads Coder until the workspace appears. | Wait. |
+| `WaitingForBuild`, `WaitingForAgents` | The start build runs, or an agent is not yet `connected` and `ready`. | Wait. The message names the agent. |
+| `AgentsReady`, `DeletingWorkspace` | Every agent was ready. The controller deletes the workspace. | Wait. |
+| `CoderUnavailable` | A Coder request failed. After HTTP 429, the controller retries with backoff. | Check Coder and its logs. |
+| `CoderAnswerMismatch` | Coder answered about another object than the one asked for. The controller logs it and retries. | Check proxies between the controller and Coder. |
+
+### Failure reasons
+
+A failed test has `Ready=False` and `Stalled=True` with the same reason. It does not run again: create a new test.
+
+| Reason | Meaning | What to do |
+| --- | --- | --- |
+| `DeadlineExceeded` | The test did not finish within `spec.timeoutSeconds`. | Fix the last wait in the message, or raise `spec.timeoutSeconds`. |
+| `TemplateDeprecated` | The template is deprecated and accepts no new workspaces. | Test another template. |
+| `TemplateVersionArchived`, `TemplateVersionImportFailed` | The version is archived, or its import did not succeed. | Push a working version. |
+| `TemplateVersionMismatch` | The version in `spec.version.id` belongs to another template. | Fix `spec.version` or `spec.template`. |
+| `CreateRejected` | Coder rejected the create request. The message holds Coder's answer. No workspace exists. | Fix the cause, for example `spec.parameters`, and create a new test. |
+| `CreateOutcomeUnknown` | No workspace from this test appeared after an uncertain create request. | Check Coder, then create a new test. |
+| `WorkspaceNameConflict` | A workspace with the test's name exists, but nothing proves that this test created it. | Check the workspace in Coder. See the escape hatch below. |
+| `BuildFailed`, `BuildCanceled` | The start build failed, or someone else canceled it. | Read the build logs in Coder. |
+| `NoAgents` | The workspace has no agents, so nothing proves that it works. | Add an agent to the template. |
+| `AgentConnectionTimeout`, `AgentStartError`, `AgentStartTimeout`, `AgentStopped` | An agent did not connect in time, its startup script failed or timed out, or it stopped. | Read the agent logs in Coder. The message names the agent. |
+| `WorkspaceChangedExternally`, `WorkspaceDeletedExternally` | Someone else started a build of the workspace, or deleted it. | Leave test workspaces alone, and create a new test. |
+| `DeleteBuildFailed` | Every agent was ready, but the delete build failed. The controller retries the delete. | Read the delete build logs in Coder. |
+| `ControlPlaneGone` | The `CoderControlPlane` was deleted while the workspace could exist. | Look for the workspace in Coder and delete it there. |
+
+### `WorkspaceDeleted` reasons
+
+The condition `WorkspaceDeleted` tracks the workspace after the test ends or while the test is being deleted. The controller keeps its finalizer until the condition is `True`, or the reason is `Retained` or `ControlPlaneGone`.
+
+| Status | Reasons | Meaning |
+| --- | --- | --- |
+| `True` | `Deleted`, `DeletedExternally`, `NotCreated` | No workspace of this test exists. `ttlSecondsAfterFinished` applies. |
+| `False` | `CleanupPending`, `Deleting`, `DeleteRetrying` | The controller deletes the workspace. |
+| `False` | `ControlPlaneUnavailable`, `CoderAnswerMismatch` | Coder is not usable, or answered about another object. The controller retries. |
+| `False` | `Retained` | `retain` is set and allowed, so the workspace stays in Coder. |
+| `Unknown` | `CreateOutcomeUnknown` | The controller reads Coder to learn whether a workspace exists. |
+| `Unknown` | `OwnershipUnknown`, `ControlPlaneGone` | The workspace can exist, and the controller cannot delete it. See the escape hatch below. |
+
+## Limit the cost
+
+Each test is a real workspace: it uses compute and provisioner time until the controller deletes it. To cap the number of tests in a namespace, use a `ResourceQuota`. The quota counts finished tests too, until they expire or someone deletes them:
+
+```yaml
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+  name: template-tests
+  namespace: coder
+spec:
+  hard:
+    count/codertemplatetests.coder.com: "10"
+```
+
+## Run nightly checks
+
+A `CronJob` can test the active version every night. It needs only `create` on `codertemplatetests`. `ttlSecondsAfterFinished` removes old results:
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: nightly-template-test
+  namespace: coder
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: nightly-template-test
+  namespace: coder
+rules:
+  - apiGroups: ["coder.com"]
+    resources: ["codertemplatetests"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: nightly-template-test
+  namespace: coder
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: nightly-template-test
+subjects:
+  - kind: ServiceAccount
+    name: nightly-template-test
+    namespace: coder
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: nightly-template-test
+  namespace: coder
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          serviceAccountName: nightly-template-test
+          restartPolicy: Never
+          containers:
+            - name: create-test
+              image: <an image with sh and kubectl>
+              command:
+                - sh
+                - -ec
+                - |
+                  kubectl create -f - <<'TEST'
+                  apiVersion: coder.com/v1alpha1
+                  kind: CoderTemplateTest
+                  metadata:
+                    generateName: docker-nightly-
+                    namespace: coder
+                  spec:
+                    controlPlaneRef:
+                      name: coder
+                    template: default.docker
+                    version:
+                      active: true
+                    ttlSecondsAfterFinished: 604800
+                  TEST
+```
+
+To find failed tests, run `kubectl -n coder get codertemplatetests`, or alert on the condition `Stalled=True`. To gate promotions instead, see [Gate template promotion with GitOps](gitops.md).
+
 ## Cleanup and the escape hatch
 
 The controller keeps a finalizer on each test until it has deleted the workspace or proved that none exists. These cases need an admin:
diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md
index b09d6468a..45694426e 100644
--- a/docs/how-to/troubleshooting.md
+++ b/docs/how-to/troubleshooting.md
@@ -219,3 +219,53 @@ See [Promote a template version](../reference/aggregated-api-behavior.md#promote
 ## Aggregated requests return `400` or `409`
 
 These errors often come from the rules for names or for `resourceVersion`. See [Aggregated API behavior](../reference/aggregated-api-behavior.md).
+
+## A `CoderTemplateTest` stays `Pending`
+
+The test waits for something outside it. `status.reason` names it:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.status.reason}: {.status.message}{"\n"}'
+```
+
+Usual causes:
+
+1. `OwnerNotConfigured` or `OwnerNotEligible`: the control plane has no tester, or the tester is not allowed. The message names the problem. See [Create the tester user](test-templates.md#1-create-the-tester-user).
+2. `OperatorAccessNotReady` or `ControlPlaneNotReady`: the control plane in `spec.controlPlaneRef` is missing or not ready. See [`CoderControlPlane` stays `Pending`](#codercontrolplane-stays-pending).
+3. `TemplateNotFound` or `TemplateVersionNotFound`: `spec.template` or `spec.version` names nothing in Coder. The spec is immutable, so create a new test with the right names.
+4. `CoderUnavailable`: a Coder request failed. Check Coder and its logs.
+
+A test without any status means that the controller has not reconciled it. See [The controller runs but nothing reconciles](#the-controller-runs-but-nothing-reconciles). Every wait ends at `spec.timeoutSeconds` with `DeadlineExceeded`. For every reason, see [Read the result](test-templates.md#4-read-the-result).
+
+## A `CoderTemplateTest` does not finish deleting
+
+The test keeps the finalizer `coder.com/template-test-cleanup` until the condition `WorkspaceDeleted` is `True`, or its reason is `Retained` or `ControlPlaneGone`. Read the condition:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> \
+  -o jsonpath='{range .status.conditions[?(@.type=="WorkspaceDeleted")]}{.status} {.reason}: {.message}{"\n"}{end}'
+```
+
+- `Deleting` or `DeleteRetrying`: the delete build runs or failed. Read its logs in Coder. The controller retries.
+- `ControlPlaneUnavailable` or `CoderAnswerMismatch`: Coder is not usable. Fix Coder, and the controller continues.
+- `OwnershipUnknown`: a workspace with the test's name exists, but nothing proves that the test created it. The controller never touches it.
+
+If the controller cannot finish, use the [escape hatch](test-templates.md#cleanup-and-the-escape-hatch). As a last resort, check Coder for a workspace named `status.workspaceName`, then remove the finalizer:
+
+```bash
+kubectl patch codertemplatetest <name> -n <namespace> --type json \
+  -p '[{"op":"remove","path":"/metadata/finalizers"}]'
+```
+
+CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwise the workspace, and the tester's session key, stay in Coder.
+
+## A namespace stays `Terminating`
+
+1. Tests in the namespace wait for their cleanup. List them with `kubectl get codertemplatetests -n <namespace>`, then see [A `CoderTemplateTest` does not finish deleting](#a-codertemplatetest-does-not-finish-deleting).
+2. With the aggregated API server installed, a namespace without an eligible `CoderControlPlane` never finishes deleting ([#209](https://github.com/coder/coder-k8s/issues/209)). Its condition `NamespaceDeletionContentFailure` is `True` with the message `no eligible CoderControlPlane instances found in namespace "<namespace>"`. The aggregated API answers the namespace controller's LIST with `503`. Read the conditions:
+
+    ```bash
+    kubectl get namespace <namespace> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
+    ```
+
+    If `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` are `False`, nothing is left in the namespace, and only #209 holds it.

From 2447be24ebdd4651528f23212517f64d2dc62ff6 Mon Sep 17 00:00:00 2001
From: Thomas Kosiewski <tk@coder.com>
Date: Sat, 3 Oct 2026 15:52:08 +0000
Subject: [PATCH 2/4] =?UTF-8?q?=F0=9F=A4=96=20docs:=20remove=20only=20the?=
 =?UTF-8?q?=20test=20finalizer,=20and=20fix=20two=20how-to=20reasons?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The last-resort patch in troubleshooting now removes only coder.com/template-test-cleanup, guarded by a JSON Patch test operation, instead of the whole finalizer list. A missing or empty operator token Secret is listed under OperatorAccessNotReady, as the controller reports it. Readiness claims now say top-level agents, because the controller skips devcontainer sub-agents.

Refs #152

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

Change-Id: I4cef0bfae938870169b3f18b00b1607662ee8584
---
 docs/how-to/test-templates.md  | 10 +++++-----
 docs/how-to/troubleshooting.md | 18 +++++++++++++-----
 2 files changed, 18 insertions(+), 10 deletions(-)

diff --git a/docs/how-to/test-templates.md b/docs/how-to/test-templates.md
index b0e394c93..401658dc2 100644
--- a/docs/how-to/test-templates.md
+++ b/docs/how-to/test-templates.md
@@ -1,6 +1,6 @@
 # Test a template version with CoderTemplateTest
 
-This guide shows how to check that a Coder template version can start a workspace whose agents become ready. A `CoderTemplateTest` creates a throwaway workspace from one template version, waits until every agent is `connected` and `ready`, records `Succeeded` or `Failed`, and deletes the workspace. For every field, see the [`CoderTemplateTest` reference](../reference/api/codertemplatetest.md).
+This guide shows how to check that a Coder template version can start a workspace whose agents become ready. A `CoderTemplateTest` creates a throwaway workspace from one template version, waits until every top-level agent is `connected` and `ready`, records `Succeeded` or `Failed`, and deletes the workspace. For every field, see the [`CoderTemplateTest` reference](../reference/api/codertemplatetest.md).
 
 You need:
 
@@ -72,8 +72,8 @@ While the test waits, the condition `Reconciling` is `True` with the same reason
 | Reason | Meaning | What to do |
 | --- | --- | --- |
 | `Initializing` | The controller started the test. | Nothing. |
-| `ControlPlaneNotReady` | The `CoderControlPlane` does not exist, is being deleted, has no usable `status.url`, or its operator token Secret has no token yet. | Check `spec.controlPlaneRef` and the control plane status. |
-| `OperatorAccessNotReady` | `status.operatorAccessReady` of the control plane is not `true`. | See [Deploy the controller](deploy-controller.md). |
+| `ControlPlaneNotReady` | The `CoderControlPlane` does not exist, is being deleted, or has no usable `status.url`. | Check `spec.controlPlaneRef` and the control plane status. |
+| `OperatorAccessNotReady` | `status.operatorAccessReady` of the control plane is not `true`, or its operator token Secret is missing or has no token. | See [Deploy the controller](deploy-controller.md), and check the Secret in `status.operatorTokenSecretRef` of the control plane. |
 | `OwnerNotConfigured` | The control plane has no `spec.templateTests.ownerUserID`. | Set it, as in step 1. |
 | `OwnerNotEligible` | The tester does not exist, is not `active`, has a person's login type, has a role that is not allowed, or is not a member of the template's organization. | Fix the tester as described in step 1. The message names the problem. |
 | `TemplateNotFound` | The organization or the template in `spec.template` does not exist. | Check `spec.template` (`<organization>.<template>`). |
@@ -82,7 +82,7 @@ While the test waits, the condition `Reconciling` is `True` with the same reason
 | `CreatingWorkspace`, `CreateRetrying` | The controller sends the create request, or sends it again after a request that had no effect. | Wait. |
 | `ConfirmingCreate` | The answer to the create request was lost. The controller reads Coder until the workspace appears. | Wait. |
 | `WaitingForBuild`, `WaitingForAgents` | The start build runs, or an agent is not yet `connected` and `ready`. | Wait. The message names the agent. |
-| `AgentsReady`, `DeletingWorkspace` | Every agent was ready. The controller deletes the workspace. | Wait. |
+| `AgentsReady`, `DeletingWorkspace` | Every top-level agent was ready. The controller does not check devcontainer sub-agents. The controller deletes the workspace. | Wait. |
 | `CoderUnavailable` | A Coder request failed. After HTTP 429, the controller retries with backoff. | Check Coder and its logs. |
 | `CoderAnswerMismatch` | Coder answered about another object than the one asked for. The controller logs it and retries. | Check proxies between the controller and Coder. |
 
@@ -103,7 +103,7 @@ A failed test has `Ready=False` and `Stalled=True` with the same reason. It does
 | `NoAgents` | The workspace has no agents, so nothing proves that it works. | Add an agent to the template. |
 | `AgentConnectionTimeout`, `AgentStartError`, `AgentStartTimeout`, `AgentStopped` | An agent did not connect in time, its startup script failed or timed out, or it stopped. | Read the agent logs in Coder. The message names the agent. |
 | `WorkspaceChangedExternally`, `WorkspaceDeletedExternally` | Someone else started a build of the workspace, or deleted it. | Leave test workspaces alone, and create a new test. |
-| `DeleteBuildFailed` | Every agent was ready, but the delete build failed. The controller retries the delete. | Read the delete build logs in Coder. |
+| `DeleteBuildFailed` | Every top-level agent was ready, but the delete build failed. The controller retries the delete. | Read the delete build logs in Coder. |
 | `ControlPlaneGone` | The `CoderControlPlane` was deleted while the workspace could exist. | Look for the workspace in Coder and delete it there. |
 
 ### `WorkspaceDeleted` reasons
diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md
index 45694426e..413121198 100644
--- a/docs/how-to/troubleshooting.md
+++ b/docs/how-to/troubleshooting.md
@@ -250,12 +250,20 @@ kubectl get codertemplatetest <name> -n <namespace> \
 - `ControlPlaneUnavailable` or `CoderAnswerMismatch`: Coder is not usable. Fix Coder, and the controller continues.
 - `OwnershipUnknown`: a workspace with the test's name exists, but nothing proves that the test created it. The controller never touches it.
 
-If the controller cannot finish, use the [escape hatch](test-templates.md#cleanup-and-the-escape-hatch). As a last resort, check Coder for a workspace named `status.workspaceName`, then remove the finalizer:
+If the controller cannot finish, use the [escape hatch](test-templates.md#cleanup-and-the-escape-hatch). As a last resort, check Coder for a workspace named `status.workspaceName`, then remove only the controller's finalizer. Other controllers can have their own finalizers on the test, so do not remove the whole list:
 
-```bash
-kubectl patch codertemplatetest <name> -n <namespace> --type json \
-  -p '[{"op":"remove","path":"/metadata/finalizers"}]'
-```
+1. Find the position of `coder.com/template-test-cleanup` in the list. The first entry has position 0.
+
+    ```bash
+    kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.metadata.finalizers}'
+    ```
+
+2. Remove the entry at that position. The `test` operation makes the patch fail if the entry at `<position>` is a different finalizer.
+
+    ```bash
+    kubectl patch codertemplatetest <name> -n <namespace> --type json -p \
+      '[{"op":"test","path":"/metadata/finalizers/<position>","value":"coder.com/template-test-cleanup"},{"op":"remove","path":"/metadata/finalizers/<position>"}]'
+    ```
 
 CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwise the workspace, and the tester's session key, stay in Coder.
 

From 2c8c019e02b0386fa9edb029f199d36d4057e905 Mon Sep 17 00:00:00 2001
From: Thomas Kosiewski <tk@coder.com>
Date: Sat, 3 Oct 2026 16:07:49 +0000
Subject: [PATCH 3/4] =?UTF-8?q?=F0=9F=A4=96=20docs:=20say=20top-level=20ag?=
 =?UTF-8?q?ents=20for=20NoAgents,=20and=20stop=20nightly=20Job=20retries?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

NoAgents means that the workspace has no top-level agents: the controller ignores devcontainer sub-agents, so the row and the sample now ask for a top-level agent. The nightly CronJob sets backoffLimit 0, because a retry after a lost create response makes a second test with generateName.

Refs #152

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

Change-Id: I1a43ba8c48730e461c27b9120cc77f7aee236ee9
---
 config/samples/coder_v1alpha1_codertemplatetest.yaml | 4 ++--
 docs/how-to/test-templates.md                        | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/config/samples/coder_v1alpha1_codertemplatetest.yaml b/config/samples/coder_v1alpha1_codertemplatetest.yaml
index b23378e9b..b4624f7da 100644
--- a/config/samples/coder_v1alpha1_codertemplatetest.yaml
+++ b/config/samples/coder_v1alpha1_codertemplatetest.yaml
@@ -1,6 +1,6 @@
 # Tests one Coder template version: the controller creates a workspace from
-# it, waits until every agent is ready, records the result, and deletes the
-# workspace. See docs/how-to/test-templates.md.
+# it, waits until every top-level agent is ready, records the result, and
+# deletes the workspace. See docs/how-to/test-templates.md.
 #
 # Prerequisites:
 # - CoderControlPlane codercontrolplane-sample with status.operatorAccessReady
diff --git a/docs/how-to/test-templates.md b/docs/how-to/test-templates.md
index 401658dc2..b4c7c3848 100644
--- a/docs/how-to/test-templates.md
+++ b/docs/how-to/test-templates.md
@@ -100,7 +100,7 @@ A failed test has `Ready=False` and `Stalled=True` with the same reason. It does
 | `CreateOutcomeUnknown` | No workspace from this test appeared after an uncertain create request. | Check Coder, then create a new test. |
 | `WorkspaceNameConflict` | A workspace with the test's name exists, but nothing proves that this test created it. | Check the workspace in Coder. See the escape hatch below. |
 | `BuildFailed`, `BuildCanceled` | The start build failed, or someone else canceled it. | Read the build logs in Coder. |
-| `NoAgents` | The workspace has no agents, so nothing proves that it works. | Add an agent to the template. |
+| `NoAgents` | The workspace has no top-level agents, so nothing proves that it works. Devcontainer sub-agents do not count. | Add a top-level agent to the template. |
 | `AgentConnectionTimeout`, `AgentStartError`, `AgentStartTimeout`, `AgentStopped` | An agent did not connect in time, its startup script failed or timed out, or it stopped. | Read the agent logs in Coder. The message names the agent. |
 | `WorkspaceChangedExternally`, `WorkspaceDeletedExternally` | Someone else started a build of the workspace, or deleted it. | Leave test workspaces alone, and create a new test. |
 | `DeleteBuildFailed` | Every top-level agent was ready, but the delete build failed. The controller retries the delete. | Read the delete build logs in Coder. |
@@ -179,6 +179,7 @@ spec:
   concurrencyPolicy: Forbid
   jobTemplate:
     spec:
+      backoffLimit: 0 # a retry after a lost create response makes a second test
       template:
         spec:
           serviceAccountName: nightly-template-test

From 7be4853ae32267258710ed332ea0ef7bdb840c92 Mon Sep 17 00:00:00 2001
From: Thomas Kosiewski <tk@coder.com>
Date: Sat, 3 Oct 2026 16:24:13 +0000
Subject: [PATCH 4/4] =?UTF-8?q?=F0=9F=A4=96=20docs:=20split=20WorkspaceNam?=
 =?UTF-8?q?eConflict=20by=20WorkspaceDeleted,=20and=20explain=20the=20firs?=
 =?UTF-8?q?t=20reconcile?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

WorkspaceNameConflict has two outcomes. With WorkspaceDeleted NotCreated, the test never created a workspace and releases its finalizer by itself, so the workspace belongs to someone else. Only OwnershipUnknown needs the escape hatch. The troubleshooting page now says that the first reconcile only adds the finalizer and the next one writes the status, and shows how to tell the two cases apart.

Refs #152

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

Change-Id: I4c6ec127bba9a3b078a6729d469518b626f92124
---
 docs/how-to/test-templates.md  |  2 +-
 docs/how-to/troubleshooting.md | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/docs/how-to/test-templates.md b/docs/how-to/test-templates.md
index b4c7c3848..6f02bf4d1 100644
--- a/docs/how-to/test-templates.md
+++ b/docs/how-to/test-templates.md
@@ -98,7 +98,7 @@ A failed test has `Ready=False` and `Stalled=True` with the same reason. It does
 | `TemplateVersionMismatch` | The version in `spec.version.id` belongs to another template. | Fix `spec.version` or `spec.template`. |
 | `CreateRejected` | Coder rejected the create request. The message holds Coder's answer. No workspace exists. | Fix the cause, for example `spec.parameters`, and create a new test. |
 | `CreateOutcomeUnknown` | No workspace from this test appeared after an uncertain create request. | Check Coder, then create a new test. |
-| `WorkspaceNameConflict` | A workspace with the test's name exists, but nothing proves that this test created it. | Check the workspace in Coder. See the escape hatch below. |
+| `WorkspaceNameConflict` | A workspace with the test's name exists, and this test did not create it, or nothing proves that it did. | Read the `WorkspaceDeleted` condition. `NotCreated` means that this test never created a workspace: the test releases its finalizer by itself, and the workspace belongs to someone else, so leave it alone. For `OwnershipUnknown`, see the escape hatch below. |
 | `BuildFailed`, `BuildCanceled` | The start build failed, or someone else canceled it. | Read the build logs in Coder. |
 | `NoAgents` | The workspace has no top-level agents, so nothing proves that it works. Devcontainer sub-agents do not count. | Add a top-level agent to the template. |
 | `AgentConnectionTimeout`, `AgentStartError`, `AgentStartTimeout`, `AgentStopped` | An agent did not connect in time, its startup script failed or timed out, or it stopped. | Read the agent logs in Coder. The message names the agent. |
diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md
index 413121198..60a841730 100644
--- a/docs/how-to/troubleshooting.md
+++ b/docs/how-to/troubleshooting.md
@@ -235,7 +235,15 @@ Usual causes:
 3. `TemplateNotFound` or `TemplateVersionNotFound`: `spec.template` or `spec.version` names nothing in Coder. The spec is immutable, so create a new test with the right names.
 4. `CoderUnavailable`: a Coder request failed. Check Coder and its logs.
 
-A test without any status means that the controller has not reconciled it. See [The controller runs but nothing reconciles](#the-controller-runs-but-nothing-reconciles). Every wait ends at `spec.timeoutSeconds` with `DeadlineExceeded`. For every reason, see [Read the result](test-templates.md#4-read-the-result).
+A new test has no status for a moment: the first reconcile only adds the finalizer `coder.com/template-test-cleanup`, and the next reconcile writes the status. If the status stays empty, check the finalizer:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.metadata.finalizers}{"\n"}'
+```
+
+Without the finalizer, the controller has not reconciled the test. See [The controller runs but nothing reconciles](#the-controller-runs-but-nothing-reconciles). With the finalizer, the controller reconciled the test once: read the controller logs for errors about the test.
+
+Every wait ends at `spec.timeoutSeconds` with `DeadlineExceeded`. For every reason, see [Read the result](test-templates.md#4-read-the-result).
 
 ## A `CoderTemplateTest` does not finish deleting