Re-register the runner scale set when it is gone from the Actions service

[diogotorres97]

Partially addresses #4370.

New scale sets — and any whose listener has not been created yet — recover automatically. An AutoscalingListener that is already crash-looping on a dead ID is not healed on its own; deleting it with kubectl delete autoscalinglistener <name> -n <controller-namespace> makes the controller rebuild it against a revalidated ID. Before this change the only recovery was deleting the whole AutoscalingRunnerSet. See "Follow-up worth considering" below for why, and what would close the gap.

The problem

A runner scale set can disappear from the Actions service out of band: deleted from the organization settings, or dropped and registered again with a fresh ID. The ID recorded in the runner-scale-set-id annotation then points at nothing.

Nothing notices:

• shouldCreateScaleSet registers a scale set again only when the annotation is missing, unparseable, or non-positive (autoscalingrunnerset_controller.go:798-805). A valid-but-dead ID survives every reconcile.
• The ID is baked into the AutoscalingListener spec once, when it is built (resourcebuilder.go:137), and each listener pod reads it on startup and exits on any error. Every pod fails with RunnerScaleSetNotFoundException and the listener crash-loops.

Restarting pods does not help, because the ID is baked into the AutoscalingListener spec. Correcting the annotation by hand does not help either. The only way out today is to delete the AutoscalingRunnerSet and let it be created again, which several people in #4370 report is not viable in production.

Deleting it gets stuck too: DeleteRunnerScaleSet returns not-found for a scale set that is already gone, deleteRunnerScaleSet propagates that error, the finalizer never completes, and the resource sits in Terminating until the finalizer is removed by hand.

The fix

Confirm the recorded scale set still exists before creating a listener with its ID, and register it again when it does not. createRunnerScaleSet already reuses an existing scale set with the same name, so a scale set that came back with a new ID is adopted rather than duplicated.

Only a definitive not-found triggers this. Any other failure is returned to the caller, so a transient Actions service outage cannot cause a second scale set to be registered — the failure mode reported in #3942. There is a test for exactly this.

The lookup runs only when a listener is about to be created, not on every reconcile, so steady state costs no extra API calls.

The delete path now treats an already-deleted scale set as the desired state, so an AutoscalingRunnerSet whose scale set is gone no longer needs its finalizer stripped by hand.

A note on the dependency bump

Both paths need a not-found sentinel from github.com/actions/scaleset. wrapResponseErrorType on main maps HTTP 404 to scaleset.NotFoundError, which is exactly what is needed, but it is not in v0.4.0 — the latest tag — so this PR moves the dependency to a pseudo-version of main.

Happy to re-point it at a tag if you cut a scaleset release, or to rework this if you would rather not carry a pseudo-version. Without the sentinel the alternative is matching on the error string, which is out of step with how the rest of the controllers detect these conditions (errors.Is(err, scaleset.JobStillRunningError) and friends).

Tests

Three specs in autoscalingrunnerset_controller_test.go:

• a stale scale set is registered again, and the listener is created with the new ID — fails on main, where the listener is created with the dead ID
• a transient Actions service failure does not re-register the scale set, and creates nothing
• deleting an AutoscalingRunnerSet whose scale set is already gone removes the finalizer instead of hanging in Terminating — fails on main, where deletion never completes

Also adds WithGetRunnerScaleSetByIDFunc to the fake client, mirroring the existing Func options, so a test can answer per-ID.

The full TestAPIs suite passes locally, including the three new specs.

Follow-up worth considering

This heals a listener at creation time. A listener that is already crash-looping is not healed automatically, because its spec never changes and so it is never rebuilt — deleting the AutoscalingListener is now enough to recover, where before the whole AutoscalingRunnerSet had to go. Making the controller notice a persistently failing listener and revalidate would close that gap, but it is a larger change and I left it out of this PR.

#4561 looks like the same shape with a different field, and may already be resolved on main now that the EphemeralRunnerSet is patched in place under a deterministic name.

[Copilot]

Pull request overview

This PR hardens the AutoscalingRunnerSet controller against out-of-band deletion/recreation of runner scale sets in the Actions service by re-validating the recorded scale set ID before creating a listener, and by treating “already deleted” as success during finalization. This prevents listener crash-loops on stale IDs and avoids stuck Terminating resources when the scale set is missing.

Changes:

• Add a pre-listener existence check (GetRunnerScaleSetByID) and re-register the scale set when the recorded ID is definitively not found.
• Treat scaleset.NotFoundError from DeleteRunnerScaleSet as desired state to allow finalizers to complete.
• Add fake client plumbing + Ginkgo specs covering stale ID recovery, transient failure behavior, and deletion when the scale set is already gone.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
go.mod	Bumps github.com/actions/scaleset to a pseudo-version to gain a not-found sentinel; indirect dependency bumps.
go.sum	Updates module checksums for the dependency changes.
controllers/actions.github.com/multiclient/fake/client.go	Adds WithGetRunnerScaleSetByIDFunc and dispatch logic to support per-ID behavior in tests.
controllers/actions.github.com/autoscalingrunnerset_controller.go	Adds runnerScaleSetGone check before listener creation; handles not-found on delete as success.
controllers/actions.github.com/autoscalingrunnerset_controller_test.go	Adds test coverage for stale scale set IDs, transient failures, and deletion when scale set is already missing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated no new comments.

[diogotorres97]

@nikola-jokic @Okabe-Junya can you take a look?