fix: allow paused actions to reach terminal phases - #7797
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a phase-guard bug in the runs/actions repository that prevented condition actions from leaving PAUSED due to enum ordinal ordering, and makes rejected/ignored phase updates observable to callers so downstream writes (e.g., signal output) don’t persist against an unchanged action phase.
Changes:
- Replaces ordinal phase comparison in
UpdateActionPhasewith an explicit allowed-source transition table (includingPAUSED -> terminal). - Returns an explicit error on rejected transitions (vs silently succeeding) and short-circuits
UpdateActionStatusoutput persistence on rejected transitions. - Adds/updates repository and service tests around paused terminal transitions and rejected updates.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| runs/service/internal_run_service.go | Treats rejected phase transitions as no-ops to prevent persisting output/run info for stale events. |
| runs/service/condition_test.go | Adds a test ensuring rejected condition transitions don’t persist output. |
| runs/repository/interfaces/action.go | Introduces ErrPhaseTransitionRejected for explicit transition rejection signaling. |
| runs/repository/impl/action.go | Implements explicit allowed-source phase transition guard + distinguishes rejected transition vs missing action. |
| runs/repository/impl/action_test.go | Adds coverage for PAUSED -> terminal transitions and tightens backward-transition assertions. |
Suppressed comments (1)
runs/repository/impl/action.go:35
PAUSEDis not included as an allowed source for transitioning intoFAILED, even though the PR description calls outPAUSEDas a valid source forFAILED. As written, a paused action still cannot be marked failed viaUpdateActionPhase.
common.ActionPhase_ACTION_PHASE_FAILED: {common.ActionPhase_ACTION_PHASE_UNSPECIFIED, common.ActionPhase_ACTION_PHASE_QUEUED, common.ActionPhase_ACTION_PHASE_WAITING_FOR_RESOURCES, common.ActionPhase_ACTION_PHASE_INITIALIZING, common.ActionPhase_ACTION_PHASE_RUNNING, common.ActionPhase_ACTION_PHASE_FAILED, common.ActionPhase_ACTION_PHASE_TIMED_OUT},
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| common.ActionPhase_ACTION_PHASE_QUEUED: {common.ActionPhase_ACTION_PHASE_UNSPECIFIED, common.ActionPhase_ACTION_PHASE_QUEUED, common.ActionPhase_ACTION_PHASE_FAILED, common.ActionPhase_ACTION_PHASE_TIMED_OUT}, | ||
| common.ActionPhase_ACTION_PHASE_WAITING_FOR_RESOURCES: {common.ActionPhase_ACTION_PHASE_UNSPECIFIED, common.ActionPhase_ACTION_PHASE_QUEUED, common.ActionPhase_ACTION_PHASE_WAITING_FOR_RESOURCES, common.ActionPhase_ACTION_PHASE_FAILED, common.ActionPhase_ACTION_PHASE_TIMED_OUT}, | ||
| common.ActionPhase_ACTION_PHASE_INITIALIZING: {common.ActionPhase_ACTION_PHASE_UNSPECIFIED, common.ActionPhase_ACTION_PHASE_QUEUED, common.ActionPhase_ACTION_PHASE_WAITING_FOR_RESOURCES, common.ActionPhase_ACTION_PHASE_INITIALIZING, common.ActionPhase_ACTION_PHASE_FAILED, common.ActionPhase_ACTION_PHASE_TIMED_OUT}, | ||
| common.ActionPhase_ACTION_PHASE_RUNNING: {common.ActionPhase_ACTION_PHASE_UNSPECIFIED, common.ActionPhase_ACTION_PHASE_QUEUED, common.ActionPhase_ACTION_PHASE_WAITING_FOR_RESOURCES, common.ActionPhase_ACTION_PHASE_INITIALIZING, common.ActionPhase_ACTION_PHASE_RUNNING, common.ActionPhase_ACTION_PHASE_FAILED, common.ActionPhase_ACTION_PHASE_TIMED_OUT}, |
| if errors.Is(err, interfaces.ErrPhaseTransitionRejected) { | ||
| return nil | ||
| } | ||
| logger.Warnf(ctx, "UpdateActionStatus: failed to update action %s: %v", req.GetActionId().GetName(), err) | ||
| return connect.NewError(connect.CodeInternal, err) |
| for _, phase := range []common.ActionPhase{ | ||
| common.ActionPhase_ACTION_PHASE_SUCCEEDED, | ||
| common.ActionPhase_ACTION_PHASE_TIMED_OUT, | ||
| common.ActionPhase_ACTION_PHASE_ABORTED, | ||
| } { |
|
Do you have an example that I can repro? |
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
589f04d to
b12aeab
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (2)
runs/service/internal_run_service.go:276
sql.ErrNoRows(action not found) is treated asconnect.CodeNotFound, but it is logged as a warning first. This will create noisy warning logs for an expected NotFound outcome; handle NotFound before logging the warning (or log at a lower level).
logger.Warnf(ctx, "UpdateActionStatus: failed to update action %s: %v", req.GetActionId().GetName(), err)
if errors.Is(err, sql.ErrNoRows) {
return connect.NewError(connect.CodeNotFound, err)
}
return connect.NewError(connect.CodeInternal, err)
runs/repository/impl/action.go:457
allowedActionPhaseSources[phase]is indexed without checking presence. If the ActionPhase enum gains a new value and the map isn’t updated, this will silently reject all transitions to that phase (via an emptyANY('{}')), returningErrPhaseTransitionRejectedand making the failure hard to diagnose. Add an explicit guard for unknown target phases.
allowedSources := allowedActionPhaseSources[phase]
allowedSourceValues := make([]int32, len(allowedSources))
for i, source := range allowedSources {
allowedSourceValues[i] = int32(source)
}
|
@pingsutw thanks for the review. i also addressed the remaining findings: paused actions can enter retry phases and failed, missing actions return not found, and the transition matrix has direct coverage. the minimal repro is paused followed by succeeded, failed, or timed_out; previously the rpc returned ok while the stored phase remained paused. rebased onto the latest main and the focused repository and service tests pass. |
Tracking Issue
NA
Why are the changes needed?
actionRepo.UpdateActionPhaseguards writes withphase <= $requested OR phase = ANY(retryablePhases), where the retryable set isFAILEDandTIMED_OUT.ACTION_PHASE_PAUSEDis enum value 9, whileACTION_PHASE_SUCCEEDEDis 5 andACTION_PHASE_TIMED_OUTis 8, so a paused condition action can never leavePAUSED: both the signal path and the timeout path fail the predicate. The statement updates zero rows, the repository still returnsnil, andUpdateActionStatusreports success while the actions informer goes on to persist the signal output against a row that is stillPAUSED.The practical effect is that human-in-the-loop conditions stay listed and filtered as awaiting input after they have been answered or timed out, no end time is ever recorded, and historical action details plus paused-run listings are permanently wrong.
Reproduction: persist a condition action, move it
QUEUED -> PAUSED, then callUpdateActionStatuswithSUCCEEDEDorTIMED_OUT. The RPC returns OK, butGetActionstill returnsPAUSED.What changes were proposed in this pull request?
UpdateActionPhasewith an explicit map of allowed source phases per target phase. Forward progress through the running phases and the existing retry-from-FAILED/TIMED_OUTbehavior are preserved,PAUSEDbecomes a valid source forSUCCEEDED,TIMED_OUT,ABORTED,FAILEDand the retry phases, and completed actions can no longer be moved backwards.ErrPhaseTransitionRejected(guard rejected the transition) orsql.ErrNoRows(no such action) instead ofnil.updateSingleActionStatustreatsErrPhaseTransitionRejectedas a no-op and returns before writing the output and run info, so a stale or out-of-order event can no longer overwrite the output of an action it was not allowed to transition.PAUSED -> SUCCEEDED/TIMED_OUT/ABORTEDincluding the persisted end time, tightened assertions on the existing backwards-transition tests, and a condition service test asserting that a rejected transition does not persist the signal output.Note for reviewers
The explicit table also rejects a few transitions that the ordinal check happened to permit, notably
SUCCEEDED -> FAILED/ABORTED/TIMED_OUTand moving intoRECOVEREDfrom a terminal phase. That follows from treating completed actions as final, but if any of those are relied on in practice I am happy to add the specific entries back.How was this patch tested?
go test ./runs/repository/impl ./runs/serviceThe new repository test drives the real
QUEUED -> PAUSED -> terminalsequence against the test database and asserts both the stored phase and thatended_atis set. The service test uses the mocked action repo to assert that a rejected transition short-circuits before the output write and still returns an OK status to the caller.Setup process
No setup or migration changes; the fix is confined to the update predicate and its error handling.
Screenshots
NA
Check all the applicable boxes