fix task token invalidation after resetting an activity by spkane31 · Pull Request #11097 · temporalio/temporal

spkane31 · 2026-07-15T22:07:59Z

What changed?

Added a monotonic activity attempt stamp (activity_attempt_stamp) to task tokens for standalone activities. CHASM activity task-token validation checks that non-legacy stamped tokens match the current attempt stamp, so a token from before an attempt reset cannot heartbeat or complete the reset attempt after the visible attempt counter is rewound.

Workflow activity behavior is unchanged: workflow and by-ID synthesized activity tokens pass 0 for the new field, preserving existing validation semantics.

Why?

Standalone activity reset can rewind the visible attempt count back to 1. Without a monotonic generation in the task token, a stale worker holding an older attempt-1 token can later match the reset attempt-1 state and incorrectly heartbeat or complete obsolete work.

The new stamp fences those stale tokens while keeping existing/pre-upgrade tokens usable. A missing stamp decodes as 0 and skips the new check for backward compatibility.

How did you test it?

Potential risks

This change is intended to be wire-compatible: older tokens decode with activity_attempt_stamp = 0, and older servers should ignore the unknown field. During rolling upgrades, stale-token protection only applies when a stamped token is validated by upgraded CHASM activity code. Existing in-flight tokens remain accepted by design.

fretz12 · 2026-07-15T23:48:36Z

 	if token.Attempt != ByIDTokenAttempt && token.Attempt != a.LastAttempt.Get(ctx).GetCount() {
 		return serviceerror.NewNotFound("activity task not found")
 	}
+	if token.GetActivityAttemptStamp() != 0 && token.GetActivityAttemptStamp() != a.LastAttempt.Get(ctx).GetStartedStamp()+1 {


During rollout upgrade, we should handle the old activity state that doesn't yet have StartedStamp. We could assume it's 0 for protobuf default, and handle it like:

Suggested change

if token.GetActivityAttemptStamp() != 0 && token.GetActivityAttemptStamp() != a.LastAttempt.Get(ctx).GetStartedStamp()+1 {

startedStamp := a.LastAttempt.Get(ctx).GetStartedStamp()

// A zero started stamp indicates state persisted before this field was introduced.

if token.GetActivityAttemptStamp() != 0 && startedStamp != 0 && token.GetActivityAttemptStamp() != startedStamp+1 {

fretz12 · 2026-07-16T00:00:26Z

We shoud test out this case an update happens and doesn't bump StartStamp and only Stamp
Specifically

Start + poll an activity so the worker holds a token

UpdateActivityExecutionOptions on the running activity

Assert the Stamp changed ( and not StartStamp)

With the original token: RecordActivityTaskHeartbeat and RespondActivityTaskCompleted → both succeed

dandavison

Nice catch! I can't think of a better solution.

dandavison · 2026-07-16T02:49:06Z

+	componentRef := task.event.GetData().GetComponentRef()
+	activityAttemptStamp := int32(0)
+	if len(componentRef) > 0 {
+		activityAttemptStamp = task.event.Data.GetStamp() + 1


It would be nice if we can avoid doing the +1 comparison: the logic is split over History and Matching, and recapitulated in tests. Can you construct a test that fails if we do not do it?

also it feels a little weird that setting activityAttemptStamp is conditional on a different field.. what happens if we just do this all the time? non-chasm activity tokens get slightly larger?

dandavison · 2026-07-16T02:50:58Z

@@ -1804,6 +1804,9 @@ func (a *Activity) validateActivityTaskToken(
 	if token.Attempt != ByIDTokenAttempt && token.Attempt != a.LastAttempt.Get(ctx).GetCount() {
 		return serviceerror.NewNotFound("activity task not found")
 	}


After the transitional phase of rollout, would we drop the naive attempt comparisons? Maybe leave a comment explaining that the attempt stamp comparison is sufficient now.

We have to wait until all activities created without this stamp are finished which would require a full table on every cell

spkane31 · 2026-07-16T19:40:40Z

 	s.LessOrEqual(expectedRange, s.taskManager.getQueueDataByKey(tlID).rangeID)
 }

+func (s *matchingEngineSuite) TestCreatePollActivityTaskQueueResponse_StandaloneAttemptStamp() {


@dandavison this is the test that validates the +1 is needed, it will fail if that is removed.

Looks like this test verifies that the +1 happens, but doesn't show that it's needed.

dnr · 2026-07-16T21:35:26Z

+  // The attempt stamp captured when the current worker started this attempt.
+  // Unlike stamp, this does not change on options updates while the worker owns the token.
+  int32 started_stamp = 14;


put this field right under stamp so readers will see them both next to each other and it's easier to compare+contrast the semantics.

also, what does "worker owns the token" mean?

dnr · 2026-07-16T21:42:23Z

+	componentRef := task.event.GetData().GetComponentRef()
+	activityAttemptStamp := int32(0)
+	if len(componentRef) > 0 {
+		activityAttemptStamp = task.event.Data.GetStamp() + 1


also it feels a little weird that setting activityAttemptStamp is conditional on a different field.. what happens if we just do this all the time? non-chasm activity tokens get slightly larger?

dnr · 2026-07-16T23:14:31Z

-	if len(componentRef) > 0 {
-		activityAttemptStamp = task.event.Data.GetStamp() + 1
-	}
+	activityAttemptStamp := task.event.Data.GetStamp() + 1


could just inline that now?

Implements Pause, Unpause, Reset, and UpdateActivityExecutionOptions RPCs for standalone CHASM activities. Adds matching frontend and history service handlers in chasm/lib/activity that dispatch to either the workflow-activity path or the standalone-activity path based on the presence of a workflow ID. Also adds PAUSE_REQUESTED and RESET_REQUESTED state machine states for standalone activities, the original_options snapshot for restore-on-reset semantics, the activity-options merge helpers (common/activityoptions), and the supporting proto schema changes (start_request_id, sdk_name, sdk_version, etc.). Squashed PRs: #9693 RPC boilerplate and code generation #9850 Implement UpdateActivityExecutionOptions #9851 Implement Pause/UnpauseActivityExecution for SAA #9852 Implement ResetActivityExecution for SAA #10001 Pause returns FailedPrecondition when already paused #10025 Validator function fixes #10265 Replace PauseState flag with PAUSE_REQUESTED state #10364 Add RESET_REQUESTED state to standalone activity #10745 Add support for updating start_delay via UpdateActivityOptions #10913 Preserve SAA retry backoff across unpause re-dispatch #10914 Fix SAA unpause for keep-paused reset #10917 Reject stale legacy SAA schedule-to-close timers #10920 Preserve NextRetryDelay across unrelated activity option updates #10921 Validate merged SAA retry policy updates #10923 Fix SAA reset attempt dispatch time reporting #11010 Use default retry policy when updating with nil #11020 Fix: de-duplicate pause even if activity is unpaused #11013 Fix: reject restore_original when a standalone activity has no original options snapshot #11032 Reject unsupported and empty update_mask paths in UpdateActivityExecutionOptions #11030 Fix: describe only provides heartbeat/failure when opted-in #11016 Preserve pending activity next retry delay #11068 Add SAA operator APIs to DC redirection whitelist #10954 Remove reset_heartbeats field in ResetActivityExecutionRequest #11087 Fix standalone activity heartbeat flag precedence #11117 Validation to reject negative activity jitter #11120 Fix bogus StartToCloseLatency for non-running standalone activities #11097 Fix task token invalidation after resetting an activity #11067 SAA bug fix: update start delay while paused #11123 SAA: cleanup

spkane31 added 2 commits July 15, 2026 16:10

fix task token invalidation after resetting an activity

9006230

additional functional test

b2be8eb

spkane31 force-pushed the spk/stale-task-token branch from df1919e to b2be8eb Compare July 15, 2026 22:11

spkane31 marked this pull request as ready for review July 15, 2026 22:12

spkane31 requested review from a team as code owners July 15, 2026 22:12

test fix

593407b

spkane31 requested review from dandavison and fretz12 July 15, 2026 22:54

github-actions Bot mentioned this pull request Jul 15, 2026

🦞 OpenClaw 生态日报 2026-07-16 junlinzhao327-oss/big_model_radar#70

Open

fretz12 reviewed Jul 16, 2026

View reviewed changes

dandavison approved these changes Jul 16, 2026

View reviewed changes

comments

2582af8

spkane31 requested a review from fretz12 July 16, 2026 16:18

fretz12 approved these changes Jul 16, 2026

View reviewed changes

spkane31 commented Jul 16, 2026

View reviewed changes

dnr reviewed Jul 16, 2026

View reviewed changes

davids comments

c79a7f0

dnr approved these changes Jul 16, 2026

View reviewed changes

remove +1 logic in task token stamping

01b4fd1

spkane31 merged commit 1bdb18c into feature/activity-operator-cmds Jul 17, 2026
46 checks passed

spkane31 deleted the spk/stale-task-token branch July 17, 2026 16:53

-	if token.GetActivityAttemptStamp() != 0 && token.GetActivityAttemptStamp() != a.LastAttempt.Get(ctx).GetStartedStamp()+1 {
+startedStamp := a.LastAttempt.Get(ctx).GetStartedStamp()
+	// A zero started stamp indicates state persisted before this field was introduced.
+	if token.GetActivityAttemptStamp() != 0 && startedStamp != 0 && token.GetActivityAttemptStamp() != startedStamp+1 {

Uh oh!

Conversation

spkane31 commented Jul 15, 2026

What changed?

Why?

How did you test it?

Potential risks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dandavison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants