Extract supervisor heartbeat logic into a standalone Heartbeater - #71102
Draft
jason810496 wants to merge 1 commit into
Draft
Extract supervisor heartbeat logic into a standalone Heartbeater#71102jason810496 wants to merge 1 commit into
jason810496 wants to merge 1 commit into
Conversation
1 task
jason810496
force-pushed
the
refactor/task-sdk/extract-heartbeater
branch
from
August 4, 2026 14:35
e39ae3d to
d0c29c2
Compare
A follow-up will let subprocess coordinators heartbeat while materializing a Dag bundle, before the task subprocess exists. That requires the heartbeat state to live outside ActivitySubprocess — especially the pid presented to the server, which must stay identical for the task instance's lifetime or the server rejects the heartbeat as "running elsewhere" and the task is killed. Reactions to fatal heartbeat outcomes (killing the process, recording SERVER_TERMINATED) remain in ActivitySubprocess, injected as callbacks. No behavior change.
jason810496
force-pushed
the
refactor/task-sdk/extract-heartbeater
branch
from
August 5, 2026 07:06
d0c29c2 to
65fd39e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dag_bundle_namematerialization step this heartbeat refactor is preparing to support during coordinator startup)Why
A planned follow-up will let subprocess coordinators heartbeat while materializing a Dag bundle, before the task subprocess exists. That requires the heartbeat state to live outside
ActivitySubprocess, especially the pid presented to the server: it must stay identical for the whole task instance lifetime, otherwise the server rejects the heartbeat with 409running_elsewhereand the task is killed. This PR is the purely mechanical foundation with no behavior change.What
Heartbeaterclass insupervisor.pyowning the heartbeat state (_last_successful_heartbeat,_last_heartbeat_attempt,failed_heartbeats) and a fixedpid, with the moved methods:send_heartbeat_if_needed()and_handle_heartbeat_failures()(moved fromActivitySubprocess)send_heartbeat(): the ungated primitivesend_heartbeat_if_needed()wraps, for callers that own their own pacing (the coordinator startup thread in the follow-up)compute_max_wait_time()(the select-timeout formula moved from_monitor_subprocess)record_successful_heartbeat()(the task-start call already counts as a beat on the server)ActivitySubprocessto compose aHeartbeater, injecting its reactions as callbacks:on_server_terminated: terminate the process and recordSERVER_TERMINATEDon_fatal_failures: kill the process afterMAX_FAILED_HEARTBEATS; optional (Nonedisables the cap and retries indefinitely, for callers with no process to kill)_send_heartbeat_if_neededstays as a thin wrapper keeping the overtime guardHEARTBEAT_TIMEOUT,MIN_HEARTBEAT_INTERVAL,MAX_FAILED_HEARTBEATS) stay module level, so existing config handling and test monkeypatching are unaffectedHeartbeaterin the module__all__test_supervisor.pyreferences to the moved attributes; the behavior tests (test_regular_heartbeat,test_no_heartbeat_in_overtime,test_state_conflict_on_heartbeat) pass unchangedWas generative AI tooling used to co-author this PR?