feat: Blueapi plan pause - #1589
Open
Alexj9837 wants to merge 11 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1589 +/- ##
==========================================
+ Coverage 95.96% 96.21% +0.25%
==========================================
Files 45 45
Lines 3317 3437 +120
==========================================
+ Hits 3183 3307 +124
+ Misses 134 130 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…ead signals Route resume() and cancel_active_task() through the worker thread's signal queue instead of mutating state directly from the caller, so state stays consistent across pause/resume/cancel transitions. Includes tests and a fix for cancel_active_task() when the RunEngine is already in a paused state.
- Resolve a paused task before stopping the worker, instead of leaving the RunEngine paused and the task incomplete forever with no thread left to resolve it. - Apply the latest cancel_active_task() request while paused instead of a stale queued one. - Prevent a race between the caller thread and worker thread when recording a cancelled task's outcome.
racing the worker thread's own finalization, which could report a completed task with no result.
Alexj9837
force-pushed
the
plan-pause
branch
2 times, most recently
from
July 14, 2026 11:04
5f85341 to
f6f82cb
Compare
Alexj9837
marked this pull request as ready for review
July 14, 2026 11:33
fajinyuan
reviewed
Aug 5, 2026
| return self._current.task_id | ||
| self._task_channel.put(CancelSignal(failure=failure, reason=reason)) | ||
| add_span_attributes( | ||
| {"Task aborted" if failure else "Task stopped": reason or ""} |
There was a problem hiding this comment.
should 'default_reason' be used instead of ""? the default reason for abort and stop are different
tpoliaw
requested changes
Aug 10, 2026
tpoliaw
left a comment
tpoliaw
left a comment
Contributor
There was a problem hiding this comment.
Taking a while to figure out what is going on with the task worker in general and what needed to change. There are a couple of blocking things (resumed plan results and exceptions) but feel free to ignore/question the rest.
Might need you to go over why the CancelSignal is needed when it wasn't before - I'm not sure I follow the logic.
| finally: | ||
| if self._current_task_otel_context is not None: | ||
| if ( | ||
| self._current_task_otel_context is not None |
Contributor
There was a problem hiding this comment.
Should the context be left here or recreated when the task is resumed? I'm not sure how to check this but wouldn't this leave the resumed part of plan running with the telemetry of the original submit and lose trace of the resume happening? Maybe that is what we want?
| if self._current is not None: | ||
| try: | ||
| result = self._ctx.run_engine.resume() | ||
| self._current.set_result(result) |
Contributor
There was a problem hiding this comment.
resume returns a RunEngineResult not the return value of the plan
Suggested change
| self._current.set_result(result) | |
| self._current.set_result(result.plan_result) |
Contributor
|
Another thing to look at - running a count plan and then pausing/resuming several times, it occasionally fails with "Received multiple indices in a `stream_datum` document for one event"I'm not sure if it's related to this change or if there is a bug in the run engine/plan that has only just surfaced now pausing is possible. |
Alexj9837
force-pushed
the
plan-pause
branch
2 times, most recently
from
August 12, 2026 12:24
a5e8e0f to
70671d9
Compare
fixed a merge conflict error
…ign change due to race conditions.
…tual completion it needed, causing a race where callers woke up before _completed_tasks was updated.
… thread finished writing to _completed_tasks, causingflaky failures , now waits on that instead of returning right after abort()/stop()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bluesky's RunEngine pauses via a RunEngineInterrupted, but blueapi was treating that exception like any other plan failure , pausing a task marked it as failed instead of leaving it resumable. On top of that, resume() and cancel_active_task() mutated RunEngine state directly from the calling thread, racing with the worker thread that owns it, which could leave a paused RunEngine stuck forever, apply a stale cancel, or double-set a task's outcome.