Skip to content

Stop ImageWatcher per-file processing from blocking the event loop - #798

Open
thusser wants to merge 1 commit into
developfrom
fix/imagewatcher-event-loop-blocking
Open

Stop ImageWatcher per-file processing from blocking the event loop#798
thusser wants to merge 1 commit into
developfrom
fix/imagewatcher-event-loop-blocking

Conversation

@thusser

@thusser thusser commented Aug 22, 2026

Copy link
Copy Markdown
Member

Implements specs/plans/2026-08-20-imagewatcher-event-loop-blocking.md (Status stays proposed — the plan's own checklist flips it to implemented only once landed and verified at the site over one night).

Problem

ImageWatcher._worker processes every watched file with synchronous, never-yielding operations directly on the module's single event loop:

  1. fits.HDUList.fromstring(data) — a synchronous, CPU-bound astropy parse per file.
  2. LocalFile.read / write / close / remove / exists / find — declared async def but execute plain blocking syscalls with no executor (listdir already offloads; the others never got the treatment).
  3. LocalFile.__init__ opens the file synchronously, and VirtualFileSystem.open_file has no await point before that — so even with read/write/remove offloaded, a slow/network-mounted watch path could still stall the loop at open time.

Result: RPC replies, inotify handling, and comm keepalive freeze for seconds at a time (the 2026-08-20 MONET South incident, module.py:205/207 "Event loop stalled for 0.81s" / 2.81 s total).

Changes

  • pyobs/modules/image/imagewatcher.py: offload the FITS parse via asyncio.to_thread in _worker; add docstring notes that process_extra/cleanup_extra run on the event loop and must not block.
  • pyobs/vfs/localfile.py: route read, write, close, remove, find, exists through the default executor (module-level _find_sync/_remove_sync helpers; exists calls os.path.exists directly), mirroring the existing listdir pattern. Move the open()/makedirs work out of __init__ into an overridden __aenter__ via a _open_sync helper — path validation still raises synchronously at construction; every call site already uses async with, so nothing observes the open moving one step later. All VFS consumers inherit the fix.
  • Tests: heartbeat-style loop-responsiveness tests in tests/modules/image/test_imagewatcher.py (slow FITS parse) and tests/vfs/test_localfile.py (slow read/write/open via a fake slow fd and a monkeypatched _open_sync). Verified they fail on the old code and pass on the new.

Verification

  • pytest tests/modules/image/ tests/vfs/ → 34 passed, 1 skipped (pre-existing SSH-disabled skip).
  • Full suite: 1558 passed; the 14 failures are sandbox-environment issues unrelated to this change (read-only /opt/pyobs/storage, read-only ~/.config/sunpy, missing pyobs console script in the venv).
  • ruff clean, black clean, pyrefly 0 errors on touched files.

Out of scope (per plan)

Archive upload latency / queue backlog (already async, never freezes the loop) and step 0's live site measurement are flagged for follow-up in the plan.

ImageWatcher._worker parses every watched file with astropy's synchronous
fits.HDUList.fromstring and reads/writes/removes files through LocalFile's
async-declared-but-synchronous methods directly on the module's event loop,
freezing RPC handling, inotify processing, and comm keepalive for seconds at
a time -- the 2026-08-20 "Event loop stalled" incident on MONET South.

Offload the FITS parse via asyncio.to_thread, and route all blocking
LocalFile I/O (open/read/write/close/remove/find/exists) through the default
executor, mirroring the existing listdir pattern. The actual open()/makedirs
moves out of __init__ into an overridden __aenter__ (routed through a
module-level _open_sync helper), closing the network-mounted-watch-path gap;
path validation still raises synchronously at construction time. Every VFS
consumer inherits the fix, not just the imagewatcher.

Add heartbeat-style tests proving the event loop stays responsive during a
slow FITS parse and slow LocalFile I/O.

See specs/plans/2026-08-20-imagewatcher-event-loop-blocking.md for the full
design.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant