Long line deadlock fix - #241
Merged
hcallahan-lowrisc merged 2 commits intoAug 7, 2026
Merged
Conversation
AlexJones0
reviewed
Aug 6, 2026
AlexJones0
left a comment
AlexJones0
left a comment
Contributor
There was a problem hiding this comment.
Thanks - mostly looks good to me, I just have a couple of small suggestions.
| log.exception( | ||
| "Error while streaming subprocess output to log file for job '%s'.", | ||
| handle.spec.full_name, | ||
| ) |
Contributor
There was a problem hiding this comment.
Would recommend logging the exception here as well (except Exception as e, etc.)
But actually, I think a better fix (untested), would be to change the asyncio.gather(...) to use return_exceptions=False, to stop the exceptions being aggregated.
Contributor
Author
There was a problem hiding this comment.
log.exception prints the trace as well as the message. This is now checked as part of a new test as well:
$ python -m pytest "tests/test_runtime.py::TestLocalBackendStreaming::test_reader_error_is_logged_not_raised" -s -o log_cli=true --log-cli-level=ERROR
<...>
tests/test_runtime.py::TestLocalBackendStreaming::test_reader_error_is_logged_not_raised [E 260807 12:06:43 plugin:178] Error while streaming subprocess output to log file for job 'job'.
Traceback (most recent call last):
File "/home/harry/projects/dvsim/src/dvsim/runtime/local.py", line 109, in _log_from_pipe
handle.log_file.write(decoder.decode(chunk))
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/harry/projects/dvsim/tests/test_runtime.py", line 230, in write
raise OSError("simulated disk-full")
OSError: simulated disk-full
PASSED
…ines LocalRuntimeBackend._log_from_pipe drained subprocess stdout/stderr with `async for line in stream`, which calls StreamReader.readline(). readline() raises LimitOverrunError (surfaced as ValueError) once a single line exceeds the StreamReader limit — 64 KiB by default, as create_subprocess_exec was invoked with no `limit=`. A tool emitting a very long line without a newline (e.g. Jasper's cov-unr UNR report, observed at 116 398 chars) crashed the reader task. With no consumer, asyncio paused the transport, the OS pipe filled, the tool blocked writing to it, and `await process.wait()` hung forever. The legacy launcher wrote tool output straight to the log fd and could not deadlock this way, so the async rewrite was the regression. Read fixed-size chunks via StreamReader.read() instead, which imposes no line-length limit. An incremental UTF-8 decoder (surrogateescape) handles multibyte characters that straddle a chunk boundary. Harden the reader against a recurrence: - Broaden the `except` so an unexpected error is logged via log.exception rather than silently killing the reader task (which would re-deadlock the pipe), while still passing asyncio.CancelledError through untouched. - Pass an explicit `limit=` to create_subprocess_exec. Keep the two concerns as separate constants: SUBPROCESS_STREAM_LIMIT is the StreamReader/readline buffer cap (64 KiB, matching the asyncio default) and SUBPROCESS_READ_CHUNK_SIZE is the read() chunk size used in _log_from_pipe. Signed-off-by: Harry Callahan <hcallahan@lowrisc.org>
The existing runtime tests only exercised the backend registry and never launched a subprocess, so regressions in _log_from_pipe (empty logs, a NameError, or a re-introduced deadlock) passed CI unnoticed. Add TestLocalBackendStreaming: - streams stdout to the log and passes. The sentinel is emitted via chr() codes so it cannot appear in the "[Executing]" command preamble the monitor writes, i.e. the assertion only holds if real subprocess output was captured. - a single line far larger than the 64 KiB StreamReader limit is logged without deadlocking. Regression test for the original readline() LimitOverrunError bug; @timeout turns a re-regression into a failure rather than a hang. - a multibyte UTF-8 character split across a read-chunk boundary decodes correctly (incremental decoder). - an unexpected log-write error is caught and reported via log.exception rather than propagating (a dead reader would re-deadlock the pipe). - normal cancellation is silent (no error logged). Confirmed these fail on both the original readline deadlock and a bare-name NameError in the read loop. Signed-off-by: Harry Callahan <hcallahan@lowrisc.org>
hcallahan-lowrisc
force-pushed
the
long_line_deadlock_fix
branch
from
August 7, 2026 11:11
926b4ee to
f47e824
Compare
Contributor
Author
|
Addressed review comments and added 5 unit tests covering the failure mode / new implementation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #240
Commit messages describe the fixes, and 5 new unit tests are added in support of this.
Checklist
git commit -s), indicating acceptance of the CLA<type>[(<scope>)][!]: <description>)!or aBREAKING CHANGE:footer