SIGSEGV in continuous batching when a streaming client disconnects mid-generation (block_manager.hpp:633 assertion, GPU, 2026.2.1)

[lusoris]

Summary

A client that hard-closes its HTTP connection during a streaming
/v3/chat/completions generation reliably crashes the whole OVMS process within
seconds on Intel Arc GPUs. All other in-flight generations in the batch die with
it. Reproduced on demand; also observed 43× in 24 h under an agent fleet whose
callers time out and disconnect.

Environment

• Image: docker.io/openvino/model_server:2026.2.1-gpu (stock, unmodified)
• GPU: Intel Arc Pro B60 (xe driver) and Arc B580 — both affected, VM passthrough
• Model: Qwen3.6-27B INT4 OpenVINO IR (also seen with a 9B on the second card)
• Serving args:
  --task=text_generation --target_device=GPU --plugin_config='{"ENABLE_CPU_PINNING":false}' --tool_parser qwen3coder --enable_tool_guided_generation true --reasoning_parser qwen3 --kv_cache_precision u8 --cache_size 4 --enable_prefix_caching true --cache_interval_multiplier 64 --max_num_seqs 16 --max_num_batched_tokens 4096

Reproducer (deterministic)

1. POST a streaming chat completion ("stream": true, long generation).
2. Read a few SSE chunks.
3. Hard-close the client socket mid-stream (socket.close(), no graceful shutdown).
4. Within ~90 s the server logs the assertion below and the process exits.

Observed on the very first attempt; kernel gained exactly one segfault entry.

Logs

[llm_executor][error][llm_executor.hpp:132] Error occurred in LLM executor:
Check 'm_block_table.count(seq_id) > 0' failed at
../../../../../repos/openvino.genai/src/cpp/src/continuous_batching/cache/block_manager.hpp:633

Kernel (55+ occurrences over days, all error 4 read faults):

ovms[…]: segfault at … ip …6f3 … error 4 in ovms
libstdc++.so.6.0.33[…] from mediapipe/NNNNN threads

Symbolized against the shipped binary (BuildID a1b35868566ed20139449c9674ff41b1104ec73c):
the faulting IP lands in spdlog::logger::sink_it_+0x43 — i.e. the process dies
while logging, consistent with the executor thread's error path racing the
request-drop path that has already freed the sequence's block table entry.

Impact

• --max_num_seqs 16 means one disconnected client kills up to 15 innocent
  concurrent generations (All requests: 3; Scheduled requests: 3 logged at death).
• On a GPU deployment the recompile/reload takes 1–6 minutes per crash.
• Any latency spike becomes self-amplifying: slow generations → client timeouts →
  disconnects → crash → colder cache → slower generations.

Notes

• The xe "CAT error → engine reset" events sometimes seen after the crash FOLLOW
  the segfault by 300–460 ms and are absent for many crashes — downstream cleanup,
  not the cause.
• Not OOM (memory.events oom_kill 0), not concurrency-proportional (a second
  card at 2.3× the concurrent load crashes 51× less per stream-second — the
  discriminating variable is the mid-stream disconnect rate).
• 2026.1 release notes mention cancellation of requests from disconnected clients
  (NPU); this looks like the GPU continuous-batching cancellation path freeing
  seq_id from m_block_table while the executor still steps the sequence.

[lusoris]

Correction to the reported trigger — it is not the client disconnect.

I originally attributed this to a mid-stream client abort. That was wrong, and I want to correct it rather than leave a misleading report.

I patched our proxy so a vanished client can no longer cancel the upstream generation (the response body is drained in the background instead of being closed early, and the upstream request context is decoupled from the client's). That patch demonstrably works — it logs client left mid-stream; draining upstream response so the generation completes — and the server still crashed with the identical assertion. My earlier "deterministic repro" was coincidental timing against a ~1.8 crashes/hour background rate.

The actual trigger is KV-cache exhaustion. With the child's own stdout captured, every crash is preceded by several seconds at 100% cache:

All requests: 3; Scheduled requests: 2; Cache type: static, cache usage: 100.0% of 3.9 GB
All requests: 3; Scheduled requests: 2; Cache type: static, cache usage: 100.0% of 3.9 GB
[error] Check 'm_block_table.count(seq_id) > 0' failed at
        continuous_batching/cache/block_manager.hpp:633

Conditions: --cache_size 4 (static) with --max_num_seqs 16 on a GPU where the model weights leave only ~3.6 GB free. Two concurrent long sequences already fill the cache, so the third request forces preemption — and the block manager then asserts on a sequence whose blocks it has already freed.

So the reproducer is simply sustained concurrent long generations against an under-sized static KV cache, no disconnect required. That also explains why a second card running the same image and flags at higher concurrency but with a comfortable cache/weights ratio crashes ~51x less per stream-second.

The remaining upstream question stands: exhausting the KV cache should degrade (queue or preempt cleanly), not abort the process and take every other batched generation with it.

Our mitigation is to size --max_num_seqs to the measured cache rather than the copied default.

[mzegla]

@apaniukov , @popovaan - this looks like an issue on the GenAI Continuous Batching layer. Could you take a look?

[lusoris]

Cross-reference from downstream. The llama-swap proxy that fronts OVMS in this deployment hits the same crash and tracks it at mostlygeek/llama-swap#980. I opened a client-side mitigation there — mostlygeek/llama-swap#981 (surviveClientAbort): it forwards the upstream request with context.WithoutCancel(...) and drains the client body under a bound, so a client that hangs up mid-stream no longer propagates cancellation into the OVMS /v3 generation. That keeps the batch alive in practice — but it's a workaround around this crash, not a fix for it.

The llama-swap maintainer (@mostlygeek) places the root cause here:

This sounds like a bug in OpenVino. OpenVino should be handling client disconnects gracefully and not discard a whole batch when it happens.

Which matches the diagnosis above: the continuous-batching cancellation path frees the sequence's m_block_table entry while the executor still steps it, so a single mid-stream disconnect faults the whole batch instead of dropping just that one sequence. The correct behavior is to cancel only the disconnected sequence and leave the rest of the batch running — never assert/segfault.

Happy to validate a candidate fix against the deterministic reproducer above on the Arc B60/B580.