Skip to content

cuda: direct-I/O fallback observability -- counters, always-on disable log, report line - #689

Open
nexus-cw wants to merge 1 commit into
antirez:mainfrom
nexus-cw:directio-observability
Open

cuda: direct-I/O fallback observability -- counters, always-on disable log, report line#689
nexus-cw wants to merge 1 commit into
antirez:mainfrom
nexus-cw:directio-observability

Conversation

@nexus-cw

@nexus-cw nexus-cw commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Addresses #687.

An EINVAL/EFAULT/ENOTSUP/EOPNOTSUPP on the O_DIRECT model read path permanently disables direct I/O for the process with only a DS4_CUDA_WEIGHT_CACHE_VERBOSE-gated stderr line as evidence -- the streaming fast path can silently degrade to buffered I/O forever. This makes the state and history observable, with no behavior change on the I/O path itself:

  • Counters (ds4_cuda.cu): fallback events by errno class (einval / efault / enotsup / other-transient), engaged/disabled/unavailable state + the disabling errno, and widened-read stats (count + total wasted alignment bytes). Plain non-atomic counters on the single CUDA-dispatch thread, same convention as the surrounding code; readers may see a beat-stale value, fine for monitoring.
  • Always-on log on the permanent-disable event (a permanent behavior change should not be verbose-only). The engaged/unavailable logs stay verbose-gated as before.
  • Surfaces: ds4_gpu.h accessors (ds4_gpu_direct_io_state / _disable_errno / _counters) and a self-report line appended to the existing --memory-report output.

Evidence from a GB10 streaming deployment (DeepSeek V4 Flash, --ssd-streaming) running this instrumentation:

ds4: CUDA direct I/O: state=engaged disable_errno=0 fallbacks einval=0
efault=0 enotsup=0 other=0 widened_reads=69166 widen_wasted=270.179 MiB
avg_waste=4096.0 B/read

i.e. exactly one 4 KB alignment page of waste per widened read (~0.03% of a typical expert fetch) -- and a silent downgrade to buffered I/O now reports itself.

Build: make cpu clean (x86_64), ds4_cuda.o + ds4.o compile clean with nvcc (arm64 GB10). Companion proposal: #688 / its PR.

🤖 Generated with Claude Code

An EINVAL/EFAULT/ENOTSUP on the O_DIRECT model read path permanently
disables direct I/O for the process, previously with only a
DS4_CUDA_WEIGHT_CACHE_VERBOSE stderr line as evidence: on any box or
filesystem where the fallback trips, the streaming fast path silently
degrades to buffered I/O forever and nothing reports it.

Track fallback events by errno class, the permanent-disable state and
errno, and widened-read stats (count + total wasted alignment bytes).
The permanent-disable event now always logs. The state is exposed
through ds4_gpu.h accessors and appended to the --memory-report output.

Measured on a GB10 streaming deployment: the engaged path shows ~70k
widened reads at exactly 4096 B waste each (~0.03% of a typical expert
fetch), confirming the widening overhead is negligible -- and making a
silent downgrade to buffered I/O immediately visible.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants