cuda: direct-I/O fallback observability -- counters, always-on disable log, report line - #689
Open
nexus-cw wants to merge 1 commit into
Open
cuda: direct-I/O fallback observability -- counters, always-on disable log, report line#689nexus-cw wants to merge 1 commit into
nexus-cw wants to merge 1 commit into
Conversation
An EINVAL/EFAULT/ENOTSUP on the O_DIRECT model read path permanently disables direct I/O for the process, previously with only a DS4_CUDA_WEIGHT_CACHE_VERBOSE stderr line as evidence: on any box or filesystem where the fallback trips, the streaming fast path silently degrades to buffered I/O forever and nothing reports it. Track fallback events by errno class, the permanent-disable state and errno, and widened-read stats (count + total wasted alignment bytes). The permanent-disable event now always logs. The state is exposed through ds4_gpu.h accessors and appended to the --memory-report output. Measured on a GB10 streaming deployment: the engaged path shows ~70k widened reads at exactly 4096 B waste each (~0.03% of a typical expert fetch), confirming the widening overhead is negligible -- and making a silent downgrade to buffered I/O immediately visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #687.
An
EINVAL/EFAULT/ENOTSUP/EOPNOTSUPPon the O_DIRECT model read path permanently disables direct I/O for the process with only aDS4_CUDA_WEIGHT_CACHE_VERBOSE-gated stderr line as evidence -- the streaming fast path can silently degrade to buffered I/O forever. This makes the state and history observable, with no behavior change on the I/O path itself:ds4_gpu.haccessors (ds4_gpu_direct_io_state/_disable_errno/_counters) and a self-report line appended to the existing--memory-reportoutput.Evidence from a GB10 streaming deployment (DeepSeek V4 Flash,
--ssd-streaming) running this instrumentation:i.e. exactly one 4 KB alignment page of waste per widened read (~0.03% of a typical expert fetch) -- and a silent downgrade to buffered I/O now reports itself.
Build:
make cpuclean (x86_64),ds4_cuda.o+ds4.ocompile clean with nvcc (arm64 GB10). Companion proposal: #688 / its PR.🤖 Generated with Claude Code