Serve token_embd.weight from the host mapping under DS4_EMBD_MMAP (default OFF) - #690
Open
nexus-cw wants to merge 1 commit into
Open
Serve token_embd.weight from the host mapping under DS4_EMBD_MMAP (default OFF)#690nexus-cw wants to merge 1 commit into
nexus-cw wants to merge 1 commit into
Conversation
token_embd.weight is accessed one row (~8-16 KB) per decoded token but held fully resident on the device (~1 GiB on DeepSeek V4 Flash Q8_0). With DS4_EMBD_MMAP=1 (default OFF) the startup span preparation leaves the table out of the resident device spans and the embed path resolves it to the read-only host model mapping instead, so row lookups are served at page-cache speed. Measured on a GB10 streaming deployment: -1010 MiB resident, warm decode within run-to-run noise (4.25/4.28/4.36 vs 4.19/4.23/4.33 t/s), no cold-prefill cost, worst case a one-time ~2.5 s transient after a full page-cache drop. Caveat: this relies on unified-memory ATS/HMM pageable access (GB10 / Grace class); discrete-GPU targets would need the cudaHostRegister'd mapping path for the same idea. The env gate defaults OFF precisely so non-unified targets are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #688.
token_embd.weightis accessed one row (~8-16 KB) per decoded token but held fully resident (~1 GiB on DeepSeek V4 Flash Q8_0). WithDS4_EMBD_MMAP=1(default OFF):token_embd.weightout of the resident device spans (logged).cuda_model_range_ptrresolves thetoken_embdlabel to the read-only host model mapping instead of a resident copy, so row lookups are served at page-cache speed.Measured on GB10 (DeepSeek V4 Flash Q8_0, streaming, ctx 131072): -1010 MiB resident, warm decode within run-to-run noise (4.25/4.28/4.36 vs 4.19/4.23/4.33 t/s), no cold-prefill cost, worst case a one-time ~2.5 s first-token transient after a full page-cache drop, recovered by the next request. Full numbers in #688.
Caveat, stated plainly: this relies on unified-memory ATS/HMM pageable access (GB10/Grace class) -- the GPU reads the file-backed host pages directly. Discrete-GPU targets would need the
cudaHostRegister'd-mapping path thatcuda_model_range_ptralready has for the same idea; that is honest future work, and the env gate defaults OFF precisely so non-unified targets are unaffected.Build:
make cpuclean (x86_64),ds4_cuda.o+ds4.ocompile clean with nvcc (arm64 GB10). Companion observability work: #687 / its PR.🤖 Generated with Claude Code