feat(ds4): add DSpark speculative runtime + optimizations #496
Draft
davide221 wants to merge 10 commits into
Draft
feat(ds4): add DSpark speculative runtime + optimizations #496davide221 wants to merge 10 commits into
davide221 wants to merge 10 commits into
Conversation
howard0su
approved these changes
Jul 7, 2026
Ports the ROCmFP4/ROCmFPX ggml work (previously lucebox-ggml #36, on the old submodule pointer) into the vendored server/deps/llama.cpp tree. - ROCmFP4/ROCmFPX quant types + CPU reference conversions (ggml/rocmfp4, ggml/rocmfpx) and the ggml type-trait registrations. - CUDA/HIP dequant, copy, getrows, MMVQ vecdot, MMVF, unary and FA paths for the new types. - Fused DS4 hyper-connection op (GGML_OP_DS4_HC) with the register-resident sinkhorn kernel; inert unless emitted by the DS4 fused-decode path. - DS4 SwiGLU split op plumbing for the fused FFN matvec paths. Layered on current main, so the main-side ggml work is preserved (fp64 RoPE reduction, #497 RDNA MMQ tile, LUCE_MMQ_DP_MAX_NE1, MMVQ_MAX_MOE_BATCH_SIZE, fused dual set_rows, raw-span guard). Review fixes on top of #36: - ggml_ftype_to_ggml_type: handle the 11 new ROCmFPX ftypes (dominant-type mapping) so the enum switch is -Wswitch/-Werror clean. - FP6 MMVQ vecdot: pad qs[] to avoid a stack over-read of the last window (bit-identical; the over-read bits were already masked out).
The non-HIP fallback of rocmfp4_get_int_from_codebook_16 / rocmfp4_get_low_int_from_codebook_16 called get_int_from_table_16, which is defined in vecdotq.cuh. TUs that pull in this header without vecdotq.cuh (fattn-chunked.cu reaches it via the fattn dequant chain) failed to compile under nvcc: rocmfp4_hip_codebook.cuh: error: identifier "get_int_from_table_16" is undefined The HIP path never hit this (it uses __builtin_amdgcn_perm), so the ROCm CI and the Strix build stayed green while the sm_86 CUDA build broke. Fix: inline the generic table expander (the generic branch of get_int_from_table_16, verbatim) as a static helper in this header, so the fallback no longer depends on include order. Bit-identical; the HIP hot path is unchanged.
Server-side DeepSeek V4 Flash changes from #494, rebased onto the vendored ROCmFPX ggml tree (no submodule pointer). Byte-identical to the validated #494 server files. Correctness (default ON): - Token-by-token prefill by default; chunked prefill only fits inside the raw SWA ring, so long prompts / multi-turn degraded. Chunked stays available via DFLASH_DS4_CHUNKED_PREFILL for short-prompt benchmarking. - Clear the cache buffer at new-sequence prefill (kv_offset==0) so requests 2..N are byte-stable instead of pooling over leftover compressor state. - Route non-hybrid (all-hot) placement through the HC-complete layer-range path; deepseek4_step's non-hybrid branch is HC-less and generates garbage. - Key the cached decode-attn graph on the flush pattern; the old key collided once the compressed-KV ring filled, reusing a stale-topology graph. - Compressor decode graphs read state/comp-cache through the ggml_set_rows result tensors so the current-step writes are explicit graph dependencies. - Default DeepSeek4 chat prefix when the request has no system message, so the ROCmFPX "Src" GGUF stops behaving like a base model under bare prompts (explicit caller system prompts are preserved). Perf, byte-identical default: - Persistent HC matvec pool (row-split preserves per-row accumulation order) and f16c dot kernels with scalar-order adds. - Per-step decode scalar inputs uploaded in 2 tensor_sets instead of ~430. Opt-in, default OFF (documented as not bit-identical): - DFLASH_DS4_FUSED_DECODE single-graph decode with GGML_OP_DS4_HC. - DFLASH_DS4_FFN_RAW_MMID / DFLASH_DS4_FFN_FUSED_COMBINE accumulation reorder. - DFLASH_DS4_ROCMFPX_HC_GPU GPU HC pre-mix.
Contributor
Author
|
Validation update (2026-07-09): the converter/loader bridge in this PR was validated on lucebox2 with the converted DSpark drafter artifact. In the follow-on DeepSeek4 runtime worktree, DSpark loads successfully, prefill feature capture and generation are wired, the batched-verify causality bug is fixed, and causal batched output matched exact sequential verification in paired probes. Adaptive EWMA width and fused batched verify are working; fused verify reached approximately 20 tok/s (peak 20.1 tok/s), versus roughly 13–15 tok/s for the earlier dynamic causal path. Exact sequential verification measured 1.78–1.91 accepted candidates per 4-candidate step. These runtime changes are follow-on work in the hub-494 experiment worktree and are not claimed as already included in PR 496. |
Contributor
Author
|
Updated DSpark adaptive width in
Validation on lucebox2 / gfx1151 with the exact committed source:
The confidence policy therefore retains essentially all high-acceptance throughput while adapting verification width on harder prompts. |
Return confidence scores from the fused Markov graph in the existing token-id synchronization, then select q=2/q=3/q=4 from cumulative prefix confidence. Compatible DSpark artifacts enable the policy automatically; artifacts without a confidence head retain the existing EWMA controller. No new deployment configuration is introduced.\n\nValidated on gfx1151: GSM+Math 10/10 at 29.25 tok/s weighted, within 0.8% of fixed q=4, with adaptive behavior retained on low-acceptance prompts.
davide221
force-pushed
the
codex/deepseek4-dspark-draft
branch
from
July 10, 2026 20:27
1fc4f00 to
867c9b8
Compare
davide221
changed the title
davide221
changed the title
Remove the no-op single-wave MMVQ shared reduction, add opt-in fixed-K FP2/FP3 and partially-unrolled FP4FAST q=1 kernels, and use AMD v_perm_b32 for packed FP3 decode. A fully build-matched Lucebox3 A/B used ROCm 7.2.4, gfx1151, wave32, -O3, FA_ALL_QUANTS=ON, fused DS4 decode, expert top-k 4, and HC-GPU disabled. Five balanced GSM8K runs per binary improved mean AR throughput from 24.270 +/- 0.018 to 25.664 +/- 0.014 tok/s: +5.744% +/- 0.122% at 95% confidence. All ten measured runs remained 5/5 with exactly 583 output tokens. Generated fused/plain specializations report zero private segment, dynamic stack, SGPR spills, and VGPR spills.
This was referenced Jul 16, 2026
Closed
Closed
…odex/ds4-q4-weight-reuse
davide221
force-pushed
the
codex/ds4-rocmfpx-server
branch
from
July 17, 2026 05:33
e943f2a to
d5fae56
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Depends on #503, which depends on #502:
#502 ROCmFPX ggml -> #503 DeepSeek4 target -> #496 DSpark runtimeThis branch was rebuilt directly on #503 so it does not carry or conflict with older copies of the target and fused-decode changes.
Summary
DFLASH_DS4_SPECis enabledValidation
gfx1151;gfx1201DFLASH_DS4_FUSED_DECODE=1git diff --checkFollow-up
Remote DSpark execution and shared transport are isolated in #505.