Skip to content

perf(ggml): reuse ROCmFP4 weights across q=4 columns#534

Draft
davide221 wants to merge 1 commit into
codex/deepseek4-dspark-draftfrom
codex/ds4-q4-weight-reuse
Draft

perf(ggml): reuse ROCmFP4 weights across q=4 columns#534
davide221 wants to merge 1 commit into
codex/deepseek4-dspark-draftfrom
codex/ds4-q4-weight-reuse

Conversation

@davide221

@davide221 davide221 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • decode each packed ROCmFP4_FAST weight fragment once for dense four-column MMVQ
  • reuse the decoded weights across the four q8 activation columns while preserving each column's original DP4A and floating-point accumulation order
  • select the specialization automatically only on gfx1151, with no new runtime setting

The dispatch is shape- and backend-based rather than DS4-specific, so other
models using the same quantized dense four-column path can reuse it. Routed
MUL_MAT_ID, AR (ncols_dst == 1), other widths, and other architectures keep
the existing kernels.

Performance

Measured on lucebox2 (gfx1151, ROCm 7.2.2), pinned to the same 2.9 GHz
performance/high state for both sides. The workload used fixed DSpark q=4,
LUCE_MMVQ_MAX_NCOLS=4, top-k 4, and the accepted ROCmFPX AR specializations.

Stable matched window: the first seven identical GSM+Math requests, totaling
896 generated tokens and 263 verification steps:

Control This PR Delta
Speculative decode 30.15 tok/s 31.51 tok/s +4.50%
Verification step 112.99 ms 108.13 ms -4.30%

The candidate's ten requests ranged from 28.7 to 35.0 tok/s. The control
encountered a host-wide latency stall on requests 9-10, so those requests are
not included in the comparison above.

AR remains unchanged at 26.88 tok/s, 5/5 GSM, and 583 output tokens.

Correctness

  • 76/76 traced q=4 verifier inputs and host logits are byte-identical to control
  • zero missing/extra trace entries and zero byte mismatches
  • AR produced the same aggregate throughput, score, and token count as control

Kernel checks

A full Release HIP/gfx1151 dflash_server build succeeds. The dedicated
specializations have no register spills, private segment, or dynamic stack:

Kernel SGPR VGPR
q=4 fused specialization 47 59
q=4 unfused specialization 28 46
existing q=4 fused kernel 50 67
existing q=4 unfused kernel 32 83

The existing generic q=4 kernel metadata is unchanged from control.

Stack

This is intentionally based on codex/deepseek4-dspark-draft / #496 because
that branch owns the fixed-q=4 DSpark verification path used for validation.

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant