fix(ds4): correct layer-split HC handoff and per-device state

[Graffioh]

Summary

Fixes correctness and device-isolation issues in the DeepSeek4 HC layer-split path:

• hand off the updated full HC state between local shards
• scope HC weights and cached decode graphs to each shard
• keep independent CUDA/HIP HC scratch per GPU device, sized for the requested shapes (shared buffers used to stay at whatever dims the first call allocated)
• select the correct device at layer-range shard entry and hybrid step dispatch
• add coverage for per-device HC scratch, scratch shape capacity, HC device selection, and the stale HC boundary-state guard

Note: deepseek4_step_hybrid is unchanged — the hybrid branch only gains
the device pin at dispatch; its HC runtime stays on the thread-local
ds4_hybrid_runtime.

Implementation

Local layer split now follows the full HC flow:

token embeddings -> shard 0 -> full HC state -> shard 1 -> ... -> final shard

Runtime caches are owned by each shard and keyed by weights, context, backend, device, layer range, and output ownership. HC scratch is stored in independently locked slots indexed by CUDA/HIP device.

Follow-up

The opt-in GPU F16 HC fn mirror for quantized ROCmFPX fn weights (DFLASH_DS4_ROCMFPX_HC_GPU) was split out into #546, stacked on this PR, including the deepseek4_cuda_hc_upload_f16 / deepseek4_cuda_hc_free primitives.

Validation

• git diff --check and C++ syntax checks passed
• at an earlier revision (before the review fixes were folded in): Release CUDA H200 builds passed for test_deepseek4_unit and dflash_server; single-shard and virtual two-shard [0,22) / [22,43) runs on cuda:0 completed sequential requests without HC, CUDA, crash, or abort errors, validating HC handoff and per-shard cache reuse — re-run on the current head pending
• real multi-device device 0 -> device 1 -> device 0 validation on lucebox3/lucebox5 is still required
• a barrier-based concurrent scratch test is still required before marking ready

[davide221]

We plan to run two AMD GPUs in the same HIP process. g_scratch is currently shared across devices, so the second shard may reuse scratch memory allocated on the first GPU. Please keep one scratch buffer per HIP device, set the correct device in HC worker threads

[Graffioh]

(non-blocking) idk if needed or can be removed

[cubic-dev-ai]

1 issue found across 11 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_hc_cuda.cu">

<violation number="1" location="server/src/deepseek4/deepseek4_hc_cuda.cu:97">
P3: Concurrent HC calls remain unverified: the per-device test runs device workers sequentially, so it cannot validate simultaneous slot creation and use. Add a barrier-based two-device invocation before treating per-device scratch isolation as covered.</violation>
</file>

[cubic-dev-ai]

P3: Concurrent HC calls remain unverified: the per-device test runs device workers sequentially, so it cannot validate simultaneous slot creation and use. Add a barrier-based two-device invocation before treating per-device scratch isolation as covered.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_hc_cuda.cu, line 97:

<comment>Concurrent HC calls remain unverified: the per-device test runs device workers sequentially, so it cannot validate simultaneous slot creation and use. Add a barrier-based two-device invocation before treating per-device scratch isolation as covered.</comment>

<file context>
@@ -62,8 +81,38 @@ struct HcCudaScratch {
+// were created. Keep one independently locked slot per logical device so local
+// shards cannot reuse another device's scratch pointers.
+std::mutex g_scratch_slots_mutex;
+std::vector<std::unique_ptr<HcScratchSlot>> g_scratch_slots;
+
+HcScratchSlot * current_scratch_slot() {
</file context>

[Graffioh]

non-blocking for this PR to be merged

how do we handle concurrency tests? in integration tests? @davide221

[davide221]

We don’t currently support concurrent generation requests (support coming in August). So non blocking for now