Skip to content

fix(qwen3next): keep big-page state cache pinned across shm serialization - #1422

Merged
hiworldwzj merged 1 commit into
ModelTC:mainfrom
sufubao:fix/linear-att-pinned-shm-getstate
Aug 4, 2026
Merged

fix(qwen3next): keep big-page state cache pinned across shm serialization#1422
hiworldwzj merged 1 commit into
ModelTC:mainfrom
sufubao:fix/linear-att-pinned-shm-getstate

Conversation

@sufubao

@sufubao sufubao commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

问题

PD / 多进程(write_to_shm 共享 mem_manager)场景下,linear attention 模型(qwen3next)开启 cpu cache 后,碎页 offload 时崩溃:

ValueError: Pointer argument (at 1) cannot be accessed from Triton (cpu tensor?)
  File ".../linear_att_cpu_cache_copy.py", in copy_linear_att_state_to_linear_att_state
  File ".../operator/linear_att.py", in offload_gpu_kv_to_cpu_cache

根因

linear_att_big_page_buffersLayerCache 里以 pin_memory=TruecudaHostAlloc)分配,此时 Triton kernel 可以直接访问该 pinned host 内存(已验证 pinned tensor 可被 Triton 读写)。

write_to_shm 通过 ForkingPickler.dumps(self) 序列化 Qwen3NextMemManager 时,torch 会把其 CPU tensor 的 storage 原地迁移到共享内存(shm mmap)。这一迁移把本进程持有的大页 state cache 从 pinned 退化成了普通 unpinned 的 shm。之后碎页 offload 调用 copy_linear_att_state_to_linear_att_state(一个 Triton kernel)携带该指针启动,Triton 拒绝非 pinned 的 CPU 指针,抛出上述错误。

关键点:失败的不是「CPU 指针喂给 GPU kernel」,而是 pinned 属性在 shm 序列化过程中被丢失。单进程(不走 write_to_shm)不会触发。

修复

重写 __getstate__,序列化时把 linear_att_big_page_buffers 置为 None,避免 torch 迁移它的 storage,从而保住本进程的 pinned 属性。

def __getstate__(self):
    state = self.__dict__.copy()
    state["linear_att_big_page_buffers"] = None
    return state

为什么安全

跨进程消费方(pd trans 进程 / dp prompt cache fetch)并不使用 cpu 侧的 linear_att_big_page_buffers,序列化时剔除不会影响它们;而本进程在反序列化后仍持有原始 pinned buffer(__init__ 时分配,未经过 pickle 重建),pinned 属性得以保留。

验证

  • 单独抽象 pinned / unpinned / cuda tensor 喂给 Triton kernel 的最小探针:pinned<->pinned 与 pinned->cuda 均通过,仅 unpinned CPU 被拒——确认根因是 pin 退化。
  • 在含本修复的分支上以 PD 拓扑 + --enable_cpu_cache + linear attention 模型跑碎页流量,prefill 节点不再崩溃;upstream/main 同配置必现崩溃。

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

…tion

write_to_shm serializes this object with ForkingPickler so other
processes can share it. During that dump, torch migrates each CPU
tensor's storage in-place into shared memory, which degrades the
pinned (cudaHostAlloc) big-page state cache to unpinned shm in the
local process; the linear attention small-page copy Triton kernel then
rejects the pointer with:

  ValueError: Pointer argument cannot be accessed from Triton (cpu tensor?)

Temporarily drop linear_att_big_page_buffers around the super() call so
the dumper never sees it. The local pinned allocation is preserved, and
cross-process consumers (pd trans / dp prompt cache fetch), which do not
use the CPU-side big-page state cache, are unaffected.
@sufubao
sufubao force-pushed the fix/linear-att-pinned-shm-getstate branch from c6d1fb1 to 148c9a2 Compare August 4, 2026 08:53
@hiworldwzj
hiworldwzj merged commit 6c87700 into ModelTC:main Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants