On-device greedy sampling (-Dllama.deviceSample): GPU argmax, keep logits device-side (~500x less D2H) - #134

Merged

stratika merged 1 commit into

beehive-lab:mainfrom

mikepapadim:feat/on-device-sampling

Jul 27, 2026

Commits on Jul 14, 2026

Add on-device greedy sampling (-Dllama.deviceSample): GPU argmax over the logits appends to the FP16 logits graph so only the sampled token id (1 int) crosses to the host instead of the full vocab …
mikepapadim
committed