Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
-
Updated
Feb 20, 2026 - Python
#
speculative-decoding
Here are 141 public repositories matching this topic...
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
retrieval chatbot rag habana large-language-model chatpdf llm-inference 4-bits speculative-decoding llm-cpu streamingllm intel-optimized-llamacpp neural-chat neural-chat-7b autoround gaudi3
-
Updated
Oct 8, 2024 - Python
Lucebox: LLM inference server built for speed for specific consumer hardware.
kernel cuda cuda-kernels nvidia-cuda luce rtx3090 llama-cpp local-ai qwen speculative-decoding dflash megakernel speculative-prefill pflash lucebox
-
Updated
May 17, 2026 - C++
Large-scale LLM inference engine
-
Updated
May 8, 2026 - C++
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
audio eagle quantization diffusion vlm llm qwen speculative-decoding llm-compression hunyuan deepseek fp4 dflash
-
Updated
May 14, 2026 - Python
2.24x decode TPS increase On Qwen 3.6 27B @ temp 0.6 | Native MTP Speculative Decoding On Apple Silicon With No External Drafter.
metal mtp mlx inference-engine apple-silicon local-ai qwen speculative-decoding speculative-sampling openai-compatible qwen3-next anthropic-compatible native-mtp mtplx
-
Updated
May 17, 2026 - Python
scalable and robust tree-based speculative decoding algorithm
-
Updated
Jan 28, 2025 - Python
Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
-
Updated
Apr 13, 2026 - Python
DFlash & TurboQuant in llama.cpp with up to 3x faster generation and 7.5x more KV cache in same VRAM
inference quantization kv-cache llm llm-serving llama-cpp ggml llm-inference speculative-decoding dflash turboquant
-
Updated
May 17, 2026 - C++
Pure Rust Inference Engine
rust cuda transformers ssm mamba dgx openai-api llm-inference speculative-decoding gb10 nvfp4 dgx-spark
-
Updated
May 18, 2026 - Rust
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
-
Updated
Aug 31, 2024 - Python
LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving
machine-learning cuda inference pytorch transformer triton moe quantization language-model inference-engine kv-cache tensor-parallelism llm speculative-decoding pagedattention continuous-batching
-
Updated
Apr 24, 2026 - Python
REST: Retrieval-Based Speculative Decoding, NAACL 2024
-
Updated
Mar 5, 2026 - C
llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).
metal cpp vulkan cuda quantization mtp multimodal apple-silicon low-bit-quantization llama-cpp local-llm llm-inference gguf speculative-decoding kv-cache-compression qwen3 multi-token-prediction turboquant gemma4 nextn
-
Updated
May 14, 2026 - C++
Tree-based speculative decoding for Apple Silicon (MLX). ~10-15% faster than DFlash on code, ~1.5x over autoregressive. First MLX port with custom Metal kernels for hybrid model support.
-
Updated
Apr 15, 2026 - Python
LLM Inference on consumer devices
-
Updated
Mar 17, 2025 - Python
[ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation
-
Updated
May 19, 2025 - Python
Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.
-
Updated
Dec 2, 2024 - Python
A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation of large language models
nlp machine-learning awesome reinforcement-learning rl awesome-list knowledge-distillation model-compression post-training opd distillation self-distillation llm rlhf gkd llm-training speculative-decoding on-policy-distillation minillm llm-distillation
-
Updated
May 18, 2026
[NeurIPS'23] Speculative Decoding with Big Little Decoder
-
Updated
Feb 6, 2024 - Python
Improve this page
Add a description, image, and links to the speculative-decoding topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the speculative-decoding topic, visit your repo's landing page and select "manage topics."