A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

An Efficient and Versatile Inference Engine for Distributed LLM Serving

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

An AI inference engine from scratch. Like nanoGPT, but for serving.

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.

High-Performance LLM Inference Engine with PagedAttention & Continuous Batching in Rust

Fork of OpenAI and Anthropic compatible server for Apple Silicon. Native MLX backend, 500+ tok/s. Run LLMs and vision-language models with continuous batching, MCP tool calling, and multimodal support.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

A minimal LLM inference server with a real paged KV-cache, continuous batching and speculative decoding, by sarmalinux

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

OpenAI-compatible server with continuous batching for MLX on Apple Silicon

OpenAI-compatible LLM serving engine built from scratch on Apple Silicon. Continuous batching, paged KV-cache, prefix caching, INT8/INT4 quantization, Prometheus/Grafana observability. Benchmarked against llama.cpp and HuggingFace baselines.

Three LLM serving backends for GPT-2-124M built from scratch: naive serial, static batching, and continuous batching with paged KV-cache. Continuous+paged achieves 2.91 req/s with 0 failures vs 2.51 req/s and 6 failures for serial under mixed-length traffic at concurrency=20.

LLM inference optimization experiments: speculative decoding, continuous batching, KV-cache, quantization

High-performance discrete-event simulator (C++20/Python) for modeling agentic LLM traffic, KV cache dynamics, Prefill-Decode Disaggregation (PDD), and scheduling policies. Features roofline model analysis, K-Means request clustering, and a real-time web dashboard.