A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

SGLang is a fast serving framework for large language models and vision language models.

Minimalist vLLM implementation in Rust

A framework for efficient model inference with omni-modality models

A high-throughput and memory-efficient inference and serving engine for LLMs

A Datacenter Scale Distributed Inference Serving Framework

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Restate is the platform for building resilient applications that tolerate all infrastructure faults w/o the need for a PhD.

The financial transactions database designed for mission critical safety and performance.

Apache Iggy: Hyper-Efficient Message Streaming at Laser Speed

A cloud native embedded storage engine built on object storage.

🚀2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platfor…

Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild

Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tensor…

NVIDIA Inference Xfer Library (NIXL)

LLM inference in C/C++

Blazingly fast LLM inference.

Distributed MoE in a Single Kernel [NeurIPS '25]

Nano vLLM

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

My solutions in C++ and Python for problems on NeetCode.io

Low latency inter-thread communication library in Rust inspired by the LMAX Disruptor.

Deep learning at the speed of light.

Official Problem Sets / Reference Kernels for the GPU MODE Leaderboard!

Perplexity open source garden for inference technology

The Modular Platform (includes MAX & Mojo)

Large Language Model Text Generation Inference