mlsystems

Latency, diversity, and quality benchmark for autoregressive decoding strategies on RTX 2070: greedy, top-k, top-p, min-p, and beam search across GPT-2 models.

Analytical benchmark for speculative decoding at batch sizes 1-64. Finds the gamma crossover where speculative decoding becomes slower than greedy, and shows that batch size does not degrade performance when continuous refill is active.

Discrete-event simulation of multi-LoRA adapter serving strategies on a single GPU — comparing naive swap, hot-set preloading, and batch-by-adapter under variable VRAM pressure and arrival rates.

Comparing LLM serving under real ShareGPT traces vs synthetic Poisson workloads. Real distributions produce 42x worse P99 TTFT and earlier SLO violations — scheduling policies only differentiate under heavy-tailed real traffic.

Phase-by-phase benchmark of LLM instance cold start on RTX 2070: weight loading, PCIe transfer, CUDA init, KV cache allocation, and JIT warmup — with pre-warming savings and extrapolation to 7B+ models.

Discrete-event simulation of SLO-aware autoscaling for multi-instance LLM serving — comparing reactive, predictive, and conservative policies across workload patterns and startup delays.

Discrete-event simulation of LLM request routing across multiple serving instances: round-robin, least-load, prefix-aware, and hybrid cache-aware routing with queue-depth threshold sweep.

Simulating KV cache reuse across multi-turn LLM conversations using real ShareGPT traces. Compares sticky, least-load, hybrid, and cost-aware routing strategies for cache locality vs load balance.

A C++ Autograd Engine written without ANY frameworks that can train a pretty big GPT locally. Built to maximize performance.

Step-level simulation of chunked prefill under KV cache pressure. Block policy wastes 89.6% of prefill chunks while achieving the same throughput as upfront rejection. Chunk waste rate is the metric standard throughput metrics miss.

Benchmark comparing Flash-Decoding-style chunked KV-parallel attention against standard SDPA at decode time (q_len=1) on RTX 2070, measuring the crossover point where KV-parallelism outperforms fused attention kernels.

Benchmarking output length prediction quality and calibration for KV-cache-aware LLM serving scheduling. A noisy predictor with 1.5x safety margin captures 99.7% of oracle gains — calibration matters more than accuracy.

Comparing KV-cache-aware scheduling policies for LLM serving: greedy preempts 49% of requests under memory pressure while memory-first achieves 43% higher effective throughput with zero preemptions.