OSCAR

Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR captures Q/K/V activations on a small calibration set, estimates attention-aware K/V covariance structures offline, and derives per-layer rotations + clipping thresholds that align KV quantization with the directions attention actually consumes. By storing the bulk of the KV cache in INT2 while retaining only a small BF16 sink and recent window, OSCAR reduces KV-cache memory by approximately 8× compared with BF16. Under the same memory budget, our attention kernel enables up to 7× higher throughput at large batch sizes, and also accelerates batch-size-1 decoding by up to 3× by reducing memory-bandwidth overhead.

OSCAR is built directly into the open-source SGLang framework (main branch), llama.cpp (zhongzhu/llamacpp branch). We also provide a rotation zoo so users can download calibrated rotations directly instead of recomputing them.

🔥 Latest News

[Upcoming] OSCAR is testing MiniMax 3, GLM 5.2 and more models in long horizon agentic tasks (1M+ token context). Happy to see OSCAR used in the wild!
[2026-06-26] OSCAR is PRing into vLLM too, bringing INT2 KV cache support to another high-throughput serving stack.
[2026-06-07] OSCAR INT2 KV cache now runs 256K Gemma 4 12B under !!16GB!! and Qwen3 on the zhongzhu/llamacpp llama.cpp fork — ~8× smaller KV at near-f16 quality, with pre-built *-rot-kv.gguf on Hugging Face. RUN GEMMA 4 / QWEN3 with LONG CONTEXT on your LOCAL MAC!

MacBook M5 Max Gemma 4 12B OSCAR INT2 KV Local Run Video
[2026-06-05] OSCAR now runs its INT2 KV cache through a fused mixed-precision Flash-Attention kernel on Apple Metal in the zhongzhu/llamacpp llama.cpp fork, making long-context decode up to ~15× faster (near-BF16) at ~7× less KV memory. Try to RUN QWEN-3-32B with LONG CONTEXT in your LOCAL MAC!

MacBook M5 Max Qwen3-32B OSCAR INT2 KV Local Run Screenshot
[2026-06-04] OSCAR now supports Gemma 4 12B with SGLang INT2 KV cache on the zhongzhu/gemma4-12b branch.
[2026-05-31] OSCAR is now runnable on the zhongzhu/llamacpp branch of llama.cpp. Feedback and suggestions are very welcome!
[2026-05-23] OSCAR release the qwen3.5 4B, 35B-A3B, minimax-m2.7 229B preview results. You can use OSCAR for qwen3.5, minimax2.7 beta now! refer to branch zhongzhu/hybrid-model and set SGLANG_LLOYD_MAX=1.
[2026-05-18] Full release: paper, code, website, and RotationZoo are all live — runs out of the box on SGLang.

📖 Table of Contents

Main results

Qwen3.5-4B, Qwen3.5-35B-A3B, MiniMax 2.7 Preview

Qwen3.5

Model	Mode	GPQA (198)	Δ vs BF16
Qwen3.5-4B	baseline	75.25%	—
Qwen3.5-4B	OSCAR	74.75%	−0.50 pp
Qwen3.5-35B-A3B	baseline	80.30%	—
Qwen3.5-35B-A3B	OSCAR	82.32%	+2.02 pp

MiniMax2.7

Benchmark	BF16	OSCAR (LM_RATIO=1.16)	Δ
GPQA-Diamond	0.7828	0.7929	+1.0 pp
HumanEval	0.8817	0.8854	+0.4 pp
AIME 2025	0.7667	0.7667	0.0 pp
MATH500	0.9379	0.9279	−1.0 pp

Multi-Modal & LongBench

Use Rotation and Run Script in zhongzhu/VL branch. Baseline numbers taken from arxiv.org/abs/2605.19660 (Su et al., 2026).

OCRBench comparison

Method	Qwen3-VL-8B	Qwen3-VL-4B
16-bit Baseline	858	852
QuaRot (INT2)	722	773
RotateKV (INT2)	754	638
KIVI (INT2)	851	813
OTT (INT2)	850	831
TurboQuant+ (2.5-bit)	847	828
OSCAR (Lloyd-Max)	854	848

Omni-Modal LLMs: MMAU-Pro

Method (Qwen3-Omni-30B-A3B)	Open-ended	Good Rate	AIF
16-bit Baseline	66.2	27.8	87.4
KIVI (INT2)	65.8	27.0	78.2
OTT (INT2)	65.8	26.9	83.9
TurboQuant+ (2.5-bit)	66.6	27.0	79.3
OSCAR	67.4	33.8	89.7

LongBench-E comparison

Method	Qwen3-8B
16-bit Baseline	49.56
QuaRot (INT2)	40.13
RotateKV (INT2)	42.95
KIVI (INT2)	47.95
OTT (INT2)	48.21
TurboQuant+ (2.5-bit)	47.56
OSCAR	50.25

Setup. Each cell is the MEAN across 5 reasoning / coding benchmarks — GPQA, HumanEval, LiveCodeBench v6, AIME 25, MATH-500. To control single-seed variance, every benchmark is evaluated 5 times per (model, method) cell (3 times for GLM-4.7-FP8) and the per-seed scores are averaged before being averaged across benchmarks. TurboQuant rows are single-run (*) because its vLLM path is too slow for repeated 32K-context evaluations under our compute budget. All runs use 32K-token max generation length. BPE = effective bits per KV element at 128K context length. Higher is better; the BF16 row is the upper bound.

Method	BPE	Qwen3-4B Thinking	Qwen3-8B	Qwen3-32B	GLM-4.7-FP8 (358B)
BF16 (upper bound)	16.00	75.64	70.84	74.19	77.89
Saw-INT4	4.25	73.11	69.97	74.43	77.95
TurboQuant K3V3 *	3.25	31.74	56.88	71.99	78.15
QuaRot-INT2	2.25	1.40	10.14	7.90	75.14
Naive INT2	2.25	0.00	0.00	0.00	60.49
OSCAR (ours)	2.28	71.86	69.42	74.17	78.16
Gap of OSCAR vs BF16		−3.78	−1.42	−0.02	+0.27

Details for each task

Baseline notes — TurboQuant / QuaRot / Saw-INT4 / Naive INT2 configurations

For a fair comparison at a comparable bit-budget, TurboQuant results use vLLM's implementation (docs) modified so that all layers are quantized (no mixed precision); the original TurboQuant keeps the first, last, and selected middle layers in full precision. We run it in its K3V3 configuration (3-bit K, 3-bit V) to land near the OSCAR bit-budget.

QuaRot-INT2 is the standard 2-bit KV-quant recipe (data-free Hadamard rotation per layer). Saw-INT4 is an INT4 reference for context. Naive INT2 is per-token symmetric INT2 with no rotation.

* TurboQuant entries are single-run results because its vLLM path is too slow for repeated 32K-context evaluations under our compute budget.

Comparison with other INT2 KV-cache methods on AIME25

Most prior INT2 KV-cache methods do not provide framework-level support for efficient long-context generation, so 32K-generation evaluations are extremely slow and their papers do not report the full benchmark suite above. For this reason, we compare against the reported AIME25 setting where public numbers are available.

Method	BPE	Qwen3-8B	Qwen3-32B
Original BF16	16.00	66.00 +/- 7.33	72.59 +/- 7.41
KIVI-KV2	2.25	52.33 +/- 9.00	57.41 +/- 9.26
KIVI-KV2*	2.26	57.67 +/- 9.00	59.05 +/- 12.38
Kitty	2.39	59.67 +/- 10.33	69.26 +/- 9.26
OSCAR (ours)	2.38	66.67 +/- 3.33	74.00 +/- 5.48

OSCAR is the only INT2 method in this comparison that reaches BF16-level AIME25 accuracy at 32K generation while staying near a 2-bit KV-cache budget.

OSCAR is the only INT2 method that stays within a few pp of BF16 across every model. QuaRot-INT2 and naive INT2 collapse on reasoning + coding tasks. Saw-INT4 is a strong INT4 reference, but OSCAR matches or beats it **at roughly half the storage** (≈2 bits per KV element).

Layout

rotation/
  eval_oscar_gpqa.sh        generic GPQA eval driver
  eval_oscar_lcb.sh         generic LiveCodeBench v6 (128K) eval driver
  compute_kv_rotation.py    eigendecomposition + R·H·P_br composition
  _dump_compat/             sgl_kernel compat shim for dump
  <model>/
    save_qkv_<model>.sh     phase 1 — dump
    compute_rotation.sh     phase 2 — rotation
    eval_gpqa.sh            phase 3 — GPQA eval
    eval_lcb.sh             phase 3 — LCB v6 (128K) eval (where applicable)
    GPQA/
      seq<T>_prompt<N>_group<G>/
        qkv_dumps/          dump output
        rotations/          rotation .pt files
        _eval_gpqa_oscar/   eval results from this rotation
        _eval_lcb_v6_128k/  ...

sglang-research/            vendored sglang fork — INT2 KV eval
sglang-dump-qkv/            vendored older sglang fork — QKV dump (loaded via shim)
third_party/simple_evals/   git submodule — eval harness (needs git clone --recursive)

Setup

OSCAR vendors a snapshot of the upstream SGLang main fork (~May 2026; torch==2.9.1, transformers==5.3.0, flashinfer==0.6.7.post3). Both sglang-research/ (INT2 eval) and sglang-dump-qkv/ (QKV dump) ship in the repo — no separate SGLang install is needed.

Requirements

1 × H100 80 GB (for 4B/8B), 4 × H100 (for 32B / MiniMax-M2.7), 8 × H100 (for GLM-4.7-FP8)
CUDA 12.8 or 12.9 (nvcc on $PATH)
Python 3.12 + Conda
HuggingFace access for the relevant model weights

Clone

git clone --recursive https://github.com/FutureMLS-Lab/OSCAR.git
cd OSCAR

Conda env (single env, dump + eval)

OSCAR uses one conda env for both dump and eval. The dump-side sglang (vendored as sglang-dump-qkv/) was originally built against an older sgl_kernel; OSCAR ships a thin rotation/_dump_compat/ shim that stubs the dropped legacy symbols at import time and falls back to PyTorch for the runtime sampling kernels it references, so a single eval-side env suffices.

conda create -n oscar python=3.12 -y
conda activate oscar

# Eval-side sglang (editable so future patches stick)
pip install -e sglang-research/python

# CUDA-12.8/12.9 compatible flashinfer + sgl_kernel build
# (see https://github.com/sgl-project/sglang for matching wheels)

If nvcc and PyTorch's CUDA versions diverge (e.g. nvcc 12.6 but torch built for 12.8), the JIT kernels in flashinfer may fail to compile. Pin CUDA_HOME to the matching cuda-12.x directory before launching.

Quick start (Qwen3-8B example)

End-to-end on a single H100, ~20 minutes total.

cd OSCAR

# Phase 1 — dump Q/K/V (TP=1, default DUMP_KVCACHE_TOKENS=30000)
bash rotation/qwen3-8B/save_qkv_8b.sh
# → writes rotation/qwen3-8B/GPQA/seq30000_prompt<N>_group128/qkv_dumps/

# Phase 2 — fit the calibrated rotation
bash rotation/qwen3-8B/compute_rotation.sh
# → writes rotation/qwen3-8B/GPQA/seq30000_prompt<N>_group128/rotations/{k,v}_rotation_qqt_r_h_pbr.pt

# Phase 3 — GPQA eval against the rotation we just produced
ROT_DIR=rotation/qwen3-8B/GPQA/seq30000_prompt<N>_group128/rotations \
  bash rotation/qwen3-8B/eval_gpqa.sh
# → writes results to rotation/qwen3-8B/GPQA/seq30000_prompt<N>_group128/_eval_gpqa_oscar/

Pick the actual seq...prompt..._group... tag printed by phase 1, or:

ROT_DIR=$(ls -1d rotation/qwen3-8B/GPQA/seq*_prompt*_group*/rotations | tail -1) \
  bash rotation/qwen3-8B/eval_gpqa.sh

Model support

Where each model runs today. main = this branch with --kv-cache-dtype int2 (full-attention INT2 path); other rows point to feature branches.

Model	Backend / branch	Status
Qwen3-4B-Thinking-2507, Qwen3-8B, Qwen3-32B	SGLang `main`	✅ supported (paper results)
GLM-4.7-FP8 (358B)	SGLang `main`	✅ supported (paper results)
MiniMax-M2.7	SGLang `zhongzhu/hybrid-model` (`SGLANG_LLOYD_MAX=1`)	🧪 preview
Qwen3.5 (4B, 35B-A3B)	SGLang `zhongzhu/hybrid-model` (`SGLANG_LLOYD_MAX=1`)	🧪 preview (hybrid linear-attn)
GLM-5.1	SGLang `zhongzhu/glm-mla`	🧪 experimental (MLA latent)
Qwen3-VL (4B, 8B)	SGLang `zhongzhu/VL`	🧪 preview
Qwen3-32B, Qwen3-4B-Thinking, Gemma-4-12B	llama.cpp `zhongzhu/llamacpp` + GGUF	✅ supported (Mac / Metal)

INT2 on main targets full-attention models. MLA models (GLM-5.1 / DeepSeek-style) and hybrid linear-attention models (Qwen3.5 GatedDeltaNet) are not on main yet — use the branches above.

All configured models

Calibration-pipeline folders included on this branch (rotation/<model>/):

Folder	HF model	TP (dump)	TP (eval)	Notes
`rotation/qwen3-4B-thinking-2507/`	`Qwen/Qwen3-4B-Thinking-2507`	1	1	thinking model
`rotation/qwen3-8B/`	`Qwen/Qwen3-8B`	1	1
`rotation/qwen3-32B/`	`Qwen/Qwen3-32B`	2-4	4
`rotation/GLM-4.7/`	`zai-org/GLM-4.7-FP8`	8	8	FP8 weights, 92 layers

MiniMax-M2.7 / Qwen3.5 calibration scripts live on zhongzhu/hybrid-model; pre-fit rotations for several models are available on the RotationZoo.

How the rotation is fit (spectral covariance)

For each transformer layer, given calibration (Q, K, V) activations, OSCAR estimates two attention-aware covariance matrices and uses their eigenspectra to derive rotations:

K covariance (qqt) — average attention-query covariance seen by K: Σ_K = (1/H_kv) · Σ_h Q_h^T Q_h / n_tokens (GQA-aware: query heads grouped under the matching KV head)
V covariance (sst) — score-weighted V-side covariance: Σ_V = (1/H_kv) · Σ_h V_h^T diag(w_h) V_h / n_tokens where w_h[t] = K_h[t] · (Q^T Q) · K_h[t]^T is the per-token attention-score weight derived from K and the Q covariance
torch.linalg.eigh(Σ) → orthogonal eigenvectors R plus the eigenvalues (used for ordering, not for scaling)
Composition r_h_pbr: R_loaded = R · H_d · P_br
- H_d — head-dim Hadamard
- P_br — bit-reversal permutation, sorted by eigenvalue magnitude; this interleaves high-variance directions evenly across quant groups so no single group concentrates outliers

Saved as fp32 per-layer (head_dim, head_dim) orthogonal matrices in <calib_dir>/rotations/{k,v}_rotation_qqt_r_h_pbr.pt.

Serving with the rotation

The eval driver eval_oscar_gpqa.sh and eval_oscar_lcb.sh set everything for you. The underlying sglang server flags are:

SGLANG_ENABLE_MIXED_KV_WINDOWS=1 \
SGLANG_OSCAR_K_ROTATION_PATH=.../k_rotation_qqt_r_h_pbr.pt \
SGLANG_OSCAR_V_ROTATION_PATH=.../v_rotation_sst_r_h_pbr.pt \
SGLANG_OSCAR_K_CLIP_RATIO=0.96 \
SGLANG_OSCAR_V_CLIP_RATIO=0.92 \
SGLANG_OSCAR_ABSORB_V_ROTATION=1 \
SGLANG_MIXED_KV_PREFIX_TOKENS=64 \
SGLANG_MIXED_KV_RECENT_TOKENS=256 \
SGLANG_MIXED_KV_HP_MAX_SPLITS=8 \
SGLANG_MIXED_KV_HP_DTYPE=bfloat16 \
SGLANG_MIXED_KV_SCALE_DTYPE=float32 \
python -m sglang.launch_server \
  --model-path <model> \
  --tensor-parallel-size <tp> \
  --kv-cache-dtype int2 \
  --kv-cache-quant-group-size 128 \
  --prefill-attention-backend fa3 \
  --decode-attention-backend triton \
  --trust-remote-code

Sink (PREFIX_TOKENS) and recent window (RECENT_TOKENS) stay in BF16; the rest of the cache is INT2-quantized into 128-element groups along head-dim.

Skip calibration — use a pre-fit rotation from RotationZoo

To serve without running phases 1–2, download a calibrated rotation from the RotationZoo and point the env vars at it:

huggingface-cli download Zhongzhu/OSCAR-RotationZoo --include "Qwen3-8B/*" --local-dir rotzoo
ROT=$(ls -1d rotzoo/Qwen3-8B/seq*_prompt*_group128 | tail -1)
export SGLANG_OSCAR_K_ROTATION_PATH=$ROT/k_rotation_qqt_r_h_pbr.pt
export SGLANG_OSCAR_V_ROTATION_PATH=$ROT/v_rotation_sst_r_h_pbr.pt

Calibration knobs

Override per bash rotation/<model>/save_qkv_<model>.sh ENV=val:

Env	Default	Effect
`DUMP_KVCACHE_TOKENS`	30000	Total token budget for calibration
`GROUP_SIZE`	128	KV quant group size, encoded in output dir name
`DATASET`	GPQA	Calibration dataset name
`MODEL`	per-model HF id	HuggingFace model id
`TP_SIZE`	per-model	Tensor parallel size for dump
`GPU`	per-model	CUDA_VISIBLE_DEVICES
`HF_HOME`	`/shared/huggingface`	HF cache (set to `$HOME/.cache/huggingface` on a fresh machine)

Troubleshooting

Garbled / mixed-language output in long-context agent sessions. Update to the latest main. Older checkouts (before the mixed-KV slot-accounting fix) could free KV slots still referenced by other in-flight requests under concurrency, corrupting reads.
--kv-cache-dtype int2 only supports full-attention models. MLA models (GLM-5.1 / DeepSeek-style) and hybrid linear-attention models (Qwen3.5 GatedDeltaNet) are not supported on main yet — see Model support.
Hybrid / preview models (Qwen3.5, MiniMax-M2.7). Use the zhongzhu/hybrid-model branch with SGLANG_LLOYD_MAX=1.
Running locally on a Mac. Use the zhongzhu/llamacpp branch or the pre-built *-rot-kv.gguf files on Hugging Face.

Citation

@misc{zhou2026oscarofflinespectralcovarianceaware,
      title={OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization},
      author={Zhongzhu Zhou and Donglin Zhuang and Jisen Li and Ziyan Chen and Shuaiwen Leon Song and Ben Athiwaratkun and Xiaoxia Wu},
      year={2026},
      eprint={2605.17757},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.17757},
}

License & acknowledgements

Released under the MIT License.
Built on top of sglang.

Name		Name	Last commit message	Last commit date
Latest commit History 40 Commits
materials		materials
rotation		rotation
sglang-dump-qkv		sglang-dump-qkv
sglang-research		sglang-research
third_party		third_party
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

OSCAR

Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

🔥 Latest News

📖 Table of Contents

Main results

Layout

Setup

Requirements

Clone

Conda env (single env, dump + eval)

Quick start (Qwen3-8B example)

Model support

All configured models

How the rotation is fit (spectral covariance)

Serving with the rotation

Skip calibration — use a pre-fit rotation from RotationZoo

Calibration knobs

Troubleshooting

Citation

License & acknowledgements

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

OSCAR

Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

🔥 Latest News

📖 Table of Contents

Main results

Layout

Setup

Requirements

Clone

Conda env (single env, dump + eval)

Quick start (Qwen3-8B example)

Model support

All configured models

How the rotation is fit (spectral covariance)

Serving with the rotation

Skip calibration — use a pre-fit rotation from RotationZoo

Calibration knobs

Troubleshooting

Citation

License & acknowledgements

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages