A Kubernetes controller for unified LLM inference orchestration, supporting both monolithic and prefill/decode (PD) disaggregated serving topologies.
FusionInfer provides a single InferenceService CRD that enables:
- Monolithic deployment: Single-pod inference handling full request lifecycle
- PD disaggregated deployment: Separate prefill and decode roles for better GPU utilization
- Multi-node deployment: Distributed inference across multiple nodes using tensor parallelism
- Gang scheduling: Atomic scheduling via Volcano PodGroup integration
- Intelligent routing: Gateway API integration with EPP (Endpoint Picker) for request scheduling
fusioninfer-demo.mp4
fusioninfer-multi-node.mp4
┌─────────────────────────────────────────────────────────────────┐
│ InferenceService CRD │
│ (roles: worker/prefiller/decoder, replicas, multinode) │
└─────────────────────────────────┬───────────────────────────────┘
│
┌───────────────────────────────┐
│ InferenceService Controller │
└─────────────┬─────────────────┘
│
┌─────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ PodGroup │ │ LeaderWorkerSet │ │ Router (EPP) │
│ (Volcano) │ │ (LWS) │ │ InferencePool │
│ │ │ │ │ HTTPRoute │
└───────────────┘ └─────────────────┘ └─────────────────┘
FusionInfer requires the following components:
1. LeaderWorkerSet (LWS) - For multi-node workload management
kubectl create -f https://github.com/kubernetes-sigs/lws/releases/download/v0.7.0/manifests.yamlReference: LWS Installation Guide | Releases
2. Volcano - For gang scheduling
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.13.1/installer/volcano-development.yamlReference: Volcano Installation Guide | Releases
3. Gateway API - For service routing
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yamlReference: Gateway API Installation Guide | Releases
4. Gateway API Inference Extension - For intelligent inference request routing
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.2.1/manifests.yamlReference: Inference Extension Docs | Releases
Set the Kgateway version and install the Kgateway CRDs:
KGTW_VERSION=v2.1.0
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crdsInstall Kgateway:
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=trueDeploy the Inference Gateway:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml# 1. Create a kind cluster (optional)
kind create cluster --name fusioninfer
# 2. Install FusionInfer CRDs
make install
# 3. Run the controller locally
make runapiVersion: fusioninfer.io/v1alpha1
kind: InferenceService
metadata:
name: qwen-inference
spec:
roles:
- name: router
componentType: router
strategy: prefix-cache
httproute:
parentRefs:
- name: inference-gateway
- name: inference
componentType: worker
replicas: 1
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:v0.11.0
args: ["--model", "Qwen/Qwen3-8B"]
resources:
limits:
nvidia.com/gpu: "1"# You can use minikube tunnel to assign IP address to an LoadBalancer Type Service
GATEWAY_IP=$(kubectl get gateway inference-gateway -o jsonpath='{.status.addresses[0].value}')
curl -X POST "http://${GATEWAY_IP}/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"messages": [
{"role": "user", "content": "Hello, how are you?"}
]
}'