LLM Inference Benchmarking Dashboard

Real-time TTFT · TPOT · ITL · E2EL benchmarking for vLLM

The four metrics that actually tell you if your LLM API is healthy — live, as requests happen.

---

The four metrics that matter

Most teams monitor their LLM API with generic HTTP metrics (response time, error rate). Those tell you something is wrong. These four tell you what is wrong and where in the inference pipeline the problem is.

Metric	Full name	What it measures	When it degrades
TTFT	Time to First Token	How long users wait before seeing any output	Prefill queue backed up; model loading; GPU contention
TPOT	Time Per Output Token	Speed of the generation stream	GPU compute-bound; context too long; batch too large
ITL	Inter-Token Latency	Consistency of the token stream	Memory bandwidth; KV cache pressure; batch size variance
E2EL	End-to-End Latency	Total request time from send to last token	Combination of TTFT + (tokens × TPOT)

---

Dashboard layout

┌─────────────────────────────────────────────────────┐
│  TTFT P50/P95/P99    │  TPOT P50/P95/P99            │
│  Live line chart     │  Live line chart              │
├──────────────────────┼──────────────────────────────┤
│  ITL distribution    │  E2EL histogram               │
│  Animated bar chart  │  Percentile breakdown         │
├──────────────────────┴──────────────────────────────┤
│  Token throughput (prompt + generation tokens/sec)   │
│  Queue depth (num_requests_running / waiting)        │
├─────────────────────────────────────────────────────┤
│  GPU utilization %   │  GPU memory used/free         │
│  DCGM_FI_DEV_GPU_UTIL│  DCGM_FI_DEV_FB_USED         │
└─────────────────────────────────────────────────────┘

---

Metrics source

All metrics are scraped from vLLM's /metrics Prometheus endpoint and NVIDIA DCGM:

vLLM metrics (via /metrics)
├── vllm:time_to_first_token_seconds_bucket    → TTFT histogram
├── vllm:time_per_output_token_seconds_bucket  → TPOT histogram
├── vllm:inter_token_latency_seconds_bucket    → ITL histogram
├── vllm:e2e_request_latency_seconds_bucket    → E2EL histogram
├── vllm:generation_tokens_total               → generation throughput
├── vllm:prompt_tokens_total                   → prompt throughput
├── vllm:num_requests_running                  → active requests
└── vllm:num_requests_waiting                  → queue depth

NVIDIA DCGM (via dcgm-exporter)
├── DCGM_FI_DEV_GPU_UTIL      → GPU utilization %
├── DCGM_FI_DEV_FB_USED       → GPU memory used (MiB)
├── DCGM_FI_DEV_FB_FREE       → GPU memory free (MiB)
└── DCGM_FI_DEV_GPU_TEMP      → GPU temperature (°C)

---

How to read the dashboard

TTFT spiking while TPOT is stable → prefill bottleneck. Too many long prompts queuing up. Reduce --max-num-seqs or scale out engine replicas.

TPOT degrading while TTFT is stable → generation bottleneck. GPU compute-bound during decoding. Check GPU utilization — if < 80%, look at batch size and --max-num-seqs.

ITL variance high → inconsistent generation speed. Usually caused by KV cache pressure (check vllm:gpu_cache_usage_perc) or competing requests with very different context lengths.

E2EL growing linearly with requests → queue saturation. num_requests_waiting will confirm. HPA should scale out — check if it's hitting maxReplicas.

---

Quick start

git clone https://github.com/ArchanaChetan07/LLM-Inference-Benchmarking-Dashboard
cd LLM-Inference-Benchmarking-Dashboard

# Configure your vLLM endpoint
export VLLM_METRICS_URL=http://localhost:8000/metrics
export PROMETHEUS_URL=http://localhost:9090

# Start backend + dashboard
docker compose up -d

# Open dashboard
open http://localhost:3001

For full GPU metrics (DCGM), deploy NVIDIA DCGM Exporter:

helm install dcgm-exporter nvidia/dcgm-exporter -n monitoring

---

Project structure

LLM-Inference-Benchmarking-Dashboard/
├── dashboard/          # HTML/JS live dashboard
├── backend/            # Prometheus metrics scraper + API
├── dashboards/         # Grafana dashboard JSON
├── configs/            # Prometheus + DCGM config
├── tests/              # pytest suite
├── docker-compose.yml  # Full stack (dashboard + backend + Prometheus)
└── requirements.txt

---

Part of the vLLM Observability Ecosystem

Project	What it does
LLM Benchmarking Dashboard ← you are here	Live TTFT/TPOT/ITL/E2EL charts · DCGM GPU metrics · inference diagnostics
AI Inference Observability Platform	FastAPI proxy · TTFT/TBT/E2E in every response · Prometheus · Grafana · 48 tests
KubeInfer	Production K8s deployment · queue-depth HPA · GitOps · 12 alert rules
KV Cache Profiler	Real-time GPU KV cache hit rate · eviction · memory pressure
AI Infrastructure Copilot	Conversational assistant for GPU capacity planning and K8s config

---

License

MIT

---

Archana Suresh Patil — ML Platform & MLOps Engineer · Sunnyvale, CA
LinkedIn · GitHub · Open to full-time · No sponsorship needed

⭐ Star this repo if it helps your LLM monitoring stack.