LMCache Lab

LMCache: reuse KV cache to speed up AI inference Long-context and multi-turn LLM apps often recompute the same tokens again and again. LMCache reduces that wasted work. Why this matters: - Lower time-to-first-token for long prompts and RAG - Better GPU utilization under repeated or similar requests How LMCache works: - Reuses KV cache for repeated text, not just prefixes - Stores KV cache on GPU, CPU, or disk - Allows cache reuse across different serving instances - Integrates with vLLM and SGLang for KV cache offloading - Works well for multi-turn chat and RAG workloads - It’s a practical way to treat KV cache as shared infrastructure instead of a per-request artifact. ♻️ Share it with anyone who’s running LLMs in production :) I share tutorials on how to build + improve AI apps and agents, on my newsletter 𝑨𝑰 𝑨𝒈𝒆𝒏𝒕 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈: https://lnkd.in/gaJTcZBR Link to repo: https://lnkd.in/ekp-uBjy #LLMs #RAG #GenAI

🔥 Interesting research conducted on top of LMCache on supporting agentic workloads!

Great time working with the AMD team!

Live from LMCache Lab office hours 🚀 We’re diving into the science of KV caching and how LMCache enables large-scale offloading to external storage — straight from the maintainers.

You don't need expensive GPU clusters to handle long-context LLMs anymore. The LMCache team just open-sourced a KV cache management system that stores and reuses caches across GPU, CPU, and disk: 👉 3-10× faster response times in multi-round QA and RAG 👉 Reduces GPU cycles through intelligent cache reuse 👉 Supports vLLM v1, SGLang, with CPU/Disk/P2P storage How it works: • Distributes cache across GPU, CPU DRAM, and local disk • Reuses KV caches across serving instances • Handles non-prefix text (not just prompt prefixes) • Install: pip install lmcache Relevant if you're running multi-round QA, RAG pipelines, or anything with reusable context.

We ran a tiny one-shot experiment from a one-shot SWE-bench task with Claude Code to study the Context Engineering & Reuse Pattern: • 92 LLM calls invoked • ~2M input tokens • 13 minutes runtime • 92% prefix reuse rate With prefix caching, this single task drops from $6.00 → $1.15 in input cost (≈ 81% savings) and dramatically reduces TTFT. This trace shows Claude Code is essentially a prefix-reuse machine with warm-up calls to prime the cache, parallel multi-agent system, and a ReAct-style execution loop — all optimized for KV cache reuse. Blog post: https://lnkd.in/g3bzMqcj Raw trace: https://lnkd.in/gBR4kjUH Trace visualizer: https://lnkd.in/g2RPSXpk If you care about context engineering, agent architecture, and KV-cache economics, this is a concrete, end-to-end look under the hood. You may paste the raw trace to the trace visualizer to peek more.

⚡ In cybersecurity, slow AI is as bad as no AI. When you’re building AI-first SOC copilots, agentic pentesting, identity risk engines, or AI triage bots, latency and cost quickly become the real attackers. That’s why ideas like LMCache – a KV cache layer / “Knowledge Delivery Network” for LLMs – are super interesting. --- 🔍 What LMCache does (in simple words) LMCache sits between your model and your infra and: Reuses KV caches (the internal attention states) across requests Avoids recomputing long, shared prefixes Speeds up generation and reduces GPU/CPU cost Plays nicely with inference engines like vLLM Think of it as Cloudflare, but for LLM context. --- 🛡️ Why this matters for Cybersecurity AI 💡 1. Faster SOC copilots & IR assistants Security workflows are long and multi-turn: > “Show me alerts on this asset → correlate with previous incidents → explain root cause → suggest actions.” LMCache lets you reuse all that shared context instead of recomputing it every time. Result: faster investigations, lower infra burn. 💡 2. Agentic pentesting that doesn’t feel laggy Your red-team / pentest agents often: Reuse the same recon info Iterate on similar payloads Call the model multiple times per target Caching the shared parts of prompts means your agent feels real-time, not “thinking…” for ages. 💡 3. More context for less cost Good cyber reasoning needs: Log snippets TI reports Past incident notes Asset relationships With LMCache, you can keep rich context in prompts and still make it affordable to run at scale across many analysts / agents. 💡 4. Scale to thousands of alerts & users As soon as you move from “demo” to “product”, the questions hit: Can we handle 10x more alerts? Can every analyst get an AI copilot? Can we run this on-prem with fixed GPUs? A caching layer gives you the throughput and cost profile needed for real enterprise deployments. 💡 5. Perfect fit for SLM + Agentic architectures For security, we increasingly prefer smaller, specialized models (SLMs) + agents over giant LLMs. LMCache makes those SLMs: Snappier Cheaper to run Better at handling repeated, structured security workflows --- 🧠 Bottom Line If you’re building Autonomous SOC, agentic pentesting, AI-IR, or identity risk engines, you don’t just need good models – you need fast, cached, cost-efficient inference. A layer like LMCache helps you: ✅ Cut latency in complex security workflows ✅ Reuse shared security context across calls ✅ Serve more analysts/agents on the same hardware ✅ Move from “cool PoC” to production-grade AI security platform Bigger models are optional. Fast, efficient, cached models are not.

😦 What if 99% of your requests could subsidize themselves with just 1% cache reuse? ✅ That's not a hypothetical. It's the actual break-even threshold for KV caching in LLM inference. Here's what most teams miss are missing: 📉 You're recomputing the same tokens over and over. Processing one NVIDIA annual report costs $3 in LLM inference. Now multiply that by thousands of users per second. We built the LMCache ROI Calculator to answer one critical question: "When does caching actually save money?" 🌟 The answer: at just 1.06% cache hit rate, you break even. That means if just 1 out of every 94 requests reuses cached data, your storage investment pays for itself. 🔗 Read the full ROI breakdown: https://lnkd.in/gGKU88XZ 🧮 Calculate your savings: https://lnkd.in/gkCwAmJy 🚀 Get $100 free GPU time on Tensormesh (no credit card required): https://lnkd.in/gmKa8chj #AI #LLM #MachineLearning #InfrastructureOptimization #CostOptimization