C. Antonio Sanchez

You're in a ML Inference Engineer interview at NVIDIA, and the interviewer asks: "Our team wants to switch from API to a self-hosted. Which serving framework and why?" Here's how you answer: Don't say: "vLLM is fastest" or "Ollama is easiest." Wrong framing. The real answer isn't about features - it's about matching serving philosophy to your constraints. Local prototype vs. production scale vs. complex workflows = completely different frameworks. NOTE - if you want to read this kinda content daily, consider subscribing my free newsletter - https://lnkd.in/g8ZJGsWj Most teams pick based on benchmarks, not bottlenecks Your serving choice matters less than your use case. PagedAttention isn't magic - it's just better memory management "Fastest" framework can be slowest for YOUR workload The infrastructure reality everyone misses: Single user prototype = Ollama/LLaMA.cpp wins Multi-user production = vLLM dominates AI agent workflows = SGLang is your only option Edge deployment = LLaMA.cpp or bust Architecture drives the decision, not features. "But what about performance?" Interviewer: "How do you handle 1000 concurrent users?" Performance without context is meaningless. > vLLM's PagedAttention gives you 24x throughput, but only if memory fragmentation was your bottleneck. > SGLang's RadixAttention only helps with cache reuse scenarios. By the way, just reading about inference frameworks won't help you land a job. I personally learnt a lot by doing stuff. If I were to do all over, I would rent a gpu from Hyperbolic for dirt cheap, and play with some open source model with different frameworks. The business framework that matters: > Rapid prototyping + Local dev = Ollama > Known high load + GPU budget = vLLM > Complex multi-step LLM workflows = SGLang > Constrained environments + Portability = LLaMA.cpp Match the tool to the business constraint. The evolution path most teams miss: > Start: Ollama (prototype fast) > Scale: vLLM (handle users) > Optimize: SGLang (complex workflows) > Edge: LLaMA.cpp (deploy anywhere) It's not replacement - it's graduation based on needs. The answer that gets you hired: "Framework choice isn't about better vs worse. It's about constraints vs capabilities. Ollama gives you speed-to-market. vLLM gives you user scale. SGLang gives you workflow complexity. Pick based on your biggest constraint, not your biggest benchmark." That's it for today. See ya tomorrow! And if you want to read more about these frameworks, checkout this blog: https://lnkd.in/eiH2sphA #llm #machinelearning #gpu #chatgpt #ai #agents

398

17 Comments

My colleague and friend Hartmut Neven once told me the best way to think about quantum computing was to imagine chopping an onion. You could do it the ‘traditional’ way: peeling the onion, later by layer, and chopping each layer in turn. That’s a classical computer.  But what if there was a different way? What if you could hold the whole onion and run your knife through it at once, getting the result but so much faster. That’s a quantum computer. Right now, ‘doing the whole onion' means quantum computer chips that can complete computations in under 5 minutes that would take today’s fastest supercomputers 10 septillion years. (To save you a Google: a septillion has 24 zeros ) When you combine computers of that speed and scale with AI, the possibilities are endless - from discovering new medicines quicker, to accelerating new types of energy like fusion. That’s the potential the Nobel judges recognised yesterday, when they awarded Google Quantum AI's Michel Devoret, his former Google colleague John Martinis, and Berkeley’s John Clarke the 2025 Nobel Prize for Physics for their pioneering discoveries in quantum. We already have their breakthroughs to thank for technologies we use every day, from mobile phones to cameras. Now they’re steering us towards the future with quantum - and it’s incredibly exciting to think that these are still just the early days. Huge congratulations to Michel, John and John! More in Harmut's blog: https://lnkd.in/dx5havXE (And glad to have the chance to revisit this photo with Hartmut, Demis and Eileen 🙂)

166

8 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,023

8 Comments