Xuan Han

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Reading Papers with Claude Code: Build First, Read Later Most research papers are hard to internalize by reading alone. You can follow the equations and still walk away unsure how the system actually behaves. But lately, I’ve been using Claude Code to flip the process: instead of starting with proofs, I start by building. The approach is straightforward. 1. First, extract the core algorithms from the paper, usually the pseudocode blocks or decision logic.  2. Then, with Claude Code, implement a toy version that runs on CPU with lightweight components. Large models become simple probabilistic stand-ins, learned rewards become rule-based checks, and large benchmarks become small synthetic tasks. As long as the interfaces match, the algorithm’s structure stays intact. The key is observability. Every candidate, score, and decision is printed or logged. You can see why the algorithm accepts one path and rejects another. This turns abstract objectives into something concrete and debuggable, and it quickly reveals which parts of the paper actually matter. Only after the toy version works do I go back to the paper. At that point, the math explains behavior I’ve already seen, rather than the other way around. You’re no longer trusting the paper, you’re verifying it. I tried this recently with a paper by Cemri et al., using Claude Code to build a simplified, fully inspectable implementation to understand the core mechanism. Even at toy scale, the central ideas became obvious in a way they never did from reading alone. Massive thanks to the authors. Hope you found this helpful! Any other paper implementation tips? Please leave a comment below. Notebook: https://lnkd.in/gVg9-Wgz Paper: https://lnkd.in/gKqFPRPN #ResearchPapers #ClaudeCode #LearningByDoing #DataScience #MachineLearning #Pseudocode #Debugging #Observability #AIResearch

1,330

34 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

New paper "Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation" with Google DeepMind colleagues Don Metzler and Zhen Qin is now available on arXiv: https://lnkd.in/d3A7AXAJ 🎓 The paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. 📊 We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. 🤔 Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content.  🗺️ We offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.

116

It’s interesting to see the shift around reinforcement learning — it’s no longer about whether RL is the right approach, but how we can train it more efficiently and at scale. I’m expecting to see a lot more domain-specific applications pop up, both in public and private (enterprise) spaces. https://lnkd.in/gjc3RdcP

18

1 Comment

Among other things, my team has been focusing on AI research for agents. We just blogged about a recent project: the Test-Time Diffusion Deep Researcher (TTD-DR). This framework uses a Deep Research agent to draft and revise its own drafts using high-quality retrieved information. This approach achieves new state-of-the-art results in writing long-form research reports and completing complex reasoning tasks. The work achieves better results than any other deep research agent we compared to. It also shows that self-evaluation and self-improvement are must-have components of a successful agent. https://lnkd.in/gBZpVMQy

297

8 Comments

Exciting news! We have open-sourced the 7B sigmoid attention model (and its softmax baseline)! In Sigmoid attention, we replace the softmax operation with sigmoid (which is a lot faster). We confirm that with hybrid-norm, we can scale it up to 7B parameters and get equal performance vs the softmax baseline. Now you can get your hands on these models, and even their intermediate checkpoints. Both are trained with deterministic data, so at each intermediate step, they have seen the exact same data — perfect for further research. See our paper for the results, or directly run the notebook that shows how to do inference on the 7B models using AXLearn: https://lnkd.in/eEWYzC8U

287

9 Comments