Papers

Exploring interesting problems (e.g. algorithms on simulated systems).

Machine Learning

• Language Models
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Emergence Abilities of Large Language Models

Systems

• Fault Tolerance
  • Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models with Adaptive Expert Placement
  • Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
• Compilers
  • Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• Job Scheduling
  • Starburst: A Cost-aware Scheduler for Hybrid Cloud
  • Efficient Strong Scaling Through Burst Parallel Training
• ML Scheduling
  • Efficient Memory Management for Large Language Model Serving with PagedAttention
  • Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• Pipeline Parallelism
  • GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
• Model Architecture
  • LoRA: Low-rank Adaptation Of Large Language Models