Why Inference Matters 12x More Than Training | YC Paper Club

Y Combinator

Summary

This YC Paper Club session explores why AI inference has become the critical bottleneck in scaling machine learning systems, surpassing training in cost and complexity. Experts from Together AI, Ollama, Luminal, and SGLang discuss how optimizations like KV cache management, kernel innovation, compiler-level search, and model routing are essential to serving LLMs profitably at scale. The session highlights that efficiency in inference, rather than just training power, is currently the primary driver of the AI economic landscape.

Chapters

  1. 0:30The Strategic Importance of Inference
  2. 0:44Inference as the main bottleneck
  3. 0:48The cost of goods sold (COGS)
  4. 2:10Training vs. Inference scale
  5. 4:46Performance Metrics and Hardware Utilization
  6. 6:36KV Cache and Memory Management
  7. 8:34Optimizing Model and Kernel Layers
  8. 16:44Routing and Model Aggregation
  9. 1:05:07Equality Saturation in Compilers

Topics covered

  • Inference Economics
  • Arithmetic Intensity

+4 more topics (full content when you follow the channel)

Transcript excerpt

[music] How's everyone doing today? Woo! All right, it's inference night. Come on, you got to get excited. Okay, I didn't really know how to uh represent an inference in a pictorial, so I did Halloween. So, I made it uh spookier as my 3-year-old would say, pookie. Um this is my wife over here. Okay, so why inference? Who cares about inference? Um well, I think we do a lot of inference investments at YC. We look at this a ton. Um I used to run an AI company where I would train and we would do a lot of inference and inference was always the bottleneck. It's so hard to do inference well at scale and without losing your shirt on cost of goods sold, uh a term called COGS as some people in software are unaware what that word means. Um and so far in just the last 10 years, it's produced 220 billion dollars of market cap um and we have two really cool companies Sorry, we have about three four re…