Why Inference Matters 12x More Than Training | YC Paper Club

Resumen

This YC Paper Club session explores why AI inference has become the critical bottleneck in scaling machine learning systems, surpassing training in cost and complexity. Experts from Together AI, Ollama, Luminal, and SGLang discuss how optimizations like KV cache management, kernel innovation, compiler-level search, and model routing are essential to serving LLMs profitably at scale. The session highlights that efficiency in inference, rather than just training power, is currently the primary driver of the AI economic landscape.

Temas tratados

• Inference Economics
• Arithmetic Intensity

+4 temas más (contenido completo al seguir el canal)

Extracto de la transcripción

[music] How's everyone doing today? Woo! All right, it's inference night. Come on, you got to get excited. Okay, I didn't really know how to uh represent an inference in a pictorial, so I did Halloween. So, I made it uh spookier as my 3-year-old would say, pookie. Um this is my wife over here. Okay, so why inference? Who cares about inference? Um well, I think we do a lot of inference investments at YC. We look at this a ton. Um I used to run an AI company where I would train and we would do a lot of inference and inference was always the bottleneck. It's so hard to do inference well at scale and without losing your shirt on cost of goods sold, uh a term called COGS as some people in software are unaware what that word means. Um and so far in just the last 10 years, it's produced 220 billion dollars of market cap um and we have two really cool companies Sorry, we have about three four re…