Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

Y Combinator

Summary

This YC Paper Club special session focuses on the evolution of hardware and software co-design in the era of LLMs. Experts from Stanford and the industry discuss the critical transition from general-purpose data centers to highly specialized chips, kernels, and heterogeneous architectures. The conversation covers breakthroughs in multi-GPU optimization, the metrics of intelligence efficiency, and the potential for running AI locally on personal hardware to optimize compute and energy costs.

Chapters

  1. 0:44Data Center Specialization
  2. 0:46Training vs Inference needs
  3. 4:21The role of batch size
  4. 7:22Multi-GPU Kernel Design
  5. 8:30Overcoming networking bottlenecks
  6. 14:19Parallel Kittens principles
  7. 21:31Intelligence Per Watt
  8. 23:48The shift to local compute
  9. 26:45Efficiency gains in local hardware
  10. 31:09AI in Kernel Development
  11. 34:31The KernelBot ecosystem
  12. 39:00Combating reward hacking
  13. 47:21Heterogeneous Infrastructure
  14. 47:36Phase-based design
  15. 55:28SRAM machine advantages
  16. 1:04:35GPU-Accelerated Simulation
  17. 1:05:51Batch simulators
  18. 1:07:50Applying ECS patterns

Topics covered

  • Chip Specialization
  • Kernel Optimization

+4 more topics (full content when you follow the channel)

Transcript excerpt

[music] Welcome to uh YC Mountain View special edition of YC Paper Club. I'm thinking about some of the feedback was like kind of making it theme related. So today it's the YC kernel and chip club. So the trend that I see in lab with Stu and John that we talk about all the time is this like specialization that's happening. Uh we saw it with um TPU8 the I and the T version zebra fish and sunfish. Is that what it's called? It's like to make an ASIC you need such activation energy. Um and now and there wasn't enough demand for there to be such a split. But we just see that at the chip level there's going to be massive specialization. The specs that you would need for a training data center are going to be very different than the specs for a uh inference data center. You don't need any bandwidth in and out for a training data center. You can literally send a spaceship to the sun, come back w…