Building makemore Part 4: Becoming a Backprop Ninja

Andrej Karpathy

Summary

This lecture focuses on demystifying backpropagation by manually implementing the backward pass for a two-layer MLP with batch normalization, moving away from PyTorch's 'loss.backward()'. The author argues that backpropagation is a 'leaky abstraction' that requires deep understanding to debug effectively. By breaking down the neural network into individual atomic operations and tensors, the viewer gains a rigorous intuition for how gradients flow through layers like linear transformations, tanh nonlinearities, and batch normalization, ultimately resulting in a stronger understanding of deep learning internals.

Chapters

  1. 0:00Introduction to Manual Backpropagation
  2. 6:44Historical Context of Backpropagation
  3. 11:55Manual Backward Pass Strategy
  4. 20:18Derivative of Cross Entropy Loss
  5. 35:08Backpropagating Through Softmax
  6. 1:10:08Matrix Multiplication Backpropagation
  7. 1:29:01Batch Normalization Derivation
  8. 1:48:00Verifying Gradients and Final Training

Topics covered

  • The Leaky Abstraction of Backprop
  • Historical Perspective on Manual Gradients

+4 more topics (full content when you follow the channel)

Transcript excerpt

hi everyone so today we are once again continuing our implementation of make more now so far we've come up to here montalia perceptrons and our neural net looked like this and we were implementing this over the last few lectures now I'm sure everyone is very excited to go into recurring neural networks and all of their variants and how they work and the diagrams look cool and it's very exciting and interesting and we're going to get a better result but unfortunately I think we have to remain here for one more lecture and the reason for that is we've already trained this multilio perceptron right and we are getting pretty good loss and I think we have a pretty decent understanding of the architecture and how it works but the line of code here that I take an issue with is here lost up backward that is we are taking a pytorch auto grad and using it to calculate all of our gradients along th…