Building makemore Part 5: Building a WaveNet

Andrej Karpathy

Summary

This lecture continues the 'makemore' series by moving from a simple multi-layer perceptron to a hierarchical architecture inspired by the WaveNet paper. The process involves refactoring the codebase to use PyTorch-like modules and containers, fixing a bug in the BatchNorm layer to handle three-dimensional inputs correctly, and experimenting with deeper, tree-like structures to fuse information more effectively. The instructor demonstrates the development process of deep learning, from prototyping in Jupyter notebooks to validating model performance and scaling architecture, ultimately improving the character-level language model's predictive capabilities.

Chapters

  1. 0:00Introduction to WaveNet architecture
  2. 4:02Module and layer building blocks
  3. 10:50Fixing graphs and batch normalization
  4. 21:43Transitioning to PyTorch Sequential
  5. 30:42Scaling up block size
  6. 38:52Matrix multiplication and tensor shapes
  7. 52:12Implementing flattened layer modules

Topics covered

  • Hierarchical Architecture
  • WaveNet Concepts

+4 more topics (full content when you follow the channel)

Transcript excerpt

hi everyone today we are continuing our implementation of make more our favorite character level language model now you'll notice that the background behind me is different that's because I am in Kyoto and it is awesome so I'm in a hotel room here now over the last few lectures we've built up to this architecture that is a multi-layer perceptron character level language model so we see that it receives three previous characters and tries to predict the fourth character in a sequence using a very simple multi perceptron using one hidden layer of neurons with 10ational neuralities so we'd like to do now in this lecture is I'd like to complexify this architecture in particular we would like to take more characters in a sequence as an input not just three and in addition to that we don't just want to feed them all into a single hidden layer because that squashes too much information too quic…