Building makemore Part 2: MLP

Andrej Karpathy

Summary

This video continues the 'makemore' series by moving from a simple bigram model to a more sophisticated Multi-Layer Perceptron (MLP) architecture. The instructor explains the limitations of bigram models, specifically how context windows cause the parameter count to explode exponentially, and introduces embeddings as a way to generalize knowledge across similar tokens. The lecture covers the step-by-step implementation of the MLP in PyTorch, including the construction of the dataset, the embedding lookup table, the hidden layer, and the use of mini-batch stochastic gradient descent. Finally, it demonstrates how to tune hyperparameters, track training progress with validation splits, and sample new names from the trained model.

Chapters

  1. 0:00Bigram and MLP Concepts
  2. 4:37Building the Neural Network
  3. 10:05Embedding Lookup Table
  4. 20:15Constructing the Model Layers
  5. 38:48Tensor Operations and View
  6. 51:15Loss Calculation and Optimization
  7. 1:11:12Mini-batch Training and Learning

Topics covered

  • MLP Architecture
  • Context Window Scaling

+6 more topics (full content when you follow the channel)

Transcript excerpt

hi everyone today we are continuing our implementation of makemore now in the last lecture we implemented the bigram language model and we implemented it both using counts and also using a super simple neural network that had a single linear layer now this is the jupyter notebook that we built out last lecture and we saw that the way we approached this is that we looked at only the single previous character and we predicted the distribution for the character that would go next in the sequence and we did that by taking counts and normalizing them into probabilities so that each row here sums to one now this is all well and good if you only have one character of previous context and this works and it's approachable the problem with this model of course is that the predictions from this model are not very good because you only take one character of context so the model didn't produce very n…