Building makemore Part 5: Building a WaveNet

Andrej Karpathy

Resumen

This lecture continues the 'makemore' series by moving from a simple multi-layer perceptron to a hierarchical architecture inspired by the WaveNet paper. The process involves refactoring the codebase to use PyTorch-like modules and containers, fixing a bug in the BatchNorm layer to handle three-dimensional inputs correctly, and experimenting with deeper, tree-like structures to fuse information more effectively. The instructor demonstrates the development process of deep learning, from prototyping in Jupyter notebooks to validating model performance and scaling architecture, ultimately improving the character-level language model's predictive capabilities.

Capítulos

  1. 0:00Introducción y objetivos del modelo
  2. 5:54Configuración del Batch Normalization
  3. 12:00Optimización del flujo de datos
  4. 22:00Implementación de modelos modulares
  5. 32:00Arquitectura y capas convolucionales
  6. 42:00Gestión de formas y tensores
  7. 52:00Refinamiento y evaluación final

Temas tratados

  • Hierarchical Architecture
  • WaveNet Concepts

+4 temas más (contenido completo al seguir el canal)

Extracto de la transcripción

hi everyone today we are continuing our implementation of make more our favorite character level language model now you'll notice that the background behind me is different that's because I am in Kyoto and it is awesome so I'm in a hotel room here now over the last few lectures we've built up to this architecture that is a multi-layer perceptron character level language model so we see that it receives three previous characters and tries to predict the fourth character in a sequence using a very simple multi perceptron using one hidden layer of neurons with 10ational neuralities so we'd like to do now in this lecture is I'd like to complexify this architecture in particular we would like to take more characters in a sequence as an input not just three and in addition to that we don't just want to feed them all into a single hidden layer because that squashes too much information too quic…