Let's reproduce GPT-2 (124M)

Andrej Karpathy

Resumen

This video is a hands-on, deep-dive technical guide on reproducing the GPT-2 (124M) model from scratch using PyTorch. The expert walks through the entire pipeline: loading original model weights, building the Transformer architecture step-by-step (embedding layers, self-attention, MLP blocks), implementing optimization techniques like AdamW with weight decay, and integrating high-performance practices such as BF16 mixed precision, Torch Compile, and Flash Attention. Finally, the analysis covers data scaling, training at scale using Distributed Data Parallel (DDP) on multi-GPU setups, and benchmarking with standard tasks like HellaSwag.

Capítulos

  1. 0:00Introducción y objetivos
  2. 41:40Arquitectura y forward pass
  3. 1:23:20Carga de pesos y parámetros
  4. 2:05:00Optimización y rendimiento en GPU
  5. 2:46:40Entrenamiento con múltiples GPUs
  6. 3:28:20Evaluación HellaSwag y conclusiones

Temas tratados

  • Model Architecture
  • Weight Loading and Porting

+4 temas más (contenido completo al seguir el canal)

Extracto de la transcripción

hi everyone so today we are going to be continuing our Zero to Hero series and in particular today we are going to reproduce the gpt2 model the 124 million version of it so when openi released gpt2 this was 2019 and they released it with this blog post on top of that they released this paper and on top of that they released this code on GitHub so open a/ gpt2 now when we talk about reproducing gpt2 we have to be careful because in particular in this video we're going to be reproducing the 124 million parameter model so the thing to realize is that there's always a miniseries when these are releases are made so there are the gpt2 miniseries made up of models at different sizes and usually the biggest model is called the gpt2 but basically the reason we do that is because you can put the model sizes on the x-axis of plots like this and on the Y AIS you put a lot of uh Downstream metrics th…