Let's build the GPT Tokenizer

Andrej Karpathy

Resumen

This video provides a deep dive into tokenization, the essential yet often overlooked process of converting text into numerical sequences for Large Language Models. Andrej Karpathy explains how character-level tokenization is insufficient for state-of-the-art models, leading to the use of Byte Pair Encoding (BPE). He demonstrates the technical pitfalls, such as 'foot guns' related to spelling, arithmetic, and non-English performance, while showing how to implement a tokenizer from scratch. The video also covers industry-standard tools like OpenAI's Tiktoken and Google's SentencePiece, explaining their specific configurations, trade-offs, and why tokenization is critical for both model performance and efficiency.

Capítulos

  1. 0:00Introducción a la tokenización
  2. 8:00Funcionamiento de la tokenización
  3. 20:00Limitaciones de GPT-2 y codificación
  4. 40:00Algoritmo Byte Pair Encoding
  5. 1:00:00Entrenamiento y merges
  6. 1:30:00Patrones Regex y lógica BPE
  7. 1:50:00Integración con Llama 2 y Tiktoken
  8. 2:00:00Desafíos y aplicaciones avanzadas

Temas tratados

  • Tokenization Basics
  • Byte Pair Encoding (BPE)

+5 temas más (contenido completo al seguir el canal)

Extracto de la transcripción

hi everyone so in this video I'd like us to cover the process of tokenization in large language models now you see here that I have a set face and that's because uh tokenization is my least favorite part of working with large language models but unfortunately it is necessary to understand in some detail because it it is fairly hairy gnarly and there's a lot of hidden foot guns to be aware of and a lot of oddness with large language models typically traces back to tokenization so what is tokenization now in my previous video Let's Build GPT from scratch uh we actually already did tokenization but we did a very naive simple version of tokenization so when you go to the Google colab for that video uh you see here that we loaded our training set and our training set was this uh Shakespeare uh data set now in the beginning the Shakespeare data set is just a large string in Python it's just te…