Titans: Learning to Memorize at Test Time (Paper Analysis)

Yannic Kilcher

Resumen

This video provides a critical analysis of the 'Titans' paper by Google Research, which proposes a new architecture that allows language models to memorize information at test time. The presenter explains how the model tries to overcome traditional transformer context window limitations by using a neural network as a dynamic memory that updates during inference. While the presenter praises the direction of the research and the necessity of test-time learning, they express skepticism regarding some of the paper's marketing, arguing that several 'novel' concepts are actually reinterpretations of existing techniques like gradient descent or basic parameter tuning.

Capítulos

  1. 0:00Análisis de Titans
  2. 2:06Limitaciones de los Transformers
  3. 8:00Funcionamiento de la atención
  4. 11:18Transformers lineales y kernels
  5. 17:00Arquitectura de memoria en Titans
  6. 22:12Entrenamiento durante el test
  7. 27:30Crítica sobre la terminología
  8. 31:30Conclusión y reflexiones finales

Temas tratados

  • Context Window Limitations
  • Historical Memory Approaches

+5 temas más (contenido completo al seguir el canal)

Extracto de la transcripción

[snorts] Hello. Today we're going to look at Titans learning to memorize at test time. This is a paper by Google research and has been pushed as part of their Nurips publications. So there's a blog post about it and people posted on social media about it and um I have to admit I just fell for the marketing. So here we are. I got interested in this paper and it is a good paper. uh it so it proposes a architecture of a um model u so think of like let's say a language model right that learns to memorize at test time so it can go beyond the current context you can go over a very very long piece of text for example you can slice that into parts and you can go over these parts and you can use this memory to connect to remember stuff from earlier parts and carry them to the next part thereby overcoming the context window limitations uh that are typically associated with something like transform…