Titans: Learning to Memorize at Test Time (Paper Analysis)

Yannic Kilcher

Summary

This video provides a critical analysis of the 'Titans' paper by Google Research, which proposes a new architecture that allows language models to memorize information at test time. The presenter explains how the model tries to overcome traditional transformer context window limitations by using a neural network as a dynamic memory that updates during inference. While the presenter praises the direction of the research and the necessity of test-time learning, they express skepticism regarding some of the paper's marketing, arguing that several 'novel' concepts are actually reinterpretations of existing techniques like gradient descent or basic parameter tuning.

Chapters

  1. 0:00Introduction to Titans Paper
  2. 1:00Context Window Limitations
  3. 4:00Memory Mechanisms in Transformers
  4. 8:00Linear Transformers and Kernels
  5. 14:00Memory as a Neural Network
  6. 20:00Training Memory with Surprise
  7. 26:00Persistent Memory and Parameters
  8. 30:00Final Thoughts on Memorization

Topics covered

  • Context Window Limitations
  • Historical Memory Approaches

+5 more topics (full content when you follow the channel)

Transcript excerpt

[snorts] Hello. Today we're going to look at Titans learning to memorize at test time. This is a paper by Google research and has been pushed as part of their Nurips publications. So there's a blog post about it and people posted on social media about it and um I have to admit I just fell for the marketing. So here we are. I got interested in this paper and it is a good paper. uh it so it proposes a architecture of a um model u so think of like let's say a language model right that learns to memorize at test time so it can go beyond the current context you can go over a very very long piece of text for example you can slice that into parts and you can go over these parts and you can use this memory to connect to remember stuff from earlier parts and carry them to the next part thereby overcoming the context window limitations uh that are typically associated with something like transform…