How to compress text to 1 bit per character | Compression is Intelligence Part 3

3Blue1Brown

Summary

Este vídeo explora cómo comprimir texto a niveles inferiores a un bit por carácter utilizando modelos de lenguaje. Se explica cómo la compresibilidad está intrínsecamente ligada a la predictibilidad, conectando la teoría de la información de Shannon con el funcionamiento de los modelos de inteligencia artificial actuales. A través de la codificación aritmética, el autor demuestra que un buen modelo de lenguaje no solo predice, sino que es fundamentalmente una herramienta de compresión. Finalmente, se reflexiona sobre si la compresión puede servir como una definición matemática válida de la inteligencia al considerar la eficiencia, la capacidad de generalización y la velocidad de aprendizaje.

Topics covered

  • Límite de Shannon y predictibilidad
  • Funcionamiento de la codificación aritmética

+6 more topics (full content when you follow the channel)

Transcript excerpt

This animation is previewing an algorithm that can compress large pieces of text down to less than just one bit per character, which is kind of mind-boggling if you think about it. There's actually an open cash prize for text compression (it's known as the Hutter prize) available to anyone who can beat the current best record for compressing a gigabyte of Wikipedia as small as possible. I promise we'll build up to what exactly this animation is showing in due time, the core idea that it conveys This underlies well I was going to say it underlies several of the winning submissions here but that's not true it's actually the backbone for all of them every single one and the method is not even specific to text it is a fully general technique that you could use to compress images or videos audio really any data type and this algorithm also got me thinking differently about that phrase compres…