The Base Model Is Dead — Varun Singh, Arcee AI

AI Engineer

Summary

El vídeo analiza cómo el concepto tradicional de 'modelo base' está cambiando. Antes, estos modelos se entrenaban principalmente con grandes cantidades de texto web para acumular conocimiento general. Ahora, el enfoque se desplaza hacia modelos de razonamiento y agentes, priorizando el entrenamiento en código, matemáticas y datos sintéticos estructurados desde la fase de pre-entrenamiento. El aprendizaje por refuerzo ya no es un añadido, sino un pilar fundamental que compite en importancia con el aprendizaje supervisado, obligando a los desarrolladores a replantear cómo construyen las bases para que los modelos sean más eficientes y capaces de realizar tareas complejas.

Chapters

  1. 0:00Speaker Intro & Traditional Base Models
  2. 2:56Old Post-Training Paradigm
  3. 4:03The Rise of Advanced RL
  4. 5:52Contrasting Data Strategy Trends
  5. 9:41Synthetic Data's Role and Impact
  6. 11:24MOEs, Mid-Training, Data Distribution
  7. 13:17Rebalancing Supervised Learning and RL
  8. 14:40Supervised Learning for RL Skills
  9. 16:47The Evolving Base Model Prior

Topics covered

  • Evolución de los modelos base
  • Reducción de texto web en pre-entrenamiento

+5 more topics (full content when you follow the channel)

Transcript excerpt

Hi everyone. My name is Varon. I'm the pre-training lead at RCAI. And the talk I'm going to be giving today is called the base model is dead. Um but not really. Um the idea of the base model that we have um kind of is like built on uh this idea of like training on super large-scale web text and the base model kind of being a reflection of like the whole knowledge of like the uh human internet. Um you can see in like uh these I've I've taken these uh from a bunch of uh different papers on like the entire LM training process. Um our own model RC uh training large thinking, um the process looked kind of like the simplified diagram on the left. I've taken the top one from uh GLM 4.5, the bottom one from GLM 5. Um all these have a pre-training phase and uh pre-training is like the stage where the model look accumulates world knowledge, builds useful representations uh all through next token p…