Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

AI Engineer

Summary

Will Brown de Prime Intellect explica cómo escalar el aprendizaje por refuerzo (RL) cuando no existen recompensas verificables, un desafío común en tareas del mundo real. En lugar de reglas deterministas, propone usar técnicas como la creación de entornos dinámicos, el uso de LLMs como jueces y el aprovechamiento de trazas de producción para generar señales de entrenamiento. El objetivo es democratizar la capacidad de entrenar agentes que aprendan continuamente de sus errores, cerrando el ciclo entre despliegue, corrección y refinamiento del modelo.

Chapters

  1. 0:00Introduction to the RL Problem
  2. 1:57Reinforcement Learning Fundamentals
  3. 4:02Prime Intellect's Platform
  4. 6:46Real-World RL Challenges
  5. 11:40Developing Autonomous Agents
  6. 13:45Manufacturing Signal & Tasks
  7. 17:29Backwards Task Generation

Topics covered

  • Conceptos básicos de RL
  • Limitaciones de la verificabilidad

+6 more topics (full content when you follow the channel)

Transcript excerpt

Uh thanks all for coming to AI engineer and checking out the post training session. Um hopefully lots of fun stuff today and throughout the conference. Um I'm Will Brown. I lead applied research at Primordial AI and today I want to talk about reinforcement learning without verifiable rewards. And so many of people may have been learning about RLVR over the past uh year or so uh year and a half as this stuff has really taken off and become the main way that we think about scaling reinforcement learning. Um but often we don't actually have verifiable rewards. And so messy real world tasks often we're kind of figuring out as we go. We're having our agents run around and we kind of in hindsight maybe can like look at what they did and say like okay this is good this is bad. Um but sometimes sitting down and just like specifying okay this is the rule this is the goal is not always so straight…