Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

AI Engineer

Summary

En esta charla, Ross Taylor y Chengxi Taylor de General Reasoning exploran cómo escalar agentes de IA a tareas de "horizonte largo". Ross relata su experiencia desde Meta AI, destacando el impacto del refuerzo por retroalimentación humana (RLHF) tras el fracaso de Galactica frente a ChatGPT, y la importancia de modelos base de alta calidad. Chengxi profundiza en los desafíos de las tareas de horizonte largo, como la escasez de contexto y la varianza del gradiente, proponiendo soluciones algorítmicas con modelos de valor y herramientas de gestión de contexto. Presenta el benchmark Kelly bench, donde los modelos actuales fracasan en tareas de trading, enfatizando la necesidad de ir más allá de las tareas de codificación. Finalmente, abordan el impacto en el cómputo y la infraestructura, presentando OpenReview como una solución.

Chapters

  1. 0:00Introduction: GR and Speakers
  2. 0:41Ross's Journey at Meta AI
  3. 1:54RLHF: Galactica vs ChatGPT
  4. 3:57Galactica's Key Innovations
  5. 6:23RL Reasoning and The Bitter Lesson
  6. 8:52Long Horizon Tasks: A Mindset
  7. 10:32Tackling Long Horizon Challenges
  8. 12:24Kelly Bench: Frontier Models Fail
  9. 14:28Compute and Infrastructure Optimization
  10. 16:48Conclusion: Long Horizon Mission

Topics covered

  • Introducción y objetivos de la charla
  • Experiencia personal en Meta AI y modelos de lenguaje

+8 more topics (full content when you follow the channel)

Transcript excerpt

Uh so this talk is called scaling to long horizons. My name is Ross. Uh I'm the CEO of GR. We're a London-based reinforcement learning company. Uh before GR, I was the reasoning lead at Meta AI working on Llamas, uh Galactica, lots of other models back in the day. I'm joined by Chengxi, uh co-founder and president of GR. Uh and yeah, hit today we're going to talk about algorithms, environments, compute, all the things you need to do to get agents scaling uh to kind of long with tasks. So we're going to have two parts of this talk today. I'm going to first of all start with a personal perspective about, you know, the early days, the golden age of language modeling in between like maybe 2020 and 2023. Uh I'll talk about, like I said, all these models and some of our early reinforcement learning efforts for LLMs. And then Chengxi is going to talk about, you know, what's ahead, you know, wha…