Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

AI Engineer

Summary

Los fundadores de Theta Software analizan el reto de definir y medir entornos de trabajo para agentes de IA en tareas de largo horizonte temporal. Explican cómo métricas como el tiempo humano o el consumo de tokens son indicadores útiles pero limitados, y proponen un marco donde la complejidad, la coordinación de herramientas, la ambigüedad y el uso de modelos juez son fundamentales para evaluar el progreso real de los agentes. También critican la falta de profundidad en los benchmarks actuales y abogan por entornos de entrenamiento más rigurosos y granulares.

Chapters

  1. 0:00Introduction to Long-Horizon Tasks
  2. 6:36Measuring Model Capabilities
  3. 10:03Challenges in Reward Assignment
  4. 13:26Principles for Effective Judge Design
  5. 17:54Critique of Current Benchmarks

Topics covered

  • Definición de tareas de largo horizonte
  • Métricas basadas en humanos vs. modelos

+6 more topics (full content when you follow the channel)

Transcript excerpt

It's uh great to see all of you here today. We're super excited to talk about one of our favorite topics uh here at Theta. Um before we get started, we just want to introduce ourselves. Um so, hi, I'm a co-founder and CTO at Data Software. >> Hi, I'm Ryan. I'm a co-founder and CEO at Thata Software. Prior to this, I was previously a founding engineer at Deep Silken where we did research into turnary models. >> Awesome. So, I can get us started with the topic today. Um, we're going to be talking about oral environments uh within the context of long horizon tasks. And I think the most important thing for us to start with at the beginning is just talk about the trends and what long horizon actually means. So, you know, we all know that the horizon at which AI agents can work autonomously is accelerating really fast. Um this is just some of the metrics that you can look at to see how this pr…