Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

AI Engineer

Summary

Ali Khial analiza la crisis actual de los benchmarks en IA, argumentando que los estándares actuales no reflejan el trabajo real de los ingenieros. Explica problemas críticos como las instrucciones irreales, la fragilidad de los evaluadores y el hackeo de recompensas por parte de los modelos. Finalmente, propone un nuevo marco de principios basado en la autoría humana, la evaluación holística y la transparencia de los resultados para que los benchmarks sean realmente útiles en entornos de producción.

Chapters

  1. 0:00Introduction to Benchmarks
  2. 1:45Understanding Benchmark Mechanics
  3. 3:37Unrealistic Benchmark Instructions
  4. 5:18Weak Benchmark Verifiers
  5. 7:00Reward Hacking and Trust Gap
  6. 8:35Framework for Better Benchmarks
  7. 11:58Engage with Benchmarks

Topics covered

  • Problemas con las instrucciones
  • Limitaciones de los evaluadores

+6 more topics (full content when you follow the channel)

Transcript excerpt

Hello everyone. Um this is the last talk of this session. So hopefully it's going to be short. I know that you guys had to go through a long day. So try to keep it short and light for you all. Um I'm going to present myself. Um I'm Ali. I'm the director of AI and ML at G2I. Um I have zero experience in ML. So I don't know why they put the ML in my title. I'm a software engineer uh at heart. And to prove that I have more than 50 abandoned side projects in my machine. So uh you can know. So uh I'm going to make a disclaimer. The the title of the the presentation is a little bit misleading. Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find a dichotomy of the the bad, the ugly, and and the good. So um let's start with um I want to grab your attention. And I invite you to look at this. These…