How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
AI Engineer
Summary
This workshop provides a comprehensive guide on building and implementing AI evaluations (evals) for agents using the Arize AI platform. SallyAnn DeLucia explains why informal 'vibe checks' are insufficient for production, detailing how to move toward rigorous testing using traces and LLM-as-a-judge methodologies. The session covers defining concrete success criteria, setting up API providers, configuring eval templates, and iterating on those evals through a prompt playground to ensure reliable agent performance.
Chapters
- 0:12Introduction to Arize and goals
- 2:11The flaws of vibe checks
- 3:08AI evals vs unit tests
- 4:24Tracing as the foundation
- 5:17Types of evaluation models
- 7:46Platform setup and API integration
- 10:20Exploring trace data
- 19:05Defining concrete criteria
- 22:42Structuring your evaluation prompt
- 28:25Building the evaluation in the playground
- 33:38Closing and future steps
Topics covered
- The limitations of vibe checks
- Understanding AI traces
+5 more topics (full content when you follow the channel)
Transcript excerpt
[music] >> Um, before we get started, I just wanted to introduce myself. I'm Sally Ann, and I'm head of product at Arize. Uh, been at Arize for like 4 years now. Um, one of the first projects that I owned actually was building our AI agent Alex into our platform. So, I've been dealing evals for a little while now, and uh, even building a lot of our evals product. And so, I think the way that I think about evals is that they're our primary signal for our agents. Um, and that includes all of our evals at Arize. So, my goal today is you'll leave with three things. Um, a clear definition of what an eval is and what it isn't. Uh, being able to write your own criteria. So, I'm hopefully going to be able to provide you with like a little bit of a recipe uh, for how you can think about evals and how you can apply that for any of your real agent scenarios. And then we're actually going to build a…