Why The Harness Matters More Than The Model | YC Paper Club

Y Combinator

Summary

This YC Paper Club session explores how 'harnesses'—the scaffolding, tools, and orchestration logic surrounding a language model—are often more critical for performance and intelligence than the model weights themselves. The speakers argue that by providing models with persistent state, sandboxed execution, and self-improving feedback loops, developers can achieve significant breakthroughs in reasoning and adaptability without needing to fine-tune the core model, ultimately moving towards the creation of agentic operating systems.

Chapters

  1. 8:37Historical progression of capabilities
  2. 10:22Transitioning to tool-use and skills
  3. 12:02Reflective multi-agent architectures
  4. 12:51Evolution of Agentic Harnesses
  5. 18:42Prime Agent Architecture
  6. 19:37First principles for agent design
  7. 21:34Managing context via cache layers
  8. 28:22Inter-agent messaging protocols
  9. 37:41Open Jarvis: Local AI Stacks
  10. 38:33The case for local inference
  11. 39:51Five primitives of a personal stack
  12. 42:48Cloud-optimized local stacks
  13. 46:05QM: YC's Agentic Operating System
  14. 52:42Centralizing context in Postgres
  15. 54:54Human-in-the-loop writing workflows
  16. 59:15Handling social and privacy boundaries

Topics covered

  • The Role of Harnesses
  • Self-Improving Agents

+4 more topics (full content when you follow the channel)

Transcript excerpt

[music] Welcome to YC Harness Club. I worked really hard. It took me like 10 its with Gemini to have Harsha riding a lobster. Gemini didn't want to do it, but we figured it out. Okay, welcome to harness night. Um, first order of business. Uh, how does everyone like the new YC paper club look? Well, this is like an idea that I kind of blurted out at EV, our head of design at YC, and two days later, she came back with this, and I'm like, "This is amazing." So, Ev in the back there, please take a bow. [applause] Very exciting. All right, why harnesses? Um, I mean, it's just a rapper. This is just scaffolding. This is just like prompt engineering. Um, why would this be at all worthy of a night? Uh, here's a great little Reddit that was only a month ago, which is actually uh the most aggressive. I'm not sure this kind of prompt engineering belongs at a top tier machine learning conference. Um…