DeepSeek's Absolutely Insane AI Speed Hack

Two Minute Papers

Summary

DeepSeek has introduced a new technique called DeepSpark that improves speculative decoding in AI models. By using a junior draft model to predict tokens and a senior model to verify them, the system accelerates text generation. DeepSpark optimizes this process with three key improvements: adding memory to the draft model, filtering out unlikely tokens early, and dynamically deciding when to check words based on compute costs. This results in significant speed increases of 60-85% for specific models without needing a fundamentally new AI, provided the system architecture supports it.

Chapters

  1. 0:00Introduction to AI Speed
  2. 0:56Speculative Decoding Explained
  3. 2:24DeepSpark Three New Tricks
  4. 4:12Predictability in Different Tasks
  5. 5:25Performance and Implementation Gains

Topics covered

  • Speculative Decoding Basics
  • The Junior Writer Metaphor

+5 more topics (full content when you follow the channel)

Transcript excerpt

What if an AI could see into its own future? Not perfectly, no, just a little. DeepSeek did something absolutely amazing in their new research work. You see, when you ask an AI how to rewrite a difficult email, it answers you one word or one token at a time. That can get pretty slow. So, why not put out multiple words, like a whole sentence, at the same time? That would be much faster. And that would be a game-changer because it would boost many slow AI systems into usability, maybe one right on your phone in your pocket. Well, yes, except that it doesn't work. A big, smart AI is like a senior editor, brilliant but expensive. If you ask it to write five words one by one, it still has to think five times, slow, expensive. So, what do you do instead? The solution is unexpectedly brilliant. What is that? Well, you hire a junior writer. The junior writer writes the next few words quickly. Of…