This week
Suggest a nameObligation Marketing for AI Builders
The 7-Rung LadderThe 8-Type TaxonomyWrapper or WinnerThe Sample ColumnThe Signal
Updated through the day. Every headline links straight to the source. The two lines underneath are ours.
This matters for inference cost at scale. Speculative decoding is already deployed by major labs, and this work improves draft model training to reduce verification overhead. If you're building LLM inference infrastructure, this is worth tracking. The theory is clean and the improvement is measurable.
This is a real efficiency win if you're deploying large models in production. The insight is simple: prefill and decode have different bottlenecks, so quantizing them identically is wasteful. The numbers are solid: 32.5 points on MMLU-Pro without increasing inference cost. If you're optimizing inference for latency-sensitive workloads, this becomes part of your baseline. Start testing on your models this week.
This is the kind of work that changes agent economics in production. Routing every step to Claude or GPT-4 wastes 60-80% of compute budget on subtasks that 7B models handle fine. AgentRouter's approach is practical: a 12M-parameter classifier with <5ms overhead that learns which steps need frontier reasoning and which don't. For builders running multi-step agents, this is immediately actionable. For cost-conscious teams, this flips the economics of when agents become viable. Implementation matters less than the insight: heterogeneous routing is now table stakes.