arXiv cs.LGPaper
Disaggregated Quantization: Specializing LLM Prefill and Decode
This is a real efficiency win if you're deploying large models in production. The insight is simple: prefill and decode have different bottlenecks, so quantizing them identically is wasteful. The numbers are solid: 32.5 points on MMLU-Pro without increasing inference cost. If you're optimizing inference for latency-sensitive workloads, this becomes part of your baseline. Start testing on your models this week.