arXiv cs.LGPaper
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
This is infrastructure troubleshooting with real teeth. A remote KV cache sounds efficient but the paper shows it can wreck p99 latency: 17.3s instead of 3.96s. The cost model (hit pays off only above B* = sR threshold, plus fixed per-hit engine cost) gives operators a way to decide when remote tiers make sense. If you're scaling long-context serving across multiple engines, this reframes the cache strategy conversation.