The Signal

Everything that matters in AI, with our take.

Updated through the day. Every headline links straight to the source. The two lines underneath are ours.

agents162 research126 infra70 claude69 capital49 labs43 culture39 safety38 openai33 regulation30 anthropic24 security24 reasoning19 evaluation18 robotics13 frontier-models12

arXiv cs.LGPaper17h ago

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

This is infrastructure troubleshooting with real teeth. A remote KV cache sounds efficient but the paper shows it can wreck p99 latency: 17.3s instead of 3.96s. The cost model (hit pays off only above B* = sR threshold, plus fixed per-hit engine cost) gives operators a way to decide when remote tiers make sense. If you're scaling long-context serving across multiple engines, this reframes the cache strategy conversation.

infra serving llms latency kv-cache

arXiv cs.CLPaper3d ago

Hashrope: A Persistent Data Structure for Mechanisms of LLM Context Management

This is a systems contribution with real performance numbers. If you run an LLM serving stack, this consolidates specialists and removes redundancy. The validation is rigorous and pre-registered. Evaluate it against your current stack, especially if you're juggling multiple context-management libraries.

research infra serving

arXiv cs.CLPaper8d ago

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Token-level routing beats query-level routing on cost-quality, but only if you can actually implement it without tanking throughput. TokenRouter solves the hard part: the systems engineering. If you're running a multi-model serving infrastructure or building one, this moves from research idea to viable production pattern. The architecture choices matter more than the algorithm.

infra serving routing research