Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
arXiv:2604.22782v1 Announce Type: cross
Abstract: Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is signifi…