From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
arXiv:2601.11020v3 Announce Type: replace
Abstract: Advances in mechanistic interpretability have identified special attention heads, known as retrieval heads, that are responsible for retrieving information from the context. However, the role of thes…