How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models
arXiv:2605.06826v1 Announce Type: new
Abstract: We study the spectral properties of sample covariance matrices constructed from pooled sequence representations, where token embeddings are drawn from a fixed two-class Gaussian mixture table and pooled …