A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
arXiv:2604.14722v1 Announce Type: new
Abstract: Transformers commonly exhibit an attention sink: disproportionately high attention to the first position. We study this behavior in GPT-2-style models with learned query biases and absolute positional em…