Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
arXiv:2601.12145v2 Announce Type: replace
Abstract: Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass disperses as sequence …