Inside the Softmax Bottleneck: Engineering Hardware-Aware Attention Mechanisms

How a single algorithmic deadlock in the attention equation nearly strangled the entire LLM era, and what a 2022 Berkeley PhD student did…