For a decade, we asked if RNNs can represent what Transformers represent. We proved they can. We forgot to ask how expensively. That…Continue reading on Towards AI »
There is a quiet difference between seeing a sentence and moving through it.A fixed-window neural language model still sees language a little like a camera sees a street. At each step, it captures a local view. It can be a very intelligent camera. It c…
Once language modeling moved beyond pure counting, something important changed.The question was no longer only:“How often did these words appear together before?”It became:“Can a model learn what kinds of words tend to fit together, even when it has no…