Tokens: The Atom of Everything in Large Language Models
Series: An Engineer’s Explorations in LLM Concepts“It started with a simple question: What exactly is a token? Three hours later, I had filled six browser tabs, youtube videos and had somehow ended up reading about some 5–6 concepts of LLM.”If you’ve w…