Tokens and Transformers: Inside a Large Language Model

Before the AI Reads Your Message, It Breaks It Apart

Every time you type a message to Claude, ChatGPT, or Gemini, the first thing that happens is not reading. It's tokenizing. The model breaks your text into units called tokens.

A token is roughly four characters of text — about three-quarters of a word on average. Common short words like "the," "is," or "of" are each a single token. Longer or rarer words get broken into multiple tokens. The word "understanding" might be two tokens: "under" and "standing." The word "ChatGPT" might be three.

Why tokens instead of words? Because the model operates on fixed-sized chunks, and words vary too much in length. Tokens let the system handle language at a consistent, predictable scale. A typical message of 100 words is roughly 130 tokens. A long document of 10,000 words is roughly 13,000 tokens.

Every token gets converted into a long list of numbers — its embedding. This numerical representation captures the token's meaning and its relationship to other tokens. The classic demonstration: the embedding for "king" minus "man" plus "woman" produces a vector very close to "queen." The geometry of these number spaces encodes genuine semantic relationships.

The Transformer: Paying Attention to What Matters

The architecture that powers all major language models today is called the transformer. It was introduced by Google researchers in 2017 in a paper called "Attention Is All You Need" — and it genuinely changed everything.

The key innovation is the attention mechanism. When generating each new token, the model doesn't just look at the token immediately before it. It looks at every previous token in the context and assigns a weight to each one based on how relevant it is to the current decision.

Consider the sentence: "The bank by the river was steep." When the model processes the word "bank," it needs to decide what kind of bank this is — financial institution, or river bank? The attention mechanism causes the model to heavily weight "river" when processing "bank," resolving the ambiguity. If the sentence were "The bank charged me a fee," the same mechanism would weight "charged" and "fee" heavily instead.

Attention is the model asking, at every step: given everything I have seen so far, what is most relevant right now?

How Responses Are Generated, One Token at a Time

Here is the exact process that produces every response you receive from an AI chatbot:

1. Your message is tokenized — broken into numerical chunks.
2. The model processes all your tokens through its layers, using attention to weigh relationships.
3. The model outputs a probability distribution over its entire vocabulary — roughly 100,000 possible next tokens. "Morning" might have 12% probability. "Evening" might have 8%. "Banana" might have 0.001%.
4. The model samples from this distribution — typically selecting a high-probability token with a bit of controlled randomness. This randomness is the "temperature" setting.
5. That token is appended to the sequence, and the process repeats from step 2 — now with the new token included.
6. This continues until the model generates a stop token or reaches a length limit.

That's it. The entire response is produced token by token, each one chosen based on a probability distribution shaped by everything that came before. There is no "plan" for the response, no outline generated in advance. The model is always doing one thing: predicting what comes next.

Why This Matters for How You Use AI

Understanding the token-by-token generation process explains several things you may have noticed:

• Why responses sometimes go off-track partway through: Each token conditions the next. If an early token leads the model toward a certain interpretation, subsequent tokens follow that path — even if a better answer would have required a different start.
• Why longer, clearer prompts usually produce better results: More context tokens give the attention mechanism more signal to work with. A vague two-word prompt provides less to attend to than a well-specified paragraph.
• Why the temperature setting affects creativity: High temperature increases randomness in token sampling — the model picks less probable tokens more often, producing more varied and creative output. Low temperature makes it more deterministic — it sticks to the highest-probability choices, which feels more focused and precise.

The model is, at its core, a very sophisticated next-token predictor operating on numerical representations of language. Everything it can do — reasoning, creativity, translation, coding — emerges from this simple mechanism, scaled enormously.