Inside an LLM

From input to next token.

L2 · IntermediateFast-movingKnown~14 min

Reviewed by YuvarajPublished 22 Aug 2026

You have met the pieces, tokens, embeddings, attention. Now let us trace a single request end to end and watch a large language model turn your text into a response.

The journey of one prompt

Suppose you type: "AI learns from" and ask the model to continue.

1TokenizeText is split into tokens and mapped to integer ids
2Embed + positionEach id becomes a vector; position info is added so order is known
3

Check your understanding

Answer from memory before revealing, retrieval practice is what builds durable recall.

1.
At its core, a large language model is trained to:
2.
The "context window" of an LLM is:
3.
An LLM's weights change every time it answers a user during inference.

Ask the tutor

Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.

Finished this lesson?

Mark it complete to earn XP, keep your streak, and schedule a review.

Transformer blocks

Attention shares info across tokens; feed-forward layers transform each

4LogitsThe final layer scores every token in the vocabulary

5ProbabilitiesSoftmax turns those scores into a distribution summing to 1

6Pick oneA token is chosen from the distribution, maybe 'data'

Stacking many transformer blocks is what builds an increasingly rich representation: early layers capture surface patterns, later layers capture meaning and long-range structure.

From scores to a choice, with numbers

Step 4 produces one logit per vocabulary token, a raw, unbounded score. Suppose, after "AI learns from," the top three logits are:

data: 4.0
water: 2.0
rocks: 1.0

Softmax exponentiates and normalizes them into probabilities:

\mathrm{softmax}([4, 2, 1]) \approx [0.84,\ 0.11,\ 0.04]

The model now assigns data an 84% chance of being next, not because it looked up a fact, but because that continuation fit the patterns it learned. How one token gets picked from this distribution is the next lesson.

The autoregressive loop

Here is the key move: the chosen token is appended to the input, and the whole process runs again to produce the next token.

AI learns from → data → AI learns from data → by → AI learns from data by → ...

The model generates one token at a time, each step conditioned on everything so far. This is called autoregressive generation. A long answer is just this loop run many times.

Prediction, not retrieval

An LLM is not looking up an answer in a database. At each step it is predicting a plausible next token from patterns learned in training. That is why it can be fluent and wrong at the same time, and why "next-token prediction" is the single most important idea about how these models work.

The context window

Everything the model considers, your prompt plus the tokens generated so far, must fit inside its context window, measured in tokens.

Why length has a limit

Attention compares tokens against each other, so its cost grows quadratically as the sequence gets longer: double the tokens, roughly quadruple the attention work. The context window is the model's hard limit on how much text it can attend to at once. Exceed it and the earliest tokens must be dropped or summarized.