Understanding Next-Token Probabilities, Sampling, and Truth

Hello. In the previous lesson, we traced how training adjusts a neural network’s parameters to reduce a loss. For a language model, the training target is usually the token that actually came next in text. This lesson examines what the trained model produces at inference time: not a single answer, but a probability distribution over possible next tokens.

By the end, you should be able to explain why two identical prompts can produce different completions, why a high-probability completion is not necessarily a true one, and what decoding settings such as temperature, top-$k$, and top-$p$ do—and importantly, do not do.

A language model predicts a distribution, not a sentence

An LLM does not compose an answer in one indivisible act. Given the tokens already present in the prompt and conversation, it estimates a conditional probability for every token in its vocabulary:

A vocabulary may contain tens or hundreds of thousands of tokens. Tokens are fragments of text, rather than necessarily whole words; the next token could be a word, a suffix, punctuation, whitespace, or a common phrase fragment.

The probabilities have two defining properties:

and

The probability assigned to a token is relative to the model’s other available continuations for this exact context. It is not a statement that the token is factually true, morally desirable, or suitable for the user’s real-world decision.

This is closely connected to the previous lesson. During next-token training, the model saw a context and was penalized when it assigned too little probability to the token that appeared next in the training text. Repeated gradient-based updates make the model better at assigning probability mass to continuations that resemble patterns in its data.

The model’s immediate numerical outputs are usually called logits: one unbounded score per vocabulary token. A final transformation called softmax converts those scores into probabilities. You do not need to calculate softmax to use this mental model. The practical point is that a larger logit becomes a larger probability after comparison with all the other candidate tokens.