How Large Language Models Generate Responses with Tokens, Context, and Probability

Welcome back. In the previous lesson, you separated rule-based software, predictive machine learning, and generative AI by what each system primarily does. We ended with an important apparent paradox: an LLM is generative AI because it creates new text, yet its core technical operation is prediction.

This lesson resolves that paradox. You will build a practical mental model of how a tool such as Claude produces a response: it breaks text into tokens, uses the available context to estimate probabilities for possible next tokens, selects one, and repeats until the response is complete. This model will make later prompt-design techniques much more intuitive.

---

The central idea: a response is built one token at a time

A large language model is not a database of prewritten answers that it retrieves intact. At generation time, it repeatedly answers a narrower question:

Given the text available so far, which token is a plausible next piece of text?

A token is a unit of text that the model processes. A token can be a whole word, a part of a word, punctuation, or another short text fragment. The exact token boundaries depend on the model’s tokenizer, so it is safer to say “token” than “word” when describing how an LLM works.

For English text, a rough rule of thumb is that one token averages about four characters, or roughly three quarters of a word. It is only an estimate: uncommon words, code, numbers, whitespace, punctuation, and languages other than English can produce very different counts.

A model has a fixed vocabulary: a large set of tokens it can select from. It can form many words that are not single vocabulary entries because it can combine subword tokens. For instance, a tokenizer may divide a word such as unwatched into meaningful smaller pieces. This is why LLMs can usually handle unfamiliar terms, names, and word variations without needing every possible word stored as one unit.

The token concept matters professionally for three reasons:

• Generation is incremental. A paragraph is assembled from many token choices, not written all at once.
• Limits and cost are token-based. Long files, detailed instructions, and long outputs consume tokens.
• Prompt wording matters. Changing even a few words changes the token sequence, which can change the model’s estimate of an appropriate continuation.

---

Probability is not knowledge or certainty

Suppose a prompt ends:

“Please send the revised report by”

The model might estimate a distribution like this. These values are illustrative, not values from any particular model.

Candidate next token	Illustrative probability
Friday	0.40
noon	0.18
tomorrow	0.13
the	0.08
All other tokens combined	0.21

The model assigns a probability to every candidate token in its vocabulary. Together, those probabilities sum to one:

$$\sum _{t\in V}P(t\mid C)=1$$

Here, $V$ is the model’s vocabulary, $t$ is a candidate next token, and $C$ is the context available so far. The notation reads: “the probability of token $t$, given context $C$.”

The image shows the key distinction between a probability distribution and the selected output. At each step, the model first considers a very large set of possible tokens. It then uses a selection method to choose one.

Two common selection approaches are:

• Highest-probability selection: choose the most likely token. This tends to be more repeatable, but can become overly rigid or repetitive.
• Sampling: select from plausible candidates with some randomness. This can produce more varied wording and creative alternatives.

The system’s settings influence how much sampling occurs. In developer interfaces, a setting often called temperature is one way to control variety: lower settings generally favor the most likely continuations; higher settings allow more lower-probability candidates to be selected. We will return to generation settings when working with the API.

However, a high probability does not mean a statement is true. It means that, given the current text and the model’s learned patterns, the token is a likely continuation. A model can therefore produce a fluent but false claim, a concern you will address directly in the evaluation and responsible-AI module.

---

Context gives token probabilities their meaning

The same token sequence can mean different things in different situations. Consider the word bank:

• “She sat on the bank and watched the river.”
• “She visited the bank to deposit her paycheck.”

A useful system must treat bank differently in these two sentences. It does so by taking account of context: the relevant text surrounding the point where it must predict the next token.

In a chat application, the context can include:

• system-level instructions set by the application or developer;
• your current prompt;
• relevant earlier user and assistant messages;
• reference material or files that the application has placed into the request;
• tokens the model has already generated in its current response.

Context is not simply “everything ever said to the model.” It is the limited set of text supplied and available for the current prediction.

Modern LLMs use a transformer architecture. At a conceptual level, its attention mechanism helps the model weigh which parts of the available context matter most for a particular interpretation or prediction. In the image, the model’s representation of “it” can be shaped strongly by “animal,” rather than treating every earlier token as equally important.

Do not read an attention diagram as proof that a model understands language exactly as a person does. It is a simplified visualization of numerical relationships within the model. Still, the practical lesson is sound: relevant context changes the probabilities of possible next tokens.

During normal response generation, the model can use the prompt and tokens it has already produced. It cannot inspect words that have not yet been generated. This is why its output is called autoregressive: each selected token becomes part of the context used to select the next one.

---

The generation loop

We can now describe a response without pretending that the model writes an entire answer in a single act of thought.

When you send a message to an LLM, the application and model conceptually perform this cycle:

1. Assemble the available context. The application provides applicable instructions, your message, relevant conversation history, and possibly attached source material.

2. Tokenize the text. The text is divided into tokens from the model’s vocabulary. Internally, these tokens are represented numerically.

3. Interpret tokens in context. The transformer processes the token sequence, using learned relationships and attention to identify patterns relevant to the next prediction.

4. Calculate next-token probabilities. The model produces a probability distribution over its vocabulary.

5. Select and append one token. A selection method chooses a token. That token becomes part of the growing response, and the cycle runs again.

The process stops when the model produces a special end-of-response token, meets a configured stopping condition, or reaches its output limit.

For example, imagine the prompt:

“Write a polite one-sentence reminder to submit the timesheet.”

The model may first select a token corresponding to “Please,” then use the expanded context to select something like “remember,” then “to,” and so on. At every point, the earlier generated wording influences what is likely next. That repeated conditional prediction can yield a coherent sentence, an email, a code sample, or a long explanation.

This is why an LLM can appear to plan or reason across a response: its token choices reflect sophisticated patterns learned during training and the constraints in the immediate context. But the operational mechanism is still repeated next-token prediction, not a guarantee that it has checked facts, followed every instruction correctly, or reasoned flawlessly.

---

Context windows: why a model can lose track of a conversation

Models have a maximum number of tokens they can consider at one time, called a context window. Both the supplied input and the generated output count toward that limit.

If an application has a limited context window, a very long conversation may require it to remove old messages, summarize them, or decline a request that is too large. Even when a long document fits, the model may not give every part equal attention. A large context window increases capacity; it does not guarantee accurate retrieval, correct interpretation, or perfect use of every detail.

This has immediate implications for good AI practice:

Situation	Useful response
You need the model to follow a specific rule	Put the rule clearly in the current instructions rather than assuming it remembers an earlier chat.
You are working with a large document	State the task, identify the relevant section, and ask for quotes or citations where accuracy matters.
The conversation has become long or unfocused	Start a new chat with a short, accurate summary of the essential context.
You need an exact fact	Treat generated text as a candidate answer to verify, not as proof.

Notice that “context” here refers to what the model can use in the current request. The patterns encoded during its earlier training are a different source of behavior. The next lesson will separate those two ideas carefully: what a model learned during training versus what you supply in the conversation now.

---

A compact explanation you can use professionally

When explaining LLM behavior to a colleague or learner, avoid saying merely, “It predicts the next word.” That is directionally correct but incomplete. A more useful explanation is:

A large language model converts the current conversation into tokens and uses the available context to assign probabilities to possible next tokens. It selects one token, adds it to the response, and repeats the process. Its learned patterns help it produce fluent language, but the current prompt and conversation steer which continuations are likely.

That explanation includes all three core concepts:

• Tokens are the units the model reads and generates.
• Context is the relevant text currently available to shape the prediction.
• Probability is how the model ranks possible next tokens before choosing one.

---

Key takeaways

An LLM generates text token by token. For each step, it uses the available context to calculate a probability distribution over possible next tokens, selects one token, incorporates it into the growing response, and repeats.

Context is powerful because it changes what the model is likely to say. It includes current instructions, user input, relevant conversation history, supplied material, and the response generated so far. Yet context is limited by a token-based context window, and a probable continuation is not automatically a factual or trustworthy one.

Next, you will distinguish two sources that are easy to confuse: the broad patterns a model acquired during training and the specific information you provide in the current conversation.