Welcome back. In the previous lesson, you distinguished supervised, unsupervised, and reinforcement learning by the kind of feedback they learn from and the business capability they produce. Large language models are different in the immediate experience they create: they can draft, summarize, answer, and converse. But their basic runtime mechanism is precise.
This lesson explains how an LLM produces that output. By the end, you should be able to describe—in business-friendly but technically accurate terms—how tokens, context, and inference turn a prompt into a response. This is essential when qualifying generative-AI opportunities: it clarifies why prompt quality, document access, response time, and output variability matter.
The core model: a next-token prediction engine
An LLM does not write an entire answer, retrieve truth from the world by itself, or follow a hidden finished script. At each moment, it estimates which token is most plausible to come next, given what it has seen so far. It selects one token, adds it to the running text, and repeats until it reaches a stopping point.
A token is a chunk of text that a specific model’s tokenizer recognizes. A token may be:
- a whole word, such as “invoice”;
- part of a longer or less common word;
- punctuation, a space pattern, or a number;
- occasionally a single character.
The important correction to everyday language is this: an LLM does not literally operate on “words.” It operates on tokens, and different models can split exactly the same text differently.
A token ID is simply a numerical label from the model’s vocabulary. The ID does not mean that the number itself contains an English definition; it lets the model look up the learned numerical representation associated with that token.
Read Google’s LLMs: What’s a large language model? for the foundational definition of tokens and the reason context matters.
LLMs: What's a large language model? | Machine Learning
Google’s Machine Learning Crash Course gives a compact, reliable explanation of LLMs as token predictors, then introduces self-attention as the mechanism that makes context useful.
In the opening discussion before the architecture material, read from the token definition through the two reasons LLMs improve on older language models. Focus on why “token” is a more accurate unit than “word.” Then, in the subsection “What is self-attention?”, read the full animal-and-street example, beginning at the ambiguity example. Notice how changing “tired” to “wide” changes what the pronoun “it” refers to.
For an enterprise conversation, a concise explanation is:
An LLM converts the available text into tokens, considers how those tokens relate to one another, predicts a probability distribution for the next token, selects one, and repeats.
The phrase “available text” deserves attention, because it is what we mean by context.
Context: the material shaping this particular answer
At inference time, context is the bounded set of tokens the model can consider while producing its next token. In a chat application, context commonly includes some combination of:
- system-level instructions that set the assistant’s role or boundaries;
- the user’s current prompt;
- selected earlier messages;
- relevant business content supplied by the application, such as policy excerpts, a product catalogue, or case details;
- tokens the model has already generated in its current response.
This runtime context is different from the model’s parameters. Parameters are the large set of numerical values learned during training. They encode broad statistical patterns from training data. Context is the specific information supplied for this one request.
| Concept | What it is | Example |
|---|---|---|
| Parameters | Learned model settings, established before the request | General patterns of language, formatting, and domain associations |
| Context | Text and instructions available for the current response | “Use the following approved policy and summarize it for a customer” |
| Output | Newly selected tokens | The generated summary |
Self-attention is a central Transformer mechanism for interpreting context. Rather than treating every earlier token as equally important, the model calculates which tokens are most relevant to interpreting the current position. In the Google example, the phrase “too tired” makes animal more relevant than street when resolving “it.” If the phrase is “too wide,” street becomes the stronger interpretation.
For business use, the same principle has practical consequences. Consider these two prompts:
- “Write an email explaining our cancellation policy.”
- “Using only the approved policy excerpt below, write a 120-word cancellation email for a customer whose renewal date is 15 June. Do not add terms not present in the excerpt.”
The second prompt supplies a clearer task, audience, constraint, and grounding material. It does not guarantee correctness, but it gives the model a much better context from which to predict suitable tokens.
Two limitations follow directly from this mechanism:
- A model can only use information placed within its effective context or otherwise made available by the application. Owning data in a CRM, document repository, or finance system does not automatically make that data available to the model.
- Context is not the same as verified truth. If the supplied document is outdated, incomplete, or wrong, a fluent response can still reflect that problem.
The enterprise systems that select and provide relevant internal information will be covered later in the course. For now, the commercial takeaway is simple: when a buyer says, “The AI should know our policies,” a useful follow-up is to ask which source is authoritative, who may access it, and what content should be included for each request.
Inference: generating a response without retraining the model
Inference is the process of running a pre-trained model on a new input to generate an output. During inference, the model’s learned parameters are not being updated. It is applying what it already learned to the context now provided.
This differs sharply from training:
| Stage | What happens | Does the model’s learned parameters change? |
|---|---|---|
| Training | The model processes vast quantities of examples and adjusts itself to improve prediction | Yes |
| Inference | The deployed model processes a new request and produces output tokens | No |
IBM’s What is LLM Inference? explains the runtime stages with useful production terminology.
IBM’s overview distinguishes inference from training and explains why an apparently simple chat response has implications for speed, computing capacity, and cost.
In “What is LLM inference?”, read the core definition. Focus on the fact that generation happens one token at a time without updating learned parameters. Then read all three numbered stages in “How does LLM inference work”: tokenization, prefill, and decoding. Relate prefill to understanding the initial prompt and decoding to producing the visible answer.
In a production LLM application, inference has two broad phases.
1. Prefill: process the supplied context
The model receives the input tokens and processes the prompt, instructions, and any other supplied material. This establishes an internal representation of the context and is often computationally intensive for long inputs.
Longer context can be valuable: a contract clause, several customer emails, or a detailed product specification may help the response fit the situation. But it also tends to consume more processing time, memory, and money. This is why “put every company document into every prompt” is not a credible enterprise design.
2. Decoding: generate tokens sequentially
The model then generates the output one token at a time. For every next position, it produces a probability distribution across its vocabulary.
For example, after a context ending with:
“The customer’s invoice is overdue because…”
the model might assign higher probability to tokens such as “payment,” “the,” or “they,” and much lower probability to unrelated tokens. This is a simplified illustration: the model evaluates a very large vocabulary, not merely three choices.
A decoding method then chooses a token from that distribution. The selected token becomes part of the context for the next prediction. The process continues until the system reaches an end token, a requested length limit, or another configured stopping condition.
This sequential nature explains several visible behaviors:
- A longer response generally takes longer to generate than a short one.
- The model can revise its direction only through the tokens it generates next; it has not necessarily planned the entire paragraph in advance.
- A small change early in an answer can influence later wording because the generated tokens become part of the context.
- The same prompt can yield slightly different answers when the system is configured to sample among plausible options.
Watch LearnThatStack’s explanation for a visual, end-to-end account of these stages. It goes slightly beyond the core outcome by introducing sampling controls, but that is useful for understanding why generated output can vary.
How LLMs Actually Generate Text (Every Dev Should Know This)
In “How LLMs Actually Generate Text,” LearnThatStack walks through tokenization, contextual attention, next-token probabilities, selection, and the repeated generation loop.
Watch tokens first for the five-stage overview and the explanation of token IDs. Then skip the detailed embeddings segment and continue with context and probabilities, focusing on attention and the probability distribution for the next token. Finish with selection and looping to see why temperature affects variability and why each output token becomes part of the next step’s context.
A complete walkthrough: from request to draft
Imagine an internal sales-assistant application. A user asks:
“Draft a follow-up to a procurement lead. Use a professional tone, mention the agreed security review, and do not promise delivery dates.”
The application may assemble a context containing its system instructions, the user’s request, relevant account notes, and any approved product information. The exact mechanics vary by product, but the LLM generation process is broadly:
- The application’s text is divided into tokens and translated into token IDs.
- During prefill, the model processes the full available context and uses attention to identify relevant relationships: procurement lead, security review, professional tone, and do not promise delivery dates.
- The model calculates probabilities for the first response token.
- The configured decoding method selects one token.
- The model appends that token to the context and predicts the next one.
- This repeats until a complete-enough email draft is formed or the output limit is reached.
- The application converts the generated token IDs back into readable text and displays the draft.
The output may be useful, polished, and well aligned to the prompt. Yet the mechanism is still probabilistic token generation, not a guarantee that the model checked the sales agreement, understood the legal implications of a delivery commitment, or accessed a live project plan. Those would require appropriate source access, workflow design, controls, and review.
Why this mechanism matters in an AI opportunity
Understanding this process lets you replace vague claims such as “the model understands everything” with credible design and discovery questions.
| Mechanism | Commercial implication | Useful discovery focus |
|---|---|---|
| Tokens | Input and output size affect limits, usage cost, and response time | How much text will each task require? What output length is useful? |
| Context | Response quality depends on instructions and relevant supplied information | Which sources should inform the response? Which must be excluded? |
| Inference | Each response consumes computing resources and may vary with settings | How many users, requests, and peak periods must the solution support? |
| Probabilistic generation | Fluent wording is not evidence of factual accuracy | What requires source citations, human approval, deterministic rules, or validation? |
A sound business-development explanation can therefore be:
“The model is not searching a fixed answer database. For each request, the application provides a bounded context, the model interprets relationships among the tokens, and it generates a response one token at a time based on probability. The quality of the result depends on the model, the instructions, the information supplied, and the controls around the workflow.”
That explanation is accurate without requiring the buyer to understand neural-network mathematics.
Key takeaways
An LLM processes tokens, not simply words. Tokenization turns text into model-readable units and numerical IDs; token count influences context capacity, cost, and latency.
Context is the information available for a specific response, including instructions, the user’s input, selected prior conversation, supplied business material, and previously generated output. It is distinct from the model’s learned parameters.
Inference runs the pre-trained model without updating it. The model processes the initial context, calculates probabilities for the next token, selects one according to a decoding method, then repeats token by token.
This mechanism explains why well-scoped prompts and relevant source material improve results, why long responses take time, and why a confident answer is not automatically a verified one.
Next, you will examine the practical limitations of generative AI in a proposed business use case: accuracy, hallucinations, context limits, variability, cost, security, and the continued need for workflow controls.
Can't find a good explanation? Sign up and we'll make it for you
Sign up