From Prompt to Text: Tracing the Language Model Inference Pipeline

Hello, and welcome to the first lesson in Inference Engineering. This course follows an LLM request from a developer’s API call through local and GPU-backed serving, benchmarking, optimization, reliability, and deployment.

This first module establishes the operational mental model you will use throughout the course: an LLM does not “write text” in one action. It loads a trained model into memory, converts a prompt into token IDs, processes the full prompt, then generates new token IDs one at a time before converting them into readable text. By the end of this lesson, you should be able to trace that lifecycle precisely and identify which work happens once per model startup versus once per request versus once per generated token.

---

The request lifecycle at a glance

An inference server has two timescales:

1. Model startup happens when the server launches or switches models. It may take seconds or minutes, depending on model size and storage speed.
2. Request execution begins when a user sends a prompt. This is where prompt processing and text generation occur.

The diagram deliberately separates three broad concerns:

• Text conversion: tokenization at the beginning and detokenization at the end.
• Prefill: processing all input tokens to establish context.
• Decode/generation: repeatedly selecting one new token while using the accumulated context.
• GPU-resident state: model weights and, during generation, cached information about the sequence.

The central distinction is worth memorizing now:

Prefill processes the prompt as a whole; decode produces the response one token at a time.

---

Before the first request: loading a model

An inference engine cannot process a prompt until it has loaded a compatible collection of model artifacts. In a typical open-weight deployment, these include:

• Model configuration, describing the architecture: number of layers, hidden size, attention-head layout, context limit, and related settings.
• Tokenizer assets, including the vocabulary and rules for splitting text into tokens.
• Model weights, the learned numeric parameters that determine the model’s behavior.
• Optionally, adapters such as LoRA weights that alter the base model’s behavior for a specialized task.

The weights are generally the largest artifact by far. An inference runtime often reads them from local disk or remote storage, may temporarily stage them in system memory, and places them in accelerator memory if the model will run on a GPU. On an Apple Silicon system, CPU and GPU share unified memory, but the capacity constraint is still real: the weights must fit alongside runtime memory such as the KV cache.

Model loading is not normally part of a single request’s latency. If a server is already warm and serving a model, the next user request does not reload billions of parameters. However, loading affects:

• startup time,
• cold-start behavior,
• memory planning,
• model-switching time,
• and the first benchmark measurements after launch.

This is why sound performance testing includes a warm-up phase before collecting results.

A useful operational separation

When diagnosing a slow system, ask this question first:

Observation	Likely category
Server takes a long time to become ready after launch	Loading, memory allocation, weight transfer, compilation
First request is unusually slow; later ones improve	Warm-up or one-time runtime setup
Long input delays the first visible output	Prefill
Output arrives slowly after the first token	Decode
Returned text contains odd fragments or malformed spacing	Tokenizer, prompt formatting, or detokenization issue

This classification will become the foundation for the metrics and benchmarking work in the next module.

---

Step 1: tokenization turns text into model input

Suppose a client sends:

“Summarize this incident report in two bullets.”

The model cannot directly operate on characters or English words. The server first applies the model’s tokenizer, which breaks text into pieces called tokens and maps each piece to an integer ID.

A token is not reliably the same thing as a word. Depending on the tokenizer and text, a token may represent:

• a whole common word,
• a word fragment,
• a leading space plus a word fragment,
• punctuation,
• code characters,
• or a special control marker.

For a chat model, the input is usually more than the user’s visible sentence. The serving stack applies a chat template that adds model-specific structural tokens representing roles such as system, user, and assistant. So the actual sequence may conceptually contain:

1. a system instruction,
2. the user’s message,
3. role delimiters,
4. a marker telling the model it should now generate an assistant response.

The output of tokenization is a sequence of integer IDs. The model then uses each ID to look up a learned embedding vector from its embedding matrix. Those vectors, rather than raw text or integer IDs themselves, enter the transformer layers.

For inference engineering, tokenization matters because request size is measured in input tokens, not characters or words. Two inputs with the same character length can have different token counts, particularly across languages, code, structured data, and unusual strings. A server also must use the exact tokenizer family intended for the model’s weights.

---

Step 2: prefill processes the entire prompt

Once the token IDs are embedded, the model performs its first major request-specific computation: prefill, also called initial prompt processing.

During prefill, the transformer processes all prompt tokens through all its layers. The attention mechanism allows each position to incorporate relevant information from earlier positions in the sequence. Because decoder-only LLMs are causally masked, each position can attend only to tokens at or before its own position, never to future tokens.

At a high level, the model turns the final prompt position into a vector of scores, called logits, over every token in its vocabulary. A selection method then chooses the first generated token. Depending on configuration, selection might be deterministic greedy selection or controlled sampling using settings such as temperature and top-p.

It is common to say “prefill generates the first token.” More precisely:

• prefill processes the prompt and produces the logits used for the first selection;
• selecting that token marks the boundary between prefill and decode;
• subsequent selected tokens belong to the decoding loop.

The model also calculates and saves attention keys and values for the prompt tokens. This saved state is the beginning of the KV cache. You will calculate its memory cost in a later lesson; for now, its purpose is straightforward: it prevents the model from recomputing the full prompt on every generated token.

Why can prefill be expensive? A longer prompt means more tokens must pass through every layer, and attention has more relationships to consider. Yet prefill often uses hardware efficiently because many prompt positions can be processed together in parallel.

A simple trace might look like this:

Phase	Sequence known to the model	Work performed
Tokenization	Raw prompt text	Convert text into token IDs
Prefill	All prompt token IDs	Process the prompt, populate KV cache, produce first-token logits
First selection	Prompt plus first output token	Choose the initial generated token
Decode step 1	Prompt plus first output	Predict the next token
Decode step 2	Prompt plus first two outputs	Predict another token
Stop	Full sequence	Return final text and release request state

The phrase “all prompt tokens are processed at once” refers to parallel tensor computation, not a claim that the model can see future prompt positions. Causal masking preserves the left-to-right language-model objective while allowing efficient parallel processing of the supplied prompt.

---

Step 3: autoregressive decoding grows the response

After the first output token has been selected, the model enters autoregressive decoding. Autoregressive means that each new token is conditioned on the prompt and all previously generated output tokens.

Consider a deliberately simplified completion:

• Prompt: “The capital of France is”
• First selected token: “ Paris”
• Next selected token: “.”
• Next selected token: an end-of-sequence marker

The visible response seems to arrive as a short sentence, but internally the sequence grows one selected token at a time. The newly selected token becomes part of the context for the next selection.

Without caching, producing each new token would require reprocessing every previous token from scratch. That would become increasingly wasteful as the response grows. With a KV cache, the runtime retains prior keys and values. At each decode step, it processes primarily the newly introduced token, attends to the cached context, appends the new key and value information to the cache, and computes the next-token logits.

This does not make generation free. Each new token still passes through all model layers, reads a growing cache, and requires a vocabulary-level output computation. Decode is inherently sequential for one request: the choice of token $t+1$ depends on token $t$. That dependency is why a single response cannot be fully generated in parallel in the way prefill can.

A useful but intentionally high-level view of one decode step is:

1. The runtime supplies the latest token and prior cached state to the transformer.
2. The transformer computes representations for that newest position.
3. Attention uses the new query against cached keys and combines cached values.
4. The final layer produces logits across the vocabulary.
5. The sampler selects a token ID according to the generation settings.
6. The runtime adds the new token’s state to the cache.
7. The process continues unless a stopping condition has been reached.

Common stopping conditions are:

• an end-of-sequence token,
• a configured maximum number of output tokens,
• a matched stop string or stop-token sequence,
• user cancellation,
• or a server-imposed timeout.

Streaming changes when the client receives text, not the underlying generation logic. With streaming enabled, the server can detokenize and send incremental text fragments as tokens are selected rather than waiting for the entire completion to finish.

---

Step 4: detokenization makes output readable

The model’s direct output is a token ID, not a word. Detokenization converts one or more generated IDs back into text using the same tokenizer’s vocabulary and decoding rules.

This is why streamed chunks can sometimes look surprising:

• A token may begin with a space, so the first visible chunk may include leading whitespace.
• One word may span multiple tokens.
• Unicode characters may require multiple pieces before they form valid displayable text.
• Internal special tokens should usually be hidden from the user.

A robust serving system does not assume that one token equals one word, one character, or one client-visible chunk. At the API boundary, its job is to turn a sequence of selected token IDs into correctly decoded text while preserving streaming behavior and stop conditions.

---

Trace a complete request

Here is the whole lifecycle for a warmed inference server receiving a chat request:

1. The model is already loaded. Configuration, tokenizer, and weights are resident in the runtime’s intended memory locations.
2. The server validates and prepares the request. It applies the chat template, generation settings, and any input limits.
3. The tokenizer converts prepared text into token IDs.
4. Embedding lookup converts IDs into learned numeric vectors.
5. Prefill processes every prompt token through the transformer. It builds initial KV-cache entries and produces logits for the first output token.
6. The runtime selects the first output token.
7. Autoregressive decode repeats. Each iteration incorporates the latest token, reuses cached context, computes new logits, and selects one additional token.
8. Detokenization converts generated IDs into text. In streaming mode, this can occur incrementally.
9. Generation stops. The server returns a final response and frees the request’s temporary state, including its KV-cache allocation.

This is the causal story you should be able to narrate in an interview, while reading engine logs, or when explaining why a long document delays the first token but a long completion continues to consume generation time afterward.

---

Key takeaways

• Model loading is a startup concern: architecture, tokenizer, and weights must be compatible and available in memory before requests can run.
• Tokenization converts input text into model-specific integer IDs; tokens are not necessarily words.
• Prefill processes the whole prompt, builds initial cached attention state, and provides the logits for the first generated token.
• Decode is autoregressive: each newly selected token becomes context for the next one.
• KV caching avoids recomputing the entire prior sequence at each decode step, at the cost of memory.
• Detokenization translates output token IDs back into readable text, often incrementally during streaming.

Next, you will quantify the largest persistent part of this picture: how much memory model weights require at different parameter counts and numerical precisions.