Tracing a Generative AI Request End to End

Hello, and welcome to the course. You are beginning with the architectural habit that underlies nearly every production AI system: following a single user request end to end, rather than treating “the LLM” as the whole application.

By the end of this lesson, you should be able to trace a generative-AI request through:

• the client where a user initiates it,
• the application service that owns policy and orchestration,
• optional data stores used to retrieve organization-specific information,
• the model provider that generates output,
• and the response path that validates, records, and returns that output.

The concrete technologies will vary across companies. A frontend could be a web app, Slack bot, or mobile app; the model could be OpenAI, Bedrock, Vertex AI, or a model running locally. The responsibilities in the request path, however, remain remarkably stable.

---

A request is a journey, not one model call

Imagine a user asks an internal IT assistant:

“My laptop cannot connect to the company VPN after the latest update. What should I do?”

A demo application might put this text directly into an LLM and display the answer. A deployable application does more. It must establish who is asking, determine what data they may access, retrieve relevant company guidance, protect provider credentials, control cost, and make failures diagnosable.

A useful mental model is to divide the system into five responsibility boundaries:

Boundary	Typical components	Core responsibility
Client	Web UI, mobile app, Slack/Teams bot	Collect input and render the result
Application service	API backend, orchestration layer	Authenticate, validate, retrieve context, build prompts, call providers, enforce policies
Data stores	Relational DB, vector store, cache, object storage	Persist application data, documents, embeddings, histories, and sometimes reusable results
Model provider	Hosted LLM API or self-hosted inference service	Generate model output from the supplied request
Response and operations path	Output filters, logs, traces, analytics	Screen, format, record, and return the result

The application service is the control point. It is not merely a thin proxy to an LLM. In production, it makes the decisions that belong to your organization rather than to a third-party model provider: which user may call which model, which documents that user may retrieve, how much context may be sent, what output format is expected, and what gets recorded.

Watch the opening of Tech With Tim’s video for the key architectural reason that client applications should call your backend, not a model provider directly.

A browser is controlled by its user. Anything shipped to it, including a provider key, can be inspected and copied. If that key grants model access, an attacker can spend your budget or access data under your account. The client may hold a user session token, but the protected backend holds provider credentials and uses them on the client’s behalf.

---

The minimal request trace

First consider a non-RAG application: one that does not need to retrieve private documents. The request still crosses several boundaries.

1. The client creates an application request

The client gathers the user’s message and sends an HTTPS request to an endpoint owned by the application, such as POST /chat.

The request generally contains:

• the current message;
• a conversation or session identifier, if there is a continuing conversation;
• the user’s identity credential, usually sent in an authorization header;
• client metadata such as application version or locale.

At this point, the client has not called the LLM and should not know the LLM provider key. It waits for the backend’s response, or, in a streaming interface, begins displaying partial text as it arrives.

2. The application service admits and validates it

The backend receives the request and commonly performs several checks before paying for a model call:

1. Authentication: Is the caller’s identity token valid?
2. Authorization: Is this user allowed to use this feature and its underlying data?
3. Validation: Is the payload structurally valid, within size limits, and appropriate for the endpoint?
4. Rate and budget checks: Has this user, tenant, or service exceeded a request or spending limit?
5. Request correlation: Assign or accept a request ID so all subsequent events can be tied to the same user action.

A request that fails at this stage should return a clear application error, such as “not authenticated,” “not permitted,” or “request too large.” It should not reach the model provider. Early rejection protects latency, cost, and security.

3. The application constructs a provider request

For a simple chat feature, the backend now turns the client’s request into the form expected by the provider. It may assemble:

• system-level instructions that define the assistant’s role;
• selected conversation history;
• the current user message;
• model parameters such as maximum output length;
• an expected response format.

The result is a provider request. Crucially, it is not necessarily identical to the text received from the user. The backend owns the full prompt contract and chooses what context to include.

4. The backend invokes the model provider

The application sends an authenticated request from its protected environment to the model provider. The provider processes the prompt and returns generated content plus useful metadata, often including model identifier, token counts, stop reason, and safety information.

A provider may be:

• a cloud-hosted API;
• a model endpoint within a company cloud account;
• a local or private inference service.

The deployment location changes networking, latency, compliance, and operations, but not the conceptual boundary: the application service requests inference; the model service supplies a result.

5. The application processes the result and replies

The raw provider output is not automatically a user-facing response. The application may:

• validate that the output has the expected structure;
• apply safety or policy checks;
• add citations or presentation metadata;
• record a trace of the call;
• transform provider-specific fields into a stable API response.

Finally, the backend returns a response to the client. The UI renders text, citations, buttons, cards, or an error state. To the user, this appears as one chat interaction. Architecturally, it was a coordinated sequence of calls across trust boundaries.

---

Adding organizational knowledge: the RAG request trace

A model’s general training does not contain reliable, current knowledge of a company’s internal VPN policy, product catalog, contracts, or support runbooks. Retrieval-augmented generation (RAG) adds a retrieval step so the application can supply relevant approved material at request time.

The key distinction is worth making now:

• Ingestion path: Documents are prepared before a user asks a question.
• Serving path: A user request queries the prepared data and uses retrieved material to generate an answer.

Do not confuse the two. The serving request usually does not parse every PDF or generate embeddings for every company document. It searches an index created earlier.

Read the architecture overview and serving flow in the Google Cloud reference. Treat the named Google services as examples of general roles: frontend, application backend, embedding model, vector store, LLM provider, response filter, and observability systems.

Here is the same IT question traced through a RAG-capable service.

1. Client to application service

The employee enters: “My laptop cannot connect to the company VPN after the latest update. What should I do?”

The web client sends the message, conversation ID, and employee authentication token to the company backend. The backend verifies that this is an employee and identifies the employee’s organization, role, and possibly device group or region.

Those identity attributes matter because retrieval must respect permissions. A contractor should not receive internal incident notes simply because their question resembles an employee’s question.

2. Application service to the embedding model

The backend sends the user query to an embedding model. Instead of generating prose, this model converts the text into a vector: a numerical representation intended to place semantically similar texts near one another.

The query “VPN cannot connect after update” can match a document entitled “Network client troubleshooting,” even if the exact phrase “cannot connect” never appears in that document. This is why semantic retrieval is useful.

3. Application service to the vector store

The backend searches a vector store using the query embedding. The store contains embeddings for chunks of approved documents, along with metadata such as:

• document and chunk identifiers;
• document title and source location;
• page or section reference;
• tenant or organization identifier;
• access-control attributes;
• ingestion version and timestamp.

The vector store returns the most relevant permitted chunks, perhaps a troubleshooting guide for the current VPN client version and a notice about a known update issue.

A vector store is not the model. It does not write an answer. It returns candidate evidence for the application to evaluate and include.

4. Application service constructs grounded context

The backend selects the returned chunks that fit its context budget and creates a contextualized prompt. It typically includes:

1. Instructions defining how to use the evidence.
2. The user’s current question.
3. Retrieved passages, labeled with source metadata.
4. Output requirements, such as “cite the supplied sources” or “say that you do not know when the sources are insufficient.”

The model sees this assembled request, not a direct database connection. The application decides what information reaches it.

5. Application service to model provider

The backend calls the LLM provider with the contextualized prompt. The LLM generates an answer based on its behavior and the supplied context. It may still make mistakes, which is why later modules will cover grounded generation, citations, abstention, and evaluation.

For now, the important trace is:

• the embedding model turns a query into a search representation;
• the vector store returns relevant source chunks;
• the LLM turns the original question plus chosen context into natural-language output.

6. Model response through safeguards and back to the client

The provider returns generated content to the backend. The backend can check the response before showing it to the employee. It may reject malformed structured output, remove disallowed content, attach citations, or replace a low-confidence answer with a safe fallback such as: “I could not find an approved procedure for this version; contact IT support.”

It then returns a stable response object to the client, for example:

{
  "answer": "Restart the VPN client, then install version 5.2.1...",
  "citations": [
    {
      "title": "VPN Client Troubleshooting",
      "section": "After a client update"
    }
  ],
  "request_id": "req_7f3c..."
}

The client should not need to know which provider generated the answer, what embedding model was used, or how retrieval was implemented. Those are backend implementation details behind an application contract.

---

Data stores are not all the same

“Database” is too broad to be useful when tracing an AI request. Distinguish stores by why the request touches them.

Store type	What it may hold	Why the serving request uses it
Application database	User profiles, tenant settings, conversation records, usage limits	Identify the user, retrieve session state, enforce product rules
Vector store	Text chunks, embeddings, source metadata, permissions	Retrieve semantically relevant and authorized knowledge
Cache	Reusable model responses or retrieval results	Reduce latency and repeated model cost
Object storage	Original uploaded PDFs, spreadsheets, images	Usually accessed during ingestion or when a source file must be opened
Observability store	Logs, metrics, traces, analytics events	Diagnose behavior, measure performance, and improve the service

A request need not touch every one of these. A simple general-purpose chat endpoint might only call an application database and an LLM provider. A knowledge assistant typically uses an application database, vector store, and model provider. A mature service may also check a cache and emit telemetry.

The GitHub architecture overview gives useful examples of the additional components surrounding a model call: vector retrieval and filtering on the request side, plus output cache, content filtering, and telemetry on the response side.

---

Trace the control plane, data plane, and observability plane

When diagnosing a system, it helps to separate three overlapping views of one request.

Control: who is allowed to do what?

Control includes authentication, authorization, rate limits, tenant policy, provider credentials, and model selection. These checks decide whether the request may proceed and what resources it may use.

For the VPN assistant, control decides whether the caller is a valid employee and whether they can retrieve a particular internal support document.

Data: what information moves and where?

Data includes the user message, conversation history, query embedding, retrieved document chunks, constructed prompt, model output, citations, and API response.

A useful security question is: Which data crosses each boundary? The application might send a user message and selected support passages to a hosted model provider. It should not automatically send the entire employee profile, every prior conversation, or unrestricted internal documents.

Observability: how can we reconstruct what happened?

A production service emits a correlated record of key events. A request_id enables an engineer to connect:

• the incoming client request;
• authorization result;
• retrieval latency and the IDs of returned chunks;
• prompt size;
• provider model name, latency, and token usage;
• output-screening decision;
• final HTTP status and latency.

Avoid logging secrets or unrestricted sensitive content. The goal is to make the system diagnosable without creating a new sensitive data repository.

This is particularly important for forward-deployed work. When a stakeholder says, “The assistant gave an unhelpful answer,” you need evidence to distinguish among several very different causes:

Observed problem	Plausible location in the trace
User receives an immediate permission error	Client authentication or backend authorization
The answer ignores current internal policy	Retrieval, context construction, or stale ingestion data
The answer is relevant but slow	Vector search, model inference, or a downstream integration
The answer is blocked unexpectedly	Input or output policy filter
The answer format breaks the UI	Prompt requirements, provider output, or backend response validation
Calls suddenly become expensive	Missing rate limits, cache misses, oversized context, or excessive model usage

You do not have to solve these failures yet. You do need to name the stage where each one occurs. That turns vague reports into engineering investigations.

---

A practical tracing template

When you encounter an unfamiliar AI application, do not begin by asking, “Which framework is it using?” First map a request using this template:

1. Entry: Which client or channel creates the request?
2. Identity: How does the application establish who the caller is?
3. Backend owner: Which service receives and orchestrates the request?
4. State: Which application data is read or written?
5. Knowledge: Does the service retrieve documents, call APIs, or use neither?
6. Model: Which model provider or inference service is invoked?
7. Guardrails: What checks occur before and after generation?
8. Return: What response contract is sent back to the client?
9. Evidence: Which correlated logs, metrics, and traces make the path inspectable?

Apply it to the VPN assistant:

• Entry: Employee web chat.
• Identity: Employee session token.
• Backend owner: Internal chat API.
• State: Conversation/session record and user permissions.
• Knowledge: Embedding service plus vector-store retrieval of permitted IT documentation.
• Model: Hosted or private LLM endpoint.
• Guardrails: Input size and authorization checks; output screening and citation formatting.
• Return: Answer, citations, and request ID sent to the web UI.
• Evidence: Correlated events around retrieval, model use, and final response.

A good architecture explanation names both components and the data or decision that crosses between them. Saying “the frontend talks to the backend” is incomplete. Saying “the frontend sends an authenticated chat message; the backend validates it, retrieves allowed context, invokes the LLM, validates the output, and returns an answer with citations” is a trace.

---

Key takeaways

A production generative-AI feature is an application system with an LLM inside it, not merely an LLM with a UI attached.

• The client collects input and displays results, but should not hold model-provider credentials.
• The application service is the protected control point for validation, authorization, orchestration, policy, and provider calls.
• A RAG request uses an embedding model and vector store to find relevant, permitted evidence before generation.
• The model provider generates output; it does not own your business rules, permissions, or source-selection policy.
• The response path may validate, filter, annotate, log, and then return the result.
• A correlation ID and structured operational events make each request traceable when quality, latency, security, or cost problems arise.

Next, you will turn a user-facing AI feature into measurable functional and non-functional requirements. The request path you mapped today will provide the structure: every component creates requirements around behavior, latency, reliability, privacy, cost, and observability.