How Context, Decoding, and Evidence Shape Model Performance

Welcome back. In the last lesson, you separated tasks an LLM should handle from tasks that must remain deterministic. That distinction is necessary, but not sufficient: even a well-chosen support-copilot feature can fail when the model receives too much context, samples too freely, or lacks the evidence needed to answer safely.

This lesson gives you an architect’s way to predict those failures before production. You will learn to reason about three levers:

• Context-window management: what the model can consider in one request;
• Decoding settings: how predictably it chooses its next tokens;
• Evidence availability: whether the retrieved material can actually support an answer.

For the SaaS support copilot, the target is not merely a fluent response. It is a response that is grounded, relevant, timely, repeatable enough for the task, and safely abstains when evidence is missing.

---

Context windows are a finite request budget

A context window is the maximum number of tokens a model can process in a request. It is best understood as the model’s working set for one generation, not as durable memory.

The application must send any information that should influence the answer: system instructions, prior conversation turns, the current user request, retrieved knowledge-base chunks, tool results, and a reservation for the model’s response. When an earlier conversation turn is no longer included, the model cannot reliably use it.

For an architect, a request should be treated as a token ledger. Let $W$ be the model’s context-window size:

$$T_{\mathrm{input}}+T_{\mathrm{reservedoutput}}\le W$$

Where:

$$T_{\mathrm{input}}=T_{\mathrm{system}}+T_{\mathrm{history}}+T_{\mathrm{query}}+T_{\mathrm{retrievedevidence}}+T_{\mathrm{toolresults}}$$

The output reservation matters. A model cannot produce an answer of the requested maximum length if the prompt has already consumed nearly the entire window. Platforms may also add formatting or service tokens, so production systems should leave a margin rather than planning to use every advertised token.

A support-copilot token-budget example

Assume a model provides an $8,192$-token context window. Your support copilot reserves:

Prompt component	Budget
System instructions and safety rules	400 tokens
Conversation history	1,200 tokens
User question	100 tokens
Desired response allowance	1,200 tokens

The amount left for retrieved evidence is:

$$8,192-(400+1,200+100+1,200)=5,292$$

If retrieval returns six chunks of roughly 900 tokens each, they consume 5,400 tokens. The request already exceeds the budget, before allowing for provider overhead. A naive system might silently truncate the beginning or end of the prompt, possibly removing instructions, the user’s question, or the most relevant evidence.

This is why “retrieve as much as possible” is not a RAG strategy. It is an uncontrolled input-growth strategy.

---

More context can reduce answer quality

The obvious failure mode is exceeding the hard context limit. But a more subtle problem arises well before that limit: the model may receive enough information to process, yet fail to use the right information.

The image below illustrates the Lost in the Middle effect. Across several models and input sizes, question-answering accuracy is typically better when the relevant document appears near the beginning or end of the prompt than when it sits in the middle of a long collection of documents.

For a SaaS support copilot, imagine a user asks:

“Why does SSO setup fail with SYNC_403 after we rotated our certificate?”

A weak retrieval pipeline might send:

• three generic SSO overview chunks;
• two outdated setup guides;
• the relevant certificate-rotation troubleshooting chunk;
• several near-duplicate error-code pages;
• a full tenant conversation history.

The model may produce a plausible response about SSO configuration while failing to use the one chunk that explains the actual certificate issue. The answer can sound competent, cite a generally relevant source, and still be operationally wrong.

The key point is:

A context window measures capacity, not attention quality, evidence quality, or factual correctness.

The Microsoft Learn RAG guidance provides the core controls for treating context as a budgeted evidence set rather than a document dump.

Context-management choices and their predictable effects

Design choice	Likely quality effect	Latency and cost effect	Reliability risk
Increase the number of retrieved chunks	May improve completeness if additional chunks are genuinely relevant; often adds noise	More input processing and cost	Relevant evidence can be diluted or buried
Use a tuned top-$k$ and relevance threshold	Improves relevance by excluding weak matches	Usually reduces prompt size	An overly strict threshold may omit needed evidence
De-duplicate overlapping chunks	Reduces repeated evidence and contradictory emphasis	Reduces input size	Poor de-duplication can discard distinct details
Truncate lowest-ranked chunks	Keeps the request within the hard limit	Efficient	Retrieval ranking errors can remove the decisive source
Summarize chunks before generation	Can increase information density	Adds at least one model call	Summaries can omit qualifications, exceptions, or precise steps
Use map-reduce or iterative refinement	Can cover large evidence sets systematically	Multiple calls increase latency and cost	Intermediate errors can affect the final synthesis

There is no universally correct $k$, chunk size, or history length. These are evaluation-driven choices. For the initial support copilot, use a small, well-ranked evidence set; reserve response tokens; and measure whether answers are both complete and cited.

Two particularly useful rules are:

1. Spend tokens on evidence, not repetition. Remove duplicate chunks, stale conversation turns, and generic content that does not help answer the request.
2. Keep relevance ahead of volume. Adding a weakly related document can lower answer quality even though it technically gives the model “more information.”

Context ordering can matter, but it is not a safety control. Put the strongest, most relevant evidence where the model is likely to use it, and evaluate alternative orderings. Do not assume that a model will always obey an instruction merely because the instruction appears first; later modules will address prompt injection and trust boundaries explicitly.

---

Decoding settings control variation, not knowledge

Once the model has its prompt, it generates text one token at a time. At each position, it assigns probabilities to possible next tokens. Decoding settings determine how it chooses among those candidates.

These settings can change a response from stable and repetitive to varied and exploratory. They cannot make unsupported claims true, repair poor retrieval, or turn an LLM into a deterministic policy engine.

Temperature

Temperature changes how strongly the model favors high-probability tokens.

• Lower temperature makes the model more likely to choose the most probable continuation. Responses tend to be more stable and less creative.
• Higher temperature gives lower-probability alternatives more chance of being selected. Responses become more diverse, but can drift, vary in wording, or include less likely details.
• A temperature of $0$ is often described as deterministic, but architecture should treat it as mostly deterministic, not perfectly repeatable. Provider infrastructure, model updates, hidden implementation details, and other sampling controls can still introduce variation.

For a cited support answer, the desired quality is ordinarily not creative variation. It is consistent use of retrieved evidence, clear instructions, and a safe fallback. Low-variation decoding is usually the sensible experiment. For drafting a friendly email subject line or generating several knowledge-base article titles, more variation might be useful.

Top-$p$, top-$k$, and the candidate pool

Top-$p$, sometimes called nucleus sampling, limits candidates to the smallest group whose cumulative probability reaches the selected probability mass.

A lower top-$p$ creates a narrower candidate pool and generally less variation. A higher top-$p$ permits a broader pool and generally more variation.

Some APIs also expose top-$k$, which limits selection to the $k$ most probable candidate tokens. Unlike top-$p$, it keeps a fixed number of candidates even when the probability distribution is very concentrated or very spread out.

Temperature, top-$p$, and top-$k$ all influence sampling. Changing all of them at once makes outcomes hard to interpret. In an evaluation, change one variable at a time and compare the same request-and-evidence set.

Output length and stopping controls

Maximum output tokens limits how long the response may become. It has a direct operational effect:

• A lower cap generally reduces generation time and variable output-token cost.
• A cap that is too low can cut off a troubleshooting step, a required citation, or a JSON response.
• A cap that is too high can permit unnecessary verbosity, higher latency, and more opportunities for the answer to wander beyond its evidence.

Stop sequences instruct the model to stop when it emits a specified string. They can help delimit a response format, but they are not a substitute for output validation. A stop sequence that accidentally appears in normal content can prematurely truncate a useful answer.

Frequency and presence penalties discourage repetition. They are potentially useful for creative generation, but in factual support work they need careful testing. Repeating an error code, product name, or prerequisite may be necessary for clarity; suppressing it merely to create lexical diversity can reduce answer quality.

A practical decoding stance for the first release

For the support copilot, choose a documented baseline configuration and test it against a small evaluation set. The baseline should favor:

• lower variation for grounded, operational answers;
• explicit response-length limits;
• response schemas or format contracts where downstream code depends on structure;
• evidence citations;
• fallback behavior when sources are inadequate.

Avoid claiming that a low temperature “prevents hallucinations.” It does not. It can make the same unsupported behavior more repeatable.

---

Missing evidence is a reliability event, not a prompt-writing defect

Consider this user request:

“Can Enterprise customers retain audit logs for seven years after downgrading to Pro?”

Suppose the retrieval system finds a Pro plan overview and a generic security page, but no document covering post-downgrade audit retention. The model has several possible behaviors:

1. Invent an answer based on generic SaaS conventions.
2. Make a vague but confident statement.
3. Answer only the part supported by evidence.
4. State that the approved documents do not contain sufficient information and route the request to the right support channel.

Only the latter two are acceptable for a production support copilot. The goal is not to force an answer; it is to make the system behave usefully under uncertainty.

Microsoft Learn’s grounding guidance turns this into explicit prompt and product behavior.

Distinguish four evidence states

A useful architecture does not treat all retrieval failures as one generic “low confidence” condition.

Evidence state	What it means	Appropriate support-copilot behavior
Sufficient evidence	Relevant, authorized sources support the material claims	Answer concisely and cite sources
No relevant evidence	Retrieval returns nothing useful or nothing clears the relevance threshold	Say the available documentation is insufficient; offer escalation or clarification
Partial evidence	Sources answer one part of a multi-part question but not the rest	Answer the supported portion; name the unanswered portion explicitly
Conflicting or stale evidence	Sources disagree, or version dates make the answer unclear	Surface the conflict with citations; do not silently choose a policy or entitlement outcome

This leads to a critical reliability principle:

A confident answer is not evidence of a supported answer.

A model can generate a self-reported confidence field, but that value should not be treated as a factual guarantee. More dependable signals come from system-controlled checks: retrieval scores, source coverage, document freshness, citation validation, source authorization, and evaluation results.

For the portfolio support copilot, define an answer policy in plain language before implementing it:

Answer only from retrieved, tenant-authorized sources. Cite each material claim. If sources do not support the answer, say so clearly. If only part of the request is supported, answer that part and identify the gap. If sources conflict, present the conflict rather than choosing silently.

This policy is both a prompt instruction and a product requirement. The user experience must make escalation practical: for example, create a categorized support ticket, show the sources considered, or ask a focused clarification question.

---

Predicting failure before it reaches a customer

An architect should be able to connect a design change to its likely consequences.

Change	Output-quality prediction	Latency prediction	Reliability prediction
Add entire knowledge-base articles to every request	More generic and distracted answers; critical detail may be overlooked	Higher input-processing time and cost	Greater chance of irrelevant or conflicting claims
Reduce retrieved chunks aggressively	Concise answers but potentially incomplete troubleshooting	Lower latency	Higher abstention or omission risk if relevant evidence is excluded
Increase temperature for support answers	More varied wording; potentially more drift	Usually little direct latency change	Less repeatable behavior; unsupported statements may vary more
Lower maximum output tokens sharply	Shorter answers, but instructions may be incomplete	Faster generation and lower output cost	Truncated responses can fail response contracts or omit citations
Omit explicit “insufficient evidence” behavior	Fluent answers may appear more helpful	No meaningful gain	Higher hallucination and unsafe-assurance risk
Add source citations and required fallback states	Slightly more prompt and output tokens	Small latency increase	Better auditability, easier evaluation, safer abstention
Summarize large evidence sets first	Can improve focus when documents are long	Extra model call increases latency	Summary may lose exceptions, dates, or contractual details

This table also shows why quality, latency, and reliability cannot be optimized independently. A longer context may improve completeness in some cases, but degrade relevance and latency. A shorter answer may improve perceived speed, but become unusable if it omits the final troubleshooting action. A low temperature may make outputs more consistent, but does nothing if the relevant document was never retrieved.

A compact diagnostic sequence

When a support-copilot answer is wrong or weak, investigate in this order:

1. Was the user’s request clear and in scope?
2. Did retrieval locate authorized, relevant, current evidence?
3. Did the evidence fit within the token budget without harmful truncation?
4. Was the decisive evidence placed in a usable prompt position and clearly labeled?
5. Did the model follow grounding and citation instructions?
6. Did decoding and output-length settings cause drift, repetition, or truncation?
7. Did the application correctly recognize insufficient, partial, or conflicting evidence?

This order matters. Teams often tune temperature first because it is easy to change. But a decoding adjustment cannot repair missing evidence, bad retrieval ranking, or a context budget that buries the required source.

---

Architect’s decision record: inference behavior

Add a small decision record to your portfolio project. It will later support both implementation choices and interview answers.

Decision area	Initial support-copilot position
Context budget	Reserve capacity for system instructions, current query, bounded history, evidence, and the response; keep a safety margin
Retrieval input	Use a tuned top-$k$, relevance threshold, source dates where relevant, and de-duplication
Overflow behavior	Remove lowest-ranked chunks first; evaluate summarization or multi-step patterns only when needed
Decoding	Prefer low-variation settings for cited operational guidance; test settings systematically rather than relying on intuition
Response length	Set a cap appropriate for concise support guidance, while reserving enough room for steps, citations, and fallback text
Evidence policy	Answer only from authorized sources; distinguish sufficient, absent, partial, and conflicting evidence
Evaluation	Measure groundedness, completeness, citation validity, abstention correctness, response latency, and token usage

In an interview, this gives you a concise defense:

“I treat the context window as a finite evidence budget. I reserve output tokens, retrieve only a small high-quality set of tenant-authorized chunks, and test ordering rather than assuming more context helps. For support guidance, I use low-variation decoding and explicit fallback behavior. If the evidence is absent or partial, the system must surface that gap and escalate rather than fabricate an answer.”

---

Key takeaways

A context window is the model’s finite working set for a request. System instructions, history, retrieved chunks, tool outputs, the user query, and generated output all compete for the same token budget. More context can increase cost and latency while reducing answer quality when relevant evidence is drowned out or placed poorly.

Decoding controls such as temperature, top-$p$, output-token limits, and repetition penalties influence variation and response shape. They do not create factual grounding. For a SaaS support copilot, start from low-variation, bounded outputs and validate behavior on representative requests.

Finally, missing evidence must produce a safe, explicit product behavior: abstain, answer only the supported portion, surface conflicts, ask for clarification, or escalate. Reliability is not making the model answer every question; it is making the system behave predictably when it cannot support an answer.

Next, you will use these constraints to choose among prompt-only generation, retrieval-augmented generation, tool use, and fine-tuning for specific SaaS scenarios.