Designing Atomic Evaluation Cases for Isolated Agent Behaviors

Good to see you again. In the previous lesson, you turned the payments-support agent’s specification into an evaluation inventory: a list of behaviours that need evidence, from grounded payment-status help to safe handovers for refund, fraud, and explicit human-support requests.

An inventory entry is not yet runnable. This lesson takes the next step: writing the conversation core of an evaluation case. The aim is precision rather than realism for its own sake: a case should create one clear situation in which one agent behaviour can succeed or fail. In the following lesson, you will write the pass criteria that judge the reply.

---

What makes an evaluation case atomic?

An atomic evaluation case tests one primary agent behaviour under a fixed, reproducible conversational situation.

For a support agent, “handles a refund request well” is not atomic. It potentially combines several behaviours:

• recognizing the request as out of scope;
• avoiding a promise that money will be returned;
• initiating the correct handover;
• explaining the next step;
• preserving context for the specialist.

If that case fails, you would not know what to fix. A missing handover route, an overpromising prompt, and an incomplete handover payload are different defects with different owners.

Instead, reduce the case to one behavioural proposition:

When a customer explicitly requests a human, the agent initiates the payments-specialist handover.

That proposition has one trigger, one expected agent action, and one primary failure family: Handoff.

The LangChain Agent Evaluation Readiness Checklist gives a useful test for this level of precision: if two reviewers cannot agree on whether a task succeeded, the task or its success definition needs further refinement.

Atomic does not mean trivial. A multi-step payment transfer can be a legitimate evaluation target, but only when the thing being tested is the ordered transfer behaviour as a whole. Conversely, a simple one-turn handover request becomes non-atomic if you also try to test fraud recognition, refund rules, account lookup, and response tone in the same conversation.

A practical diagnostic is this:

If the case fails, can you name one primary behaviour that was not demonstrated and one likely place to investigate first?

If the answer is “several,” split the case.

---

The anatomy of a focused conversation

For the workbook used in this course, a conversation is a fixed sequence of lines:

• each line begins with user: or agent:;
• the final line is always a user: message;
• the agent’s response to that final line is the output being evaluated.

A one-turn case therefore has the simplest possible form:

user: I need to speak to a real person right now.

The message is short because it needs to expose only one decision: whether an explicit request for a person triggers the handover route. It does not mention an overdue payment, a suspected scam, a refund, or an account identifier. Those additions would introduce competing reasons to hand over and make the eventual failure harder to interpret.

Use the following construction method.

1. Begin with one inventory item

Choose one row from the previous lesson’s inventory. For example:

• I-05: Immediately route when the customer requests a human.

Do not start by brainstorming customer messages. Start with the behaviour you already decided requires evidence.

2. Express the primary behaviour in one sentence

Write it in a form that has a condition and an observable action:

When the customer explicitly requests a human, the agent initiates the payments-specialist handover.

This sentence is not yet the final pass criterion. It is a design constraint on the conversation: every included detail must help establish this situation.

3. Include only the context needed to make the behaviour possible

Ask: What must the agent know from the conversation in order for the final response to be fairly judged?

For the explicit-human-request case, the answer is: nothing beyond the request itself. It should be a single-turn case.

For a case about the second unsuccessful attempt to gather information, previous turns are essential. Without them, the agent has no basis for recognizing that the escalation threshold has been reached.

4. Make the final customer message natural, but not noisy

A fixed test input should resemble how customers speak, including concise or imperfect wording where appropriate. But naturalness is not a reason to add irrelevant complications.

Compare these alternatives:

Conversation ending	What it actually tests
user: I need a real person right now.	Explicit human-request handover
user: I need a real person, my payment is late and someone may have stolen my card. Can you refund it?	Human request, payment status, fraud handling, refund boundary, and possibly tool avoidance
user: real person please, this bot is useless	Explicit human-request handover under frustration; useful later as a wording variant, but not required for the base case

The first version is the best initial case. It proves the core requirement with the least ambiguity.

5. Check for a competing correct response

A good case is solvable: the agent has enough information and a permitted route to succeed.

For example, a status-lookup test that says only “Where is my payment?” may be broken if the agent requires a payment reference or another identifier that the case does not supply. In that situation, asking a clarifying question could be correct; calling a tool could be wrong; handover might also be permitted. The case cannot cleanly identify the intended behaviour.

Do not repair such a case by inventing account details, tool parameters, or policy facts that the specification has not defined. Record the missing dependency and choose a different inventory item until the relevant design decision exists.

---

Start simple: single turn before multi-turn

The staged approach below is particularly useful when building a new suite. Early cases should establish stable, easy-to-diagnose evidence before they attempt realistic conversational complexity.

For the payments-support agent, apply the same progression even though the initial channel is web chat:

Stage	Appropriate case shape	What it establishes
Crawl	Synthetic, single-turn conversation	The core routing or answer behaviour works under a clear signal
Walk	More natural customer wording and channel-specific input	The same behaviour survives realistic phrasing, ambiguity, or noise
Run	Multi-turn conversation with necessary history	The agent uses prior context, follows a threshold, or recovers from a failure

Do not begin with “Run” cases merely because real customer conversations are multi-turn. A long transcript creates many possible causes of failure. The point of a multi-turn case is not realism alone; it is to establish state that the final user message cannot express by itself.

The Google Cloud Tech video distinguishes component-level checks from full trajectory and human-review evaluation. That distinction helps decide whether your case needs only a final reply or evidence from the agent’s entire course of action.

---

Worked case: direct request for a person

Here is an atomic case derived from inventory item I-05.

Workbook field	Value
Case name	explicit-human-handover
Conversation	user: I need to speak to a real person right now.
Primary behaviour	Initiate the payments-specialist handover when a customer explicitly requests a human
Primary suite	Handoff
Expected evidence later	Final reply and handover record or trace

The case is well formed for four reasons.

First, its trigger is unambiguous. “Speak to a real person” is not a vague dissatisfaction signal that could reasonably be handled by offering more help. It directly invokes the handover rule.

Second, it contains no payment facts. The agent cannot be distracted into answering a payment-status or fee question.

Third, it does not test the entire handover design. A separate case will test whether the handover payload includes summary, intent, reason, verified context, and prior actions. If the agent routes correctly but loses the context, you want that to be a distinct, diagnosable failure.

Finally, the conversation is frozen. Each future agent version sees the same words, so a difference in result can be attributed to changes in the agent or its environment rather than a changed test.

An overloaded version to reject

Consider this apparently realistic alternative:

user: I need a person now. My payment still hasn't arrived, I think it may be fraud, and I want the money reversed today.

This should not be one case. A failure could reflect any of the following:

• failure to honor an explicit human request;
• failure to recognize suspected unauthorized activity;
• an improper reversal promise;
• a prohibited date promise;
• an unnecessary or missing status lookup;
• an incorrect escalation route.

The response might correctly hand over but still make a prohibited promise. Or it might fail to hand over while correctly refusing the reversal. One PASS or FAIL verdict would obscure the diagnosis.

Split it into at least these cases:

1. Explicit human request — tests handover on request.
2. Suspected unauthorized activity — tests the security-sensitive route.
3. Request to reverse a payment — tests the money-movement boundary.

Each gets its own conversation and eventually its own pass criteria.

---

When a multi-turn conversation is necessary

The reference specification states that the agent should hand over after two unsuccessful in-scope attempts. That behaviour cannot be tested fairly in one turn, because the agent needs prior history to know how many attempts have occurred.

A focused multi-turn case might be:

user: Where is my payment?
agent: Please provide an identifier or details that let me verify the payment.
user: I do not have any of those details.
agent: Is there any other payment information you can share?
user: No, I do not have any payment information.

The last user line is the scored input. The preceding messages exist for one reason only: they establish two unsuccessful attempts to obtain verifiable information. The target behaviour is now:

After two unsuccessful attempts to obtain information needed for an in-scope payment-status request, the agent initiates handover rather than repeating the request or guessing.

Notice what this case deliberately avoids:

• a named tool or invented parameter requirements;
• a specific payment amount, date, or reference;
• an additional refund or fraud request;
• a claim about what the payment status actually is.

Those details would introduce rules not needed to test the threshold-based handover.

Prior agent messages are scenario context, not the result being graded

In this kind of case, the agent: lines represent the fixed history provided to the agent. They are not earlier outputs you are judging in this particular row. The evaluation assesses the reply to the last user line.

If you also need to test whether the agent’s first clarifying question was appropriate, write a separate single-turn case. Do not attempt to grade both the first question and the escalation after a second failure in one row.

---

A final atomicity check before adding a case

Before entering a conversation into the evaluation workbook, review it against this checklist.

Check	Question to ask
One primary behaviour	Can you complete “This case tests whether the agent…” with one clear verb and object?
Minimal context	Does every prior message establish information necessary for the final decision?
Single-turn by default	Could the same behaviour be tested without history? If yes, remove the history.
No invented facts	Have you avoided made-up policy values, payment details, tool requirements, or specialist procedures?
No competing objective	Would a failure identify one main failure family rather than several unrelated ones?
Runnable ending	Is the last line a customer message that the agent can answer?
Fixed wording	Will every future iteration receive the identical conversation?

One nuance matters: a case can still run global safeguards such as a No PII check alongside its primary behaviour. That does not make the conversation non-atomic. The distinction is between the case’s central diagnostic purpose and universal checks that apply to all agent outputs.

For now, leave the Match, Pass criteria, and detailed grader fields incomplete if you are drafting outside the workbook. You have succeeded at this lesson once the conversation cleanly creates the situation in which the intended behaviour can be observed.

---

Key takeaways

An atomic evaluation case is a fixed conversation designed to expose one primary agent behaviour. Start from an inventory item, formulate the specific behaviour to prove, include only the context required for that behaviour, and ensure the final line is the user message the agent must answer.

Use a single turn whenever possible. Use multi-turn context only when the target behaviour depends on earlier state, such as the payments agent’s escalation rule after two unsuccessful attempts. Avoid overloaded “realistic” messages that merge handover, fraud, refunds, tool use, and policy answers into one ambiguous verdict.

Next, you will turn these focused conversations into reliable tests by writing self-contained pass criteria: criteria that state what the reply must do and the precise condition that makes it fail.