Deriving an Evaluation Inventory from Specification Requirements

Good to see you again. In the previous lesson, you defined the Retail Payments Support Agent as an authenticated web-chat agent that provides grounded payment-status and published-fee information, while handing money-moving, security-sensitive, unsupported, and explicitly human-requested matters to a payments specialist.

That specification is now a source of test obligations. This lesson turns its prose into an evaluation inventory: a structured list of everything the eventual suite must prove or catch. We are not yet writing full conversations or pass criteria. Instead, we are making sure no important requirement disappears between specification and testing.

By the end, you should be able to read a specification systematically and derive coverage from its must, never, tool, knowledge-source, and out-of-scope statements.

---

From a specification to an evaluation inventory

An evaluation suite answers a deceptively simple question: did the agent become better after a change? That question is impossible to answer if the team has not first decided what behaviours count as success and which failures matter.

The route from a business specification to a runnable test suite has three distinct artifacts:

1. Specification — the operational contract: what the agent may do, must do, and must not do.
2. Evaluation inventory — the coverage plan: the requirements that need evidence, the failure family they belong to, and the kinds of scenarios needed.
3. Evaluation cases — runnable records containing a specific conversation, a match method, pass criteria, and any required graders.

Keeping the inventory separate is valuable. If you skip directly to writing cases, you tend to write the memorable “happy path” examples first, then discover much later that nobody tested a failed handover, an unsupported question, or a tool called when it should have remained unused.

The Components of Evaluations for Agents diagram shows the distinction visually. A task contains the inputs and definition of success; a trial is one execution; graders inspect the final outcome and, where relevant, the full trajectory of messages and tool calls. The inventory is what ensures the collection of tasks actually represents the agent’s contract.

An inventory therefore does not ask, “What prompts shall we try?” It asks:

• Which requirement could be violated?
• What would a customer say that makes the violation observable?
• What evidence would distinguish success from failure?
• Is this a normal success case, a boundary case, or a deliberately tempting failure case?
• Which failure family should own it when it fails?

This is particularly important for agents because an agent can produce plausible text while doing the wrong thing internally: consulting the wrong source, skipping a required handover, calling a tool twice, or promising an outcome it cannot authorize.

The video’s central point applies directly here: the difficult work is converting “we would recognize a bad reply when we saw one” into requirements that can be checked consistently. The inventory is the bridge.

---

Read requirements as coverage obligations

Start by extracting statements from the specification without trying to invent tests yet. Each statement belongs to one of five sources of evaluation demand.

Requirement source	What it means	What the inventory must include
Must	A required behaviour or outcome	At least one case where the agent must demonstrate it
Never	A prohibited claim, action, or disclosure	A case that tempts the prohibited behaviour
Tool	A live lookup or external action	One case requiring the tool and one case where using it would be wrong
Knowledge source	A document or approved searchable source	One covered question and one question the source does not cover
Out of scope	A request that must leave autonomous handling	A case proving correct handover, refusal, or safe fallback

The key term is observable. “Be helpful” cannot be inventoried because it does not identify an outcome. By contrast, “when the customer asks for a human, route to the payments specialist” is observable: the handover action either occurred or did not.

Do not confuse compound prose with one behaviour

Specifications often compress several obligations into one sentence:

“The agent does not initiate, approve, reverse, refund, or dispute payments.”

This sentence contains related prohibitions, but it may imply several independently important failures. An agent could correctly decline a refund request while incorrectly implying that it can reverse a payment. If those failures would be diagnosed or fixed differently, they deserve separate inventory entries.

There is an important restraint, however: do not create a new entry merely because a sentence contains several verbs. Split it when the following differ:

• the customer intent or phrasing;
• the correct safe response;
• the tool or workflow that must not occur;
• the risk or business impact;
• the evidence needed to grade it.

For example, a refund request, a dispute request, and suspected fraud all lead out of the agent’s autonomous scope, but they are not interchangeable:

• A refund request tests whether the agent avoids promising money or an approval.
• A dispute request may require a distinct specialist workflow.
• A fraud report tests urgency and a security-oriented handover boundary.

An inventory preserves these differences before they become blurred into a generic “escalation” test.

Positive, negative, and boundary coverage

For every requirement, decide which orientation the eventual case needs.

Orientation	Purpose	Example for the payments agent
Positive	Prove that a correct agent can do its authorized job	Answer a payment-status question using verified account data
Negative	Tempt a specific failure	Ask the agent to “just refund the two charges” and check that it does not promise a refund
Boundary	Test the point where authority or evidence runs out	Ask for a fee not covered by the published fee source
Failure recovery	Test what happens when the expected path cannot complete	Make the payments-specialist route unavailable

A mature suite needs all four. A suite of only positive cases shows that the agent can work in ideal conditions; it says little about whether it stops safely.

Out of scope is not the same as “never”

This distinction is easy to miss.

A never requirement says the agent must not perform or claim something:

Never approve a refund.

An out-of-scope requirement says what it must do instead when asked:

Route refund requests to a payments specialist and explain the next step.

The first guards against a harmful act or promise. The second guards against a dead end. A reply such as “I cannot help with that” might technically avoid approving a refund, yet still fail the handover requirement.

So, for important out-of-scope requests, the inventory normally includes both ideas:

• a case checking that the agent did not make the prohibited promise;
• a case checking that it provided the required route or fallback.

---

Build the inventory ledger

A practical inventory is a spreadsheet or workbook tab that sits beside the specification. It is not yet the evaluation workbook’s full case sheet. Its purpose is traceability: every obligation has an owner in the future suite.

Use these fields:

Field	Purpose
Inventory ID	A stable label, such as H-01 or K-02
Specification requirement	The exact rule, quoted or closely paraphrased
Requirement type	Must, never, tool, knowledge source, or out of scope
Behaviour to prove	One observable agent behaviour
Scenario orientation	Positive, negative, boundary, or failure recovery
Primary suite	The failure family used for reporting and prioritisation
Evidence needed	Reply text, tool trace, skill trace, source citation, or handover record
Dependency or gap	Any missing tool name, knowledge source, policy, or workflow detail

The last column is crucial. An inventory should expose ambiguity rather than conceal it with invented detail.

For example, the previous lesson specifies that payment-status answers use verified account data, but it does not yet name the status-lookup tool, its required inputs, or its response fields. Do not fabricate a tool name just to make the inventory look complete. Record a dependency such as:

“Status lookup interface is required; tool name, authentication assumption, and minimum input fields are not yet specified.”

That gap will be resolved in the Data and Agent layers. It is still legitimate to reserve two coverage obligations now: one where the status lookup is required and one where it must not be called.

---

Inventory for the reference payments-support agent

Below is a first inventory derived from the working specification. It has 15 coverage obligations, enough to meet the initial-suite floor while still reflecting the agent’s narrow scope.

Some eventual cases will share context, but each row has one primary behaviour. That discipline prepares us for the next lesson, where each runnable case will isolate one behaviour.

ID	Specification requirement	Type	Primary behaviour to prove	Orientation	Primary suite	Evidence needed
I-01	Answer payment-status questions using verified account data	Must / tool	Retrieves and communicates a supported status answer	Positive	Wrong tool call	Reply and status-lookup trace
I-02	Answer published payment-fee questions using approved knowledge	Must / knowledge source	Answers a fee question from the approved fee material	Positive	Knowledge gap	Reply and source citation
I-03	Do not fill unsupported information from memory	Must / knowledge source	States that an uncovered fee or policy detail cannot be verified and offers the approved route	Boundary	Knowledge gap	Reply and retrieval record
I-04	A conversation completes only with a grounded answer or completed handover	Must	Does not treat an unanswered request for required information as resolved	Boundary	Dead-end	Conversation and final reply
I-05	Immediately route when the customer requests a human	Out of scope	Initiates the payments-specialist handover rather than continuing autonomous support	Negative	Handoff	Reply and handover record
I-06	Route suspected unauthorized activity, fraud, or account compromise	Out of scope	Performs the security-sensitive handover	Negative	Handoff	Reply and handover record
I-07	Route refund requests; do not promise an outcome	Out of scope / never	Hands over without stating or implying that a refund will happen	Negative	Overpromise	Reply and handover record
I-08	Route dispute requests	Out of scope / never	Hands over a request to challenge a payment rather than attempting to resolve it	Negative	Handoff	Reply and handover record
I-09	Do not reverse or initiate payments	Never	Declines a request to change money movement and uses the approved route	Negative	Handoff	Reply and action-tool trace
I-10	Do not make financial, legal, or regulatory decisions	Never	Avoids presenting an authorization or decision as final	Negative	Overpromise	Reply
I-11	Route information that cannot be verified from approved sources	Out of scope	Escalates rather than guessing when evidence is unavailable	Boundary	Knowledge gap	Reply and source or tool trace
I-12	Hand over after two unsuccessful in-scope attempts	Out of scope	Recognises the second unsuccessful attempt and routes rather than looping	Failure recovery	Dead-end	Multi-turn conversation and handover trace
I-13	Send summary, intent, reason, verified context, and actions attempted with every handover	Must	Preserves the required minimum context for the specialist	Positive	Handoff	Handover payload or trace
I-14	If the normal route is unavailable, provide the approved alternative and make no unverified promise	Must	Communicates a truthful fallback after failed handover	Failure recovery	Dead-end	Reply and failed-route trace
I-15	A status tool is not needed for a published fee question	Tool	Does not call the live status lookup for a fee-policy question	Negative	Wrong tool call	Tool trace

Why this is an inventory, not yet a test suite

Consider I-07, the refund-request item. It already tells us:

• the relevant requirement;
• the failure family;
• that the scenario should tempt the agent to overpromise;
• that a correct outcome includes a handover;
• that reply text and a handover record matter.

But it deliberately does not yet contain:

• a realistic customer message;
• a complete conversation;
• exact pass criteria;
• the match type;
• a specific grader configuration.

Those elements belong in the next lessons. Separating the work prevents a common mistake: drafting a plausible conversation first, then quietly allowing it to test three unrelated requirements at once.

Count requirements before inventing a round number

A useful starting rule for a first suite is:

• one initial case for each distinct must;
• one initial case for each meaningful never;
• two cases per tool;
• two cases per knowledge source;
• one case per out-of-scope category;
• additional phrasing variants for high-risk requirements, including frustrated and vague customers.

This is not a license to produce dozens of near-duplicates. The purpose is to reveal the suite’s implied size from the specification rather than deciding in advance that “twenty tests sounds thorough.”

For this agent, the highest-risk requirements are money-related promises, security-sensitive reports, and failed handovers. These deserve later variants because wording can change behaviour:

• “Refund me now.”
• “Can you sort out these charges?”
• “I think somebody used my card.”
• “I need a real person, not a chatbot.”
• “You have not helped me twice already.”

The inventory identifies the underlying requirement once; the suite later tests selected phrasings of that requirement.

---

Audit the inventory before writing cases

Before proceeding, conduct a short traceability audit.

1. Every specification obligation has a home

Read the specification sentence by sentence and mark the inventory row that covers it. If a rule has no row, either the inventory is incomplete or the rule is too vague to test.

For the reference agent, the phrase “using approved knowledge sources and verified account data” maps to I-01 through I-03 and I-11. The phrase “does not initiate, approve, reverse, refund, or dispute payments” maps to I-07 through I-10.

2. Each tool and source has both sides tested

For every tool:

• Include a case where it must be called.
• Include a case where it must not be called.

For every knowledge source:

• Include a question it does cover.
• Include a question it does not cover.

The negative side matters because a system can appear correct while taking unnecessary calls, using the wrong source, or confidently answering beyond the source’s scope.

3. High-impact risks have evidence beyond the reply

A reply alone cannot show whether a tool was called, whether it was called twice, or whether handover context was preserved. Mark behavioural evidence early.

For example:

• I-01 needs the status-lookup trace, not only a convincing sentence.
• I-13 needs the handover payload, not only “I’ve transferred you.”
• I-15 needs a trace proving the status tool was not used.

This is why agent evaluation is broader than judging prose quality.

4. Missing design decisions become explicit gaps

An inventory is successful when it exposes questions such as:

• What precisely is the normal payments-specialist route?
• What is the approved fallback if that route is unavailable?
• Which source contains published fee information?
• Which live system owns payment status?
• What customer context may be included in a handover?
• What counts as one “unsuccessful attempt”?

Those are not administrative details. They determine what a correct agent can truthfully do and what the suite can test.

---

Key takeaways and next step

An evaluation inventory is the structured coverage plan between an agent specification and individual runnable tests. Derive it from five requirement sources: must statements, never statements, tools, knowledge sources, and out-of-scope boundaries.

For the payments-support agent, the inventory must cover grounded answers, supported and unsupported knowledge questions, required and forbidden tool use, money-related non-promises, direct human requests, security-sensitive cases, repeated failed attempts, and handover recovery when the normal route is unavailable.

Most importantly, do not invent missing implementation details. Record them as design dependencies. A trustworthy suite begins by making ambiguity visible.

Next, you will take one inventory item and write an atomic evaluation case: a focused conversation that tests one agent behaviour without mixing it with several others.