Mocking Jev Responses with Deterministic Test Fixtures

Hello again. In the previous lesson, you made an authenticated Python call with uv, kept TYPESAFE_API_KEY outside the project, and consumed a live Choice result through response.choices["category"].

A live call proves that your integration works, but it is the wrong dependency for ordinary automated tests: it requires credentials and network access, costs money, adds latency, and may not return identical values every time. This lesson makes the consumer of a Jev result testable by supplying a fixed, typed response fixture instead.

By the end, you will have a committed Python fixture that represents a complete Jev response for one Choice question, plus tests that run without an SDK client, an API key, or a network request.

---

A fixture freezes the boundary, not the model

A test fixture is known input data kept under source control. Here, it represents a specific Jev response that your application might receive. The important word is specific: the fixture does not claim that Jev will always classify a similar ticket the same way. It lets you verify what your code does when it receives this result.

There are three useful kinds of checks in a Jev application:

Check	Uses live Jev?	Needs credentials?	What it establishes
Local unit test with a fixture	No	No	Your routing, display, thresholds, and fallback logic behave as intended
Manual smoke test	Yes	Yes	The installed SDK, key, and basic request path work
Controlled integration or evaluation run	Yes	Yes	The real model performs acceptably on representative data

The previous lesson was a smoke test. This lesson focuses on the first row.

This separation matters in CI. A pull request that changes a deterministic routing rule should not fail merely because a provider is temporarily unreachable, an API key is unavailable to a forked repository, or a model response differs from a previous call.

For a concise explanation of why network-dependent tests become slow and inconsistent, watch the opening of How to Mock Fetch in Jest Manually by Leigh Halliday. Although its code uses JavaScript, the testing principle applies equally to Python.

A fixture test is not a replacement for real evaluation. It tests the deterministic part of your system: given this typed judgment, what action follows?

---

Start with the documented response contract

The TypeSafe API reference describes a response as:

• a model identifier;
• an answers map keyed by the question IDs you supplied;
• usage metadata;
• an answer whose fields depend on its type.

For a Choice answer, the selected choice is the highest-probability option, probabilities contains all defined options, and confidence is derived from that distribution.

A complete fixture does not need to include every Jev answer type. It needs to accurately represent one complete response for the particular request being tested. Since the previous request had one Choice question, this fixture has one Choice answer.

One detail needs care: the API reference shows the wire-style answers map, while the Python SDK result used in the prior lesson exposes convenience collections such as:

response.choices["category"]

Do not pretend these are the same Python object. Instead, keep your business decision code independent of either representation. Production code can pass fields from the SDK object; a test can pass the corresponding fields from the documented response fixture.

---

Create a typed, deterministic response fixture

In your existing jev-python-demo project, install pytest as a development dependency:

uv add --dev pytest

Create jev_fixtures.py at the project root:

from typing import Literal, TypedDict


Department = Literal["billing", "technical", "account", "other"]


class CategoryAnswer(TypedDict):
    type: Literal["choice"]
    choice: Department
    probabilities: dict[Department, float]
    confidence: float


class CategoryAnswers(TypedDict):
    category: CategoryAnswer


class Usage(TypedDict):
    input_tokens: int
    output_tokens: int


class CategoryResponseFixture(TypedDict):
    model: str
    answers: CategoryAnswers
    usage: Usage


TECHNICAL_CATEGORY_RESPONSE: CategoryResponseFixture = {
    "model": "jev-latest",
    "answers": {
        "category": {
            "type": "choice",
            "choice": "technical",
            "probabilities": {
                "billing": 0.08,
                "technical": 0.85,
                "account": 0.05,
                "other": 0.02,
            },
            "confidence": 0.82,
        }
    },
    "usage": {
        "input_tokens": 128,
        "output_tokens": 12,
    },
}


LOW_CONFIDENCE_CATEGORY_RESPONSE: CategoryResponseFixture = {
    "model": "jev-latest",
    "answers": {
        "category": {
            "type": "choice",
            "choice": "technical",
            "probabilities": {
                "billing": 0.31,
                "technical": 0.36,
                "account": 0.19,
                "other": 0.14,
            },
            "confidence": 0.18,
        }
    },
    "usage": {
        "input_tokens": 128,
        "output_tokens": 12,
    },
}

These are deliberately synthetic fixtures. Their token usage is not a measurement, their model name is not a version-selection policy, and their values should not be used to estimate production cost. They exist to exercise code paths predictably.

The TypedDict definitions serve two practical purposes:

1. They document the contract immediately beside the test data.
2. A type checker can detect obvious mistakes, such as writing "techincal" instead of "technical" or omitting confidence.

The fixture also preserves important Choice invariants:

• The question ID is exactly "category", matching the previous request.
• technical is the option with the highest probability.
• The option probabilities sum to $1$.
• Confidence lies between $0$ and $1$.

Do not invent a formula for confidence in tests. The API documents it as derived from the probability distribution, but your application should consume the reported value rather than attempt to recalculate it.

---

Put the application decision behind a pure function

Now create routing.py:

from typing import Literal

from jev_fixtures import Department


RoutingDecision = Literal[
    "billing_queue",
    "technical_queue",
    "account_queue",
    "manual_review",
]

MIN_ROUTING_CONFIDENCE = 0.70


def decide_category(
    choice: Department,
    confidence: float,
) -> RoutingDecision:
    if confidence < MIN_ROUTING_CONFIDENCE:
        return "manual_review"

    if choice == "billing":
        return "billing_queue"

    if choice == "technical":
        return "technical_queue"

    if choice == "account":
        return "account_queue"

    return "manual_review"

This function has no client, no environment-variable lookup, no I/O, and no model dependency. It is deterministic: the same choice and confidence always produce the same decision.

In the live script from the previous lesson, the production call site would use it like this:

answer = response.choices["category"]

decision = decide_category(
    choice=answer.choice,
    confidence=answer.confidence,
)

The live SDK remains responsible for authentication and transport. Your deterministic application code is responsible for deciding what to do with the typed output.

The specific 0.70 threshold is only a placeholder policy for this exercise. Later in the course, you will choose thresholds using labeled data, confidence calibration, and the relative cost of incorrect automation versus review. For now, the key design point is that the threshold is explicit, named, and testable.

---

Test the fixture and the behavior it drives

Create test_routing.py at the project root:

from pytest import approx

from jev_fixtures import (
    LOW_CONFIDENCE_CATEGORY_RESPONSE,
    TECHNICAL_CATEGORY_RESPONSE,
)
from routing import decide_category


def test_choice_fixture_respects_its_contract() -> None:
    answer = TECHNICAL_CATEGORY_RESPONSE["answers"]["category"]

    most_likely_option = max(
        answer["probabilities"].items(),
        key=lambda item: item[1],
    )[0]

    assert answer["choice"] == most_likely_option
    assert sum(answer["probabilities"].values()) == approx(1.0)
    assert 0.0 <= answer["confidence"] <= 1.0


def test_high_confidence_technical_category_routes_to_technical_queue() -> None:
    answer = TECHNICAL_CATEGORY_RESPONSE["answers"]["category"]

    decision = decide_category(
        choice=answer["choice"],
        confidence=answer["confidence"],
    )

    assert decision == "technical_queue"


def test_low_confidence_category_is_sent_to_manual_review() -> None:
    answer = LOW_CONFIDENCE_CATEGORY_RESPONSE["answers"]["category"]

    decision = decide_category(
        choice=answer["choice"],
        confidence=answer["confidence"],
    )

    assert decision == "manual_review"

Run the suite:

uv run pytest

The expected result has the general form:

3 passed

Notice what is absent:

• No TypeSafeClient()
• No system_one(...)
• No TYPESAFE_API_KEY
• No network request
• No dependency on a particular live model output

This is the desired test boundary. The fixture supplies a stable representation of a response; the test verifies your deterministic policy.

---

Keep fixtures realistic and safe

A fixture is code-adjacent test data, so it deserves the same review as source code. An unrealistic fixture can make tests pass while hiding a real integration problem.

Preserve response invariants

For a Choice fixture:

Field	Fixture rule
type	Must be "choice"
choice	Must be one of the allowed options
probabilities	Must include every allowed option
Probabilities	Must sum to $1$
Selected choice	Must have the highest probability
confidence	Must be between $0$ and $1$
Answer key	Must match the question ID used by the request

The first test in test_routing.py protects several of these rules. That test may seem redundant because you authored the fixture yourself, but it prevents a later edit from silently creating an impossible test case.

Keep secrets and sensitive evidence out

A response fixture should never contain:

• TYPESAFE_API_KEY
• Authorization headers
• copied request logs containing credentials
• customer emails, account identifiers, payment data, or access tokens
• production ticket text unless it has been properly sanitized and approved for test use

The fixtures above contain only bounded category labels, numbers, a model identifier, and synthetic token counts. They are safe to commit.

Your .gitignore should still exclude local secret material and environments:

.venv/
.env

Unlike .env, jev_fixtures.py should normally be committed. Shared deterministic fixtures make local development and CI agree on the behaviors that matter.

Prefer named scenarios over mutating imports

Avoid changing a shared fixture in place inside a test. A mutation can leak into another test and create order-dependent failures.

Prefer separate, intention-revealing fixtures such as:

• TECHNICAL_CATEGORY_RESPONSE
• LOW_CONFIDENCE_CATEGORY_RESPONSE
• BILLING_CATEGORY_RESPONSE

A reviewer should be able to understand the scenario from the fixture name before reading its internal values.

---

Fixture tests versus live tests

It is tempting to use a recorded live response as a fixture. That can be useful, but only after a careful review.

A sensible workflow is:

1. Use a synthetic, non-sensitive state in a manual live call.
2. Copy only the response fields required for the test.
3. Remove headers, credentials, request payloads, and identifying metadata.
4. Check that question IDs, options, and probabilities still satisfy the documented contract.
5. Commit the sanitized fixture.
6. Run unit tests only against the committed fixture.

Do not make a unit test call Jev simply to “refresh” its expected answer. That turns a deterministic test into a remote integration test and makes it impossible to tell whether a failure came from your code, the network, authentication, service availability, or a changed model judgment.

---

Completion checklist

Before continuing, confirm that:

• uv add --dev pytest updated pyproject.toml and uv.lock.
• jev_fixtures.py contains a complete, typed response for the category question.
• The fixture contains no API key, request headers, customer text, or other sensitive data.
• routing.py makes its decision without importing the SDK or reading environment variables.
• uv run pytest passes without a live Jev call.
• Your Choice fixture has probabilities that sum to $1$, with the selected option as the most probable option.

---

Key takeaways

A deterministic Jev fixture freezes a possible typed response, not a claim about how the model will always behave. It allows you to test the business logic that follows a Jev decision without credentials, cost, network access, or model variability.

For the Python SDK integration from the previous lesson, keep the boundary clear:

• Live code reads the SDK’s response.choices["category"].
• A fixture can preserve the documented API-style answers["category"] structure.
• Both pass the same bounded fields, such as choice and confidence, into a pure application function.
• Fixture data must preserve Choice invariants and exclude secrets and sensitive customer evidence.

Next, you will move upstream from the response and design the state payload itself: what evidence a Jev decision should receive, what to omit as irrelevant, and what must never be sent.