Testing AI Services with Mocked Model Responses

Hello. In the previous lesson, you moved configuration and secrets out of source code, validated them at startup, and exposed settings through FastAPI dependencies. That dependency seam is exactly what makes an AI service testable without a real model API key or network request.

This lesson focuses on isolated tests: tests that exercise your application’s own behavior while replacing the model provider with a controlled stand-in. By the end, you will be able to test a successful generated answer, a provider failure, and a FastAPI endpoint without invoking a real LLM.

---

What isolation means for an AI service

An AI endpoint normally depends on systems outside your repository:

• a model provider and its credentials;
• the network;
• provider availability, rate limits, and latency;
• variable model outputs and changing model behavior.

Those are important to test eventually, but they should not determine whether your ordinary test suite passes. A test that calls a live provider can be slow, costly, flaky, and potentially send test data outside your environment.

Instead, isolate the part you own. For an answer endpoint, that means checking questions such as:

• Does the service send the intended user message and system instruction to its provider boundary?
• Does it convert a provider result into the typed AnswerResponse contract?
• Does the HTTP endpoint validate input and serialize the expected output?
• When the provider fails, does the service follow its current failure policy?

The supplied Mocked model-provider boundary diagram shows the desired shape: the test makes an in-process request to FastAPI, while a mock layer prevents its nominal HTTP client from reaching an external service. For this lesson, we will place the mock one level earlier, at a Python dependency boundary. That is usually simpler and more durable than intercepting raw HTTP requests.

A useful rule is:

Mock the boundary you do not own; assert the observable behavior you do own.

Do not mock your AnswerService when testing AnswerService: then the test would only prove that your mock behaves as configured. Similarly, do not mock FastAPI’s request handling when your goal is to verify the endpoint contract.

Before implementing the pattern, read FastAPI’s concise explanation of dependency overrides. It is the mechanism that lets an endpoint use a test provider without changing production code.

---

Create a small model-provider seam

A test becomes easy when production code accepts its dependencies rather than constructing them deep inside business logic. Define the smallest interface your answer service needs—not an entire provider SDK.

The code below assumes the previous lesson’s config.py has Settings and get_settings(). If your existing request and response model names differ, preserve their roles and adapt the names.

# app/providers.py
from typing import Protocol


class ModelProvider(Protocol):
    async def generate(
        self,
        *,
        system_instruction: str,
        user_message: str,
    ) -> str:
        """Return generated answer text."""

A Protocol describes the behavior required by the service. Any object with a compatible async generate() method can be used: a real provider adapter in production, an AsyncMock in a unit test, or a small fake in a local development environment.

Now keep application-specific prompt construction and response shaping in an AnswerService.

# app/services.py
from app.providers import ModelProvider
from app.schemas import AnswerResponse

SYSTEM_INSTRUCTION = (
    "You are a support assistant. Give concise, accurate answers."
)


class AnswerService:
    def __init__(
        self,
        provider: ModelProvider,
        model_name: str,
    ) -> None:
        self._provider = provider
        self._model_name = model_name

    async def answer(self, question: str) -> AnswerResponse:
        generated_text = await self._provider.generate(
            system_instruction=SYSTEM_INSTRUCTION,
            user_message=question,
        )

        return AnswerResponse(
            answer=generated_text,
            model=self._model_name,
        )

The service knows what it needs from a model: generated text for a specified instruction and message. It does not know whether the provider is OpenAI, Bedrock, another vendor, or a controlled test double.

For now, represent the production adapter as a separate implementation. Its actual SDK call belongs in the next module on reliable LLM API engineering.

# app/providers.py
from app.config import Settings


class HostedModelProvider:
    def __init__(self, settings: Settings) -> None:
        self._settings = settings

    async def generate(
        self,
        *,
        system_instruction: str,
        user_message: str,
    ) -> str:
        # Replace this body with the chosen provider SDK call later.
        raise NotImplementedError("Model-provider integration is not configured")

Finally, connect this provider to FastAPI through dependencies.

# app/dependencies.py
from typing import Annotated

from fastapi import Depends

from app.config import Settings, get_settings
from app.providers import HostedModelProvider, ModelProvider
from app.services import AnswerService


def get_model_provider(
    settings: Annotated[Settings, Depends(get_settings)],
) -> ModelProvider:
    return HostedModelProvider(settings)


def get_answer_service(
    provider: Annotated[ModelProvider, Depends(get_model_provider)],
    settings: Annotated[Settings, Depends(get_settings)],
) -> AnswerService:
    return AnswerService(
        provider=provider,
        model_name=settings.model_name,
    )

Your route remains thin. It validates HTTP input, delegates work to the service, and returns a typed response.

# app/main.py
from typing import Annotated

from fastapi import Depends, FastAPI

from app.dependencies import get_answer_service
from app.schemas import AnswerRequest, AnswerResponse
from app.services import AnswerService

app = FastAPI()


@app.post("/v1/answers", response_model=AnswerResponse)
async def create_answer(
    request: AnswerRequest,
    service: Annotated[AnswerService, Depends(get_answer_service)],
) -> AnswerResponse:
    return await service.answer(request.question)

This organization gives you two valuable test targets:

Target	What it verifies	What stays real
AnswerService test	Prompt arguments, response mapping, failure behavior	Your business logic
Endpoint test	FastAPI validation, dependency wiring, HTTP response contract	Routing, validation, serialization, AnswerService
Neither test	Whether an LLM vendor actually responds correctly	Nothing: replace the provider

Install the test runner and async-test plugin in your project environment:

python -m pip install pytest pytest-asyncio

Run the suite from the project root:

pytest

---

Control an asynchronous model result with AsyncMock

A normal Mock models ordinary synchronous calls. Your service uses:

await provider.generate(...)

so its replacement must return something awaitable. Python’s standard-library AsyncMock is designed precisely for this.

Create tests/test_services.py:

from unittest.mock import AsyncMock, Mock

import pytest

from app.providers import ModelProvider
from app.services import AnswerService, SYSTEM_INSTRUCTION


@pytest.mark.asyncio
async def test_answer_returns_provider_text_in_typed_response() -> None:
    provider = Mock(spec=ModelProvider)
    provider.generate = AsyncMock(
        return_value="You can reset your password from Settings."
    )

    service = AnswerService(
        provider=provider,
        model_name="test-model",
    )

    result = await service.answer("How do I reset my password?")

    assert result.answer == "You can reset your password from Settings."
    assert result.model == "test-model"
    provider.generate.assert_awaited_once_with(
        system_instruction=SYSTEM_INSTRUCTION,
        user_message="How do I reset my password?",
    )

This is isolated because provider has no credentials, no network client, and no connection to a vendor. Its return value is controlled entirely by the test.

The test has two categories of assertion:

1. Output assertion: the service returns the answer text and model name in the expected typed structure.
2. Boundary assertion: the service awaited the provider once with the important contract values.

The second assertion matters. A test that asserts only the output could still pass if a future change sends the wrong user question or drops the system instruction.

Use AsyncMock, not Mock, for the coroutine method. If you write:

provider.generate = Mock(return_value="Some answer")

then await provider.generate(...) attempts to await a plain string and raises a TypeError. There is also an important distinction between calling an async mock and awaiting it: assert_called_once_with() does not prove the coroutine was awaited. Use assert_awaited_once_with() for an async provider method.

Read the relevant portions of the standard-library documentation now. It is worth learning these built-in tools rather than relying on a particular third-party mocking wrapper.

---

Test failures as deliberately as successes

A mock can return a value, raise an exception, or provide a sequence of outcomes. The side_effect property controls the latter two cases.

At this point in the course, AnswerService deliberately lets a provider error propagate. Later you will add timeouts, retries, and HTTP error mapping. Test the behavior that exists today, so a future resilience change is an intentional, visible update to the test.

# tests/test_services.py
from unittest.mock import AsyncMock, Mock

import pytest

from app.providers import ModelProvider
from app.services import AnswerService


class ProviderTimeout(Exception):
    pass


@pytest.mark.asyncio
async def test_answer_propagates_provider_timeout() -> None:
    provider = Mock(spec=ModelProvider)
    provider.generate = AsyncMock(
        side_effect=ProviderTimeout("provider did not respond")
    )

    service = AnswerService(
        provider=provider,
        model_name="test-model",
    )

    with pytest.raises(ProviderTimeout, match="did not respond"):
        await service.answer("Summarize this document")

    provider.generate.assert_awaited_once()

This test does not claim that surfacing ProviderTimeout directly to an API client is the final product behavior. It establishes the current service-level contract: the service does not silently turn a provider outage into invented answer text.

When you later introduce bounded retries and a user-safe error response, update this test to assert the new policy instead.

For a compact visual demonstration of patching an external dependency with a controlled return value, watch this segment.

---

Test the FastAPI endpoint with a dependency override

The service test is narrow and fast. It does not confirm that FastAPI parses a request correctly or produces the promised HTTP response. Add one endpoint-level test that leaves the framework and service real, but substitutes the model provider.

First, ensure your existing input schema rejects an empty question:

# app/schemas.py
from pydantic import BaseModel, Field


class AnswerRequest(BaseModel):
    question: str = Field(min_length=1)


class AnswerResponse(BaseModel):
    answer: str
    model: str

Now write tests/test_api.py:

from unittest.mock import AsyncMock, Mock

import pytest
from fastapi.testclient import TestClient

from app.config import get_settings
from app.dependencies import get_model_provider
from app.main import app
from app.providers import ModelProvider
from app.services import SYSTEM_INSTRUCTION


@pytest.fixture
def client_and_provider(monkeypatch):
    # Required only because the previous lesson validates settings at startup.
    # This is a non-secret placeholder and is never sent to a model provider.
    monkeypatch.setenv("SUPPORT_PROVIDER_API_KEY", "test-only-value")
    monkeypatch.setenv("SUPPORT_MODEL_NAME", "test-model")
    get_settings.cache_clear()

    provider = Mock(spec=ModelProvider)
    provider.generate = AsyncMock(
        return_value="Reset your password from Settings."
    )

    def override_model_provider() -> ModelProvider:
        return provider

    app.dependency_overrides[get_model_provider] = override_model_provider

    try:
        with TestClient(app) as client:
            yield client, provider
    finally:
        app.dependency_overrides = {}
        get_settings.cache_clear()


def test_post_answer_returns_mocked_model_response(
    client_and_provider,
) -> None:
    client, provider = client_and_provider

    response = client.post(
        "/v1/answers",
        json={"question": "How do I reset my password?"},
    )

    assert response.status_code == 200
    assert response.json() == {
        "answer": "Reset your password from Settings.",
        "model": "test-model",
    }
    provider.generate.assert_awaited_once_with(
        system_instruction=SYSTEM_INSTRUCTION,
        user_message="How do I reset my password?",
    )


def test_post_answer_rejects_empty_question_without_calling_model(
    client_and_provider,
) -> None:
    client, provider = client_and_provider

    response = client.post(
        "/v1/answers",
        json={"question": ""},
    )

    assert response.status_code == 422
    provider.generate.assert_not_awaited()

The fixture has several jobs:

• It supplies minimum valid configuration for the startup validation introduced previously.
• It creates a new provider mock per test, so call history does not leak.
• It registers an override using the original get_model_provider function as the key.
• It opens TestClient as a context manager, which correctly runs FastAPI lifespan startup and shutdown.
• Its finally block clears the global override registry even if an assertion fails.

The second endpoint test is particularly valuable. It checks not only that invalid input produces $422$, but also that invalid input never reaches the model boundary. That protects cost and avoids processing malformed user input.

Notice what the endpoint test does not do:

• It does not assert internal private attributes of AnswerService.
• It does not assert every incidental implementation detail.
• It does not use a live API key.
• It does not make a real model call.
• It does not expect a particular response from a probabilistic model.

It checks stable contracts: HTTP input/output and the relevant provider interaction.

---

Prefer dependency injection; patch legacy lookups correctly

Dependency overrides are generally the cleanest solution for FastAPI because the dependency is explicit and can be replaced at the application boundary.

Sometimes, however, you will inherit code that constructs a provider internally:

# Do not use this pattern for new code.
from app.providers import HostedModelProvider


class LegacyAnswerService:
    async def answer(self, question: str) -> str:
        provider = HostedModelProvider(...)
        return await provider.generate(
            system_instruction="...",
            user_message=question,
        )

You can still test it with patch(), but the patch target must be the name as looked up by the code under test:

from unittest.mock import AsyncMock, patch

import pytest

from app.services import LegacyAnswerService


@pytest.mark.asyncio
async def test_legacy_service_patches_local_lookup() -> None:
    with patch(
        "app.services.HostedModelProvider",
        autospec=True,
    ) as provider_class:
        provider_instance = provider_class.return_value
        provider_instance.generate = AsyncMock(
            return_value="Controlled result"
        )

        result = await LegacyAnswerService().answer("Question")

    assert result == "Controlled result"
    provider_instance.generate.assert_awaited_once()

The target is app.services.HostedModelProvider, not app.providers.HostedModelProvider, because LegacyAnswerService looks up the imported name in app.services.

This is a frequent source of apparently broken mocks:

# Incorrect if app.services imported the name directly:
patch("app.providers.HostedModelProvider")

That patch may modify the definition module, while app.services still holds its earlier reference to the real class. The Python documentation calls this “patch where the object is looked up.”

Use autospec=True when patching a concrete class or function where possible. It catches mistakes such as calling a method with unsupported parameters. A very permissive mock can make a test pass even after production code drifts away from the real provider API.

For new application code, however, the earlier ModelProvider protocol and FastAPI override pattern remove most need for class patching. The test can inject a provider instance directly, without depending on import details.

---

A practical testing checklist

Before trusting a test involving an LLM provider, check the following:

• The test runs with no valid provider credential.
• The provider object is an AsyncMock or a compatible fake when the production method is awaited.
• The expected provider result is fixed in the test.
• The test asserts application behavior, not a live model’s wording.
• Important provider arguments are asserted when they form part of your prompt contract.
• Provider errors are tested with side_effect.
• Every test gets a fresh mock or resets its call history.
• app.dependency_overrides is cleared after endpoint tests.
• A unit test targets the service directly, while at least one endpoint test verifies the HTTP boundary.

---

Key takeaways

Isolated model-provider tests are the foundation of a fast, deterministic AI-service test suite.

• Introduce a small provider interface and inject it into your application service.
• Use AsyncMock for async model methods and assert that calls were awaited.
• Set return_value for a controlled generation result and side_effect for controlled failures.
• Test AnswerService directly for prompt and response-mapping behavior.
• Use FastAPI’s app.dependency_overrides to test real HTTP validation and serialization without real model calls.
• Always clean up dependency overrides so tests cannot affect one another.
• If patching is unavoidable, patch the namespace where the code under test looks up the dependency.

Next, the course begins Reliable LLM API Engineering. You will first learn how to select an LLM using capability, context-window, latency, privacy, and cost constraints—criteria that should inform the provider boundary you created here.