Regression Testing for Decision Logic, Thresholds, and Fallbacks

Last lesson used live sensitivity tests to find brittle behavior in question wording, payload shape, and threshold boundaries. The next engineering step is to preserve what you learned without making every test run depend on Jev availability, latency, or changing live outputs.

This lesson builds an offline regression suite for the deterministic part of a Jev workflow. You will record carefully reviewed, typed Jev responses from real calls, then use those records to test:

• decision routes at exact probability and confidence boundaries;
• policy changes such as threshold updates;
• safe fallback behavior for timeouts, rate limits, and invalid responses;
• the distinction between a reviewed fixture update and an accidental snapshot refresh.

The result is a fast test suite suitable for every pull request, while live Jev evaluation remains a separate, intentional activity.

---

1. What a recorded-response regression test proves

A live evaluation answers a model-quality question:

Given this state and question contract, how does the current Jev model behave?

A recorded-response regression test answers a software-behavior question:

Given this already-reviewed typed model result, does our application still make the intended deterministic decision?

Those are related, but they must not be conflated.

Consider an urgency policy for support tickets:

Condition	Application route
Probability is high and confidence is sufficient	Automatic urgent action
Probability is uncertain, or confidence is too low	Human review
Probability is low and confidence is sufficient	Pass
Jev is unavailable or returns invalid data	Safe fallback, usually review

The policy code owns these routes. Jev supplies typed evidence, such as a Noul probability and confidence. A regression test should therefore inject recorded evidence and test the policy without a network call.

This is not a substitute for real Jev usage. You still make real calls when you:

• build and expand the evaluation dataset;
• investigate a production failure;
• test a new question contract or model version;
• deliberately record a new fixture after human review.

But CI should not repeatedly call a production AI service merely to verify that $0.90$ remains on the intended side of an application threshold.

---

2. Choose regression cases from actual failure modes

Your regression corpus should grow from meaningful evidence, not from arbitrary synthetic JSON. The most useful initial cases are usually:

1. Known production or evaluation failures
   A ticket that was wrongly escalated, passed, or routed to the wrong reviewer.

2. Threshold boundaries
   Recorded or constructed typed responses just below and exactly at deployed thresholds.

3. Sensitivity failures from the previous lesson
   For example, a case where irrelevant metadata unexpectedly changed the live result.

4. Fallback events
   A timeout, rate limit, malformed typed response, or unavailable service that must never become an accidental “pass.”

5. Policy exceptions
   Cases where product policy requires review even though the probability alone might permit automation.

The Jev jaggedness guide is useful here because it identifies classes of behavior that should lead to explicit application safeguards rather than wishful prompt wording.

Two implications matter especially for this course’s support workflow:

• Keep arithmetic, timestamps, rate-limit windows, and threshold comparisons in code. Do not record a fixture that merely preserves a model attempt to do deterministic computation.
• Do not expect mathematical identities between separate Jev questions. A Noul result and a yes/no Choice result are different contracts, so they need separate fixture types and separately tuned thresholds.

---

3. Treat fixtures as versioned evidence

A fixture is not just a convenient mock object. It is a small, reviewable historical record of a real model interaction.

For an urgency workflow based on Noul, a fixture should preserve:

• a stable case identifier;
• the model name;
• question-contract and state-schema versions;
• a de-identified input state sufficient to understand the case;
• the typed Jev result used by the policy;
• the recording date and a short review note.

It should not contain:

• API keys, authorization headers, or request signatures;
• customer names, email addresses, phone numbers, payment details, or raw production identifiers;
• volatile metadata such as request IDs, wall-clock latency, or tracing spans;
• unrestricted raw provider payloads that couple tests to irrelevant SDK details.

A small fixture can look like this:

{
  "fixtureSchemaVersion": 1,
  "caseId": "checkout-outage-active-001",
  "recordedAt": "2025-03-08",
  "reviewNote": "Verified active checkout outage; used to protect automatic escalation.",
  "jev": {
    "model": "jev-1.13",
    "questionContractVersion": "urgent-intervention-v3",
    "stateSchemaVersion": "support-ticket-v2"
  },
  "input": {
    "question": "Based only on the supplied state, what is the probability that this ticket requires urgent human intervention within one hour?",
    "state": {
      "checkoutStatus": "failing",
      "affectedUsers": "multiple customers",
      "durationMinutes": 25,
      "knownIncidentCount": 1
    }
  },
  "typedResponse": {
    "noul": 0.94,
    "confidence": 0.96
  }
}

The fixture records observed Jev behavior, not the business decision. Your test derives the business decision from the fixture using the current policy code. That distinction is crucial.

If you later change the action threshold from 0.90 to 0.95, this fixture remains valuable. It lets you see which historical cases change route because of the policy revision.

A disciplined fixture lifecycle

Use this lifecycle:

Stage	What happens	May CI call Jev?
Investigate	Run a real request against a de-identified case	No, local/manual only
Review	Confirm input, contract version, response, and intended policy behavior	No
Record	Commit a sanitized typed fixture with a meaningful name	No
Regress	Run deterministic tests using the fixture	Never
Revise contract	Run live evaluations for the candidate question or model	Deliberately, outside ordinary CI
Promote	Add newly reviewed live outcomes as new fixtures	No

A fixture is therefore an artifact of a live evaluation, not a random value invented to make a test pass.

---

4. Separate the Jev boundary from decision logic

The cleanest design is to make your core decision function accept a typed result or a typed failure. Your HTTP and SDK adapter translates the real external outcome into that narrow application type.

This creates three independently testable layers:

Layer	Responsibility	Test style
Jev adapter	Call SDK, map API response and errors	Small integration tests and manually run live checks
Typed boundary	Validate Noul and confidence are finite values from 0 through 1	Unit tests with malformed records
Decision policy	Apply thresholds, confidence gate, routes, and fallbacks	Fast deterministic regression tests

Here is a compact decision module for the support workflow:

// src/urgency/decision.ts

export type Route = "action" | "review" | "pass";

export type NoulAssessment = {
  noul: number;
  confidence: number;
};

export type FailureReason =
  | "timeout"
  | "rate-limit"
  | "unavailable"
  | "invalid-response";

export type AssessmentAttempt =
  | {
      kind: "ok";
      assessment: NoulAssessment;
    }
  | {
      kind: "failure";
      reason: FailureReason;
    };

export type UrgencyPolicy = {
  reviewThreshold: number;
  actionThreshold: number;
  minimumConfidence: number;
};

export type UrgencyDecision = {
  route: Route;
  reason: string;
  automated: boolean;
};

function assertUnitInterval(value: number, label: string): void {
  if (!Number.isFinite(value) || value < 0 || value > 1) {
    throw new Error(`${label} must be a finite number from 0 through 1.`);
  }
}

export function validatePolicy(policy: UrgencyPolicy): void {
  assertUnitInterval(policy.reviewThreshold, "reviewThreshold");
  assertUnitInterval(policy.actionThreshold, "actionThreshold");
  assertUnitInterval(policy.minimumConfidence, "minimumConfidence");

  if (policy.reviewThreshold > policy.actionThreshold) {
    throw new Error(
      "reviewThreshold must not be greater than actionThreshold.",
    );
  }
}

export function validateAssessment(
  assessment: NoulAssessment,
): NoulAssessment {
  assertUnitInterval(assessment.noul, "noul");
  assertUnitInterval(assessment.confidence, "confidence");
  return assessment;
}

export function decideUrgency(
  attempt: AssessmentAttempt,
  policy: UrgencyPolicy,
): UrgencyDecision {
  validatePolicy(policy);

  if (attempt.kind === "failure") {
    return {
      route: "review",
      reason: `fallback-${attempt.reason}`,
      automated: false,
    };
  }

  const assessment = validateAssessment(attempt.assessment);

  if (assessment.confidence < policy.minimumConfidence) {
    return {
      route: "review",
      reason: "low-confidence",
      automated: false,
    };
  }

  if (assessment.noul >= policy.actionThreshold) {
    return {
      route: "action",
      reason: "urgent-probability-at-or-above-action-threshold",
      automated: true,
    };
  }

  if (assessment.noul >= policy.reviewThreshold) {
    return {
      route: "review",
      reason: "urgent-probability-in-review-band",
      automated: false,
    };
  }

  return {
    route: "pass",
    reason: "urgent-probability-below-review-threshold",
    automated: true,
  };
}

This module has no SDK import, no network operation, and no environment variable. It is therefore fully deterministic.

The external adapter should convert a timeout or rate-limit response into:

{
  kind: "failure",
  reason: "timeout"
}

rather than leaking arbitrary provider error objects into the policy layer. This makes fallback behavior explicit and testable.

---

5. Build deterministic Vitest tests around recorded responses

Place fixtures next to the tests that consume them:

src/
  urgency/
    decision.ts
tests/
  urgency/
    fixtures/
      checkout-outage-active-001.json
    decision.test.ts

If Vitest is not already in your TypeScript project:

pnpm add -D vitest

Add a test command to package.json:

{
  "scripts": {
    "test": "vitest run"
  }
}

Now write tests that check the policy’s observable behavior. Explicit assertions are preferable for safety-critical routes because they make the intended contract visible in code review.

// tests/urgency/decision.test.ts

import { describe, expect, it } from "vitest";

import activeOutageFixture from "./fixtures/checkout-outage-active-001.json";
import {
  decideUrgency,
  type AssessmentAttempt,
  type FailureReason,
  type UrgencyPolicy,
} from "../../src/urgency/decision";

const policy: UrgencyPolicy = {
  reviewThreshold: 0.55,
  actionThreshold: 0.9,
  minimumConfidence: 0.7,
};

describe("urgency decision policy", function () {
  it("automatically acts on the recorded active-outage assessment", function () {
    const attempt: AssessmentAttempt = {
      kind: "ok",
      assessment: activeOutageFixture.typedResponse,
    };

    expect(decideUrgency(attempt, policy)).toEqual({
      route: "action",
      reason: "urgent-probability-at-or-above-action-threshold",
      automated: true,
    });
  });

  it.each([
    {
      label: "acts exactly at the action threshold",
      assessment: { noul: 0.9, confidence: 0.7 },
      route: "action",
    },
    {
      label: "reviews immediately below the action threshold",
      assessment: { noul: 0.899, confidence: 0.7 },
      route: "review",
    },
    {
      label: "reviews exactly at the review threshold",
      assessment: { noul: 0.55, confidence: 0.7 },
      route: "review",
    },
    {
      label: "passes below the review threshold",
      assessment: { noul: 0.549, confidence: 0.7 },
      route: "pass",
    },
    {
      label: "reviews high probability with insufficient confidence",
      assessment: { noul: 0.98, confidence: 0.69 },
      route: "review",
    },
  ])("$label", function ({ assessment, route }) {
    const decision = decideUrgency(
      {
        kind: "ok",
        assessment,
      },
      policy,
    );

    expect(decision.route).toBe(route);
  });

  it.each<FailureReason>([
    "timeout",
    "rate-limit",
    "unavailable",
    "invalid-response",
  ])("safely falls back to review on %s", function (reason) {
    const decision = decideUrgency(
      {
        kind: "failure",
        reason,
      },
      policy,
    );

    expect(decision).toEqual({
      route: "review",
      reason: `fallback-${reason}`,
      automated: false,
    });
  });
});

Run the suite:

pnpm test

These tests establish several important invariants:

• Equality with a threshold is deliberate because the comparison uses >=.
• A score near a threshold is not rounded before routing.
• High probability does not bypass the confidence gate.
• Every declared service failure has a safe deterministic outcome.
• No test needs a Jev key, network access, or a live API response.

---

6. Use snapshots selectively, not as an approval mechanism

Snapshots are useful when a decision report has enough structure that repeated manual assertions become noisy. They are poor when used as a way to approve an opaque, unreviewed external response.

For example, you might snapshot a compact policy matrix used in documentation or a release-review report:

it("keeps the reviewed urgency policy matrix stable", function () {
  const cases: AssessmentAttempt[] = [
    {
      kind: "ok",
      assessment: { noul: 0.94, confidence: 0.96 },
    },
    {
      kind: "ok",
      assessment: { noul: 0.73, confidence: 0.88 },
    },
    {
      kind: "ok",
      assessment: { noul: 0.12, confidence: 0.91 },
    },
    {
      kind: "failure",
      reason: "timeout",
    },
  ];

  const report = cases.map(function (attempt) {
    return {
      input: attempt,
      decision: decideUrgency(attempt, policy),
    };
  });

  expect(report).toMatchSnapshot();
});

Use explicit assertions for core safety rules, such as:

• timeout always reaches review;
• 0.90 acts and 0.899 reviews;
• malformed probability is rejected;
• low confidence cannot automatically act.

Use snapshots for broader, readable artifacts where a diff is useful, such as a matrix of decisions across many fixtures.

Never bulk-update snapshots after a policy change

A snapshot update command is a tool, not an approval workflow. Before accepting an update:

1. Inspect the diff.
2. Identify which contract changed: fixture, threshold, confidence gate, fallback rule, or formatting only.
3. Confirm the new behavior against product and safety requirements.
4. Update the version or release note that explains the change.
5. Run the whole regression suite and relevant live evaluation suite.

If a snapshot changes because a new question contract gave different live Jev results, do not simply update it. First evaluate the candidate contract on the labeled dataset and sensitivity suite from prior lessons.

---

7. Record fixtures through a deliberately gated live script

Keep fixture capture out of ordinary tests. A capture script should require an explicit environment flag and use the real Jev adapter you already built.

The following example intentionally leaves the actual SDK call behind assessUrgencyLive. That adapter is the appropriate place for your authenticated Jev request. The capture workflow, schema, and review controls remain independent of SDK details.

// scripts/capture-urgency-fixture.ts

import { writeFile } from "node:fs/promises";

type State = Record<string, unknown>;

type NoulAssessment = {
  noul: number;
  confidence: number;
};

type CaptureInput = {
  caseId: string;
  reviewNote: string;
  questionContractVersion: string;
  stateSchemaVersion: string;
  question: string;
  state: State;
};

type RecordedNoulFixture = {
  fixtureSchemaVersion: 1;
  caseId: string;
  recordedAt: string;
  reviewNote: string;
  jev: {
    model: "jev-1.13";
    questionContractVersion: string;
    stateSchemaVersion: string;
  };
  input: {
    question: string;
    state: State;
  };
  typedResponse: NoulAssessment;
};

type AssessUrgencyLive = function (
  question: string,
  state: State,
): Promise<NoulAssessment>;

function validateUnitInterval(value: number, label: string): void {
  if (!Number.isFinite(value) || value < 0 || value > 1) {
    throw new Error(`${label} must be a finite number from 0 through 1.`);
  }
}

export async function captureFixture(
  input: CaptureInput,
  assessUrgencyLive: AssessUrgencyLive,
): Promise<RecordedNoulFixture> {
  if (process.env.JEV_CAPTURE !== "1") {
    throw new Error(
      "Fixture capture is disabled. Set JEV_CAPTURE=1 after review.",
    );
  }

  const typedResponse = await assessUrgencyLive(
    input.question,
    input.state,
  );

  validateUnitInterval(typedResponse.noul, "typedResponse.noul");
  validateUnitInterval(
    typedResponse.confidence,
    "typedResponse.confidence",
  );

  return {
    fixtureSchemaVersion: 1,
    caseId: input.caseId,
    recordedAt: new Date().toISOString().slice(0, 10),
    reviewNote: input.reviewNote,
    jev: {
      model: "jev-1.13",
      questionContractVersion: input.questionContractVersion,
      stateSchemaVersion: input.stateSchemaVersion,
    },
    input: {
      question: input.question,
      state: input.state,
    },
    typedResponse,
  };
}

export async function writeFixture(
  fixture: RecordedNoulFixture,
  destination: string,
): Promise<void> {
  const content = `${JSON.stringify(fixture, null, 2)}\n`;
  await writeFile(destination, content, "utf8");
}

A practical capture command might be:

JEV_CAPTURE=1 pnpm tsx scripts/capture-urgency-fixture.ts

Before committing a generated fixture, review it as carefully as code:

• Is the state safely de-identified?
• Does the question precisely match the deployed contract?
• Is the model version recorded?
• Does the typed response conform to the expected primitive?
• Is the case worth protecting in the suite?
• Does the test assert the intended route independently of the fixture?

---

8. A compact starting regression suite

For the support-automation capstone, begin with this minimum set:

Case	Fixture or constructed input	Expected protection
Clearly urgent checkout outage	Recorded Noul response	Automatic action remains intentional
Review-band incident	Recorded response	No accidental pass or action
Clearly non-urgent ticket	Recorded response	No unnecessary escalation
Exact action threshold	Constructed typed result	Boundary comparison remains stable
Exact review threshold	Constructed typed result	Boundary comparison remains stable
Low-confidence high-risk result	Constructed typed result	Confidence gate prevents automation
Timeout	Typed failure result	Safe fallback route
Rate limit	Typed failure result	Safe fallback route
Invalid probability or confidence	Malformed fixture test	Invalid data cannot become a decision
Previously observed sensitivity failure	Recorded source and variant results	Repair remains protected

The recorded fixtures preserve meaningful real Jev behavior. The constructed values protect deterministic boundaries that may be difficult or wasteful to obtain from a live service exactly.

---

Key takeaways

• Record real, reviewed, de-identified Jev typed responses once; consume them offline in ordinary regression tests.
• Keep the Jev SDK boundary separate from deterministic decision logic.
• Test threshold boundaries explicitly, including equality behavior and confidence gates.
• Model timeouts, rate limits, unavailable service, and invalid responses as typed failure states with safe fallback behavior.
• Use snapshots for readable, normalized reports, not as blind approval for raw external outputs.
• Never bulk-update fixtures or snapshots after a prompt, model, or policy change. Review each behavioral diff and rerun the appropriate live evaluation suite.
• A regression suite protects the application policy; live evaluations continue to protect the model contract.

Next, you will begin the production capstone by specifying a typed decision contract and acceptance criteria for the multi-stage support-automation workflow.