Sensitivity Testing for Robust Decision-Making

The last lesson selected action and review thresholds from labeled data, cost assumptions, and review capacity. That policy is only trustworthy if small, non-meaningful changes do not send equivalent cases down different routes. A system that escalates a ticket with one phrasing but passes the same ticket after an irrelevant field is added is brittle, even if its aggregate accuracy looks good.

In this lesson, you will build sensitivity tests for a Jev decision workflow. You will define which changes should preserve a decision, which changes should deliberately alter it, run paired requests against those variants, and turn unexpected changes into evidence for improving the question contract or state payload. The focus remains your confidence-aware support workflow, using a Noul proposition such as:

“Based only on the supplied state, this ticket requires urgent human intervention within one hour.”

---

1. Sensitivity testing asks a stronger question than accuracy

A labeled evaluation asks:

Did the system make the expected decision on this case?

A sensitivity test asks:

Does the system make a consistent decision when the meaningful facts have not changed?

Consider this support state:

const state = {
  message: "Checkout is failing for our customers. We have lost sales for 25 minutes.",
  affectedUsers: "many customers",
  checkoutStatus: "failing",
  knownIncidentCount: 1,
};

Suppose the urgent-risk assessment is $0.91$, above your automatic-action threshold. Now compare it with a version that also contains an irrelevant UI preference:

const stateWithNoise = {
  ...state,
  customerThemePreference: "dark",
};

If the new decision becomes “pass,” that is not a normal disagreement with a gold label. The evidence supporting urgency did not change. It is a warning that the decision contract may be overly sensitive to irrelevant context, formatting, ordering, or an underspecified question.

The important distinction is between two kinds of variation:

Variation	Meaning should change?	Expected testing relationship
Equivalent paraphrase of the Jev question	No	Same decision, with bounded score drift
Add irrelevant, non-sensitive state field	No	Same decision, with bounded score drift
Remove irrelevant state field	No	Same decision, with bounded score drift
Add verified evidence of wider impact	Yes	Urgency probability should not decrease materially
Add verified evidence that impact is resolved	Yes	Urgency probability should not increase materially
Remove key supporting evidence	Yes	Scenario-specific safe behavior, often review rather than automation
Replace “urgent” with “important”	Probably	Do not treat as an invariance test; you changed the proposition

The final row is a common trap. Similar words are not necessarily equivalent in a business policy. “Urgent human intervention within one hour” is a defined operational proposition; “important” may be broader and less actionable. A sensitivity suite must preserve the decision semantics, not merely surface similarity.

---

2. Think in metamorphic relations, not isolated examples

A useful formal model comes from metamorphic testing. Instead of requiring a perfect expected output for every generated variant, you define a relationship that should hold between a source case and its transformed version.

For Jev, a source input consists of two separately versioned things:

1. The question contract: question type, wording, options or proposition, and any instructions that define the judgment.
2. The state payload: the evidence passed to Jev.

A transformation modifies one of them while holding the other fixed. This separation is essential for debugging:

Test family	Change	Hold fixed	Typical diagnosis when it fails
Question sensitivity	Wording of the question contract	State evidence	The question is ambiguous or relies on accidental phrasing
State-noise sensitivity	Irrelevant state detail	Question contract	Payload contains distracting information, or model attention is brittle
Evidence ablation	A relevant evidence field is removed	Question contract	State lacks explicit missing-data semantics or policy handles uncertainty poorly
Evidence contrast	A verified fact changes	Question contract	The question is not using the most decision-relevant evidence

For an invariant transformation, define the outcome relationship before sending a request. For a Noul urgency judgment, a reasonable relation might include:

$$\left|p_{\mathrm{variant}}-p_{\mathrm{source}}\right|\le {\delta }_p$$ $$\left|c_{\mathrm{variant}}-c_{\mathrm{source}}\right|\le {\delta }_c$$

and, for cases sufficiently far from a threshold:

$$\mathrm{route}(p_{\mathrm{variant}})=\mathrm{route}(p_{\mathrm{source}})$$

Here, $p$ is the Noul probability, $c$ is Jev confidence, and the route is your deterministic action, review, or pass policy.

The tolerance values ${\delta }_p$ and ${\delta }_c$ are acceptance criteria, not universal constants. Start with values that are operationally meaningful for your product. A probability movement of $0.03$ may be harmless for a case far from thresholds but serious if it flips a ticket from automatic escalation to pass.

---

3. Build the test corpus from meaningful cases

Do not create a sensitivity suite by randomly mutating every word in production traffic. That generates many tests whose semantic relation is unclear, which leads to noisy failures and wasted review time.

Instead, begin with a small collection of human-reviewed source cases from the evaluation dataset you built earlier. Include cases from distinct operational regions:

Source-case group	Why it belongs in the suite
Clearly urgent	Detects whether equivalent wording can suppress a needed action
Clearly non-urgent	Detects whether noise creates unnecessary escalation
Review-band case	Tests whether ambiguity is consistently routed to review
Near-threshold case	Reveals operational instability, but should be reported separately
Previously misclassified case	Confirms that a repair does not regress
Sparse-evidence case	Tests missing-data and fallback behavior
Edge-case language	Covers typos, rambling descriptions, mixed signals, or irrelevant details

The Claude Platform guide gives practical examples of variations such as paraphrases, typos, rambling input, and irrelevant information. Although its examples concern generated answers, the principle transfers directly to typed Jev judgments: equivalent inputs should not produce arbitrary decision changes.

For the support workflow, define transformations deliberately.

Question-contract transformations

These are safe only when the proposition remains identical.

Variant	Example	Expected relation
Paraphrase	“Estimate the probability…” instead of “What is the probability…”	Invariant
Explicit evidence scope	Add “Based only on the supplied state”	Usually invariant; may expose an underspecified original
Concise phrasing	Remove redundant explanation while retaining the same proposition	Invariant
Developer typo	A minor accidental misspelling	Regression probe; usually expected to remain stable
Different deadline	Change “within one hour” to “within one business day”	Semantic change, not invariant

State-payload transformations

These should never introduce real customer data merely to make a test more realistic. Use de-identified, representative examples.

Variant	Example	Expected relation
Add irrelevant metadata	UI theme, request trace label, harmless campaign tag	Invariant
Remove irrelevant metadata	Omit data not needed for urgency	Invariant
Reorder equivalent evidence	Present independent factual fields in another order	Invariant, if serialization preserves the same content
Add verified urgency evidence	“Affected customers increased from 3 to 500”	Urgency should not decrease materially
Add verified resolution evidence	“Checkout restored; no current errors”	Urgency should not increase materially
Remove key evidence	Omit affected-user scope	Scenario-specific: often no automatic action without supporting proof

Do not assert that Jev confidence must always move monotonically when you add or remove evidence. Confidence summarizes Jev’s distributional certainty; it is not a generic measure of “amount of information.” Test the business decision and probability relation you care about, then record confidence as an additional diagnostic.

---

4. Write acceptance criteria before running variants

The workflow below is worth treating like ordinary test-driven development. The test relation should exist before you inspect the new output; otherwise, it is too easy to rationalize a surprising result after the fact.

For each source case, record:

type SensitivityCaseMetadata = {
  id: string;
  contractVersion: string;
  sourceDatasetVersion: string;
  humanReviewedAt: string;
  goldUrgent: boolean;
  notes: string;
};

Then give each variant an explicit expectation. For example:

type InvariantExpectation = {
  kind: "invariant";
  maximumNoulDrift: number;
  maximumConfidenceDrift: number;
  requireSameRoute: boolean;
};

type DirectionalExpectation = {
  kind: "not-less-urgent" | "not-more-urgent";
  permittedProbabilityMovement: number;
};

type RouteExpectation = {
  kind: "allowed-routes";
  allowedRoutes: Array<"action" | "review" | "pass">;
};

type SensitivityExpectation =
  | InvariantExpectation
  | DirectionalExpectation
  | RouteExpectation;

A source case far from decision thresholds is the best candidate for a strict invariance test. Define a boundary margin for those cases:

$$m=\min \left(\left|p-t_r\right|,\left|p-t_a\right|\right)$$

where $t_r$ is the review threshold and $t_a$ is the automatic-action threshold from the previous lesson.

If $m$ is small, a minor probability shift can change the route. That may still be useful information, but it should not be mixed with failures from clearly urgent or clearly non-urgent cases. Report it as boundary sensitivity.

A practical initial policy is:

• Robustness suite: only cases with sufficient boundary margin; route stability is required.
• Boundary-observation suite: cases close to review or action thresholds; route movement is measured and manually inspected.
• Evidence-ablation suite: known cases where required evidence is absent; allowed routes are specified by product and safety policy.

---

5. Implement a paired Jev sensitivity runner in TypeScript

Keep this runner independent from your UI and application routing code. It should call a narrow adapter that performs the live Jev request using the Noul contract already established in your project, then store the typed response and test metadata.

The following code intentionally leaves assessUrgency as an injected adapter. Its job is to make the Jev request and return typed values; the sensitivity logic remains deterministic and fully testable.

type Route = "action" | "review" | "pass";

type State = Record<string, unknown>;

type NoulAssessment = {
  noul: number;
  confidence: number;
};

type UrgencyPolicy = {
  reviewThreshold: number;
  actionThreshold: number;
};

type InvariantVariant = {
  id: string;
  label: string;
  question: string;
  state: State;
  maximumNoulDrift: number;
  maximumConfidenceDrift: number;
};

type InvariantSensitivityCase = {
  id: string;
  question: string;
  state: State;
  minimumBoundaryMargin: number;
  variants: InvariantVariant[];
};

type VariantResult = {
  caseId: string;
  variantId: string;
  passed: boolean;
  reasons: string[];
  source: NoulAssessment;
  variant: NoulAssessment;
  sourceRoute: Route;
  variantRoute: Route;
};

type AssessUrgency = (
  question: string,
  state: State,
) => Promise<NoulAssessment>;

function assertUnitInterval(value: number, label: string): void {
  if (!Number.isFinite(value) || value < 0 || value > 1) {
    throw new Error(`${label} must be a finite number from 0 through 1.`);
  }
}

function routeUrgency(
  probability: number,
  policy: UrgencyPolicy,
): Route {
  if (probability >= policy.actionThreshold) {
    return "action";
  }

  if (probability >= policy.reviewThreshold) {
    return "review";
  }

  return "pass";
}

function boundaryMargin(
  probability: number,
  policy: UrgencyPolicy,
): number {
  return Math.min(
    Math.abs(probability - policy.reviewThreshold),
    Math.abs(probability - policy.actionThreshold),
  );
}

function validateAssessment(
  assessment: NoulAssessment,
  label: string,
): void {
  assertUnitInterval(assessment.noul, `${label}.noul`);
  assertUnitInterval(assessment.confidence, `${label}.confidence`);
}

export async function runInvariantSensitivityCase(
  testCase: InvariantSensitivityCase,
  policy: UrgencyPolicy,
  assessUrgency: AssessUrgency,
): Promise<VariantResult[]> {
  const source = await assessUrgency(testCase.question, testCase.state);
  validateAssessment(source, `${testCase.id}.source`);

  const sourceRoute = routeUrgency(source.noul, policy);
  const sourceMargin = boundaryMargin(source.noul, policy);

  const results: VariantResult[] = [];

  for (const variant of testCase.variants) {
    const assessment = await assessUrgency(
      variant.question,
      variant.state,
    );
    validateAssessment(assessment, `${testCase.id}.${variant.id}`);

    const variantRoute = routeUrgency(assessment.noul, policy);
    const reasons: string[] = [];

    if (sourceMargin < testCase.minimumBoundaryMargin) {
      reasons.push(
        `Source is too close to a decision boundary: margin ${sourceMargin.toFixed(3)}.`,
      );
    }

    const noulDrift = Math.abs(assessment.noul - source.noul);
    if (noulDrift > variant.maximumNoulDrift) {
      reasons.push(
        `Noul drift ${noulDrift.toFixed(3)} exceeds ${variant.maximumNoulDrift}.`,
      );
    }

    const confidenceDrift = Math.abs(
      assessment.confidence - source.confidence,
    );
    if (confidenceDrift > variant.maximumConfidenceDrift) {
      reasons.push(
        `Confidence drift ${confidenceDrift.toFixed(3)} exceeds ${variant.maximumConfidenceDrift}.`,
      );
    }

    if (variantRoute !== sourceRoute) {
      reasons.push(
        `Route changed from ${sourceRoute} to ${variantRoute}.`,
      );
    }

    results.push({
      caseId: testCase.id,
      variantId: variant.id,
      passed: reasons.length === 0,
      reasons,
      source,
      variant: assessment,
      sourceRoute,
      variantRoute,
    });
  }

  return results;
}

Here is one carefully scoped test case. The base state contains material urgency evidence. The variants preserve that evidence while changing only the question wording or an intentionally irrelevant field.

const urgentCheckoutCase: InvariantSensitivityCase = {
  id: "checkout-outage-001",
  question:
    "Based only on the supplied state, what is the probability that this ticket requires urgent human intervention within one hour?",
  state: {
    message:
      "Checkout has failed for 25 minutes and multiple customers cannot pay.",
    checkoutStatus: "failing",
    affectedUsers: "multiple customers",
    knownIncidentCount: 1,
  },
  minimumBoundaryMargin: 0.1,
  variants: [
    {
      id: "question-paraphrase",
      label: "Equivalent question wording",
      question:
        "Using only the provided state, estimate the probability that this ticket needs urgent human intervention within the next hour.",
      state: {
        message:
          "Checkout has failed for 25 minutes and multiple customers cannot pay.",
        checkoutStatus: "failing",
        affectedUsers: "multiple customers",
        knownIncidentCount: 1,
      },
      maximumNoulDrift: 0.1,
      maximumConfidenceDrift: 0.15,
    },
    {
      id: "irrelevant-ui-metadata",
      label: "Irrelevant metadata added",
      question:
        "Based only on the supplied state, what is the probability that this ticket requires urgent human intervention within one hour?",
      state: {
        message:
          "Checkout has failed for 25 minutes and multiple customers cannot pay.",
        checkoutStatus: "failing",
        affectedUsers: "multiple customers",
        knownIncidentCount: 1,
        diagnosticUiTheme: "dark",
      },
      maximumNoulDrift: 0.1,
      maximumConfidenceDrift: 0.15,
    },
  ],
};

The adapter can be connected to the actual Jev call you built earlier:

const policy: UrgencyPolicy = {
  reviewThreshold: 0.55,
  actionThreshold: 0.9,
};

const results = await runInvariantSensitivityCase(
  urgentCheckoutCase,
  policy,
  assessUrgency,
);

console.table(
  results.map(function (result) {
    return {
      caseId: result.caseId,
      variantId: result.variantId,
      passed: result.passed,
      sourceProbability: result.source.noul.toFixed(3),
      variantProbability: result.variant.noul.toFixed(3),
      sourceConfidence: result.source.confidence.toFixed(3),
      variantConfidence: result.variant.confidence.toFixed(3),
      sourceRoute: result.sourceRoute,
      variantRoute: result.variantRoute,
      reasons: result.reasons.join(" "),
    };
  }),
);

The runner should write a record for every request, including:

• case ID and variant ID;
• question-contract version;
• state-schema version;
• recorded typed Jev response;
• thresholds used to derive the route;
• test expectation and pass/fail reasons;
• timestamp, latency, and request outcome.

This makes a sensitivity failure reproducible rather than a screenshot of one surprising run.

---

6. Handle evidence changes differently from invariance tests

A common failure in AI evaluation is to expect every transformed input to produce exactly the same answer. That is wrong when the transformation changes real evidence.

Suppose the original state says:

{
  checkoutStatus: "failing",
  affectedUsers: "multiple customers",
  durationMinutes: 25
}

Now add verified information:

{
  checkoutStatus: "failing",
  affectedUsers: "all EU customers",
  durationMinutes: 25,
  revenueImpact: "orders cannot be completed"
}

This is not an invariant case. The correct relation is directional: urgency should not fall substantially.

Conversely, this updated state changes the substantive facts:

{
  checkoutStatus: "restored",
  affectedUsers: "no currently affected customers",
  durationMinutes: 25,
  mitigation: "rollback completed"
}

Here, the urgency assessment should not rise substantially.

For sparse evidence, do not create a generic rule such as “missing data must lower probability.” Absence of evidence is not evidence of absence. Instead, specify an application policy. For example:

State condition	Expected application behavior
Scope is unknown but an outage is active	Human review may be acceptable
Strong evidence of active, widespread outage	Automatic urgent action may be acceptable
No impact, no active issue, and no corroborating evidence	Pass may be acceptable
Contradictory facts from trusted sources	Review or deterministic incident-system lookup

That policy is part of the deterministic application layer. Jev provides a bounded judgment; your code decides whether available evidence is sufficient for automatic action.

---

7. Diagnose failures without hiding them in threshold changes

When a sensitivity test fails, do not first adjust the action threshold. A threshold change can conceal the route flip while leaving the underlying probability instability untouched.

Inspect failures in this order:

1. Verify the transformation.
   Did the purported paraphrase actually preserve the one-hour urgency proposition? Did the supposedly irrelevant field imply a real business fact?

2. Compare the source and variant payloads.
   Use a structured diff. Check accidental omission, changed negation, altered units, or JSON serialization differences.

3. Classify the failure.
   Typical categories are:

   • ambiguous question contract;
   • irrelevant state distractor;
   • missing evidence semantics;
   • conflicting evidence;
   • threshold-boundary sensitivity;
   • intermittent request or service failure.

4. Fix one layer deliberately.
   You might refine the question wording, reduce the state to decision-relevant evidence, add an explicit unknown field, or route a class of sparse-evidence cases to review.

5. Re-run the complete evaluation suite.
   Changing the question contract or state schema changes the decision system. Re-check accuracy, calibration, threshold cost, and review capacity rather than validating only the repaired case.

6. Promote the failure to a regression test.
   The next lesson focuses on exactly this step: recording typed responses and preserving decision-logic behavior without repeatedly calling the live service.

If a failure appears only intermittently, rerun the same, already-recorded source and variant pair several times. Preserve the exact inputs. This distinguishes a consistently brittle relation from variability that needs a broader statistical treatment. Do not regenerate the paraphrase on every retry, or you will be testing a different input each time.

---

8. Report brittleness as an engineering metric

For each test family, report more than a simple pass percentage.

Metric	What it reveals
Invariant violation rate	Fraction of valid equivalent variants that break the expected relation
Route-flip rate	Operational instability in action, review, and pass behavior
Median and maximum probability drift	Typical versus worst sensitivity of the Noul result
Confidence drift	Whether equivalent changes alter Jev certainty
Boundary-case flip rate	How unstable behavior is near deployed thresholds
Failure rate by transformation type	Whether paraphrasing, irrelevant metadata, or evidence removal is the main problem
Service-error rate	Separates model-decision brittleness from timeouts or unavailable service

A report might conclude:

Equivalent question paraphrases were stable for clearly urgent and clearly non-urgent cases. Adding irrelevant metadata caused three route changes, all near the review threshold. Evidence-ablation tests correctly moved two cases to review, but one sparse-evidence case still triggered automatic action. The next revision will make impact scope explicit and add a deterministic policy rule preventing automatic action when scope is unknown.

That is actionable. “Accuracy improved by two points” is not enough to explain whether the workflow is dependable under real input variation.

---

Key takeaways

• Sensitivity testing evaluates whether equivalent inputs receive consistent Jev judgments and deterministic application routes.
• Separate question-wording transformations from state-evidence transformations so failures are diagnosable.
• Only call a transformation invariant when it truly preserves the business proposition.
• Test irrelevant additions and removals for stability; test meaningful evidence changes with directional or scenario-specific expectations.
• Track probability drift, confidence drift, and route flips separately. A small score change can still be operationally serious at a threshold.
• Do not solve brittleness by immediately moving thresholds. First investigate the question contract, state payload, and fallback policy.
• Record live typed responses and test metadata so surprising behavior can become a reproducible regression case.

Next, you will convert important sensitivity cases and known failures into deterministic regression tests using recorded Jev responses.