Evaluating Jev Workflow Performance and Abstention Behavior

Hello. In the previous lesson, you created a versioned, human-reviewed dataset for the support-intent router. Now you will use that dataset to answer the questions that matter before enabling automation:

• When the workflow acts automatically, how often is it right?
• What fraction of incoming cases does it automate?
• Which cases does it deliberately hand to a person, and why?

The goal is not to produce one flattering percentage. It is to measure the tradeoff between correct automation and safe deferral using a frozen held-out set. Plan for about 40 minutes.

---

1. Turn a Jev judgment into an explicit application policy

A Jev Choice response gives you a selected value, option probabilities, and confidence. The response itself is not yet an application decision. Your deterministic policy must decide whether that result permits an automatic action.

For the support router from the previous lesson, use this provisional policy:

1. If the request failed, timed out, or produced an invalid response, use the deterministic fallback.
2. If Jev selects other, send the ticket to human review.
3. If the selected supported intent has confidence below a threshold, send it to human review.
4. Otherwise, route it automatically to the queue associated with that intent.

This creates four mutually exclusive outcomes:

Policy outcome	Meaning	Automated?
automatic	Supported intent selected at or above the confidence threshold	Yes
review_low_confidence	A supported intent was selected, but evidence was not concentrated enough	No
review_other	The request is outside the supported routing taxonomy	No
fallback_failure	Timeout, unavailable service, malformed response, or parsing failure	No

The distinction between the two review outcomes is important:

• review_low_confidence is a confidence-based abstention. The workflow declines to make a specific automated decision.
• review_other is an intentional semantic handoff. The model may be quite certain that the request is out of scope, but your product contract says a person must handle it.
• fallback_failure is neither of these. It is an operational reliability event and must not be quietly counted as a model abstention.

Study the TypeSafe AI cookbook’s classification policy before implementing your own. It uses a coarse answer below a confidence threshold; in the support workflow, the comparable safe behavior is escalation to review.

A high winning probability alone does not always mean the choice was well separated from its alternatives. Jev confidence summarizes the concentration of the full option distribution, so it is the appropriate signal for this particular gate.

For now, treat your threshold as a fixed policy parameter, perhaps $0.85$. The next lessons will establish whether reported confidence is empirically meaningful and how thresholds should reflect the cost of mistakes versus review. Do not tune the threshold repeatedly on held-out data today.

---

2. Define the three core metrics precisely

Let $N$ be the number of valid labeled cases in the frozen held-out split. Every held-out case remains in the denominator, including cases that fail at runtime. Otherwise, a fragile workflow could appear better merely by dropping difficult requests.

Let:

• $N_{\text{auto}}$ be cases where the policy selected automatic.
• $C_{\text{auto}}$ be automatically routed cases whose selected intent equals the gold intent.
• $N_{\text{low}}$ be review_low_confidence cases.
• $N_{\text{other}}$ be review_other cases.
• $N_{\text{fail}}$ be fallback_failure cases.

Selective decision accuracy

This answers: when we automate, how often is the routing decision correct?

$${\mathrm{accuracy}}_{\text{auto}}=\frac{C_{\text{auto}}}{N_{\text{auto}}}$$

This is sometimes called selective accuracy: accuracy conditional on making an automatic decision.

For this router, exact intent accuracy is appropriate because the labels have distinct operational destinations. cancel_order versus return_policy, for example, may both concern an order, but routing one to the wrong queue is still a failure.

Automation coverage

This answers: how much of the overall workload can the workflow handle automatically?

$${\mathrm{coverage}}_{\text{auto}}=\frac{N_{\text{auto}}}{N}$$

Coverage is not accuracy. A system that automatically routes every ticket has $100\%$ coverage, even if many routes are wrong. A system that sends every ticket to review can have no automatic errors, but has $0\%$ automation coverage.

Abstention and handoff behavior

The narrow confidence-based abstention rate is:

$${\mathrm{abstention}}_{\text{low confidence}}=\frac{N_{\text{low}}}{N}$$

For operations, also report the broader review rate:

$$\mathrm{reviewrate}=\frac{N_{\text{low}}+N_{\text{other}}}{N}$$

And keep infrastructure failures visible:

$$\mathrm{failurerate}=\frac{N_{\text{fail}}}{N}$$

These outcomes must sum to the full held-out set:

$$N_{\text{auto}}+N_{\text{low}}+N_{\text{other}}+N_{\text{fail}}=N$$

That identity is a useful invariant for the evaluation runner. If it does not hold, some row has silently disappeared from your accounting.

A fourth metric worth keeping nearby

An automated error is more costly than a review in many support workflows. Measure its rate over all incoming cases:

$$\mathrm{incorrectautomationrate}=\frac{N_{\text{auto}}-C_{\text{auto}}}{N}$$

This prevents a misleading interpretation such as “$95\%$ automatic accuracy is good enough.” If automation coverage is large, the remaining $5\%$ can still mean many wrongly routed customers.

---

3. Read the coverage–accuracy tradeoff correctly

Suppose your held-out set contains 100 cases and your threshold is fixed at $0.85$:

Outcome	Count	Share of all cases
Correct automatic routes	66	66%
Incorrect automatic routes	4	4%
Review: low confidence	18	18%
Review: other	12	12%

The headline metrics are:

$${\mathrm{accuracy}}_{\text{auto}}=\frac{66}{70}=94.3\%$$ $${\mathrm{coverage}}_{\text{auto}}=\frac{70}{100}=70\%$$ $${\mathrm{abstention}}_{\text{low confidence}}=\frac{18}{100}=18\%$$ $$\mathrm{reviewrate}=\frac{18+12}{100}=30\%$$ $$\mathrm{incorrectautomationrate}=\frac{4}{100}=4\%$$

The result says: the workflow automates 70% of this workload, and 94.3% of those automated routes are correct. It does not say that the entire workflow is 94.3% correct. The remaining 30% still requires review, and the 4% incorrect automations need investigation.

To generate a curve like this, evaluate several thresholds against the same recorded held-out responses. Higher thresholds ordinarily reduce coverage because more cases go to review. A useful confidence signal should make automatic accuracy increase as coverage falls.

The graph also warns against a dangerous assumption: a threshold only works if the confidence measure orders easy and hard cases meaningfully. A poorly calibrated or brittle decision system can abstain frequently without improving automatic accuracy much. In the next lesson, you will test whether Jev’s reported confidence corresponds to observed accuracy in your own data.

For today, record the curve and inspect it; do not select the “best” threshold from a small held-out set.

---

4. Implement a deterministic measurement runner in TypeScript

Run Jev once for each held-out dataset row and store a result artifact separate from the gold dataset. The evaluation input must contain only the row’s state and contract. Never send expected, review, or reviewer rationale to Jev.

A recorded result should preserve at least:

type RecordedChoice = {
  choice: Intent;
  confidence: number;
  probabilities: Record<Intent, number>;
};

type IntentEvalRun = IntentEvalCase & {
  actual: RecordedChoice | null;
  failure: "timeout" | "rate_limit" | "unavailable" | "invalid_response" | null;
};

Here, actual: null means that no valid Jev Choice result was available. Keeping failures explicit lets your evaluation distinguish a weak decision policy from a broken dependency or response parser.

The following implementation assumes the IntentEvalCase schema from the previous lesson and measures the policy without making any new model calls. This separation is deliberate: live calls produce evidence; deterministic evaluation code interprets that evidence.

import type { IntentEvalCase } from "./intent-eval-case.js";

type Intent = IntentEvalCase["expected"]["intent"];

type RecordedChoice = {
  choice: Intent;
  confidence: number;
  probabilities: Record<Intent, number>;
};

type IntentEvalRun = IntentEvalCase & {
  actual: RecordedChoice | null;
  failure: "timeout" | "rate_limit" | "unavailable" | "invalid_response" | null;
};

type PolicyStatus =
  | "automatic"
  | "review_low_confidence"
  | "review_other"
  | "fallback_failure";

type EvaluatedRun = {
  run: IntentEvalRun;
  status: PolicyStatus;
};

function statusFor(
  run: IntentEvalRun,
  confidenceThreshold: number,
): PolicyStatus {
  if (run.actual === null) {
    return "fallback_failure";
  }

  if (run.actual.choice === "other") {
    return "review_other";
  }

  if (run.actual.confidence < confidenceThreshold) {
    return "review_low_confidence";
  }

  return "automatic";
}

function selectedIntentIsCorrect(run: IntentEvalRun): boolean {
  return run.actual !== null && run.actual.choice === run.expected.intent;
}

function countStatus(
  evaluated: EvaluatedRun[],
  status: PolicyStatus,
): number {
  return evaluated.filter(function (item) {
    return item.status === status;
  }).length;
}

function rate(numerator: number, denominator: number): number | null {
  return denominator === 0 ? null : numerator / denominator;
}

function formatRate(value: number | null): string {
  return value === null ? "n/a" : `${(value * 100).toFixed(1)}%`;
}

export function measureIntentRouter(
  runs: IntentEvalRun[],
  confidenceThreshold: number,
) {
  const heldout = runs.filter(function (run) {
    return run.split === "heldout";
  });

  const evaluated = heldout.map(function (run): EvaluatedRun {
    return {
      run,
      status: statusFor(run, confidenceThreshold),
    };
  });

  const automatic = evaluated.filter(function (item) {
    return item.status === "automatic";
  });

  const correctlyAutomated = automatic.filter(function (item) {
    return selectedIntentIsCorrect(item.run);
  });

  const responded = evaluated.filter(function (item) {
    return item.run.actual !== null;
  });

  const forcedCorrect = responded.filter(function (item) {
    return selectedIntentIsCorrect(item.run);
  });

  const total = evaluated.length;
  const automaticCount = automatic.length;
  const correctAutomaticCount = correctlyAutomated.length;
  const lowConfidenceCount = countStatus(
    evaluated,
    "review_low_confidence",
  );
  const otherCount = countStatus(evaluated, "review_other");
  const failureCount = countStatus(evaluated, "fallback_failure");

  const accountedFor =
    automaticCount +
    lowConfidenceCount +
    otherCount +
    failureCount;

  if (accountedFor !== total) {
    throw new Error("Policy outcomes do not account for every held-out run.");
  }

  const metrics = {
    total,
    responded: responded.length,
    automaticCount,
    correctAutomaticCount,
    lowConfidenceCount,
    otherCount,
    failureCount,

    automaticAccuracy: rate(correctAutomaticCount, automaticCount),
    automationCoverage: rate(automaticCount, total),
    lowConfidenceAbstentionRate: rate(lowConfidenceCount, total),
    reviewRate: rate(lowConfidenceCount + otherCount, total),
    failureRate: rate(failureCount, total),
    incorrectAutomationRate: rate(
      automaticCount - correctAutomaticCount,
      total,
    ),

    // Baseline: always accept the selected intent when a valid response exists.
    forcedAccuracy: rate(forcedCorrect.length, responded.length),
  };

  console.table([
    {
      metric: "Automatic accuracy",
      numerator: `${correctAutomaticCount}/${automaticCount}`,
      value: formatRate(metrics.automaticAccuracy),
    },
    {
      metric: "Automation coverage",
      numerator: `${automaticCount}/${total}`,
      value: formatRate(metrics.automationCoverage),
    },
    {
      metric: "Low-confidence abstention",
      numerator: `${lowConfidenceCount}/${total}`,
      value: formatRate(metrics.lowConfidenceAbstentionRate),
    },
    {
      metric: "Total review rate",
      numerator: `${lowConfidenceCount + otherCount}/${total}`,
      value: formatRate(metrics.reviewRate),
    },
    {
      metric: "Failure rate",
      numerator: `${failureCount}/${total}`,
      value: formatRate(metrics.failureRate),
    },
    {
      metric: "Incorrect automation",
      numerator: `${automaticCount - correctAutomaticCount}/${total}`,
      value: formatRate(metrics.incorrectAutomationRate),
    },
    {
      metric: "Forced-choice baseline",
      numerator: `${forcedCorrect.length}/${responded.length}`,
      value: formatRate(metrics.forcedAccuracy),
    },
  ]);

  return metrics;
}

A few details in this code deserve attention:

• The forcedAccuracy baseline ignores the confidence gate but excludes runtime failures, because there is no selected intent to force after a failure. Always report its denominator alongside the rate.
• automaticAccuracy includes only the cases where your product would take an automatic action.
• review_other is excluded from automatic coverage even if Jev selected it with confidence $1.0$. That is intentional: the support contract routes other to a person.
• A high-confidence wrong answer counts as an incorrect automation, not as a review success.
• The model-visible state remains separate from expected labels throughout the process.

---

5. Inspect abstentions rather than treating them as a single bucket

A raw abstention rate is useful, but it is not enough for diagnosis. Add a breakdown to the report after the first run.

Question to inspect	Why it matters
Which expected intents produce the most low-confidence reviews?	Reveals unclear category boundaries or missing evidence.
Which slices have low coverage?	Finds language, channel, typo, multi-intent, or long-context disparities.
Are high-confidence errors concentrated in one intent pair?	Points to a question, taxonomy, or state-design flaw.
How often does other occur?	Measures true out-of-scope demand and pressure to expand the taxonomy.
How often is a low-confidence selected label actually correct?	Shows potential automation opportunity, but does not itself justify lowering the threshold.
Are failures clustered by time or request volume?	Separates decision quality from availability, rate-limit, or timeout problems.

For example, the following function exposes accuracy and coverage by expected intent:

type IntentBreakdown = {
  expectedIntent: Intent;
  total: number;
  automatic: number;
  correctAutomatic: number;
  lowConfidenceReview: number;
  otherReview: number;
  failures: number;
};

export function breakdownByExpectedIntent(
  runs: IntentEvalRun[],
  confidenceThreshold: number,
): IntentBreakdown[] {
  const byIntent = new Map<Intent, IntentBreakdown>();

  for (const run of runs) {
    if (run.split !== "heldout") {
      continue;
    }

    const expectedIntent = run.expected.intent;
    const existing = byIntent.get(expectedIntent) ?? {
      expectedIntent,
      total: 0,
      automatic: 0,
      correctAutomatic: 0,
      lowConfidenceReview: 0,
      otherReview: 0,
      failures: 0,
    };

    existing.total += 1;

    const status = statusFor(run, confidenceThreshold);

    if (status === "automatic") {
      existing.automatic += 1;

      if (selectedIntentIsCorrect(run)) {
        existing.correctAutomatic += 1;
      }
    }

    if (status === "review_low_confidence") {
      existing.lowConfidenceReview += 1;
    }

    if (status === "review_other") {
      existing.otherReview += 1;
    }

    if (status === "fallback_failure") {
      existing.failures += 1;
    }

    byIntent.set(expectedIntent, existing);
  }

  return [...byIntent.values()];
}

Do not over-interpret a tiny slice. If you have only three held-out cancel_order cases, “100% automatic accuracy” means three correct routes, not that the policy is proven dependable. Keep both numerators and denominators in every report.

A practical first report can therefore include:

1. Overall automatic accuracy, coverage, review rate, and failure rate.
2. The same values by expected intent.
3. A list of all incorrect automatic routes, including state, predicted intent, confidence, and gold intent.
4. A list of low-confidence and other handoffs, grouped by review reason.
5. A forced-choice baseline for context.

That artifact gives you a real engineering conversation: not “is the AI good?” but “what does it automate, where does it defer, and what failure mode should we fix first?”

---

Key takeaways

A Jev evaluation should measure a policy, not merely compare selected labels with gold labels.

• Automatic accuracy measures whether decisions the application actually automates are correct.
• Automation coverage measures the share of the whole held-out workload that receives an automatic action.
• Confidence-based abstention measures how often the system declines to automate because the evidence is insufficiently decisive.
• other review and service failures are distinct from low-confidence abstention and should be reported separately.
• The incorrect-automation rate is often the clearest operational risk metric because it measures wrong automatic actions across the entire workload.
• Preserve actual Jev responses separately from gold data, then make the evaluation calculation deterministic and reproducible.

Next, you will test calibration: whether higher reported Jev confidence actually corresponds to higher empirical accuracy on your held-out dataset.