Setting Action and Review Thresholds Based on Error Costs

Hello. In the previous lesson, you tested whether Jev confidence is a useful operational signal by comparing it with held-out accuracy. That work matters here: a threshold should not be chosen because $0.80$ “feels high,” but because it produces an acceptable trade-off on representative cases.

This lesson turns that evidence into a deployable policy. You will define the cost of a wrong automatic action, a missed case, and a human review; simulate candidate action and review thresholds on labeled data; and select a policy that minimizes expected operational cost while respecting review-team capacity. Plan for about 45 minutes.

---

1. A threshold is a product decision, not a model property

Jev supplies typed assessments: a Noul probability, a Choice result and option probabilities, a Score, plus confidence. Your application decides what to do with them.

For a binary safety or escalation judgment, a useful three-path policy is:

Assessment range	Application action
High risk	Take the automated action: block, escalate, mark urgent, and so on.
Medium risk	Send to human review.
Low risk	Pass, defer, or follow the ordinary deterministic workflow.

TypeSafe’s guidance frames this as a high-, medium-, and low-confidence pattern, while emphasizing that the boundaries depend on the stakes.

For this lesson, use a risk score $p$ for one precise proposition, such as:

“This incoming support message requires urgent human intervention.”

A Noul question is especially natural here because its value directly represents the probability of that proposition. Define two thresholds:

• $t_r$, the review threshold
• $t_a$, the action threshold, where $t_a\ge t_r$

The policy is:

$$d(p)=\{\begin{array}{ll}\text{action},& p\ge t_a\\ \text{review},& t_r\le p<t_a\\ \text{pass},& p<t_r\end{array}$$

In a support workflow, “action” might mean automatically assigning an urgent queue or presenting an immediate recovery workflow. In a safety workflow, it may mean blocking content. Those actions have very different error costs, so they should not share a threshold merely because they share a model.

For a Choice workflow, you can use the same structure, but be precise about the signal:

• If the question is “Which intent is this?”, the selected option is a category, not a risk probability.
• You might gate an automatic route using the selected-option probability or Jev confidence.
• Because neither is automatically a calibrated probability of business correctness, threshold selection must rely on observed held-out error rates, as in the previous lesson.

---

2. Name the costs before choosing any numbers

A threshold produces three kinds of operational outcome. Consider an urgent-support detector where “positive” means actually urgent.

Policy outcome	Actual case	Operational result	Cost symbol
Automatic action	Urgent	Correct action	Usually $0$ or a small normal-processing cost
Automatic action	Not urgent	False positive: unnecessary urgent action	$C_{\mathrm{FP}}$
Human review	Either class	Reviewer time and delay	$C_{\mathrm{R}}$
Pass	Urgent	False negative: missed urgent case	$C_{\mathrm{FN}}$
Pass	Not urgent	Correct pass	Usually $0$

These are relative costs, not necessarily literal currency. You can express them in internal “loss units” as long as the ratios reflect a real business judgment.

For example:

Event	Example relative cost	Reasoning
Unnecessary urgent escalation	$8$	Interrupts the on-call team and delays ordinary tickets.
Human review	$1$	A reviewer spends time, but can correct the outcome.
Missed urgent case	$20$	A time-sensitive customer issue remains untreated.

The central cost function for a labeled evaluation set is:

$$L(t_r,t_a)=C_{\mathrm{FP}}\cdot FP+C_{\mathrm{FN}}\cdot FN+C_{\mathrm{R}}\cdot R$$

where:

• $FP$ is the number of non-urgent cases sent to automatic action,
• $FN$ is the number of urgent cases passed without review,
• $R$ is the total number of reviewed cases.

Dividing this by the number of cases gives average loss per case:

$$\bar{L}(t_r,t_a)=\frac{L(t_r,t_a)}{N}$$

This formulation makes an important product fact explicit:

Review is not “free safety.” It reduces costly mistakes, but it consumes finite reviewer capacity and introduces delay.

A threshold policy can be statistically cheap but operationally impossible if it routes 60% of traffic to a small review team. Treat review capacity as a constraint, not an afterthought.

---

3. Costs create a rational starting point, not a final threshold

Suppose your Noul probability $p$ is well calibrated for the proposition “this case is urgent.” Ignore review temporarily and compare only automatic action with pass:

• Automatic action has expected error cost $(1-p)C_{\mathrm{FP}}$.
• Passing has expected error cost $p{C}_{\mathrm{FN}}$.

Automatic action becomes preferable when:

$$(1-p)C_{\mathrm{FP}}\le p{C}_{\mathrm{FN}}$$

Solving gives the two-way action threshold:

$$p\ge \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}$$

This result can initially feel backward, so test it with intuition:

• If a false positive is very costly, $C_{\mathrm{FP}}$ is large. The threshold rises. You require stronger evidence before acting.
• If a false negative is very costly, $C_{\mathrm{FN}}$ is large. The threshold falls. You act earlier to avoid missing harmful cases.

With $C_{\mathrm{FP}}=8$ and $C_{\mathrm{FN}}=20$, the two-way threshold is:

$$\frac{8}{8+20}=0.286$$

That does not mean “automatically escalate every case above $0.286$.” It only says that, if you had no human-review path, automatic action becomes less costly than pass at that point.

Review adds a third alternative. Under a simplified model where review resolves the case correctly and costs $C_{\mathrm{R}}$, its cost is constant:

$$\text{review cost}=C_{\mathrm{R}}$$

The review region exists where review is cheaper than both kinds of automated error. The resulting theoretical boundaries are:

$$t_r=\frac{C_{\mathrm{R}}}{C_{\mathrm{FN}}}$$ $$t_a=1-\frac{C_{\mathrm{R}}}{C_{\mathrm{FP}}}$$

For the example costs:

$$t_r=\frac{1}{20}=0.05$$ $$t_a=1-\frac{1}{8}=0.875$$

This would produce a broad review band from $0.05$ through values below $0.875$. That may minimize loss in theory, but it may also overwhelm reviewers. Real threshold selection therefore needs empirical simulation and capacity limits.

The formulas are a sanity check under strong assumptions:

1. The score is a calibrated probability for the exact proposition.
2. False-positive and false-negative costs are stable.
3. Review outcomes are correct enough to treat review as a known fixed cost.
4. The evaluation dataset represents production traffic.

Your Jev evaluation is where those assumptions meet evidence.

---

4. Sweep candidate policies on labeled tuning data

Do not choose thresholds on the final held-out set. The workflow should be:

1. Agree on a cost model with the product owner, operations lead, or safety owner.
2. Use a tuning split to test candidate threshold pairs.
3. Select a policy based on minimum cost among policies that meet review-capacity limits.
4. Run that fixed selected policy once on the held-out split.
5. Record the threshold, costs, dataset version, and result.

This avoids repeatedly adjusting thresholds until the held-out metrics look good.

Here is an illustrative tuning result for 100 cases, including 15 truly urgent cases. The costs are $C_{\mathrm{FP}}=8$, $C_{\mathrm{FN}}=20$, and $C_{\mathrm{R}}=1$.

Review threshold	Action threshold	Auto-action FP	Passed urgent cases	Review count	Total loss
0.45	0.80	1	2	16	64
0.35	0.70	3	1	22	66
0.55	0.90	0	2	13	53

The third policy has the lowest loss:

$$0\cdot 8+2\cdot 20+13\cdot 1=53$$

It also creates the fewest reviews in this small comparison. Notice what did not select the policy:

• not the highest automatic-action accuracy;
• not the greatest number of urgent cases acted on automatically;
• not an arbitrary $0.80$ confidence convention;
• not ROC AUC.

It is the policy with the best business trade-off under an explicit cost model.

The threshold mechanics are useful to visualize before you automate the sweep.

ROC is useful when comparing the ranking quality of scoring systems, but it does not know:

• how costly a false positive is in your product;
• whether a review can correct the case;
• how much a reviewer costs;
• how many reviews the team can handle.

The rightmost plot in the following image is closer to the deployment question: evaluate an explicit objective for each threshold and choose the threshold with the best objective.

---

5. Implement a deterministic threshold evaluator in TypeScript

Keep the threshold sweep separate from Jev calls. It should consume recorded assessments and gold labels, just like the calibration report from the previous lesson.

This example uses a Noul result for one proposition: whether a case should enter the urgent path.

type RiskEvalRun = {
  id: string;
  split: "tuning" | "heldout";
  actual: {
    noul: number;
  } | null;
  expected: {
    urgent: boolean;
  };
};

type RiskPolicy = {
  name: string;
  reviewThreshold: number;
  actionThreshold: number;
};

type CostModel = {
  falsePositive: number;
  falseNegative: number;
  review: number;
};

type PolicyEvaluation = {
  policy: RiskPolicy;
  evaluatedCases: number;
  excludedFailures: number;
  actionTruePositive: number;
  actionFalsePositive: number;
  reviewUrgent: number;
  reviewNotUrgent: number;
  passFalseNegative: number;
  passTrueNegative: number;
  reviewRate: number;
  totalLoss: number;
  averageLoss: number;
};

function validateThreshold(value: number, name: string): void {
  if (!Number.isFinite(value) || value < 0 || value > 1) {
    throw new Error(`${name} must be a number from 0 through 1.`);
  }
}

function validatePolicy(policy: RiskPolicy): void {
  validateThreshold(policy.reviewThreshold, "reviewThreshold");
  validateThreshold(policy.actionThreshold, "actionThreshold");

  if (policy.reviewThreshold > policy.actionThreshold) {
    throw new Error(
      "reviewThreshold must be less than or equal to actionThreshold.",
    );
  }
}

function classifyRisk(
  risk: number,
  policy: RiskPolicy,
): "action" | "review" | "pass" {
  if (risk >= policy.actionThreshold) {
    return "action";
  }

  if (risk >= policy.reviewThreshold) {
    return "review";
  }

  return "pass";
}

export function evaluateRiskPolicy(
  runs: RiskEvalRun[],
  split: "tuning" | "heldout",
  policy: RiskPolicy,
  costs: CostModel,
): PolicyEvaluation {
  validatePolicy(policy);

  let excludedFailures = 0;
  let actionTruePositive = 0;
  let actionFalsePositive = 0;
  let reviewUrgent = 0;
  let reviewNotUrgent = 0;
  let passFalseNegative = 0;
  let passTrueNegative = 0;

  const relevantRuns = runs.filter(function (run) {
    return run.split === split;
  });

  for (const run of relevantRuns) {
    if (run.actual === null) {
      excludedFailures += 1;
      continue;
    }

    const risk = run.actual.noul;

    if (!Number.isFinite(risk) || risk < 0 || risk > 1) {
      throw new Error(`Invalid Noul value for case ${run.id}: ${risk}`);
    }

    const route = classifyRisk(risk, policy);
    const isUrgent = run.expected.urgent;

    if (route === "action" && isUrgent) {
      actionTruePositive += 1;
    } else if (route === "action" && !isUrgent) {
      actionFalsePositive += 1;
    } else if (route === "review" && isUrgent) {
      reviewUrgent += 1;
    } else if (route === "review" && !isUrgent) {
      reviewNotUrgent += 1;
    } else if (route === "pass" && isUrgent) {
      passFalseNegative += 1;
    } else {
      passTrueNegative += 1;
    }
  }

  const evaluatedCases =
    actionTruePositive +
    actionFalsePositive +
    reviewUrgent +
    reviewNotUrgent +
    passFalseNegative +
    passTrueNegative;

  if (evaluatedCases === 0) {
    throw new Error(`No valid ${split} responses to evaluate.`);
  }

  const reviewCount = reviewUrgent + reviewNotUrgent;

  const totalLoss =
    actionFalsePositive * costs.falsePositive +
    passFalseNegative * costs.falseNegative +
    reviewCount * costs.review;

  return {
    policy,
    evaluatedCases,
    excludedFailures,
    actionTruePositive,
    actionFalsePositive,
    reviewUrgent,
    reviewNotUrgent,
    passFalseNegative,
    passTrueNegative,
    reviewRate: reviewCount / evaluatedCases,
    totalLoss,
    averageLoss: totalLoss / evaluatedCases,
  };
}

A call might compare a small set of intentional policy candidates:

const costs: CostModel = {
  falsePositive: 8,
  falseNegative: 20,
  review: 1,
};

const candidates: RiskPolicy[] = [
  {
    name: "conservative-action",
    reviewThreshold: 0.55,
    actionThreshold: 0.90,
  },
  {
    name: "balanced",
    reviewThreshold: 0.45,
    actionThreshold: 0.80,
  },
  {
    name: "high-coverage",
    reviewThreshold: 0.35,
    actionThreshold: 0.70,
  },
];

const evaluations = candidates.map(function (policy) {
  return evaluateRiskPolicy(runs, "tuning", policy, costs);
});

console.table(
  evaluations.map(function (result) {
    return {
      policy: result.policy.name,
      reviewThreshold: result.policy.reviewThreshold,
      actionThreshold: result.policy.actionThreshold,
      falsePositiveActions: result.actionFalsePositive,
      missedUrgentCases: result.passFalseNegative,
      reviewRate: `${(result.reviewRate * 100).toFixed(1)}%`,
      totalLoss: result.totalLoss,
      averageLoss: result.averageLoss.toFixed(2),
    };
  }),
);

Use candidate values based on the distribution of recorded outputs. If all observed scores fall between $0.70$ and $0.95$, testing thresholds at $0.10$, $0.20$, and $0.30$ adds no useful information because all those policies behave identically.

---

6. Make review capacity a hard constraint

Assume the review team can process no more than $15\%$ of daily cases. A policy with the lowest cost but a $30\%$ review rate is not an eligible policy.

Filter candidate policies before selecting among them:

export function selectPoliciesWithinReviewCapacity(
  evaluations: PolicyEvaluation[],
  maxReviewRate: number,
): PolicyEvaluation[] {
  if (
    !Number.isFinite(maxReviewRate) ||
    maxReviewRate < 0 ||
    maxReviewRate > 1
  ) {
    throw new Error("maxReviewRate must be a number from 0 through 1.");
  }

  const feasible = evaluations.filter(function (evaluation) {
    return evaluation.reviewRate <= maxReviewRate;
  });

  if (feasible.length === 0) {
    throw new Error(
      "No candidate policy satisfies the review-capacity constraint.",
    );
  }

  const minimumLoss = Math.min(
    ...feasible.map(function (evaluation) {
      return evaluation.totalLoss;
    }),
  );

  return feasible.filter(function (evaluation) {
    return evaluation.totalLoss === minimumLoss;
  });
}

Returning all equal-cost candidates is deliberate. Do not hide a business decision behind arbitrary technical tie-breaking. If two policies have equal measured loss, compare:

1. False negatives: Does one policy leave more genuinely urgent cases unreviewed?
2. False-positive actions: Does one create more needless escalations or blocks?
3. Review volume: Does one preserve more capacity for other workflows?
4. Stability: Are the differences based on a handful of borderline cases?

If the winner changes dramatically when one or two cases change, your evaluation set is too small for a strong claim. Start conservatively, collect review outcomes, and revisit with more evidence.

---

7. Validate the chosen policy without moving the goalposts

Once the tuning split yields an acceptable candidate:

1. Freeze the cost model and selected thresholds.
2. Run the policy unchanged on the held-out split.
3. Report the same counts and average loss.
4. Report review rate and Jev failure rate separately.
5. Inspect the false-positive actions and missed urgent cases individually.

For the final report, include enough detail that a teammate can reconstruct the decision:

Field	Example
Policy version	urgent-triage-v1
Jev question contract version	urgent-proposition-v3
Dataset version	support-eval-2025-03-01
Cost model	FP $=8$, FN $=20$, review $=1$
Review limit	$15\%$
Selected thresholds	Review $=0.55$, action $=0.90$
Tuning loss	Recorded before held-out evaluation
Held-out loss	Reported once after policy selection
Failure behavior	Fallback and review behavior for unavailable Jev results

The previous lesson’s calibration report still matters. If high scores do not correspond to better real outcomes, a more elaborate cost function will not make thresholds trustworthy. Improve the question contract, the evidence in state, or the taxonomy before relying on an automated action path.

---

Key takeaways

• A Jev assessment is not a policy. Your application owns the action, review, and fallback decision.
• Use two thresholds for three paths: automatic action, human review, and pass.
• Assign relative costs to false positives, false negatives, and review work before selecting thresholds.
• Choose policies by simulated loss on a tuning dataset, subject to a review-capacity constraint.
• A theoretical threshold is a useful starting point only when the score is well calibrated and the cost assumptions are realistic.
• Keep the final held-out dataset for one fixed-policy evaluation, not repeated threshold adjustment.
• ROC and AUC describe ranking behavior, but they do not encode your false-positive cost, false-negative cost, reviewer workload, or operational capacity.

Next, you will run sensitivity tests that vary Jev question wording and state evidence, looking for threshold policies that appear good on aggregate metrics but behave brittly under small input changes.