Week 4 — Evaluation, Calibration, and Failure Analysis

Build a labeled evaluation dataset from representative states, expected decisions, and review outcomes.

Measure decision accuracy, automation coverage, and abstention behavior for a Jev workflow.

Assess whether higher reported confidence corresponds to higher empirical accuracy on an evaluation dataset.

Select action and review thresholds from the relative costs of false positives, false negatives, and human review.

Run sensitivity tests that vary question wording and state evidence to expose brittle decisions.

Create regression tests with recorded typed responses for decision logic, thresholds, and fallback behavior.