Execute a Production Release Gate Evaluation

The evaluation runner from the previous lesson can tell you how a baseline and a candidate behaved. A release gate turns that evidence into a deterministic deployment decision: either the candidate satisfies the agreed operational contract, or it does not ship.

For this capstone’s support workflow, the contract is deliberately broader than “accuracy improved.” A version must meet quality and safe-automation targets, remain within latency and estimated-cost limits, and demonstrate that its fallback paths behave safely under controlled failure conditions.

By the end of this lesson, you will have a Python gate that consumes the runner’s JSON report, emits a machine-readable decision, and exits with a non-zero status when the candidate fails. That makes it suitable for CI without putting an LLM in the release decision itself.

---

A release gate is a deterministic policy over evaluation evidence

The workflow is still probabilistic at its Jev decision points. The decision to deploy it should not be. The gate is ordinary deterministic application code with three inputs:

1. An immutable evaluation report for the candidate and baseline.
2. A versioned configuration of production targets.
3. Explicit fallback scenarios that exercise failure handling.

Its output is a list of pass or fail checks and a final release decision.

This separation is important:

• The evaluation runner observes behavior on a labeled dataset.
• The release gate applies business and operational constraints to those observations.
• Deployment automation acts only on the gate’s exit status and saved decision artifact.

A gate is not an optimizer. It should not decide that a large cost reduction compensates for an unsafe security regression. Instead, it encodes non-negotiable boundaries.

For the support workflow, use the following ordering:

Priority	Release concern	Typical hard constraint
1	Correct action and policy handling	Minimum strict decision quality
2	Correct automatic handling	Minimum safe automation coverage
3	Safe degradation	All required timeout, outage, and low-confidence fallback cases pass
4	User experience	Caller-observed p95 latency stays below the SLA
5	Economics	Estimated cost per evaluated case stays below the budget

The exact targets belong to the product and operations team, not to a generic framework. A premium account-recovery queue may choose a high quality floor and broad review behavior. A routine ecommerce queue may accept more automation, provided its safety constraints remain intact.

---

Define targets before looking at the candidate

Do not inspect the candidate report first and then choose thresholds that make it pass. Put the targets in version control, review them like any other policy change, and keep them stable for a release comparison.

Create release-targets.json beside eval-config.json:

{
  "quality": {
    "min": 0.92,
    "maxRegressionPercentagePoints": -1.0
  },
  "safeAutomationCoverage": {
    "min": 0.65,
    "maxRegressionPercentagePoints": -2.0
  },
  "callerP95LatencyMs": {
    "max": 1200
  },
  "estimatedCostPerCaseUsd": {
    "max": 0.00009
  },
  "errorRate": {
    "max": 0.0
  },
  "fallbackScenarios": [
    {
      "tag": "fault:jev_timeout",
      "minCases": 2,
      "disposition": "review",
      "allowedModelStatuses": ["timeout"],
      "allowedFallbackReasons": ["jev_timeout"],
      "maxJevRequests": 1
    },
    {
      "tag": "fault:jev_unavailable",
      "minCases": 2,
      "disposition": "review",
      "allowedModelStatuses": ["unavailable", "circuit_open"],
      "allowedFallbackReasons": ["jev_unavailable", "jev_circuit_open"],
      "maxJevRequests": 1
    },
    {
      "tag": "low_confidence",
      "minCases": 3,
      "disposition": "review",
      "allowedModelStatuses": ["completed"],
      "allowedFallbackReasons": ["low_confidence"],
      "maxJevRequests": 1
    }
  ]
}

These figures are illustrative. They are a useful initial contract for the capstone, not recommended universal values. Calibrate your own from:

• the support queue’s SLA and budget;
• prior baseline performance;
• the cost of a wrong automatic decision;
• review-team capacity;
• labeled examples, especially security and account-access cases.

The two quality constraints serve different purposes:

$$\text{candidate quality}\ge \text{absolute quality floor}$$ $$100\times (\text{candidate quality}-\text{baseline quality})\ge \text{maximum allowed regression in percentage points}$$

The absolute floor prevents release when both versions are inadequate. The baseline-relative limit prevents a candidate from quietly degrading a mature workflow while still barely clearing the floor.

The same approach applies to safe automation coverage. Recall its definition from the previous lesson:

$$\text{safe automation coverage}=\frac{\text{eligible cases that are correct and automated}}{\text{automation eligible cases}}$$

A candidate with higher raw automation but flat or lower safe coverage is not more useful. It is merely taking more unverified actions.

---

Treat fallback behavior as a first-class release requirement

The ordinary labeled dataset tests normal support inputs. A production release also needs evidence that the workflow degrades correctly when its Jev dependency is slow, unavailable, or uncertain.

A safe support fallback should be a structured decision such as:

{
  "decision": {
    "decisionKey": "manual_triage:jev_timeout",
    "outcome": "manual_triage",
    "disposition": "review"
  },
  "inspection": {
    "latency": {
      "totalMs": 803,
      "modelMs": null
    },
    "model": {
      "attempted": true,
      "status": "timeout",
      "fallbackReason": "jev_timeout"
    }
  },
  "metrics": {
    "jevRequestCount": 1,
    "estimatedJevCostUsd": 0.0
  }
}

The exact cost of a timeout depends on provider and network behavior. The key point is that the workflow reports the cost it can account for rather than assuming every failed request costs zero.

Add explicit fault scenarios to the evaluation dataset. For example:

{
  "id": "fallback-timeout-001",
  "state": {
    "status": "open",
    "subject": "Charged twice",
    "message": "I was charged twice for my subscription.",
    "accountTier": "standard"
  },
  "evalFault": "jev_timeout",
  "expected": {
    "decisionKey": "manual_triage:jev_timeout",
    "outcome": "manual_triage",
    "disposition": "review"
  },
  "automationEligible": false,
  "tags": ["fallback", "fault:jev_timeout"]
}

evalFault must be accepted only by an authenticated internal evaluation endpoint. It should activate a controlled failure at the Jev adapter boundary, not be forwarded from any customer-controlled request field.

This is fault injection, not a replacement of Jev with a fixture during ordinary evaluation. Your normal cases still execute the real workflow and real Jev calls. The controlled fault cases verify that production code handles a dependency failure safely and predictably.

For a low-confidence scenario, do not inject an outage. Instead, use a reviewed state that is expected to fall into the workflow’s configured review band and assert the resulting low_confidence fallback reason.

The cookbook’s warning matters for your cost gate: historical or benchmark pricing is not a production billing source. Use one of these approaches:

1. Preferred: have the Jev adapter attach an account-specific, versioned cost estimate to every workflow evaluation result.
2. Acceptable initially: multiply instrumented request counts by a reviewed internal price snapshot, recording the snapshot date and assumptions in the report.
3. Never acceptable for a gate: silently treat unknown cost as zero.

For a candidate evaluated on $N$ cases, the cost metric is:

$$\text{estimated cost per case}=\frac{{\sum }_{i=1}^N\text{estimated Jev cost for case }i}{N}$$

This cost includes every attempted Jev request, including retries and speculative calls. It is intentionally a per-case metric, because two workflows can handle the same number of tickets while making very different numbers of model calls.

---

Extend the evaluation contract with status, fallback reason, and cost

The previous runner already reads the decision, latency, model status, and Jev request count. Extend its record so the release gate has enough evidence to judge fallback behavior and cost.

First, update the internal evaluation endpoint contract so every completed workflow response includes:

{
  "inspection": {
    "model": {
      "attempted": true,
      "status": "completed",
      "fallbackReason": null
    }
  },
  "metrics": {
    "jevRequestCount": 1,
    "estimatedJevCostUsd": 0.000041
  }
}

A deterministic short-circuit should explicitly report zero rather than omit fields:

{
  "inspection": {
    "model": {
      "attempted": false,
      "status": "not_needed",
      "fallbackReason": null
    }
  },
  "metrics": {
    "jevRequestCount": 0,
    "estimatedJevCostUsd": 0.0
  }
}

Patch the Python runner

In runner.py, add these fields to RunRecord:

    model_status: str | None
    fallback_reason: str | None
    estimated_cost_usd: float | None

In the successful RunRecord(...) construction inside post_case, add:

            model_status=inspection["model"]["status"],
            fallback_reason=inspection["model"].get("fallbackReason"),
            estimated_cost_usd=metrics["estimatedJevCostUsd"],

In the exception branch, add the corresponding unknown values:

            model_status=None,
            fallback_reason=None,
            estimated_cost_usd=None,

Then let fault-tagged dataset rows send their internal evaluation mode. Add this field to EvalCase:

    eval_fault: str | None

Set it in load_cases:

                eval_fault=raw.get("evalFault"),

Finally, replace the current payload construction at the start of post_case with:

    payload_object: dict[str, Any] = {
        "caseId": case.case_id,
        "ticket": case.state,
    }

    if case.eval_fault is not None:
        payload_object["eval"] = {"faultMode": case.eval_fault}

    payload = json.dumps(payload_object).encode("utf-8")

After making this change, run the evaluator again against both versions. A report record for a controlled timeout should now retain all the evidence needed by the gate:

{
  "case_id": "fallback-timeout-001",
  "version": "candidate-9d03f6b",
  "correct": true,
  "observed_disposition": "review",
  "model_status": "timeout",
  "fallback_reason": "jev_timeout",
  "jev_request_count": 1,
  "estimated_cost_usd": 0.0,
  "error": null,
  "tags": ["fallback", "fault:jev_timeout"]
}

---

Implement the release gate

Create gate.py in the uv project. It deliberately uses only the standard library. It reads a report, selects one candidate, checks the configured targets, writes a decision artifact, and exits with code 1 if any check fails.

from __future__ import annotations

import argparse
import json
import math
from pathlib import Path
from typing import Any


def read_json(path: str) -> dict[str, Any]:
    return json.loads(Path(path).read_text())


def is_number(value: Any) -> bool:
    return (
        isinstance(value, (int, float))
        and not isinstance(value, bool)
        and math.isfinite(value)
    )


def check(
    checks: list[dict[str, Any]],
    name: str,
    passed: bool,
    actual: Any,
    target: Any,
    detail: str,
) -> None:
    checks.append(
        {
            "name": name,
            "passed": passed,
            "actual": actual,
            "target": target,
            "detail": detail,
        }
    )


def require_min(
    checks: list[dict[str, Any]],
    name: str,
    actual: Any,
    minimum: float,
) -> None:
    passed = is_number(actual) and actual >= minimum
    check(checks, name, passed, actual, f">= {minimum}", "minimum threshold")


def require_max(
    checks: list[dict[str, Any]],
    name: str,
    actual: Any,
    maximum: float,
) -> None:
    passed = is_number(actual) and actual <= maximum
    check(checks, name, passed, actual, f"<= {maximum}", "maximum threshold")


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--report", required=True)
    parser.add_argument("--targets", required=True)
    parser.add_argument("--candidate", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()

    report = read_json(args.report)
    targets = read_json(args.targets)

    baseline_name = report["baseline"]
    candidate_name = args.candidate
    summaries = report["summaries"]

    if candidate_name == baseline_name:
        raise ValueError("Candidate must differ from the baseline.")

    if baseline_name not in summaries or candidate_name not in summaries:
        raise ValueError("Report does not contain the requested baseline and candidate.")

    baseline = summaries[baseline_name]
    candidate = summaries[candidate_name]
    records = [
        record
        for record in report["records"]
        if record["version"] == candidate_name
    ]

    checks: list[dict[str, Any]] = []

    require_min(
        checks,
        "quality floor",
        candidate["quality"],
        targets["quality"]["min"],
    )

    quality_delta_pp = 100 * (candidate["quality"] - baseline["quality"])
    require_min(
        checks,
        "quality regression limit",
        quality_delta_pp,
        targets["quality"]["maxRegressionPercentagePoints"],
    )

    require_min(
        checks,
        "safe automation coverage floor",
        candidate["safeAutomationCoverage"],
        targets["safeAutomationCoverage"]["min"],
    )

    coverage_delta_pp = 100 * (
        candidate["safeAutomationCoverage"]
        - baseline["safeAutomationCoverage"]
    )
    require_min(
        checks,
        "safe automation coverage regression limit",
        coverage_delta_pp,
        targets["safeAutomationCoverage"]["maxRegressionPercentagePoints"],
    )

    require_max(
        checks,
        "caller p95 latency",
        candidate["callerLatencyMs"]["p95"],
        targets["callerP95LatencyMs"]["max"],
    )

    error_rate = (
        sum(record["error"] is not None for record in records) / len(records)
        if records
        else None
    )
    require_max(
        checks,
        "transport or contract error rate",
        error_rate,
        targets["errorRate"]["max"],
    )

    costs = [record["estimated_cost_usd"] for record in records]
    known_costs = all(is_number(cost) and cost >= 0 for cost in costs)
    mean_cost = sum(costs) / len(costs) if known_costs and costs else None

    require_max(
        checks,
        "estimated Jev cost per case",
        mean_cost,
        targets["estimatedCostPerCaseUsd"]["max"],
    )

    for scenario in targets["fallbackScenarios"]:
        matching = [
            record
            for record in records
            if scenario["tag"] in record["tags"]
        ]

        scenario_name = f"fallback: {scenario['tag']}"
        enough_cases = len(matching) >= scenario["minCases"]

        valid_records = [
            record
            for record in matching
            if record["correct"]
            and record["error"] is None
            and record["observed_disposition"] == scenario["disposition"]
            and record["model_status"] in scenario["allowedModelStatuses"]
            and record["fallback_reason"] in scenario["allowedFallbackReasons"]
            and is_number(record["jev_request_count"])
            and record["jev_request_count"] <= scenario["maxJevRequests"]
        ]

        passed = enough_cases and len(valid_records) == len(matching)

        check(
            checks,
            scenario_name,
            passed,
            {
                "matchingCases": len(matching),
                "validCases": len(valid_records),
            },
            {
                "minimumCases": scenario["minCases"],
                "disposition": scenario["disposition"],
                "maxJevRequests": scenario["maxJevRequests"],
            },
            "all fault or low-confidence cases must degrade as specified",
        )

    passed = all(item["passed"] for item in checks)

    decision = {
        "passed": passed,
        "baseline": baseline_name,
        "candidate": candidate_name,
        "dataset": report["dataset"],
        "checks": checks,
    }

    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
    Path(args.out).write_text(json.dumps(decision, indent=2))

    for item in checks:
        status = "PASS" if item["passed"] else "FAIL"
        print(f"{status}: {item['name']} — {item['detail']}")

    print(f"Wrote gate decision: {args.out}")

    if not passed:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Run the comparison, then the gate:

uv run python runner.py \
  --config eval-config.json \
  --out reports/triage-comparison.json

uv run python gate.py \
  --report reports/triage-comparison.json \
  --targets release-targets.json \
  --candidate candidate-9d03f6b \
  --out reports/candidate-9d03f6b-gate.json

A successful gate exits with status 0. A failed gate writes its decision artifact but exits with status 1, which is exactly what CI needs to block promotion.

---

Read a failed gate as diagnostic evidence

Suppose the candidate’s report gives:

Metric	Baseline	Candidate	Target
Quality	0.93	0.92	At least 0.92
Safe automation coverage	0.67	0.69	At least 0.65
Caller p95 latency	970 ms	1,260 ms	At most 1,200 ms
Estimated cost per case	0.000071 USD	0.000082 USD	At most 0.000090 USD
Timeout fallbacks	2 of 2 valid	2 of 2 valid	All valid

The candidate is better at safe automation and remains within cost budget, but it fails the latency gate. That should produce a failed release decision even though most checks pass.

The next engineering action is not “raise the latency threshold” by default. Inspect the high-latency records:

• Did the candidate introduce a second Jev call?
• Did a timeout consume most of the caller-side budget before fallback?
• Did deterministic ineligible tickets unexpectedly start calling Jev?
• Did a new branch add remote work to all cases rather than only the relevant subset?
• Does the latency regression occur in one high-risk segment where a slower route is intentional?

Only after understanding the mechanism should you change code, targets, or both. If the target itself changes, treat that as a business-policy decision with a recorded rationale.

---

Put the gate into the release path

A practical release workflow has three layers:

1. Fast deterministic tests run on every change. They validate schema handling, thresholds, policy composition, and recorded response fixtures.
2. Real workflow evaluation runs for a release candidate against the fixed labeled dataset and controlled fallback scenarios.
3. The release gate reads the immutable report and blocks or permits promotion.

Keep these artifacts together for each candidate:

reports/
├── triage-comparison.json
├── candidate-9d03f6b-gate.json
└── candidate-9d03f6b-metadata.json

The metadata file can record:

• candidate Git SHA;
• baseline Git SHA;
• dataset hash already emitted by the runner;
• target-config hash;
• evaluation timestamp;
• environment identity;
• Jev pricing snapshot or cost-estimation method.

That record is useful when production monitoring later reveals a problem: you can identify exactly what evidence and policy allowed a version to ship.

Before relying on the gate, verify these properties:

• Every metric required by the gate is present and finite; missing cost or latency must fail rather than pass accidentally.
• Fallback cases are marked automationEligible: false, so they do not distort the normal automation denominator.
• Every fallback scenario asserts the final business decision, disposition, model status, fallback reason, and request-count ceiling.
• A circuit-open result is treated as a structured fallback, not as a successful automated resolution.
• The candidate identifier is immutable, ideally a Git SHA or release artifact digest.
• Reports and target configuration are retained after deployment.
• New production failures become reviewed candidate cases and eventually enter the next version of the stable evaluation set.

---

You now have a deterministic release gate over the evidence produced by your real Jev workflow. It enforces strict decision quality, safe automation coverage, p95 latency, estimated cost, protocol reliability, and explicit fallback behavior rather than relying on a single attractive aggregate score.

This completes the capstone: the support automation workflow can make bounded Jev judgments, expose its evidence for inspection, be evaluated with a uv-based Python runner, and be promoted only when a versioned, reviewable release contract is satisfied.