Building a Minimal Decision Inspector

Good to see the workflow becoming operational rather than merely functional. In the previous lesson, you made Jev failure modes explicit: low-confidence results were reviewable decisions, while timeouts, rate limits, outages, and an open circuit produced deterministic fallback behavior.

Now make that behavior inspectable. A support lead or on-call engineer should be able to answer, for any ticket: what decision was made, what Jev signals were available, how long it took, and why the system escalated, deferred, or fell back. The inspector must report evidence and policy outcomes without exposing ticket text, credentials, or invented model rationale.

Plan for roughly 40 minutes: first establish the inspection contract, then wire it into the triage service and optionally correlate it with OpenTelemetry traces.

---

An inspector is an audit projection, not a debugging dump

A decision inspector should not show “everything the system knows.” It should show the small, stable set of fields needed to explain a bounded decision.

For the support workflow, that means:

Category	Inspector should display	Inspector should not display
Decision	Outcome kind and policy reason	Unstructured internal notes
Model evidence	Selected option, all option probabilities, confidence	Raw prompt or full ticket message
Timing	End-to-end latency and model-attempt latency	Guessed latency for a call that never happened
Escalation	Explicit policy code such as intent_uncertain	A fabricated natural-language explanation
Fallback	jev_timeout, jev_rate_limited, and similar codes	A claim that Jev “decided” during an outage
Correlation	Decision ID and optional trace ID	Customer identifiers in generic telemetry

The important distinction is between model evidence and application outcome.

Suppose Jev selects billing for an intent question with a high probability, but the safety check reports an ambiguous credential-risk probability. The application outcome may be manual_triage, even though the selected intent was billing. The inspector needs both facts. If it shows only the final outcome, reviewers cannot tell whether the escalation came from uncertainty, safety policy, or dependency failure.

Before implementing, take a short look at the trace model. It provides useful terminology for correlating one decision with the request that caused it.

A trace viewer is useful for cross-service diagnosis, but it is not your product-facing inspector. A trace may contain dozens of infrastructure spans. Your inspector should be a compact, domain-specific record for one support decision.

---

Define a durable inspection contract

Do not expose the provider SDK response directly to your UI. SDK response shapes can evolve, and they often contain metadata that is inappropriate to retain or display. Instead, normalize Jev results at the adapter boundary.

Create lib/decision-inspection.ts:

export type InspectorQuestionKind = "choice" | "score" | "noul";

export type ProbabilityOption = {
  value: string;
  probability: number;
};

export type QuestionInspection = {
  questionId: string;
  kind: InspectorQuestionKind;
  selectedValue: string;
  confidence: number | null;
  options: readonly ProbabilityOption[];
};

export type ModelAttemptStatus =
  | "not_needed"
  | "completed"
  | "timeout"
  | "rate_limited"
  | "unavailable"
  | "misconfigured"
  | "circuit_open";

export type DecisionDisposition =
  | "automated"
  | "review"
  | "deferred"
  | "no_action";

export type DecisionInspection = {
  decisionId: string;
  observedAt: string;
  traceId?: string;

  outcome: string;
  reason: string;
  disposition: DecisionDisposition;

  latency: {
    totalMs: number;
    modelMs: number | null;
  };

  model: {
    attempted: boolean;
    status: ModelAttemptStatus;
    questions: readonly QuestionInspection[];
  };

  escalationReason?: string;
  fallbackReason?: string;
};

A few design decisions matter here:

• selectedValue is always a string. The presentation layer should not need to understand whether the original value was a boolean, score label, or enum.
• options contains the full distribution for the bounded question. Do not display only the selected value.
• confidence is nullable. A timeout is not “zero confidence”; it is no returned confidence.
• modelMs is nullable for local eligibility decisions and circuit-open fallbacks because no model call took place.
• outcome and reason are separate. manual_triage is an outcome; intent_uncertain or jev_timeout is the reason.

For a Noul question, normalize the one proposition probability into two visible options. If the proposition is “Does this message request credential reset?” and Jev returns $0.91$, the inspector can show:

Value	Probability
true	91.0%
false	9.0%

That makes the output legible without making the reviewer mentally calculate the complement.

Use stable, versioned question identifiers rather than the full question text:

const QUESTION_IDS = {
  intent: "support.intent.v1",
  credentialRisk: "support.credential_risk.v1",
  urgency: "support.urgency.v1",
} as const;

This lets you compare behavior over time while keeping prompts and customer content out of the inspector.

---

Preserve Jev evidence before policy reduces it to a decision

Your earlier decideTriage() function correctly reduces Jev evidence to a typed application decision. But once that reduction has happened, you may no longer have the probability distribution required by the inspector.

The solution is to make the SDK adapter return both:

1. the deterministic TriageDecision;
2. a normalized list of inspectable model-question results.

Extend the transport contract from the previous lesson:

import type { TriageDecision } from "./triage-contract";
import type { QuestionInspection } from "./decision-inspection";
import type { SupportTicket } from "./triage-service";

export type EvaluatedTriage = {
  decision: TriageDecision;
  questions: readonly QuestionInspection[];
};

export interface TriageAttempt {
  run(
    ticket: SupportTicket,
    options: { signal: AbortSignal },
  ): Promise<EvaluatedTriage>;
}

Inside your real Jev adapter, normalize the provider’s successful typed response immediately. The exact field names depend on the version of the Jev SDK you receive, so isolate that version-sensitive code in one function:

function normalizeChoice(
  questionId: string,
  selectedValue: string,
  confidence: number,
  probabilities: Record<string, number>,
): QuestionInspection {
  return {
    questionId,
    kind: "choice",
    selectedValue,
    confidence,
    options: Object.entries(probabilities).map(function ([value, probability]) {
      return { value, probability };
    }),
  };
}

Use the same pattern for score and noul results. The adapter should validate three invariants before returning data to the application:

1. Every probability is within the range from 0 to 1.
2. A Choice or Score distribution contains every allowed option exactly once.
3. A Noul result exposes both true and false probabilities, which sum to approximately 1.

This is a useful boundary: the rest of the application never needs to know Jev’s raw response schema, and the UI never receives a provider object.

---

Collect one inspection record per request

The collector below is deliberately request-scoped. Create a new instance for every triage call. Do not store mutable “current decision” state on a singleton service, because concurrent support requests would overwrite one another.

Add the following to lib/decision-inspection.ts:

import { randomUUID } from "node:crypto";

type InspectableDecision = {
  kind: string;
  reason?: string;
};

type InspectionClock = {
  monotonicNow(): number;
  wallNow(): Date;
};

const FALLBACK_REASONS = new Set([
  "jev_timeout",
  "jev_unavailable",
  "jev_circuit_open",
  "jev_misconfigured",
  "jev_rate_limited",
]);

const DEFAULT_CLOCK: InspectionClock = {
  monotonicNow: function () {
    return performance.now();
  },
  wallNow: function () {
    return new Date();
  },
};

function dispositionFor(kind: string): DecisionDisposition {
  if (kind === "route") {
    return "automated";
  }

  if (kind === "defer_triage") {
    return "deferred";
  }

  if (kind === "no_action") {
    return "no_action";
  }

  return "review";
}

export class DecisionInspectionBuilder {
  private readonly startedAtMs: number;
  private readonly observedAt: string;

  private attempted = false;
  private modelStartedAtMs: number | undefined;
  private modelLatencyMs: number | null = null;
  private modelStatus: ModelAttemptStatus = "not_needed";
  private questions: readonly QuestionInspection[] = [];

  constructor(
    private readonly traceId: string | undefined,
    private readonly clock: InspectionClock = DEFAULT_CLOCK,
  ) {
    this.startedAtMs = clock.monotonicNow();
    this.observedAt = clock.wallNow().toISOString();
  }

  modelAttempted(): void {
    this.attempted = true;
    this.modelStartedAtMs = this.clock.monotonicNow();
  }

  modelCompleted(questions: readonly QuestionInspection[]): void {
    this.questions = questions;
    this.modelStatus = "completed";
    this.finishModelTiming();
  }

  modelFailed(
    status: Exclude<ModelAttemptStatus, "not_needed" | "completed" | "circuit_open">,
  ): void {
    this.modelStatus = status;
    this.finishModelTiming();
  }

  modelBypassed(): void {
    this.modelStatus = "circuit_open";
  }

  finish(decision: InspectableDecision): DecisionInspection {
    const reason = decision.reason ?? decision.kind;
    const isFallback = FALLBACK_REASONS.has(reason);

    return {
      decisionId: randomUUID(),
      observedAt: this.observedAt,
      traceId: this.traceId,
      outcome: decision.kind,
      reason,
      disposition: dispositionFor(decision.kind),
      latency: {
        totalMs: Math.round(this.clock.monotonicNow() - this.startedAtMs),
        modelMs: this.modelLatencyMs,
      },
      model: {
        attempted: this.attempted,
        status: this.modelStatus,
        questions: this.questions,
      },
      escalationReason:
        decision.kind === "manual_triage" && !isFallback
          ? reason
          : decision.kind === "security_review"
            ? reason
            : undefined,
      fallbackReason: isFallback ? reason : undefined,
    };
  }

  private finishModelTiming(): void {
    if (this.modelStartedAtMs === undefined) {
      return;
    }

    this.modelLatencyMs = Math.round(
      this.clock.monotonicNow() - this.modelStartedAtMs,
    );
  }
}

Use a monotonic clock such as performance.now() for elapsed time. Date.now() can move backward or forward if the host clock is adjusted; it is appropriate for timestamps, not duration measurement.

A completed manual-review decision might produce a record like this:

{
  "outcome": "manual_triage",
  "reason": "intent_uncertain",
  "disposition": "review",
  "latency": {
    "totalMs": 614,
    "modelMs": 587
  },
  "model": {
    "attempted": true,
    "status": "completed",
    "questions": [
      {
        "questionId": "support.intent.v1",
        "kind": "choice",
        "selectedValue": "billing",
        "confidence": 0.54,
        "options": [
          { "value": "billing", "probability": 0.44 },
          { "value": "technical", "probability": 0.39 },
          { "value": "account", "probability": 0.17 }
        ]
      }
    ]
  },
  "escalationReason": "intent_uncertain"
}

Notice that this does not say Jev “explained” why the ticket is billing-related. It records the bounded signal and the application policy that chose review.

---

Wire the collector into resilient triage

Add an observer parameter to the resilience facade from the previous lesson. The observer records events; it has no authority to alter the decision.

import type {
  QuestionInspection,
} from "./decision-inspection";

export interface TriageInspectionObserver {
  modelAttempted(): void;
  modelCompleted(questions: readonly QuestionInspection[]): void;
  modelFailed(
    status: "timeout" | "rate_limited" | "unavailable" | "misconfigured",
  ): void;
  modelBypassed(): void;
}

const NOOP_OBSERVER: TriageInspectionObserver = {
  modelAttempted: function () {},
  modelCompleted: function () {},
  modelFailed: function () {},
  modelBypassed: function () {},
};

Then make the relevant changes in ResilientTriageService.triage():

async triage(
  ticket: SupportTicket,
  observer: TriageInspectionObserver = NOOP_OBSERVER,
): Promise<ResilientTriageDecision> {
  const localDecision = localEligibility(ticket);

  if (localDecision !== undefined) {
    return localDecision;
  }

  const startMs = this.now();

  if (!this.breaker.permit(startMs)) {
    observer.modelBypassed();
    return fallback("jev_circuit_open", false);
  }

  observer.modelAttempted();

  try {
    const evaluated = await runWithDeadline(
      this.attempt,
      ticket,
      POLICY.jevDeadlineMs,
    );

    this.breaker.recordSuccess();
    observer.modelCompleted(evaluated.questions);

    return evaluated.decision;
  } catch (error) {
    if (!(error instanceof JevTransportError)) {
      throw error;
    }

    if (error.kind === "misconfigured") {
      observer.modelFailed("misconfigured");
      return fallback("jev_misconfigured", true);
    }

    this.breaker.recordFailure(this.now());

    if (error.kind === "rate_limited") {
      observer.modelFailed("rate_limited");

      return {
        kind: "defer_triage",
        reason: "jev_rate_limited",
        retryAt: new Date(
          this.now() + boundedRetryDelay(error.retryAfterMs),
        ).toISOString(),
        fallback: {
          source: "jev",
          modelAttempted: true,
          retryable: true,
        },
      };
    }

    if (error.kind === "timeout") {
      observer.modelFailed("timeout");
      return fallback("jev_timeout", true);
    }

    observer.modelFailed("unavailable");
    return fallback("jev_unavailable", true);
  }
}

The key change is that runWithDeadline() now resolves to EvaluatedTriage, rather than only TriageDecision. Its timing and cancellation behavior remains unchanged.

At the HTTP or server-action boundary, create the collector, run the workflow, persist or emit the record, and dispatch only the decision:

import { DecisionInspectionBuilder } from "./decision-inspection";

export async function triageWithInspection(
  service: ResilientTriageService,
  ticket: SupportTicket,
  traceId?: string,
): Promise<{
  decision: ResilientTriageDecision;
  inspection: DecisionInspection;
}> {
  const inspector = new DecisionInspectionBuilder(traceId);
  const decision = await service.triage(ticket, inspector);
  const inspection = inspector.finish(decision);

  return { decision, inspection };
}

Keep this separation strict:

• decision goes to the downstream dispatcher.
• inspection goes to a short-retention audit store, internal inspector endpoint, or telemetry pipeline.
• Neither object should contain the full support message unless you have made an explicit, reviewed retention decision.

---

Build a small display model for the frontend

Avoid putting policy logic in the React layer. The frontend should receive a stable view model and render it using your existing component system.

import type { DecisionInspection } from "./decision-inspection";

export type InspectorRow = {
  label: string;
  value: string;
};

export type InspectorQuestionRow = {
  questionId: string;
  selectedValue: string;
  confidence: string;
  probabilities: readonly InspectorRow[];
};

export type DecisionInspectorView = {
  summary: readonly InspectorRow[];
  questions: readonly InspectorQuestionRow[];
};

function percent(value: number): string {
  return `${(value * 100).toFixed(1)}%`;
}

function displayLatency(value: number | null): string {
  return value === null ? "not attempted" : `${value} ms`;
}

export function toDecisionInspectorView(
  inspection: DecisionInspection,
): DecisionInspectorView {
  const summary: InspectorRow[] = [
    { label: "Outcome", value: inspection.outcome },
    { label: "Reason", value: inspection.reason },
    { label: "Disposition", value: inspection.disposition },
    { label: "Total latency", value: `${inspection.latency.totalMs} ms` },
    { label: "Model latency", value: displayLatency(inspection.latency.modelMs) },
    { label: "Model status", value: inspection.model.status },
  ];

  if (inspection.escalationReason !== undefined) {
    summary.push({
      label: "Escalation reason",
      value: inspection.escalationReason,
    });
  }

  if (inspection.fallbackReason !== undefined) {
    summary.push({
      label: "Fallback reason",
      value: inspection.fallbackReason,
    });
  }

  const questions = inspection.model.questions.map(function (question) {
    return {
      questionId: question.questionId,
      selectedValue: question.selectedValue,
      confidence:
        question.confidence === null
          ? "not returned"
          : percent(question.confidence),
      probabilities: question.options.map(function (option) {
        return {
          label: option.value,
          value: percent(option.probability),
        };
      }),
    };
  });

  return { summary, questions };
}

A minimal internal inspector page needs only two regions:

1. Summary: outcome, reason, disposition, total latency, model latency, model status.
2. Question evidence: selected value, confidence, and every allowed option with its probability.

When model.status is not completed, render the summary and an explicit message such as “No Jev evidence was returned for this decision.” Do not render empty probability rows or a fake 0% confidence.

---

Correlate the inspector with traces

The inspector answers “what happened to this ticket?” OpenTelemetry answers “what happened across this request and its dependencies?” A trace ID connects the two without requiring you to make the trace UI your primary support-review interface.

Read the active-span pattern before adding manual instrumentation:

Once tracing is initialized in the application, wrap the inspection call in a support.triage span:

import {
  context,
  SpanStatusCode,
  trace,
} from "@opentelemetry/api";

const tracer = trace.getTracer("support-decision-service");

export async function tracedTriage(
  service: ResilientTriageService,
  ticket: SupportTicket,
) {
  return tracer.startActiveSpan(
    "support.triage",
    async function (span) {
      try {
        const traceId = span.spanContext().traceId;

        const result = await triageWithInspection(
          service,
          ticket,
          traceId,
        );

        span.setAttribute(
          "support.decision.outcome",
          result.inspection.outcome,
        );
        span.setAttribute(
          "support.decision.reason",
          result.inspection.reason,
        );
        span.setAttribute(
          "support.model.status",
          result.inspection.model.status,
        );
        span.setAttribute(
          "support.decision.total_latency_ms",
          result.inspection.latency.totalMs,
        );

        if (result.inspection.fallbackReason !== undefined) {
          span.addEvent("support.decision.fallback", {
            "support.fallback.reason": result.inspection.fallbackReason,
          });
        }

        return result;
      } catch (error) {
        if (error instanceof Error) {
          span.recordException(error);
        }

        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: "Unexpected triage service error",
        });

        throw error;
      } finally {
        span.end();
      }
    },
  );
}

Two status rules keep telemetry meaningful:

• A returned manual_triage or defer_triage result is an expected application outcome. The outer support.triage span can complete successfully.
• An unexpected programming error that escapes the resilience policy is an error and should mark the span accordingly.

Do not attach the full probability distribution, ticket content, email address, or account ID as trace attributes. The inspection record already owns bounded decision evidence; trace attributes should remain low-cardinality operational metadata.

---

Verify the inspector’s honest behavior

Your tests should check the record, not a particular UI framework. At minimum, cover these cases:

• A high-confidence route produces automated, completed, a non-null modelMs, and visible question distributions.
• An intent_uncertain result produces review, retains its Jev question evidence, and sets escalationReason.
• A timeout produces manual_triage, jev_timeout, model.status: "timeout", and an empty question list.
• A 429 produces defer_triage, jev_rate_limited, and a non-null model latency because a call was attempted.
• A circuit-open result produces jev_circuit_open, model.status: "circuit_open", attempted: false, and modelMs: null.
• A closed ticket produces no_action, not_needed, no model questions, and no fallback reason.

The most important invariant is simple: missing evidence must remain missing. A production inspector that fills in guessed confidence, probabilities, or latency is worse than no inspector because it creates false confidence during review and incident response.

---

You now have a minimal, typed decision inspector: it shows the outcome and policy reason, preserves Jev’s bounded probabilities and confidence when they exist, records accurate latency, and clearly distinguishes escalation from dependency fallback. The inspection record is separate from both the UI and the provider SDK, which makes it safe to test, evolve, and export to operational tooling.

Next, you will build the uv-based Python evaluation runner. It will use fixed datasets and recorded workflow outcomes to compare versions on quality, automation coverage, latency, and request volume before release.