Deterministic Candidate Re-Ranking with Jev Probabilities, Tie-Breaking, and Cutoffs

Welcome back. In the previous lesson, you used speculative fan-out: Jev answered several bounded questions in one request, while deterministic Python code decided which answers were applicable to the selected route.

Here we apply the same division of responsibility to search. A retriever supplies a bounded candidate set quickly; Jev assesses how useful each candidate is for the particular query; your Python code performs the final ordering, tie-breaking, and cutoff decisions. The goal is not to let a model “decide what to show” opaquely. It is to use Jev’s probabilities as one ranking signal inside an auditable, deterministic policy.

By the end, you will have a Python reranking function that:

• evaluates a fixed set of retrieved candidates in one Jev request,
• ranks candidates by Jev’s relevance probability,
• resolves equal probabilities deterministically,
• applies both a relevance threshold and an output limit,
• records why every candidate was selected or excluded.

---

Retrieval finds broadly; reranking selects precisely

A vector search index is designed for speed. It converts documents into compact vectors and retrieves items whose vectors are near the query vector. That is useful for finding a manageable candidate set, but a vector is a lossy representation: it may miss a detail that makes a document directly answer the user’s question.

A reranker gets to inspect the query and the candidate text together. This supports a narrower judgment:

Is this candidate directly useful for answering this query?

The usual two-stage shape is:

1. Retrieve a moderately broad, bounded set, such as 25 document chunks.
2. Ask Jev to judge each candidate’s relevance to the query.
3. Sort candidates deterministically using the returned probabilities.
4. Keep only candidates above a policy threshold and within the final output limit, such as three chunks.

The image captures an important boundary: Jev should not be asked to rank an entire corpus. First-stage retrieval controls latency and cost; reranking improves precision within the retriever’s already bounded output.

For a support knowledge base, suppose a user asks:

“Can I change the payment method for an active subscription?”

Vector retrieval may return chunks about:

• changing payment methods,
• cancelling subscriptions,
• payment failures,
• invoice history,
• general account settings.

All are somewhat semantically related. But only some are directly useful for the requested task. A relevance judgment can reorder the candidate set accordingly.

---

Define the ranking contract before calling Jev

“Relevant” is too vague unless the product gives it an operational meaning. For a RAG answer, a practical contract is:

A candidate is directly relevant when it contains information that materially helps answer the user’s query. Topical similarity alone is insufficient.

That wording deliberately rejects vague matches. A document about “subscription billing” may be topically related to changing payment methods, but it is not directly useful if it contains no instructions or policy relevant to the change.

We will use a binary Choice question for every candidate:

• directly_relevant
• not_directly_relevant

Although the selected Choice value is useful for logging, we rank using:

$$P(\text{directly relevant})$$

This retains information that would be lost if we used only the winner. For example, both candidates below may receive directly_relevant as their selected answer:

Candidate	Relevance probability	Meaning for ranking
A	$0.93$	Strong evidence that it belongs near the top
B	$0.56$	Barely more likely relevant than not
C	$0.49$	Does not meet a $0.65$ relevance cutoff

Using only the selected label would make A and B appear equivalent. The probability produces an ordering signal, while the application retains ownership of the policy: what probability is sufficient, how many candidates to retain, and what happens when Jev is unavailable.

Jev’s other output fields still matter:

• Selected value is a readable classification for logs and inspection.
• Probability is the ranking value in this lesson.
• Confidence describes how concentrated the answer distribution is. Record it for evaluation, but do not casually invent a confidence gate unless you have tested that gate on labeled retrieval examples.

For an ordered Score, the fractional score is also a probability-weighted ranking signal. TypeSafe’s Score documentation explains why a continuous score can rank items, while probabilities and confidence provide necessary context.

---

Keep candidate identity, retrieval order, and content separate

The reranker needs candidate text and enough context to judge it. Your application also needs stable candidate identity and original retrieval rank, but those are not evidence that should influence Jev’s relevance judgment.

A useful data model keeps those roles distinct:

from __future__ import annotations

from dataclasses import dataclass
from typing import Mapping, Sequence


@dataclass(frozen=True)
class Candidate:
    candidate_id: str
    title: str
    text: str
    retrieval_rank: int


@dataclass(frozen=True)
class RankedCandidate:
    candidate: Candidate
    relevance_probability: float
    confidence: float


@dataclass(frozen=True)
class RerankResult:
    selected: tuple[RankedCandidate, ...]
    excluded_below_threshold: tuple[RankedCandidate, ...]
    excluded_by_output_limit: tuple[RankedCandidate, ...]

candidate_id is a stable document or chunk identifier, not a list index. It makes an output auditable and lets the UI retrieve the final chunk content later.

retrieval_rank records the candidate’s initial order from the retriever. It is not sent to Jev. Its role is deterministic tie-breaking: if two candidates receive exactly equal Jev probabilities, retain the one the first-stage retriever originally ranked higher.

The final stable ID is a third tie-breaker. This may seem overly cautious, but it prevents unpredictable ordering if a retriever returns equal scores or if candidate construction changes upstream.

A reasonable retrieval budget for an initial implementation is:

MAX_RERANK_CANDIDATES = 25
MIN_RELEVANCE_PROBABILITY = 0.65
FINAL_OUTPUT_LIMIT = 3

These are policy constants, not model facts. You will tune them later using a labeled evaluation set. For now, they make the behavior explicit:

• Jev evaluates at most 25 candidates per request.
• A candidate with relevance probability below $0.65$ is not eligible.
• At most three eligible candidates are returned.

---

Ask Jev one atomic question per candidate

We will construct one Choice question for each retrieved candidate and send the complete question set in one system_one call. This is speculative fan-out applied to a ranked list: every candidate receives the same narrow relevance judgment independently.

from typesafe_sdk import Choice


def build_relevance_questions(
    candidates: Sequence[Candidate],
) -> dict[str, Choice]:
    questions: dict[str, Choice] = {}

    for position, candidate in enumerate(candidates):
        questions[f"candidate_{position}"] = Choice(
            instructions=(
                f"Evaluate only `candidates[{position}]` against `query`. "
                "Is this candidate directly useful for answering the user's query? "
                "Choose directly_relevant only if it contains information that "
                "materially helps answer the query. Do not treat broad topical "
                "similarity as sufficient."
            ),
            criteria={
                "directly_relevant": (
                    "Contains instructions, facts, policy, or context that "
                    "materially helps answer the query."
                ),
                "not_directly_relevant": (
                    "Does not materially help answer the query, even if it shares "
                    "general topic words."
                ),
            },
        )

    return questions

Notice two deliberate choices:

1. Every question points at one candidate.
   We are not asking Jev to create a global ordering in prose. Each answer is a typed, inspectable local judgment.

2. The options are operationally contrastive.
   “Directly useful” is contrasted with “topically related but not useful.” Without that contrast, a relevance question often becomes too permissive.

Next, build a narrow shared state. The question needs the query and candidate evidence, but no retrieval score, internal user profile, or unrelated metadata.

def build_rerank_state(
    query: str,
    candidates: Sequence[Candidate],
) -> dict[str, object]:
    return {
        "query": query,
        "candidates": [
            {
                "title": candidate.title,
                "text": candidate.text,
            }
            for candidate in candidates
        ],
    }

In a production knowledge base, candidate text should already be chunked to a size that provides enough local context without sending entire documents. If a chunk requires title, section heading, or document date to be interpretable, include those fields deliberately.

---

Turn Jev probabilities into a deterministic order

The live API call should be thin. The core behavior belongs in a pure function that can be tested using recorded Jev responses later.

def rank_candidates(
    candidates: Sequence[Candidate],
    answers: Mapping[str, object],
    *,
    min_relevance_probability: float,
    output_limit: int,
) -> RerankResult:
    scored: list[RankedCandidate] = []

    for position, candidate in enumerate(candidates):
        answer = answers[f"candidate_{position}"]

        scored.append(
            RankedCandidate(
                candidate=candidate,
                relevance_probability=answer.probabilities["directly_relevant"],
                confidence=answer.confidence,
            )
        )

    eligible = [
        item
        for item in scored
        if item.relevance_probability >= min_relevance_probability
    ]

    below_threshold = [
        item
        for item in scored
        if item.relevance_probability < min_relevance_probability
    ]

    ranked = sorted(
        eligible,
        key=lambda item: (
            -item.relevance_probability,
            item.candidate.retrieval_rank,
            item.candidate.candidate_id,
        ),
    )

    return RerankResult(
        selected=tuple(ranked[:output_limit]),
        excluded_below_threshold=tuple(below_threshold),
        excluded_by_output_limit=tuple(ranked[output_limit:]),
    )

The sort key embodies the complete ranking policy:

Sort key element	Direction	Purpose
relevance_probability	Descending	Prefer the candidate Jev considers more likely directly useful
retrieval_rank	Ascending	Preserve first-stage retriever preference when probabilities are equal
candidate_id	Ascending	Produce a stable result even if the first two keys tie

Do not round probabilities before sorting. A display may show both $0.801$ and $0.799$ as 0.80, but the underlying values are different ranking signals. Round only when presenting values to a person.

Python’s sorted() accepts a key function and supports tuple keys, which makes multi-level deterministic ranking straightforward.

Consider the following scored candidates after applying the $0.65$ threshold:

Candidate ID	Jev relevance probability	Retrieval rank	Result
chunk_18	$0.91$	7	First
chunk_04	$0.78$	1	Second
chunk_11	$0.78$	4	Third
chunk_22	$0.62$	2	Excluded below threshold

The equality between chunk_04 and chunk_11 is resolved by retrieval_rank. If those were equal too, candidate_id would settle the order. With an output limit of three, the selected set is stable and explainable.

---

Complete the bounded reranking service

Now combine validation, the Jev request, and the deterministic ranking policy.

from typesafe_sdk import TypeSafeClient


def rerank_candidates(
    query: str,
    candidates: Sequence[Candidate],
    *,
    min_relevance_probability: float = MIN_RELEVANCE_PROBABILITY,
    output_limit: int = FINAL_OUTPUT_LIMIT,
) -> RerankResult:
    if not query.strip():
        raise ValueError("query must not be empty")

    if not candidates:
        return RerankResult(
            selected=(),
            excluded_below_threshold=(),
            excluded_by_output_limit=(),
        )

    if len(candidates) > MAX_RERANK_CANDIDATES:
        raise ValueError(
            f"reranking accepts at most {MAX_RERANK_CANDIDATES} candidates"
        )

    if output_limit < 1:
        raise ValueError("output_limit must be at least 1")

    candidate_ids = [candidate.candidate_id for candidate in candidates]
    if len(set(candidate_ids)) != len(candidate_ids):
        raise ValueError("candidate_id values must be unique")

    state = build_rerank_state(query, candidates)
    questions = build_relevance_questions(candidates)

    with TypeSafeClient() as client:
        response = client.system_one(
            state=state,
            questions=questions,
        )

    return rank_candidates(
        candidates,
        response.answers,
        min_relevance_probability=min_relevance_probability,
        output_limit=output_limit,
    )

A caller can use the result without relinquishing control over what eventually reaches an LLM:

result = rerank_candidates(
    query="How can I change the payment method for my subscription?",
    candidates=retrieved_candidates,
)

for position, ranked in enumerate(result.selected, start=1):
    print(
        f"{position}. {ranked.candidate.candidate_id} "
        f"p={ranked.relevance_probability:.3f} "
        f"confidence={ranked.confidence:.3f}"
    )

The final answer-generation step, if your product has one, should receive only result.selected. It does not receive every vector-search match just because it was initially retrieved.

---

Treat cutoffs as product policy, not model behavior

There are two independent cutoffs here:

1. Probability threshold decides whether a candidate is good enough to be eligible.
2. Output limit prevents excessive context, even if many candidates appear relevant.

They solve different problems.

A threshold alone might retain twelve candidates for a broad query. An output limit alone might force three weak candidates into the result set. Combining them allows a safe “fewer than requested” outcome:

• zero selected candidates: tell the answer layer there is insufficient grounded context;
• one selected candidate: answer only if that evidence is sufficient;
• three selected candidates: use all three as the bounded context set.

Do not compensate for poor retrieval by lowering the threshold until something passes. If no candidate meets the policy, that is useful information: the query may need broader retrieval, keyword fallback, clarification, or human support.

Likewise, do not make probability thresholds appear more precise than your evidence supports. A value such as $0.65$ is a starting hypothesis, not a universal truth. In Week 4, you will use labeled examples to examine whether a cutoff improves retrieval quality while retaining enough useful automation.

For now, log enough data to make that evaluation possible:

def rerank_audit_record(
    query: str,
    result: RerankResult,
) -> dict[str, object]:
    return {
        "query": query,
        "selected": [
            {
                "candidate_id": item.candidate.candidate_id,
                "retrieval_rank": item.candidate.retrieval_rank,
                "relevance_probability": item.relevance_probability,
                "confidence": item.confidence,
            }
            for item in result.selected
        ],
        "excluded_below_threshold": [
            {
                "candidate_id": item.candidate.candidate_id,
                "relevance_probability": item.relevance_probability,
            }
            for item in result.excluded_below_threshold
        ],
        "excluded_by_output_limit": [
            {
                "candidate_id": item.candidate.candidate_id,
                "relevance_probability": item.relevance_probability,
            }
            for item in result.excluded_by_output_limit
        ],
    }

Avoid logging raw customer queries or document text unless your data-retention and privacy policy permits it. Often, stable IDs, policy version, probabilities, and aggregate outcomes are sufficient for ranking diagnostics.

---

Operational boundaries and failure behavior

Reranking should improve a retrieval result, not become an unbounded dependency that blocks your application.

Keep these boundaries explicit:

• Bound the input before Jev. Retrieve the first 25 candidates deterministically; do not send the corpus.
• Keep the relevance question atomic. Ask whether one candidate helps answer one query.
• Do not send retrieval rank to Jev. It is a deterministic secondary signal, not semantic evidence.
• Rank with raw probabilities. Use the probability of the specific directly_relevant option.
• Apply tie-breaking in application code. Jev supplies evidence; Python owns repeatable ordering.
• Separate below-threshold candidates from output-limit exclusions. They represent different reasons for omission.
• Define a fallback. If Jev times out or is unavailable, a conservative first version can return the retriever’s original top candidates, clearly marked as unrereanked. In higher-risk workflows, the correct fallback may instead be to return no grounded answer.

A particularly tempting mistake is to use Jev’s selected label as the cutoff:

# Avoid this as the primary ranking policy.
if answer.choice == "directly_relevant":
    keep(candidate)

This discards ranking information. It also creates unstable product behavior around the binary decision boundary. A candidate at $0.51$ and one at $0.99$ may both be selected as “directly relevant,” yet they should not be treated as equally strong evidence.

---

Key takeaways

Candidate reranking is a two-stage system:

• A fast retriever produces a bounded candidate set.
• Jev evaluates each candidate’s direct usefulness for the query in one batched request.
• Your Python code sorts by the probability of directly_relevant.
• A tuple sort key supplies deterministic tie-breaking through retrieval rank and stable candidate ID.
• A probability threshold controls minimum quality; an output limit controls context size.
• Record selected and excluded candidates separately so the ranking can be audited and evaluated later.

Next, you will implement bounded structured extraction in TypeScript: mapping natural-language requests to an allowlisted function and schema-validated arguments, while keeping the actual operation firmly under deterministic application control.