Implementing Weighted Composite Scoring with Independent Jev Scores

Good to see you again. In the last lesson, Jev supplied several narrow safety assessments while deterministic TypeScript policy selected the route. Here we use the same division of responsibility for a different product decision: support-ticket priority. Jev will assess a few separately defined dimensions; Python will normalize them, apply business-owned weights, and return a reproducible priority value and lane.

The key outcome is not “ask Jev for a priority.” It is to make the priority calculation inspectable: anyone on the team should be able to see which judgments contributed, how much each mattered, and why a ticket was expedited, handled normally, or sent for review.

---

One business decision, several measurable dimensions

“Which ticket should we work on first?” is a compound question. It often combines:

• How severe is the underlying product problem?
• How frustrated is the customer?
• How actionable is the report for an engineer?

Those are related, but they are not the same judgment. A customer can be extremely frustrated by a workaround-able issue; a calm customer can report a complete outage; a severe issue can be described too vaguely for immediate diagnosis.

A single question such as “How high priority is this ticket?” makes the weighting implicit inside the model assessment. You cannot tell whether it gave severity more importance than customer sentiment, nor easily alter that trade-off when your support policy changes.

Instead, ask one Score per dimension and compose them in Python.

There is an important meaning of “independent” here: the questions should be separable and interpretable, not necessarily statistically uncorrelated. Severity and frustration may correlate in real tickets. The design requirement is that a reviewer can understand each answer on its own and that the application has a defensible reason to weight it.

For our triage policy, use these dimensions:

Question ID	What it measures	Score range	Business weight
severity	Functional harm caused by the reported issue	$0$ to $2$	$0.60$
frustration	Customer’s expressed dissatisfaction	$0$ to $2$	$0.30$
report_quality	How much actionable investigation detail is present	$0$ to $3$	$0.10$

The weights sum to $1$. That is deliberate: the final priority will also stay on a $0$-to-$1$ scale.

Severity receives twice the weight of frustration, while report quality makes a small contribution. This is not a property of Jev, nor a universal formula. It is product policy encoded where it can be reviewed and versioned.

---

Normalize before you apply weights

A Score answer is a probability-weighted position on the ordered criteria. It can be fractional because Jev may distribute probability across neighboring levels.

For example, a severity answer of $1.24$ on a three-level scale lies between:

• Level $1$: broken or degraded, but a workaround exists
• Level $2$: blocking issue with no workaround

But the largest possible severity score is $2$, whereas the largest possible report-quality score is $3$. Adding raw values would silently give report quality more numeric influence simply because its scale has an extra level.

Normalize every score to the unit interval:

$$n_i=\frac{s_i}{L_i-1}$$

where:

• $s_i$ is Jev’s returned score for question $i$
• $L_i$ is the number of criteria levels for that question
• $L_i-1$ is the highest possible level number
• $n_i$ is the normalized score, from $0$ to $1$

Then calculate the weighted composite:

$$P=\sum _{i=1}^k{w}_i{n}_i$$

Here, $w_i$ is the business weight for criterion $i$, and the weights should satisfy:

$$\sum _{i=1}^k{w}_i=1$$

Using the documentation’s example values:

$$n_{\text{severity}}=\frac{1.24}{2}=0.62$$ $$n_{\text{frustration}}=\frac{1.45}{2}=0.725$$ $$n_{\text{report quality}}=\frac{3.0}{3}=1.0$$

With weights $0.6$, $0.3$, and $0.1$:

$$P=0.6(0.62)+0.3(0.725)+0.1(1.0)=0.6895$$

So this ticket’s composite priority is approximately $0.69$.

Do not round component scores before calculating the sum. A fractional score contains useful evidence about boundary cases; early rounding turns distinct tickets into artificial ties.

---

Build the Python scoring policy

The following code belongs in your existing uv-managed Python project with the TypeSafe SDK already configured. Keep the question definitions and weighting policy near each other, but preserve a hard boundary:

1. Jev evaluates the supplied ticket state against atomic questions.
2. Your Python code normalizes and combines answers.
3. Your Python code chooses the operational lane.

from __future__ import annotations

from dataclasses import dataclass
from math import isclose, isfinite
from typing import Literal, Mapping

from typesafe_sdk import Score, TypeSafeClient


TRIAGE_QUESTIONS = {
    "severity": Score(
        instructions="How severe is the product issue reported in `ticket.message`?",
        criteria=[
            "Cosmetic; no impact to functionality.",
            "A broken or degraded feature, but a practical workaround exists.",
            "A blocking issue with no practical workaround for the affected user.",
        ],
    ),
    "frustration": Score(
        instructions="How frustrated does the customer appear in `ticket.message`?",
        criteria=[
            "Calm and matter of fact.",
            "Frustrated but civil and constructive.",
            "Very angry, strongly dissatisfied, or threatening to leave.",
        ],
    ),
    "report_quality": Score(
        instructions=(
            "How much actionable information does `ticket.message` provide "
            "for an engineer to investigate the issue?"
        ),
        criteria=[
            "No useful detail beyond saying that something is broken.",
            "Names the feature or symptom, but gives no environment or reproduction steps.",
            "Provides either reproduction steps or relevant environment details.",
            "Provides both reproduction steps and relevant environment details.",
        ],
    ),
}

# Product policy, not model configuration.
TRIAGE_WEIGHTS = {
    "severity": 0.60,
    "frustration": 0.30,
    "report_quality": 0.10,
}

PriorityLane = Literal["expedited", "normal", "review"]


@dataclass(frozen=True)
class PriorityAssessment:
    composite_score: float
    normalized_components: Mapping[str, float]
    confidences: Mapping[str, float]
    lane: PriorityLane

The definitions deliberately use descriptive criteria rather than labels such as “low,” “medium,” and “high.” Jev needs concrete situations to compare against the state. In particular, do not mix severity and sentiment into the same scale; the resulting score would have no stable interpretation.

Before calling Jev, validate your policy once. A weight typo should fail during development or service startup, not quietly distort every ranking.

def validate_triage_policy() -> None:
    question_ids = set(TRIAGE_QUESTIONS)
    weight_ids = set(TRIAGE_WEIGHTS)

    if question_ids != weight_ids:
        raise ValueError(
            "Every Score question must have exactly one matching business weight."
        )

    for question_id, weight in TRIAGE_WEIGHTS.items():
        if not isfinite(weight) or weight < 0:
            raise ValueError(
                f"Weight for {question_id!r} must be a finite non-negative number."
            )

    total_weight = sum(TRIAGE_WEIGHTS.values())

    if not isclose(total_weight, 1.0, abs_tol=1e-9):
        raise ValueError(
            f"Triage weights must sum to 1.0; received {total_weight}."
        )


validate_triage_policy()

It may be tempting to “helpfully” divide every set of weights by its total. Avoid doing that silently. A failed invariant exposes an incorrect policy change immediately. If product stakeholders intentionally want to revise the weights, make the correction explicit in source control and attach it to a policy version.

---

Turn Jev Score answers into a deterministic result

Now create two small functions:

• normalized_score maps one Score answer onto $0$ to $1$.
• compose_priority applies the declared weights and chooses a lane.

def normalized_score(question_id: str, raw_score: float) -> float:
    """
    Convert a Jev Score position to the 0-to-1 interval.

    A question with three criteria has levels 0, 1, and 2,
    so its top level number is 2.
    """
    criteria = TRIAGE_QUESTIONS[question_id].criteria
    top_level = len(criteria) - 1

    if top_level < 1:
        raise ValueError(
            f"{question_id!r} needs at least two criteria levels."
        )

    if not 0.0 <= raw_score <= top_level:
        raise ValueError(
            f"{question_id!r} returned score {raw_score}, "
            f"outside the expected range 0 to {top_level}."
        )

    return raw_score / top_level


def choose_lane(
    composite_score: float,
    confidences: Mapping[str, float],
) -> PriorityLane:
    """
    Operational policy for this example.

    A low-confidence component does not change the calculated score.
    Instead, it changes whether the score may trigger an automated lane.
    """
    minimum_confidence = min(confidences.values())

    if minimum_confidence < 0.65:
        return "review"

    if composite_score >= 0.75:
        return "expedited"

    return "normal"


def compose_priority(answers) -> PriorityAssessment:
    normalized_components = {
        question_id: normalized_score(
            question_id,
            answers[question_id].score,
        )
        for question_id in TRIAGE_QUESTIONS
    }

    confidences = {
        question_id: answers[question_id].confidence
        for question_id in TRIAGE_QUESTIONS
    }

    composite_score = sum(
        TRIAGE_WEIGHTS[question_id] * normalized_components[question_id]
        for question_id in TRIAGE_QUESTIONS
    )

    return PriorityAssessment(
        composite_score=composite_score,
        normalized_components=normalized_components,
        confidences=confidences,
        lane=choose_lane(composite_score, confidences),
    )

This implementation makes a few useful guarantees:

• composite_score remains between $0$ and $1$, since it is a weighted sum of normalized values with non-negative weights totaling $1$.
• Changing a weight affects the calculation predictably and visibly.
• The original per-dimension results remain available for logging, inspection, and later evaluation.
• Confidence remains separate from the priority score.

That last point is especially important. Do not multiply each component by confidence:

$$0.6\times \text{severity}\times \text{confidence}$$

That arithmetic produces a number, but it changes the meaning of your business policy. A potentially severe issue with low model confidence does not become less severe; it becomes less suitable for an automated action. Route uncertainty to review rather than disguising it as a lower priority.

The 0.65 confidence floor and 0.75 expedited threshold are initial policy values. They are not calibrated truths. Week 4 will provide the evaluation techniques needed to tune them against labeled tickets and actual reviewer outcomes.

---

Send the atomic questions together

All three Scores can be asked in one System One request. Jev evaluates the questions against the same state, while your code preserves the separate outputs.

def assess_ticket_priority(
    state: dict[str, object],
) -> PriorityAssessment:
    """
    `state` should contain only evidence needed for the triage questions.

    Example:
    {
        "ticket": {
            "message": "...",
            "product_area": "exports",
            "affected_users": "several workspace members",
            "known_workaround": "Use CSV export in Chrome",
        }
    }
    """
    with TypeSafeClient() as client:
        response = client.system_one(
            state=state,
            questions=TRIAGE_QUESTIONS,
        )

    return compose_priority(response.answers)

A caller might use it like this:

ticket_state = {
    "ticket": {
        "message": (
            "PDF export has been stuck on a spinner since this morning. "
            "It affects our finance team before month-end reporting. "
            "Safari 17.3 on macOS. CSV export still works, but we need PDFs."
        ),
        "product_area": "exports",
        "affected_users": "finance team",
        "known_workaround": "CSV export works in some browsers",
    }
}

assessment = assess_ticket_priority(ticket_state)

print(f"priority={assessment.composite_score:.3f}")
print(f"lane={assessment.lane}")
print(f"components={assessment.normalized_components}")
print(f"confidences={assessment.confidences}")

The state includes evidence that can help distinguish a cosmetic report from a degraded workflow. It does not need unrelated metadata such as an internal user ID, IP address, payment history, or analytics profile unless one of the defined questions truly requires it.

---

Interpret the result as evidence, not a command

Suppose the returned answers produce:

PriorityAssessment(
    composite_score=0.690,
    normalized_components={
        "severity": 0.62,
        "frustration": 0.725,
        "report_quality": 1.0,
    },
    confidences={
        "severity": 0.63,
        "frustration": 0.33,
        "report_quality": 1.0,
    },
    lane="review",
)

The priority is relatively high, but frustration has confidence $0.33$. That could mean the customer language straddles “frustrated but civil” and “very angry,” or that the wording does not give enough evidence to place it cleanly.

The deterministic policy responds by preserving the $0.690$ value but selecting review. This lets a support lead see the actual breakdown rather than receiving an unexplained “high priority” label.

The broader composition pattern applies to more than ticket triage:

For any composite score, check these four questions before shipping:

1. Does each component measure one thing?
   If a criterion says “urgent, complex, and important,” split it. Those dimensions can point in different directions.

2. Do all components point in the same direction?
   In this policy, a higher normalized value always means “more reason to prioritize.” If a high raw value instead means “less desirable,” invert it deliberately and document why.

   $$n_{\text{reversed}}=1-n$$

3. Were unlike scales normalized first?
   Never let the number of criteria levels accidentally determine business importance.

4. Can you explain every weight?
   “Severity gets $0.60$ because customer harm is the primary service objective” is an explanation. “The model seemed to like it” is not.

---

Key takeaways

Composite scoring turns several Jev judgments into a transparent, application-specific priority:

• Define one Score per decision dimension, rather than asking Jev for a single opaque final priority.
• Normalize each Score by its top level number before combining scales of different lengths.
• Calculate the final value as a weighted sum whose weights are explicit business policy.
• Validate that weights are non-negative, correspond to known questions, and sum to $1$.
• Keep Score confidence separate from the composite value; use low confidence to gate automation or route to review.
• Return and log the component scores, confidence values, policy version, and final lane so the workflow can later be evaluated.

Next, you will implement speculative fan-out: ask several potentially useful Jev questions in parallel, then let deterministic code use only the answers relevant to the branch it selected.