Defining Score Questions with Observable Performance Levels

Good to see you again. In the previous lesson, you defined a Choice question that selects one operational support path. A Score solves a different problem: it places evidence on an ordered spectrum.

For support automation, this is useful when the application needs to know degree: how frustrated a customer sounds, how severe a reported defect is, or how complete an engineering report is. This lesson focuses on the design work that makes a Score trustworthy: defining ordered levels that are concrete, distinct, and grounded in evidence the state actually contains.

---

A Score is a rubric, not a vague rating

Use a Score when the answer has a meaningful low-to-high order.

Question	Appropriate primitive	Why
“Which team should handle this?”	Choice	Teams are alternative categories, not positions on a scale.
“Is the customer asking for a refund?”	Noul	This is one precise proposition.
“How frustrated is the customer?”	Score	Frustration can be described as increasing levels.

A Score question has:

• an instructions field defining one thing to rate;
• a criteria array containing ordered descriptions of the levels;
• between 2 and 10 levels, indexed by their position in the array, starting at 0.

The model returns a numeric position on that scale. It can be fractional, such as $1.4$, when the evidence lies between two levels.

The screenshot illustrates an important design principle. The message says that a Stripe connection has failed for three days and is causing lost sales. That same evidence can support several independent judgments:

• Choice: which team should handle the issue?
• Score: how frustrated does the customer appear?
• Noul: does the message convey urgency?

Do not combine these into one question such as “How urgent, severe, and frustrated is this ticket?” Those are different dimensions. A single number would be impossible to interpret consistently.

---

Anchor levels in observable situations

A weak scale names degrees:

criteria: ["low", "medium", "high"]

Those words tell Jev almost nothing. What does “medium frustration” look like in a support message? Does a polite report of three failed attempts count as medium? What about an angry message with no technical detail?

A useful level tells the model what it can observe in the state. For a customer-frustration scale, observable signals might include:

• neutral versus emotionally charged wording;
• explicit dissatisfaction;
• an account of repeated failed attempts;
• emphatic demands;
• statements about cancelling or leaving.

The difference is substantial:

Weak level	Anchored level
Moderately frustrated	Clearly dissatisfied but civil; expresses inconvenience, repeated failed attempts, or a need for help without hostile language.
High frustration	Very angry; uses strong hostile language or gives an ultimatum to cancel or leave.

The second column does not require Jev to know what “high” means in the abstract. It can compare the supplied customer message against recognizable evidence.

Four rules for an effective Score rubric

1. Measure one dimension.
“Customer frustration” is one dimension. “Customer frustration, business impact, and defect severity” is three. A customer might be calm while reporting a catastrophic outage, or highly frustrated about a cosmetic annoyance. Those combinations make a combined scale incoherent.

2. Make every level self-contained.
Jev evaluates each level against the state. Do not define level 2 as “worse than the previous level.” Instead, fully describe the situation that belongs at level 2.

3. Ensure adjacent levels have a boundary.
The most useful comparison is between neighboring levels. You should be able to explain why a particular message belongs at level 1 rather than level 2.

4. Use only as many levels as you can distinguish.
Three or four strong levels are more useful than seven labels whose boundaries nobody can explain. Add a distinct top level only when it represents a rare case the application genuinely needs to handle differently.

An unknown level is usually not a good part of a degree scale. “Unknown frustration” is not a degree of frustration. If the message lacks enough evidence, that should generally appear as uncertainty in the returned probabilities and confidence, or trigger a deterministic review policy later in the workflow.

---

Build a customer-frustration Score

Continue the support context from the preceding lessons. The application has already built a compact state payload, and it should contain only evidence relevant to the judgment.

const state = {
  customer_message:
    "I've tried to connect Stripe for three days and it keeps failing. Please help ASAP.",
  previous_contact_count_for_same_issue: 2,
};

The second field is optional evidence, but if you include it, define it precisely. It should mean repeated contacts about the same unresolved issue, not the customer’s lifetime support history.

Now define a Score question. The array order is meaningful: its first entry is level 0, the next is level 1, and so on. The object field names inside each criterion are your own structure; their value is in making every level comparable.

const customerFrustrationQuestion = {
  type: "score",
  instructions:
    "How frustrated does the customer express themselves to be? " +
    "Rate expressed frustration only. Do not rate ticket urgency, " +
    "business impact, customer value, or technical severity.",
  criteria: [
    {
      what:
        "Calm and factual. The customer makes a routine request or reports a problem without expressing dissatisfaction or emotional distress.",
      signals: [
        "neutral wording",
        "a straightforward question",
        "no complaint about repeated failure",
      ],
      examples: [
        "Could you explain how to connect my Stripe account?",
        "The connection page shows an error. What information do you need?",
      ],
    },
    {
      what:
        "Mildly dissatisfied but civil. The customer reports inconvenience or an unsuccessful attempt, but remains restrained and asks for help.",
      signals: [
        "polite complaint",
        "one or more failed attempts",
        "civil request for assistance",
      ],
      examples: [
        "I tried connecting Stripe twice, but it still fails. Please help.",
        "This has been inconvenient, although I would appreciate guidance.",
      ],
    },
    {
      what:
        "Clearly frustrated but still civil. The customer expresses exhaustion, strong dissatisfaction, or distress after repeated failure.",
      signals: [
        "explicit frustration",
        "emphatic wording",
        "repeated unsuccessful attempts",
        "language such as unacceptable or fed up",
      ],
      examples: [
        "This is incredibly frustrating. I have tried for days with no result.",
        "I am fed up with this error. I need someone to resolve it today.",
      ],
    },
    {
      what:
        "Very angry. The customer uses strongly hostile wording or makes a direct ultimatum to cancel or leave.",
      signals: [
        "hostile or insulting language",
        "strong blame",
        "explicit cancellation or departure ultimatum",
      ],
      examples: [
        "Your product is useless. Fix this now or I am cancelling.",
        "I am done with this service and will leave unless this is fixed immediately.",
      ],
    },
  ],
} as const;

This definition deliberately excludes urgency and severity from the instructions. “Please help ASAP” might express urgency without proving strong anger. Likewise, “I am losing sales” may indicate serious business impact without necessarily telling you the customer’s emotional state. Keeping those judgments separate preserves a clear meaning for the returned score.

Why these levels are ordered

The scale’s progression is based on increasingly explicit evidence of negative emotional expression:

Level	Core distinction
0	No expressed dissatisfaction
1	Civil dissatisfaction or inconvenience
2	Explicit, strong frustration after repeated difficulty
3	Strong hostility or an ultimatum to leave

There is still room for uncertainty at the boundaries. A message saying “I have tried three times; please help” could reasonably sit between levels 1 and 2. That is not a failure of the Score design. It is exactly the kind of case where a fractional score and probability distribution carry useful information.

---

Read the result as a distribution, not just a number

For $n$ levels numbered from $0$ through $n-1$, Jev’s score is the probability-weighted mean of the level indices:

$$\text{score}=\sum _{i=0}^{n-1}i\cdot p_i$$

where $p_i$ is the probability assigned to level $i$.

Suppose Jev evaluates a customer message using the four-level rubric above and returns:

{
  "type": "score",
  "score": 1.4,
  "probabilities": {
    "0": 0.0,
    "1": 0.6,
    "2": 0.4,
    "3": 0.0
  },
  "confidence": 0.6
}

The score of $1.4$ means the message falls between the “mildly dissatisfied” and “clearly frustrated” anchors, with more probability on level 1.

It does not mean that the customer is “140% frustrated,” and it does not mean that 40% of customers are at level 2. It is a position on this particular rubric.

Also, the scalar score alone is incomplete. Consider a different distribution:

{
  "score": 1.4,
  "probabilities": {
    "0": 0.3,
    "1": 0.0,
    "2": 0.7,
    "3": 0.0
  }
}

This has the same numeric score, but a different interpretation: the model sees a competition between calm and clearly frustrated, rather than a normal boundary between mild and clear frustration. In application telemetry and debugging tools, preserve the probabilities and confidence alongside the score.

Confidence describes how concentrated Jev’s answer is across the levels. It is useful evidence about ambiguity, but it is not a guarantee that the rubric itself is correct. You validate that later against labeled examples from your own support traffic.

---

Test the rubric before connecting it to automation

A score definition is an interface contract. Review it with examples before relying on it for queue prioritization, dashboards, or automatic messages.

Message	Intended level	Boundary being tested
“Where can I find the Stripe connection settings?”	0	A routine informational request should not be read as dissatisfaction.
“The connection failed once. Could you advise?”	1	A failure can be frustrating without strong emotional language.
“I have spent days retrying this and I am fed up.”	2	Repeated failure plus explicit emotional language.
“Fix this broken integration today or I am cancelling.”	3	A direct ultimatum belongs in the top anchor.
“The integration is failing and this is affecting sales.”	Boundary case	Business impact alone must not automatically become emotional frustration.

A practical review checklist:

1. Evidence check: Every signal named in a criterion must be present or inferable from the state.
2. Neighbor check: Compare levels 0 and 1, then 1 and 2, then 2 and 3. State the observable fact that separates each pair.
3. Order check: A case satisfying a higher level should represent more of the same measured dimension, not a different property.
4. Action check: If the top level receives special handling, document that policy outside the rubric. The Score should describe the evidence, while deterministic code owns the action.
5. Vocabulary check: Use language your support team would recognize when labeling real cases.

For a concise visual walkthrough of a bug-severity Score, watch this segment:

---

A Score question turns ambiguous language into a position on a carefully defined spectrum. Its quality comes from the rubric, not from the presence of numeric indices:

• measure one dimension at a time;
• order levels from low to high;
• describe observable situations at every level;
• keep adjacent levels distinguishable;
• retain probabilities and confidence with the numeric score.

Your customer-frustration question now has four evidence-based anchors, from calm factual reporting through a clear cancellation ultimatum. In the next lesson, you will define a Noul question: one precise proposition expressed as a calibrated probability.