Defining SLIs and SLOs for ML-Powered Services

Hello. In the previous lesson, you treated the model, preprocessing assets, feature definitions, and time semantics as a single prediction contract, then used replay to distinguish implementation skew from genuine drift. That diagnosis becomes operationally useful only when the service has explicit commitments: which user-visible failures matter, how they are measured, and when the team must act.

This lesson develops those commitments through service-level indicators (SLIs) and service-level objectives (SLOs) for an ML-powered service. By the end, you should be able to define a compact SLO package that covers conventional service reliability and ML-specific concerns such as feature freshness, degraded prediction paths, and delayed model-quality measurement.

---

From a metric to an operational commitment

Production systems emit many metrics: GPU utilization, model-server error rate, queue depth, p99 latency, feature null rate, model confidence, F1 on delayed labels, and so on. Most are useful for diagnosis. Very few should be elevated to an SLI.

An SLI measures a user-relevant aspect of service behavior. In a particularly practical form, it is a ratio:

$$\mathrm{SLI}(W)=\frac{\text{good eligible events during window }W}{\text{all eligible events during window }W}$$

An SLO sets the acceptable target for that SLI over a stated window:

$$\mathrm{SLI}(W)\ge \text{target}$$

For example:

Latency SLO: During each rolling 28-day window, at least $99\%$ of eligible prediction requests complete within $750$ ms, measured at the public inference gateway.

This statement is concrete enough to implement, audit, and act on. It specifies:

1. The user interaction: a prediction request.
2. The event population: eligible requests.
3. What counts as good: completion within $750$ ms.
4. The objective: $99\%$.
5. The evaluation window: rolling 28 days.
6. The measurement boundary: public gateway.

The final item matters. A model server may report that a request completed in $100$ ms while the user experienced a two-second delay caused by gateway queueing, feature retrieval, or network retries. An SLI should normally be measured as close to the user journey as is practical.

SLI specification versus implementation

A useful distinction from SRE practice is between the specification of an SLI and its implementation.

Level	Example for an ML ticket-triage service
SLI specification	The proportion of submitted tickets that receive a usable triage result quickly enough for an agent workflow.
SLI implementation	The proportion of accepted requests for which the gateway records a schema-valid triage response within $750$ ms.
Measurement source	Gateway telemetry, augmented by application events that identify whether the result was degraded.

The specification expresses the desired user outcome. The implementation makes a deliberate measurement choice, with known blind spots.

For instance, gateway logs may not detect a front-end rendering failure. Client telemetry may better capture the whole experience, but can be more expensive, subject to sampling, and unavailable for API consumers. The right choice depends on the service boundary and the operational maturity of the team.

A few things that are usually not SLIs by themselves:

• GPU utilization;
• model-server CPU saturation;
• number of pods;
• training loss;
• feature-store cache hit rate;
• a model’s raw confidence score.

These are diagnostic or capacity metrics. They can explain why an SLO is at risk, but they do not directly express whether users received the promised service.

---

Begin with the user journey, not the model endpoint

An ML system nearly always contains several service types:

• a request-driven inference API;
• a streaming or batch feature pipeline;
• artifact, feature, and prediction storage;
• monitoring and retraining workflows.

The Azure architecture below makes this visible. Although the depicted services are Azure-specific, the architectural point is cloud-neutral: the prediction endpoint is only one part of a larger data, training, deployment, and monitoring system.

A senior design discussion starts by naming the user and the critical journey. Consider a support-ticket triage system:

A support agent submits a ticket and expects a priority classification with enough freshness and reliability to decide what to do next.

That one sentence already reveals several distinct risks:

• no response arrives;
• a response arrives too slowly to be useful;
• the system silently uses a fallback classifier;
• required customer or incident features are stale;
• the response is technically valid but incorrectly classifies urgent tickets;
• the model regresses for a particular product tier or language.

Not every risk needs a user-facing SLO. But each needs either an SLO, a release gate, a diagnostic metric, or an explicit acceptance of risk.

Define the event population carefully

The denominator in an SLI is a product decision, not a logging detail. A denominator that excludes inconvenient requests can make a poor service appear healthy.

For an inference availability SLI, a reasonable denominator might be:

All authenticated prediction requests accepted by the public gateway after syntactic validation.

This definition needs explicit treatment of edge cases:

Event	Typical treatment	Why
Internal model timeout	Count as bad	The service failed after accepting work.
Feature-store timeout with fallback response	Potentially good for availability, bad for non-degraded quality	The API responded, but the user may have received an inferior result.
Client cancels before the gateway receives the request	Exclude	The service never had an opportunity to serve it.
First-party client sends malformed payload due to a client deployment	Usually include in an end-to-end user SLI	Users still experienced failure, even if the immediate bug is upstream.
Intentional rate-limit rejection	Usually count separately	It may reflect an admission-control policy rather than serving failure, but it is still relevant to the product experience.
Synthetic probes	Measure separately	Their traffic pattern and authorization path may not represent real users.

The goal is not a universal denominator. The goal is a written, stable policy that does not change silently when incidents occur.

---

The ML-specific SLI portfolio

For a conventional API, availability and latency may be sufficient at first. An ML service usually needs a slightly richer portfolio because its user value depends on data and model behavior, not merely HTTP success.

A useful starting point is five or fewer SLI types. The following set is appropriate for the support-ticket triage example.

User-relevant concern	Example SLI	What “good” means	Typical source
Availability	Successful prediction responses / eligible prediction requests	Response is delivered and conforms to the response contract	Gateway and application telemetry
Latency	Requests completed within $750$ ms / eligible requests	End-to-end completion meets the agent-workflow threshold	Gateway or client-side telemetry
Non-degraded coverage	Primary-path predictions / eligible prediction requests	Intended model, required features, and normal retrieval path were used	Inference decision log
Feature freshness	Predictions using critical features within freshness policy / predictions requiring those features	Features meet their declared age limit	Feature metadata and inference trace
Model effectiveness	Correct high-priority classifications / label-complete high-priority tickets	The model achieves the intended decision quality once ground truth is known	Joined prediction, label, and model-version data

The first two are common request-driven SLIs. The next two expose ML-dependent degradation that availability alone can conceal. The fifth recognizes that a syntactically correct, low-latency response still has little value if the model is consistently wrong.

Availability is not semantic correctness

Suppose a feature service becomes unavailable. The triage API responds with a generic model that ignores customer history. A status-code-based availability SLI may classify the response as successful. That may be appropriate: the system did not go down.

But if product behavior promises enriched triage, you should separately measure the fraction of responses delivered through the non-degraded path. Otherwise, a broad fallback can preserve a $99.99\%$ availability dashboard while most users experience a substantial quality regression.

A precise non-degraded SLI could be:

$${\mathrm{SLI}}_{\text{primary}}(W)=\frac{\text{eligible requests served by approved model and fresh critical features}}{\text{all eligible requests}}$$

The definition must say what counts as “approved” and “fresh.” For example:

• an immutable model bundle is in the production-approved registry state;
• the rendered feature schema passes validation;
• critical features are no more than 15 minutes old;
• no fallback rule, stale snapshot, or reduced feature set was invoked.

This connects directly to the training-serving skew lesson. If a service starts accepting an unknown schema version or silently substitutes stale features, its availability can remain high while the actual prediction contract is violated.

Freshness depends on the decision

Feature freshness is meaningful only relative to a business decision. A daily account segment may be perfectly adequate for one use case and unacceptable for another.

For a ticket triage system, consider two feature classes:

• Static or slowly changing: account tier, product family, support entitlement.
• Rapidly changing: number of open incidents, recent error volume, service-health state.

A sensible freshness SLI might apply only to the second class:

$${\mathrm{SLI}}_{\text{fresh}}(W)=\frac{\text{predictions with all required dynamic features no older than 15 minutes}}{\text{predictions that require dynamic features}}$$

The denominator is not necessarily every prediction. Some ticket categories may not require a live incident aggregate at all. Including them would dilute the signal and make the SLI less tied to the real dependency.

Model quality has delayed, incomplete feedback

Operational SLIs are usually observed immediately. Model quality is often not.

For classification, labels may arrive only after an agent resolves a ticket. For forecasting, error is known after the forecast horizon. For ranking and recommendation, observed clicks are influenced by exposure policy. For an LLM assistant, correctness may require human review or a carefully designed evaluation set.

When labels are trustworthy and sufficiently timely, model effectiveness can be measured directly. If missing a high-priority ticket is costly, recall may be the governing quality metric:

$${\mathrm{Recall}}_{\text{high}}=\frac{\text{true high-priority tickets predicted as high priority}}{\text{all true high-priority tickets}}$$

A quality objective might state:

For label-complete tickets evaluated in a rolling 28-day window, recall for high-priority tickets must be at least $95\%$, with separate reporting for supported product tiers and languages that meet a minimum sample-size requirement.

The minimum sample-size condition matters. A slice with four labeled tickets should not trigger the same operational response as a major slice with 20,000, although it may still justify investigation.

Where labels are delayed, use leading indicators—feature validation failures, skew scores, fallback rate, prediction-distribution shifts, agent overrides—as early warnings, not as proof that accuracy has declined. A proxy is valuable when it predicts user harm; it is not a substitute for ground truth.

---

An SLO package for a production prediction service

Here is an illustrative SLO package for the triage service. The numbers are not universal recommendations; their role is to show the level of precision expected in an actual SLO document.

SLO	Formal objective	Measurement and exclusions	Primary response when at risk
Availability	At least $99.9\%$ of eligible requests receive a schema-valid response in a rolling 28-day window.	Gateway terminal events; gateway 5XX errors, internal timeouts, and invalid server responses are bad.	Diagnose by dependency and deployment version; stabilize the serving path.
Latency	At least $99\%$ of eligible requests complete in at most $750$ ms over 28 days.	End-to-end gateway duration, including queueing and feature lookup.	Inspect queueing, feature retrieval, model execution, and autoscaling behavior.
Non-degraded coverage	At least $99.5\%$ of requests use the approved primary bundle and full required feature set.	Inference trace records model bundle, fallback reason, feature contract, and feature-age state.	Pause risky releases; repair the dependency or explicitly revise product behavior.
Dynamic-feature freshness	At least $99.5\%$ of requests requiring dynamic features use values no more than 15 minutes old.	Feature timestamp is compared with prediction time; static features are excluded.	Investigate upstream ingestion lag, watermark delay, or online-store replication.
High-priority recall	Recall is at least $95\%$ on label-complete high-priority tickets over 28 days.	Predictions are joined to final labels, model version, product tier, and language.	Investigate skew, drift, labeling changes, and affected slices; consider rollback or retraining.

Several details make this more than a collection of dashboard targets.

Measure at the right boundary

For latency, the relevant duration begins when the gateway accepts a request and ends when the gateway emits its final response. It therefore includes:

• waiting in an application or model queue;
• online feature retrieval;
• input validation;
• model inference;
• postprocessing and thresholding;
• serialization of the response.

A model-only latency metric should still be recorded, but it is a diagnostic decomposition, not the user-facing SLI.

Similarly, a $200$ response code should not automatically count as good availability. If the service returns malformed JSON, omits a required priority field, or emits a response that violates the documented schema, the user has not received a successful service outcome.

Keep quality and release gates distinct when needed

Not every quality threshold should consume an error budget in the same way as availability.

An offline benchmark threshold such as “macro F1 must exceed $0.84$ before promotion” is usually a release gate. It controls whether a candidate can be deployed. It does not tell you whether the deployed service is currently healthy.

A delayed-label recall target can be an SLO if it represents a user-facing commitment and has a meaningful response policy. But it should not page an on-call engineer at 03:00 when labels take two weeks to settle. Its operational response might instead be:

• halt automatic model promotions;
• extend a canary;
• route a high-risk ticket category to human review;
• roll back a specific model bundle;
• prioritize investigation of a detected distribution shift.

This distinction avoids both extremes: pretending model effectiveness is irrelevant to reliability, and treating every offline evaluation metric as an immediate production incident.

Preserve diagnostic dimensions without fragmenting the SLO

The customer-facing availability SLO should normally aggregate across all model versions and deployment zones. A user does not care which canary received their request.

However, every SLI event should carry safe, low-cardinality diagnostic attributes such as:

• model bundle version;
• feature-contract version;
• inference route or fallback reason;
• region;
• request class;
• product tier;
• language;
• feature freshness bucket.

This lets you ask, after an SLO burn: Did failures begin only after model bundle 2026.03.4? Are they isolated to one region? Did the fallback route activate?

Avoid putting user IDs, ticket text, or unrestricted high-cardinality data into metrics labels. Link metrics to privacy-controlled traces or logs through sampled request identifiers where necessary.

---

Error budgets turn targets into decisions

An SLO below $100\%$ implies an error budget: the amount of bad behavior the service is allowed during its evaluation window.

If the availability objective is $99.9\%$ and the service receives $2,500,000$ eligible requests over 28 days, the availability error budget is:

$$B=(1-0.999)\times 2,500,000=2,500$$

So, over that period, at most 2,500 requests may fail the availability definition before the SLO is missed.

This does not mean that 2,500 failures are acceptable in a concentrated five-minute outage. The rate at which budget is consumed matters. A fast burn calls for immediate response, while a slow sustained burn may call for planned reliability work.

Also, SLO budgets are separate. A single request can be a successful but slow response, consuming latency budget but not availability budget. A fallback response may preserve availability while consuming the non-degraded-coverage budget.

A credible SLO package needs a predefined policy, agreed with product and engineering. For the example service:

Budget state	Illustrative policy
Healthy budget, normal burn	Continue normal deployments and model promotions.
Rapid availability or latency burn	Freeze nonessential serving changes, investigate active incident, and scale or shed load if necessary.
Non-degraded coverage budget nearing exhaustion	Pause model or feature-contract promotions; prioritize restoring primary dependencies.
Quality objective missed after labels mature	Review affected slices, inspect replay parity and drift signals, limit automation for high-risk cases, and decide on rollback or retraining.
Repeated SLO misses	Reassess architecture, capacity, fallback semantics, and whether the objective reflects actual user needs.

The SLO is effective only if these actions are real. A target that nobody is empowered to enforce is a dashboard annotation, not an operational agreement.

---

Selecting targets without guessing

Setting $99.9\%$ or $99.99\%$ because it sounds standard is not senior engineering. Targets should emerge from a combination of user cost, service design, historical behavior, and the cost of improving reliability.

Ask:

1. What failure duration or frequency materially harms users?
   An internal asynchronous forecasting pipeline can tolerate different delays from an interactive fraud decision endpoint.

2. What behavior does the product promise?
   If the UI advertises “real-time incident-aware triage,” then a 24-hour feature freshness objective contradicts the product.

3. What is technically defensible?
   A target that requires heroic manual intervention is not sustainable. That may indicate missing redundancy, capacity, or an overly ambitious promise.

4. What does historical data reveal?
   Use it to understand feasible thresholds and failure modes, but do not simply set the target equal to today’s average performance. Current performance may reflect slack that users have already come to depend on.

5. Can the team act on an SLO miss?
   If there is no plausible response, the metric may be a research KPI or diagnostic signal rather than an SLO.

6. Do the SLO misses correlate with user dissatisfaction?
   Compare error-budget consumption with support tickets, agent overrides, escalation rates, and user research. If important incidents do not appear in SLO data, the SLI lacks coverage. If SLO alerts constantly occur without user impact, the objective or its implementation may be too strict.

For an ML system, this final feedback loop is particularly important. A model-quality metric may be excellent globally while failing a high-value slice; conversely, a small decline in a broad aggregate metric may not affect the decisions users actually care about.

---

A practical SLO-document template

For portfolio work or a system-design interview, define each SLO with the following fields:

• Service and user journey: What interaction is protected, and who is the user?
• SLI specification: What outcome constitutes good service?
• Eligible-event definition: What is in the denominator, including exclusions and rationale?
• Good-event definition: What exactly qualifies for the numerator?
• Measurement source and boundary: Gateway, client, application event, pipeline watermark, or joined label data.
• Objective and window: Target percentage, threshold, and rolling or calendar-aligned interval.
• Segmentation: Which model versions, regions, request classes, or user slices must be reviewed separately?
• Known blind spots: What failures the SLI cannot observe.
• Owner and review date: Who can change the target and who responds to misses?
• Error-budget policy: What changes when budget burn is fast, sustained, or exhausted?

This structure gives you a defensible answer to a common senior-level question: “How would you know that an ML service is healthy?” The answer is not “I would monitor latency, accuracy, and drift.” It is: “I would define user-centered good-event ratios, set objectives over explicit windows, instrument the boundaries that observe those outcomes, and attach response policies to the resulting error budgets.”

---

Key takeaways

An SLI is a measurable, user-relevant indicator, often expressed as good eligible events divided by total eligible events. An SLO is the target value of that SLI over a defined time window. The associated error budget makes the reliability-versus-delivery trade-off explicit.

For ML-powered services:

• measure availability and latency at a meaningful end-to-end boundary;
• separately measure degraded paths, feature freshness, or coverage when they materially change the prediction experience;
• treat model effectiveness as a production concern, while respecting delayed labels and slice-specific behavior;
• distinguish user-facing SLOs from internal diagnostic metrics and pre-deployment release gates;
• define denominators, exclusions, and measurement sources precisely;
• attach a real owner and response policy to every SLO.

Next, you will use these operational commitments to construct a failure-mode table across the data, model, infrastructure, and application layers.