Estimating ML Workload Performance and Cost Requirements

Hello. In the previous lesson, you built a failure-mode table for a production prediction flow and connected each failure to user-visible effects, telemetry, containment, and ownership. That analysis identified what can go wrong. This lesson makes the operational commitments quantitative: how fast the service must respond, how much traffic it must absorb, how much unavailability it can tolerate, and what those choices cost.

We will continue with the support-ticket triage service: agents submit tickets, the application obtains online features, a deployed model predicts priority and routing, and the result is returned with a trace. The techniques generalize to recommendation, fraud, vision, and LLM-backed systems; only the workload unit changes from requests to images, records, or tokens.

By the end, you should be able to turn a product description into a defensible initial capacity and cost envelope, state the assumptions behind it, and identify what must be benchmarked before committing to an architecture.

---

Start with a workload contract, not an instance type

A common design mistake is to begin with a serving technology or accelerator choice: “Should this be a GPU endpoint?” That reverses the decision. First define the workload’s demand and guarantees. Hardware and topology follow from those constraints.

The AWS architecture below is a useful reference shape. The application reaches a managed inference service through private connectivity; model artifacts and input data sit in object storage; serving capacity is spread across availability zones and adjusted by autoscaling. Estimation must cover the whole request path, not just the model container.

For any ML workload, write a compact workload contract with four groups of inputs.

Dimension	Questions to answer	Triage-service example
Demand	Who invokes it, how often, when do peaks occur, and how bursty are they?	Support agents submit tickets during business hours; incidents create sharp bursts.
Work unit	What drives computation and data movement?	Ticket text length, feature lookups, model size, and output schema.
User experience	Which latency percentile matters, and is a degraded response acceptable?	An agent needs a usable priority in under 500 ms at p95; high-risk tickets must not silently use a weak fallback.
Reliability and economics	What is the semantic availability target, what failures must be survived, and what spend is acceptable?	99.9% valid-response availability; survive ordinary replica loss; target a defined cost per triaged ticket.

The word semantic matters. A response is successful only if it is schema-valid, comes from an approved route, and meets the prediction contract established in the earlier lessons. HTTP 200 alone is not enough.

Before moving into formulas, watch this short refresher on back-of-the-envelope estimation. Its central discipline is useful in senior design interviews and in real planning: declare assumptions, estimate at the right order of magnitude, then validate the few assumptions that can materially change the decision.

A useful principle: an estimate is an explicit, falsifiable model of reality, not a confident guess. “Peak traffic is 35 requests per second because we assume 20,000 active agents, 12 submissions per agent per day, and an eightfold peak factor” is reviewable. “We need several GPUs” is not.

---

Estimate throughput from demand, then convert it into dependency load

For an online service, begin with average request rate:

$$R_{\text{avg}}=\frac{U\times d}{86,400}$$

where:

• $U$ is daily active users or active entities;
• $d$ is mean prediction-triggering actions per active entity per day;
• $86,400$ is the number of seconds in a day.

Then estimate a design peak:

$$R_{\text{peak}}=k_{\text{peak}}\times R_{\text{avg}}$$

The multiplier $k_{\text{peak}}$ should represent more than a predictable daily cycle. For ML systems, it may also reflect synchronized client retries, scheduled jobs, incident-driven use, marketing events, or a batch replay accidentally sharing the online path.

Worked demand estimate

Assume the ticket-triage product has:

• $20,000$ daily active support agents;
• $12$ ticket submissions per active agent per day;
• an eightfold peak multiplier;
• a further burst margin that raises the operational design point from the estimated 22 QPS peak to 35 QPS.

The daily request count is:

$$N_{\text{day}}=20,000\times 12=240,000$$

The average rate is:

$$R_{\text{avg}}=\frac{240,000}{86,400}\approx 2.8\text{ requests per second}$$

The estimated peak is:

$$R_{\text{peak}}=8\times 2.8\approx 22\text{ requests per second}$$

The service should be designed and load-tested at 35 requests per second, not 2.8 QPS and not merely 22 QPS. The extra margin captures imperfect assumptions and short bursts, but it is not a substitute for a real traffic distribution once production telemetry exists.

One client request usually produces several internal operations

External QPS is only the beginning. Translate it into load on every shared dependency.

Suppose each triage request performs:

• one authentication or authorization check;
• three online feature reads;
• one model inference;
• one trace event;
• one asynchronous prediction log write.

At 35 external QPS, the approximate dependency demand is:

Component	Calls or events per user request	Peak demand
API gateway / application	1	35 QPS
Feature service	3	105 reads per second
Model serving endpoint	1	35 inferences per second
Trace pipeline	1	35 traces per second
Prediction log pipeline	1	35 events per second

This decomposition often changes the architecture. A model endpoint might have plenty of capacity while online features, logging, a vector store, or a shared database becomes the true bottleneck.

For generative workloads, requests alone are insufficient. Track at least:

$$T_{\text{in}}=R\times \text{input tokens per request}$$ $$T_{\text{out}}=R\times \text{output tokens per request}$$

Input and output tokens stress the system differently: long prompts burden prefill, while generated tokens drive the decode phase. Later, in the high-performance LLM inference module, you will model that distinction directly. For now, the general requirement is clear: define the unit that actually consumes capacity.

---

Turn a latency objective into a critical-path budget

A latency SLO must be stated at a percentile and at a boundary.

For the triage service, an appropriate initial objective might be:

At least 95% of valid triage requests complete within 500 ms, measured from receipt at the public gateway to completion of the response at that gateway.

This says more than “the model should be fast.” It tells you:

• the population being measured: valid triage requests;
• the percentile: p95 rather than a misleading mean;
• the measurement boundary: end-to-end, not container-only;
• the budget: 500 ms.

Then allocate a provisional budget along the critical path.

Critical-path stage	Initial p95 budget	What to measure
Gateway, authentication, request validation	45 ms	Gateway span and application trace
Online feature retrieval	85 ms	Feature dependency latency and feature-age checks
Model inference	190 ms	Model-server queue time and execution time
Policy, thresholding, and response construction	35 ms	Application span and schema-validation failures
Network and serialization allowance	45 ms	Client-to-gateway and response timings
Queueing and unallocated reserve	100 ms	Queue wait, concurrency, and total end-to-end latency
Total	500 ms	Gateway-observed end-to-end latency

This budget is not a claim that p95 values from separate services add up exactly; latency distributions are correlated, and their tails may coincide during overload. It is a conservative design tool for asking the right questions:

• Is feature retrieval allowed to consume half the request budget?
• Can the model still meet its allocation under peak concurrency?
• Is queueing observable separately from inference execution?
• Which component receives the remaining budget when the system changes?

For a streaming LLM interface, split the objective further:

• time to first token for perceived responsiveness;
• inter-token latency for reading fluency;
• completion latency for end-to-end task duration.

For a non-streaming classifier, a single end-to-end latency objective is often sufficient, provided you separately monitor queueing and model execution.

Use Little’s Law carefully

Little’s Law connects arrival rate, latency, and requests in flight:

$$L=\lambda W$$

where:

• $L$ is average concurrent work in the system;
• $\lambda$ is arrival rate;
• $W$ is average time in the system.

If a service receives 35 QPS and average end-to-end latency is 300 ms, the rough average concurrency is:

$$L=35\times 0.3=10.5$$

This is useful for estimating connection pools, queue depth, worker concurrency, and—in LLM workloads—KV-cache pressure. But do not size solely from average latency. The capacity limit that matters is the maximum offered load at which the system still meets its tail-latency SLO for a realistic mixture of requests.

The following reading connects those measurements to capacity planning and reliability practice.

---

Size capacity from benchmarked sustainable throughput

The key number you need from a benchmark is not a vendor’s maximum throughput claim. It is:

Sustainable capacity per replica under the production request mix, while meeting the specified tail-latency and error-rate objectives.

Assume a representative load test shows that one model-serving replica sustains 12 inferences per second while meeting the triage service’s p95 latency budget. At higher rates, queue time pushes p95 latency above 500 ms.

At the 35-QPS design peak, the minimum normal operating capacity is:

$$n_{\text{normal}}=\lceil \frac{35}{12}\rceil =3\text{ replicas}$$

Three replicas technically provide 36 QPS. That is not a production design; it has almost no headroom for request variance, a slow replica, uneven load balancing, deployments, or telemetry overhead.

Suppose the team wants 20% operating headroom at peak:

$$n_{\text{headroom}}=\lceil \frac{1.2\times 35}{12}\rceil =4\text{ replicas}$$

Now connect capacity to availability. If the service must retain its p95 latency objective after losing one Availability Zone, the remaining zone must independently support the peak load with its intended headroom:

$$n_{\text{per zone}}=\lceil \frac{1.2\times 35}{12}\rceil =4\text{ replicas per zone}$$

For two zones, that implies eight warm replicas:

$$n_{\text{total}}=2\times 4=8$$

This looks expensive compared with “three replicas are enough,” because it is answering a different question:

• Three replicas: can the system meet normal peak demand if everything is healthy?
• Four replicas: can it meet normal peak demand with operating headroom?
• Eight replicas across two zones: can it meet the same peak objective after losing a zone, without assuming instant successful autoscaling?

The correct answer depends on what the availability objective truly requires. It may be acceptable to degrade functionality during an availability-zone failure, such as disabling optional explanations or routing lower-risk tickets to a delayed queue. It is not acceptable to claim high availability while silently assuming that a failed zone, an overloaded scaler, or cold GPUs will recover immediately.

Benchmark the workload distribution, not a single ideal request

A capacity test should include distributions of:

• ticket text lengths;
• feature payload sizes;
• model routes or versions;
• cache-hit and cache-miss behavior;
• request bursts and ramp rates;
• concurrent deployment or autoscaling events;
• normal and degraded dependency paths.

For an LLM-backed service, additionally vary:

• input-token distribution;
• output-token distribution;
• number of concurrent sequences;
• context length;
• streaming versus non-streaming clients.

The useful output is a throughput-latency curve, not one QPS number. Identify the knee: the highest offered load before queueing causes a sharp increase in tail latency or errors. Plan normal operation below that point.

---

Convert availability into failure tolerance and architecture requirements

Availability should be expressed as a user-visible SLO, measured over a defined period. For example:

At least 99.9% of triage requests return a valid, policy-compliant result within the latency objective each calendar month.

The allowed unavailability is:

$$D=(1-A)\times T$$

For a 30-day month:

Availability objective	Allowed unavailability per month
99.0%	7 hours 12 minutes
99.9%	43 minutes 12 seconds
99.95%	21 minutes 36 seconds
99.99%	4 minutes 19 seconds

A 99.9% objective is not a universal “good” target. It may be too weak for a fraud decision that blocks transactions and unnecessarily expensive for an internal analyst tool with a usable manual process. The objective must reflect the cost of downtime, degraded quality, and unsafe automation.

Availability is an end-to-end property

For mandatory, serial dependencies, a rough upper-bound model is:

$$A_{\text{path}}\approx \prod _{i=1}^m{A}_i$$

If an application service, feature store, and model-serving system each have 99.9% availability, a simplistic independence assumption gives:

$$A_{\text{path}}\approx 0.999^3\approx 0.997$$

That is roughly 99.7%, already below a 99.9% end-to-end target.

This calculation is deliberately simplified. Real failures are often correlated: a regional network failure can simultaneously affect the application, feature store, and model service. But the model is still valuable because it exposes an important design fact: a user-facing availability promise cannot exceed the reliability of its required dependencies without mitigation.

Mitigation changes the request path. For the triage system:

Dependency condition	Unsafe response	Explicit availability design
Feature store unavailable	Serve a prediction with missing critical incident features and no disclosure	For low-risk tickets, use an approved stale-feature route marked as degraded; route high-risk tickets to human review
One serving replica fails	Let queues grow until requests time out	Maintain load-balanced replicas with enough capacity after ordinary replica loss
Candidate model release fails semantic checks	Continue serving because HTTP health checks pass	Roll back the full model bundle and preserve the last approved route
One Availability Zone fails	Continue normal routing until the remaining zone saturates	Reserve or rapidly obtain verified capacity in the surviving zone; define an intentional degradation policy if full peak capacity is not economically justified

Notice that a human-review fallback can preserve decision safety while reducing automation coverage. That may count as successful semantic availability for one product and as a degraded outcome for another. Define this before incidents, not during them.

Availability estimates should therefore state the failure scope being covered:

1. Single process or pod loss.
2. Single node or accelerator loss.
3. Availability-zone loss.
4. Regional loss.
5. Loss of a critical dependency, such as a feature store or model provider.

Each wider scope is more expensive to tolerate. A senior design makes the trade-off explicit rather than presenting multi-zone deployment as a complete reliability argument.

---

Estimate cost as a range tied to a service level

Cost has two distinct questions:

1. What does the service cost under expected traffic?
2. What does it cost to uphold the stated latency and availability commitments?

The second is usually more important. A low average-cost design that cannot meet the p95 objective at peak or cannot survive an expected failure is not actually cheaper; it simply excludes the reliability cost from the calculation.

A basic monthly cost model is:

$$C_{\text{month}}=C_{\text{serving}}+C_{\text{data}}+C_{\text{network}}+C_{\text{observability}}+C_{\text{platform}}+C_{\text{engineering}}$$

For request-level economics:

$$c_{\text{request}}=\frac{C_{\text{month}}}{N_{\text{requests per month}}}$$

The support-ticket workload receives:

$$N_{\text{month}}=240,000\times 30=7.2\text{ million requests}$$

Assume, strictly for an initial planning exercise, that an appropriately sized serving replica costs $1.20 per hour. This is an input to validate against the actual cloud region, commitment type, and instance family; it is not a current pricing claim.

Scenario A: lower-cost autoscaled design

Suppose the fleet averages 3.5 replicas across the month:

$$C_{\text{serving}}=3.5\times 730\times 1.20=\$3,066$$

Assume monthly feature, logging, network, and shared-platform charges total $700:

$$C_{\text{month}}=\$3,066+\$700=\$3,766$$

Then:

$$c_{\text{request}}=\frac{\$3,766}{7.2\times 10^6}\approx \$0.00052$$

This is about $0.52 per thousand predictions.

This design may be suitable if it keeps a small multi-zone baseline, scales predictably before traffic peaks, and has an approved degraded path during a zone-level failure. It does not guarantee full peak capacity after losing a zone.

Scenario B: warm capacity for zone-loss peak protection

The earlier availability calculation called for eight warm replicas to retain 20% headroom at a 35-QPS peak after the loss of one of two zones:

$$C_{\text{serving}}=8\times 730\times 1.20=\$7,008$$

Using the same $700 monthly estimate for supporting services:

$$C_{\text{month}}=\$7,008+\$700=\$7,708$$ $$c_{\text{request}}=\frac{\$7,708}{7.2\times 10^6}\approx \$0.00107$$

This is about $1.07 per thousand predictions.

The difference between $0.52 and $1.07 per thousand predictions is not an accounting detail. It is the price of a stronger availability statement.

Include the costs that get missed

For a production ML platform, do not stop at accelerator or endpoint runtime. At minimum, track:

• Serving compute: warm replicas, autoscaled replicas, GPU or CPU nodes, and reserved capacity.
• Data path: online feature reads, cache memory, object storage, vector retrieval, and egress.
• Observability: metrics, traces, prediction logs, retained payloads, and alerting.
• Model lifecycle: training, evaluation, shadow or canary traffic, registry storage, and rollback artifacts.
• Engineering effort: benchmarking, serving optimization, incident response, compliance work, and platform maintenance.
• Opportunity cost of idle capacity: replicas held for p95 latency or fault tolerance but underused at off-peak times.

Stable, continuously used inference capacity may justify committed-use pricing. Fault-tolerant batch evaluation, embedding backfills, and offline experimentation can use interruptible capacity when checkpointing and restart behavior are designed appropriately. Do not place user-facing, latency-sensitive inference on preemptible capacity merely because the hourly rate is lower.

---

Produce an estimation sheet that survives design review

A senior-level estimate should fit on one page before it becomes a spreadsheet. The value is not the formatting; it is the chain from assumptions to consequences.

For the triage example, the initial sheet would look like this:

Area	Initial estimate	Assumption or validation needed
Monthly volume	7.2 million predictions	20,000 daily active agents, 12 tickets per agent per day
Average traffic	2.8 QPS	Usage spread over a full day
Design peak	35 QPS	Eightfold peak plus burst margin
Feature-store traffic	105 reads per second	Three online reads per prediction
Latency SLO	End-to-end p95 below 500 ms	Measured at gateway; valid, approved responses only
Per-replica sustainable throughput	12 QPS	Must be load-tested with representative ticket and feature distributions
Normal peak capacity	4 replicas	20% operational headroom
Zone-loss peak capacity	8 warm replicas across two zones	Full peak SLO retained after loss of one zone
Availability SLO	99.9% semantic availability monthly	Degraded-route and human-review policy must be explicit
Lower-cost monthly envelope	About $3,800	Autoscaling assumed; no full zone-loss capacity guarantee
Higher-availability monthly envelope	About $7,700	Warm capacity for zone-loss peak protection

The highest-risk assumptions deserve validation first. In this case, they are likely:

1. The true burst factor. Production traffic might be much more concentrated during incidents than the initial eightfold estimate.
2. The sustainable 12-QPS replica capacity. It can change sharply with model version, feature latency, request size, container configuration, and co-located workloads.
3. The degraded-path policy. A technical fallback may be inexpensive but unacceptable for high-priority tickets.
4. Accelerator availability and scale-up time. An autoscaling design is only credible if capacity can be acquired before the queue violates the SLO.
5. The real unit-cost denominator. If only a fraction of requests require the expensive model route, report unit economics separately by route rather than masking them in a fleet average.

A practical implementation habit is to keep this sheet versioned beside the architecture decision record. Update it after load tests, model releases, pricing changes, and incidents. It becomes the bridge between product planning, platform capacity, SRE objectives, and finance.

---

Key takeaways

Estimating an ML workload means converting product behavior into four linked commitments:

• Throughput: derive average demand, peak demand, burst margin, and downstream dependency load.
• Latency: specify an end-to-end percentile objective, allocate a critical-path budget, and size against benchmarked sustainable capacity rather than average response time.
• Availability: translate a semantic availability SLO into allowed downtime, required failure tolerance, and explicit fallback behavior.
• Cost: calculate both expected operating cost and the additional cost of the latency and availability guarantees being made.

For the triage example, a design peak of 35 QPS is easy to state, but the architecture differs substantially depending on whether it needs only normal-operation headroom or must preserve the same objective after an Availability Zone failure. That is exactly the kind of trade-off a senior ML engineer should surface early.

This completes the production ML architecture refresher module. Next, the course moves into transformer and LLM foundations, beginning with the tensor shapes created by multi-head self-attention.