Estimating Inference System Performance and Cost from Assumptions

Hello. In the previous lesson, you turned an ambiguous ML prompt into a design contract: a product objective, a prediction target, a metric stack, workload drivers, and explicit non-functional requirements. This lesson begins at the next interview move: turning those requirements into numbers that constrain a credible design.

You do not need precise forecasting in an interview. You need transparent assumptions, unit-safe arithmetic, and estimates accurate enough to distinguish a single service from a large fleet, gigabytes from petabytes, and a modest operating cost from an architectural concern. By the end, you will be able to estimate request and prediction throughput, construct a latency budget, interpret an availability target, size major data flows, and translate serving capacity into a rough compute-cost range.

---

Estimation is a decision tool, not a math performance

An order-of-magnitude estimate answers a design question:

• Does the serving tier need horizontal scaling?
• Is the bottleneck likely to be compute, network, or a feature dependency?
• Can we afford real-time scoring for every candidate?
• Does the availability target require multi-zone capacity and a fallback?
• Is logged training data likely to become a material storage cost?

State assumptions before calculations, round aggressively, and label every unit. A useful approximation is:

$$1\text{ day}\approx 10^5\text{ seconds}$$

This makes the core conversion quick:

$$10^6\text{ requests per day}\approx 10\text{ requests per second}$$

The estimate is only as useful as its unit of work. For an ML ranking service, distinguish:

• Inference requests per second: feed loads or API calls received by the service.
• Predictions or scores per second: individual user-item pairs evaluated by a model.
• Feature reads per second: calls or lookups needed to assemble inputs.
• Events per second: impressions, clicks, labels, and telemetry written for analysis and training.

A service handling 6,000 feed requests per second may sound manageable. If each request scores 500 candidates, it represents 3 million candidate scores per second. Those are radically different capacity numbers.

A compact estimation ledger

Before calculating, put assumptions in a visible table. It makes your reasoning easy to audit and revise.

Assumption	Example value	Why it matters
Daily active users	10 million	Establishes daily demand
Feed loads per active user per day	5	Converts users to inference requests
Peak-to-average factor	10	Sizes serving capacity for bursts
Candidates scored per request	500	Converts requests into model work
End-to-end p95 target	150 ms	Limits batching and dependency calls
Serving availability target	99.95%	Sets error budget and redundancy needs
Logged displayed items per request	20	Determines training-event volume
Event record size	200 bytes	Determines storage and bandwidth
Retention	90 days	Converts daily ingestion into stored data

In an interview, say explicitly: “I will use these as planning assumptions; the architecture changes materially if the peak factor, candidates per request, or latency target changes.”

---

A worked ML-inference estimate

Continue the signed-in feed-ranking example from the prior lesson. Assume:

• 10 million DAU;
• 5 feed loads per user each day;
• a 10-times peak-to-average traffic ratio;
• 500 eligible candidates scored for each feed load;
• a p95 end-to-end latency SLO of 150 ms.

1. Request throughput

First calculate total daily feed requests:

$$10^7\text{ users per day}\times 5\text{ requests per user per day}=5\times 10^7\text{ requests per day}$$

Average request rate is therefore:

$$R_a=\frac{5\times 10^7}{10^5}=5\times 10^2=500\text{ requests per second}$$

Using the tenfold peak assumption:

$$R_p=500\times 10=5,000\text{ requests per second}$$

Using the exact 86,400-second day would give a somewhat higher number, about 580 average requests per second and 5,800 at peak. For this interview estimate, call the design point roughly 6,000 peak requests per second. The rounding direction is sensible because this is capacity planning.

2. Model-score throughput

Now convert request traffic into work at the model tier:

$$Q_p=6,000\text{ requests per second}\times 500\text{ candidates per request}=3,000,000\text{ scores per second}$$

This is the number that informs whether:

• the ranker must be compact and highly optimized;
• candidates must be reduced before ranking;
• some features must be precomputed;
• accelerators are economically viable;
• a simpler model may be appropriate at the final ranking stage.

Do not assume that all stages see the same work. A retrieval stage may search a very large corpus but return 500 candidates; the ranking model only sees those 500. State clearly which stage your estimate refers to.

3. Latency budget

A latency SLO needs to be decomposed across the request’s critical path. Suppose the product requires 150 ms p95 end-to-end. A practical budget, rather than a prediction of exact observed latency, could be:

Component	Budget
Edge and network overhead	20 ms
API orchestration	10 ms
Candidate retrieval and feature reads, in parallel	25 ms
Ranking-model inference, including allowed batching	60 ms
Business rules and response assembly	10 ms
Queueing and jitter reserve	25 ms
End-to-end p95 budget	150 ms

When two independent calls occur in parallel, the critical-path contribution is approximately the slower one, not their sum:

$$T_{\text{parallel}}\approx \max (T_{\text{features}},T_{\text{retrieval}})$$

The full budget here is:

$$T=20+10+\max (25,25)+60+10+25=150\text{ ms}$$

Do not present this as “the p95 of the endpoint equals the sum of every component’s p95.” Percentiles do not combine so neatly, and dependency latencies may be correlated. Present it as a design budget: each owner has a target, and load tests validate whether the composed system meets the endpoint SLO.

A quick concurrency sanity check is also useful. At 6,000 requests per second and a 60 ms ranking-service budget, roughly this many ranking requests are simultaneously in flight:

$$6,000\times 0.06=360\text{ concurrent requests}$$

That number matters for request-state memory, batch queues, connection pools, and overload protection.

The efficient-frontier idea prevents a common mistake: treating a latency target and a throughput target as unrelated. Increasing batch size may improve accelerator utilization and score throughput, but it can increase queueing delay and violate the 60 ms model budget. Conversely, using tiny batches can protect latency while requiring a much larger—and more expensive—fleet. The right operating point is one that meets the user-facing SLO at the required peak load, not the configuration with the highest benchmark throughput.

---

Availability is an error budget with a product meaning

“Four nines” should never be left as a vague aspiration. Start by asking what counts as unavailable.

For a feed product, separate at least two service levels:

1. Feed availability: the user receives a usable feed, potentially from a cached, popular, or editorial fallback.
2. Personalized-ranking availability: the real-time personalized ranker successfully contributes to the response.

The first can often have a higher SLO than the second because graceful degradation protects the product experience. For high-risk decisions such as fraud approval, a fallback might be an explicit review path rather than a default decision.

For a 99.95% monthly availability target, the permitted unavailability is:

$$1-0.9995=0.0005=0.05\%$$

Over an approximate 30-day month:

$$0.0005\times 30\times 24=0.36\text{ hours}\approx 22\text{ minutes}$$

Over a year:

$$0.0005\times 365\times 24\approx 4.4\text{ hours}$$

That is the error budget. It must cover the failures included in the SLO definition: service errors, timeouts, and possibly unacceptable stale results. It does not mean every dependency may independently consume 22 minutes of outage; dependencies on the critical path compound reliability risk.

Availability affects capacity estimates. Suppose 429 model replicas are required to meet peak scoring demand at a conservative utilization target. If the fleet is evenly distributed over three availability zones, losing one zone removes approximately one-third of provisioned capacity. To retain capacity for 429 required replicas after such a loss, provision roughly:

$$429\times \frac{3}{2}\approx 644\text{ replicas}$$

This deliberately excludes other practical headroom such as deployments, uneven traffic, and autoscaling delay. The point is not that 644 is an exact fleet size; it is that reliability has a capacity cost. Mentioning this connection signals that you understand availability as a system property rather than a setting on a load balancer.

---

Estimate the data, not just the requests

For ML systems, data volume comes from at least three places:

• online request and response payloads;
• feature-store state;
• event logs retained for monitoring, experimentation, and training.

Network and feature payloads

Assume the ranking service receives an average 20 KB serialized feature payload per request and returns a 5 KB ranked response. At peak:

$$6,000\text{ requests per second}\times 20\text{ KB}=120\text{ MB per second}$$

The response egress is:

$$6,000\text{ requests per second}\times 5\text{ KB}=30\text{ MB per second}$$

These are internal service-to-service traffic estimates, not necessarily traffic from a mobile device. They tell you to inspect network paths, serialization overhead, and whether all 500 candidate feature vectors are being repeatedly transferred.

The daily ingress volume is approximately:

$$50\text{ million requests}\times 20\text{ KB}=1\text{ TB per day}$$

This does not mean one terabyte must be retained daily. Most request payloads are transient. It does establish a bandwidth and data-transfer scale.

Event logging and training data

For each feed request, perhaps 20 items are shown. Assume a compact impression event—including identifiers, ranking position, outcome fields, timestamp, and experiment metadata—uses 200 bytes.

$$50\text{ million requests per day}\times 20\text{ impressions per request}\times 200\text{ bytes per impression}=200\text{ GB per day}$$

With 90-day retention and three copies for durability:

$$200\text{ GB per day}\times 90\times 3=54\text{ TB}$$

This estimate excludes indexes, schema overhead, click and session events, derived tables, and downstream training copies. Therefore, call it a lower-bound planning number.

The logging design matters enormously. Logging all 500 candidate scores and complete feature vectors can multiply data volume by orders of magnitude. In an interview, state a policy such as: log displayed impressions and decision metadata by default; sample deeper per-candidate diagnostics; retain detailed payloads for a shorter period subject to privacy policy. That preserves observability without making the training and logging path needlessly expensive.

---

Convert serving capacity into compute cost

Cost estimates require a measured or explicitly assumed unit capacity. Avoid saying, “One GPU handles 10,000 predictions per second,” without defining:

• model version and precision;
• feature and candidate payload shape;
• batch-size policy;
• latency percentile;
• hardware type;
• utilization limit;
• whether feature fetching is included.

For the running example, assume a benchmark shows that one model-serving replica can sustain 10,000 scores per second at saturation for this model and payload. To leave headroom for burstiness, latency variation, and deployment events, target 70% of that rate:

$$C_{\text{usable}}=10,000\times 0.70=7,000\text{ scores per second per replica}$$

Required replicas for the 3 million-score-per-second peak are:

$$N=\frac{3,000,000}{7,000}\approx 429$$

If a multi-zone design provisions approximately 644 replicas to survive loss of one of three zones, and an all-in accelerator cost assumption is $3 per replica-hour, the monthly accelerator cost is roughly:

$$644\times 3\times 730\approx \$1.4\text{ million per month}$$

That is not a forecast. It is a forcing function: the result says you must investigate candidate reduction, model distillation or quantization, improved batching, caching, hardware choice, or an alternative serving pattern. It may also expose that the original assumptions are inconsistent with the intended product economics.

In an interview, say what is excluded:

“This is accelerator serving cost only. I would separately estimate CPU orchestration, feature-store reads, streaming and logging, storage, network egress, and operational overhead. The largest sensitivity is candidate count per request, followed by measured score throughput at the p95 latency target.”

That final sentence is managerial judgment. It tells the interviewer where additional measurement will have the highest value.

---

A five-minute sizing talk track

Use this sequence after clarifying the product requirements:

1. Name the assumptions. “I’m assuming 10 million DAU, five feed loads daily, a tenfold peak, and 500 candidates per feed load.”
2. Calculate request volume. “That is 50 million requests per day, about 500 to 600 average requests per second, and about 6,000 at peak.”
3. Translate into ML work. “At 500 candidates per request, the ranking tier must sustain roughly 3 million scores per second at peak.”
4. Allocate the latency SLO. “For a 150 ms endpoint p95, I reserve 60 ms for ranking, run feature and retrieval work in parallel, and leave explicit queueing headroom.”
5. Define availability precisely. “I distinguish the feed’s availability, including fallback, from real-time personalization availability; 99.95% permits only about 22 minutes of monthly unavailability.”
6. Size the major data flow. “Displayed-impression logging alone is about 200 GB per day under my assumptions, or about 54 TB for 90 days with three copies.”
7. Give a cost range and sensitivity. “Using a benchmarked 7,000 usable scores per second per replica, this is hundreds of replicas and roughly low-single-digit millions per month in accelerator spend, before adjacent infrastructure.”

Do not calculate every conceivable number. Calculate the quantities that change the architecture, then state the uncertainty and the next validation step: benchmark the real model, run a load test with realistic feature calls, and measure tail latency under peak-like traffic.

---

Key takeaways

A defensible estimate starts with declared assumptions and preserves the distinction between requests, candidate scores, feature reads, and logged events.

• Convert daily usage to average and peak requests per second; plan capacity around peak, not the daily average.
• For ranking systems, multiply requests by candidates per request to expose true model throughput.
• Treat latency as an explicit critical-path budget, including queueing headroom; parallel calls contribute roughly their maximum latency.
• Translate an availability percentage into an error budget, and distinguish product availability from model-personalization availability.
• Estimate data at its source: transient network payloads, persisted feature state, and retained logs have different implications.
• Calculate compute cost from a measured or clearly assumed per-replica capacity at the required latency, then include utilization and failure-domain headroom.
• Use estimates to identify the largest sensitivity and the next experiment or benchmark, not to claim false precision.

Next, you will use these estimates to select an inference pattern—batch, asynchronous, streaming, or synchronous online serving—based on freshness, latency, scale, and cost.