Batch, Streaming, and Online Inference: Latency, Throughput, and Consistency Trade-offs

Good to see you again. In the previous lesson, we followed an ML prediction through its full production lifecycle: source data and feature definitions, training and approval, deployment, live inference, delayed outcomes, and monitoring. That lifecycle remains the same across serving modes. What changes is the operating contract around the prediction: when it must be available, how much work must be processed, and what it means for data and outputs to be consistent.

This lesson distinguishes batch scoring, streaming inference, and online inference. By the end, you should be able to describe each mode in terms of latency, throughput, freshness, and consistency—not merely name the infrastructure commonly associated with it. This is the level of precision expected in a senior system-design conversation.

---

Serving mode is a product and systems contract

Teams sometimes choose a mode by starting with a technology: “we have Kafka, so this should be streaming,” or “we deployed an endpoint, so this must be online.” That is backward. The correct starting point is how the consuming application uses a prediction.

A serving mode answers three operational questions:

1. When is a prediction needed?
   Is it needed before a user can proceed, shortly after an event arrives, or by a scheduled business deadline?

2. What is the unit of work?
   Is it one request, an unbounded sequence of events, or a known dataset?

3. Which consistency properties actually matter?
   Must all predictions use one data snapshot? Must event ordering and stateful aggregates be correct? Is bounded feature staleness acceptable? Must duplicate outputs be prevented?

A model can be identical across modes. A malware classifier may score a historical archive overnight, classify an uploaded executable synchronously, or inspect an incoming telemetry stream. The difference lies in the surrounding execution and correctness contract.

The following compact comparison is useful, but it should not be treated as a set of rigid boundaries.

Mode	Consumer’s expectation	Primary latency measure	Primary throughput measure	Typical freshness expectation
Batch scoring	Results available by a deadline	Job completion time	Records or entities scored per job	Hours, days, or a scheduled cutoff
Streaming inference	Each incoming event is processed continuously	Event-to-output delay	Events per second and backlog drain rate	Seconds to minutes, often tied to event time
Online inference	A caller waits for the response	End-to-end request latency, usually percentile based	Requests per second or tokens per second	Current enough for the immediate decision

The next sections make the differences concrete.

---

Batch scoring: optimize for deadline, volume, and a coherent snapshot

Batch scoring applies a model to a finite, known collection of entities. The work is usually triggered on a schedule, by an upstream dataset becoming available, or by a manual backfill. Its output is typically written to a table, object store, search index, or downstream operational database.

Examples include:

• recalculating churn risk for all active accounts every night;
• producing day-ahead energy forecasts each morning;
• rescoring a corpus of files after a malware model update;
• generating embeddings for a newly ingested document collection;
• assigning a support-priority score to all open tickets every hour.

The critical fact is that no individual caller is blocked waiting for a particular prediction. If a churn score is available by 06:00 rather than 05:45, users may see no difference. If the job finishes at 10:00 and sales actions begin at 08:00, the system has violated its real SLA even if the model itself ran quickly.

Batch latency and throughput

For batch jobs, the relevant latency is generally time to complete the whole scoring run, including:

• waiting for upstream data;
• feature computation and joins;
• model loading and inference;
• output writes;
• validation and publication.

If $N$ entities must be scored within a completion window $T$, the minimum sustained processing rate is:

$$\frac{N}{T}$$

For example, scoring $10$ million customers in six hours requires an average rate of approximately:

$$\frac{10,000,000}{6\times 3600}\approx 463$$

entities per second before allowing for retries, skewed partitions, input delays, and output write overhead. A design should therefore target a rate above this theoretical minimum.

Batch systems normally favor high hardware utilization and cost efficiency. They can use large vectorized operations, distributed compute, bulk feature reads, and large inference batches. A model call that takes two seconds may be entirely acceptable when thousands of examples are processed concurrently and the business deadline is met.

Batch consistency requirements

Batch scoring usually needs snapshot consistency more than low latency. The question is:

Do all scores describe a coherent population at a defined cutoff time?

Suppose a daily risk model scores accounts using account status, balances, and transactions. If account data is read at midnight, transactions at 02:00, and balances at 04:00, the output may mix incompatible views of the world. That may be acceptable for rough segmentation, but not for financial reporting or a tightly controlled decision workflow.

A robust batch contract commonly specifies:

• Input snapshot or cutoff: which data versions and time boundary define the scoring population;
• Point-in-time-correct features: a score for time $t$ uses only information available at or before $t$;
• Immutable model and transformation versions: all rows in a run are produced by a known serving contract;
• Completeness: the job identifies missing, invalid, or unscorable entities rather than silently omitting them;
• Atomic publication: consumers see either the previous complete output or the new complete output, not a partially written mixture.

An atomic manifest, partition swap, versioned output table, or pointer update is often more important than the exact distributed-compute framework. Without controlled publication, downstream systems can consume one segment from a new scoring run and another from an old one.

---

Online inference: optimize for an immediate decision

Online inference is synchronous, request-driven prediction. An application or service sends a request and cannot proceed until it receives a response or an explicit failure.

Typical examples include:

• classifying an uploaded file before permitting it into an environment;
• ranking search results during a user query;
• approving, declining, or routing a transaction;
• selecting a recommendation as a page is rendered;
• generating an assistant response for an active user session.

The fundamental performance metric is end-to-end latency, not just model execution time. A request may spend time in a queue, retrieve online features, run preprocessing, execute the model, apply policy, and serialize the response.

For a conventional model endpoint:

$$L_{\text{end to end}}=L_{\text{queue}}+L_{\text{feature}}+L_{\text{preprocess}}+L_{\text{model}}+L_{\text{postprocess}}+L_{\text{network}}$$

The service-level objective is usually written as a percentile, such as “$p95$ response latency below 200 ms” or “$p99$ below 800 ms,” together with an availability target. An average latency is insufficient: a small fraction of very slow requests may materially damage a user workflow.

Online throughput is variable, not merely high

Online traffic is often bursty. A system may handle a low average request rate but experience sharp peaks due to a product launch, a market event, or synchronized client retries.

The relevant capacity questions are therefore:

• What request rate must be sustained at peak?
• What request size distribution is expected?
• What latency percentile must still hold at that load?
• How much queueing is acceptable before requests are rejected or degraded?
• Can the system safely shed noncritical traffic?

For LLM applications, request count alone is especially misleading. A short classification request and a long-context generation request have radically different costs. Later in the course, token-level latency, continuous batching, and GPU scheduling will make this distinction more precise.

Online consistency requirements

Online systems rarely need a globally serializable view of every data source. Requiring that can make a service unnecessarily slow and fragile. They do need a carefully stated definition of what “current enough” means for each feature and decision.

Consider a real-time ticket-routing service. It might safely use:

• account tier no more than 24 hours old;
• open-incident count no more than five minutes old;
• the ticket content supplied in the request exactly as received;
• one approved model, feature-definition, and policy version for the entire request.

That is a bounded-staleness contract: some features may lag reality, but only within a defined and monitored limit.

An inventory allocation service has a different requirement. If a prediction immediately causes a reservation, the authoritative inventory state may need a strongly consistent read, and the resulting side effect must be idempotent. A low-latency model cannot compensate for a business action that oversells stock.

For online inference, distinguish these concerns:

Concern	Typical requirement
Request payload	Schema-valid and associated with one request or trace ID
Model artifact	Immutable, approved version for the request
Feature definitions	Semantically consistent with training and versioned
Feature values	Fresh within a stated bound, unless the decision requires stronger reads
Decision side effect	Idempotent or deduplicated under retries
Response	Returned within a latency and availability objective

This makes clear why “eventual consistency” is neither universally bad nor universally acceptable. It is a choice tied to the consequence of a stale decision.

---

Streaming inference: optimize for continuous event handling and temporal correctness

Streaming inference processes an unbounded sequence of events as they arrive. The output may be an alert, a scored event, an updated entity state, or an action sent to another service.

A stream is not simply “a fast batch.” It is open-ended: there is no final dataset to finish. The system must remain correct while events arrive late, arrive twice, are reordered, or temporarily arrive faster than they can be processed.

A turbine-monitoring system offers a concrete example. Each sensor reading can be scored for anomaly risk as it arrives, but the prediction may also depend on rolling statistics over the previous ten minutes, current operating mode, and recent maintenance events. This is not just per-record model invocation; it is stateful temporal processing.

Streaming latency: measure from event to durable outcome

A streaming model may have a very fast inference call yet still deliver an operationally late result. Streaming latency is usually measured from an event’s relevant timestamp to a durable output or action.

That end-to-end delay can include:

• producer and broker delay;
• consumer lag;
• state lookup or window computation;
• feature enrichment;
• inference time;
• output publication or action dispatch.

Two metrics are especially important:

• Event-to-output latency: How long after the event occurred does the prediction become available?
• Consumer lag or backlog: How far behind the arrival rate is the system?

A pipeline that normally produces alerts in two seconds but accumulates a 30-minute backlog during peak traffic has not met a two-second operational requirement.

Event time versus processing time

A sensor can create an event at 10:00, lose connectivity, and send it at 10:08. If the pipeline processes it at 10:08, these are three distinct times:

• Event time: when the sensor observed the event;
• Ingestion time: when the messaging system accepted it;
• Processing time: when the stream processor handled it.

For rolling features and time-windowed predictions, the choice is a correctness decision. A ten-minute anomaly feature should usually use event time, because it represents when observations actually occurred. Processing-time windows may be easier and lower latency, but they can become incorrect when delivery is delayed or out of order.

A streaming design should explicitly state:

• the event-time field and its validation rules;
• how much late arrival is tolerated;
• when a window is considered complete;
• whether a late event updates a previous prediction;
• whether downstream consumers can handle corrections.

Streaming consistency and delivery semantics

Streaming inference commonly requires consistency across three connected pieces:

1. State consistency
   A rolling aggregate, deduplication store, or latest-entity view must be updated correctly during failure recovery.

2. Delivery semantics
   At-least-once processing avoids loss but can reprocess events. Exactly-once processing coordinates reads, state changes, and writes within the supported transactional boundary.

3. External side effects
   A stream processor may produce a correct prediction once, but an external alerting API, database, or ticketing tool can still receive a duplicate if a retry occurs across a failure boundary.

This last distinction matters in senior designs. “Exactly once” is not a universal property that a single framework grants to the entire business process. If inference produces a fraud-review case, a practical design often uses an idempotency key such as:

$$(\text{model version},\text{event ID},\text{decision type})$$

The downstream sink can then upsert or reject duplicate actions. This achieves effectively-once business behavior even when infrastructure delivery is at least once.

---

Consistency has several dimensions

The word consistency can obscure design discussions because different modes need different kinds of it. The table below separates the most important dimensions.

Dimension	Batch scoring	Streaming inference	Online inference
Data-time semantics	One defined snapshot or cutoff	Event time, windows, late data policy	Request time plus stated feature freshness
Feature correctness	Point-in-time joins and reproducible transformations	Correct stateful aggregates and temporal joins	Versioned transformations and bounded staleness
Output completeness	All expected entities scored or explicitly accounted for	Each event handled according to delivery policy	Every accepted request gets a response or explicit error
Duplicate handling	Idempotent reruns and atomic partition replacement	Deduplication, transactional state, idempotent sinks	Idempotency keys for retries and side effects
Artifact consistency	A run uses known model and code versions	Events are processed under traceable deployment versions	Each request is attributable to one model and policy version

Notice that all three modes need versioned model artifacts and feature semantics. The differences concern the temporal contract around their inputs and outputs.

For example, a batch job can tolerate a stale snapshot but must not silently mix snapshots. A streaming pipeline may tolerate a few seconds of processing lag but must define late-event handling. An online endpoint may tolerate a five-minute-old customer feature but cannot use a partially deployed feature schema or an unapproved model version.

---

A feature platform can support all three modes

The serving modes do not require completely separate ML definitions. The central architectural goal is to preserve feature semantics while offering different access patterns.

The Feast feature architecture illustrates an important separation:

• Request sources support per-request feature assembly for online inference.
• Stream sources continuously update feature state used by streaming or online decisions.
• Batch sources provide historical and large-scale data for training and scheduled scoring.
• The registry records feature definitions so that offline and online paths can share intended semantics.

The diagram should not be read as a guarantee of correctness by itself. A feature platform does not automatically solve point-in-time joins, late events, or duplicate side effects. Those remain design responsibilities. Its contribution is to reduce the risk that different teams implement incompatible versions of the same feature transformation.

Two useful GenAI distinctions

For GenAI systems, two terms are often confused with streaming inference:

• Streaming tokens to a user interface is usually still online inference. A user has made a synchronous request; the response is delivered incrementally to improve perceived latency. The relevant metrics include time to first token and inter-token latency.

• Continuous batching inside an LLM server is an implementation strategy for improving online-serving efficiency. The external serving mode remains online because callers are waiting for responses.

By contrast, embedding a newly ingested document corpus overnight is batch inference, while scoring every newly uploaded document from an event bus is streaming inference.

---

A concise selection discipline

When documenting a serving design or answering an interview prompt, start with a one-sentence mode statement:

“This is batch scoring because the system must score a known population by 06:00 from a consistent daily snapshot; per-entity latency is not user-visible.”

“This is streaming inference because each telemetry event must produce an anomaly result within 10 seconds of event time, including out-of-order-event handling and idempotent alert creation.”

“This is online inference because the user workflow waits for a decision; the service therefore needs a $p95$ latency objective, high availability, and explicitly bounded feature staleness.”

Then make the design testable by stating:

• the input unit and arrival pattern;
• the latency or completion objective;
• the throughput or volume target;
• the allowable staleness;
• the time model;
• the duplicate and failure behavior;
• the output publication or side-effect contract.

This prevents an architecture discussion from collapsing into a list of tools.

---

Key takeaways

• Batch scoring processes a known dataset. It prioritizes high throughput, cost efficiency, a completion deadline, snapshot consistency, and atomic publication.
• Streaming inference processes an unbounded event sequence. It prioritizes sustained event throughput, event-to-output latency, temporal correctness, managed state, and explicit delivery semantics.
• Online inference serves a caller waiting for a response. It prioritizes end-to-end percentile latency, availability, burst handling, coherent model artifacts, and defined feature freshness.
• Consistency is not one requirement. It includes snapshot or event-time semantics, feature freshness, artifact versioning, duplicate handling, and side-effect correctness.
• The same model may be used in several serving modes; the choice depends on the consuming workflow, not the model class or infrastructure brand.
• Token streaming and internal LLM batching do not, by themselves, make a workload streaming inference.

Next, we will go one level deeper: selecting an appropriate consistency model for features, predictions, and model metadata in a concrete system scenario.