Understanding Latency, Throughput, Bandwidth, and Concurrency

Good to see you again. In the previous lesson, you traced one HTTPS request through DNS, Internet routing, a public web server, application code, and a database. That path gives us the components; this lesson gives us the vocabulary for judging its performance.

By the end, you should be able to distinguish four terms that are often blurred together in interviews: latency (how long one operation takes), throughput (how much work finishes over time), bandwidth (a link’s data-transfer capacity), and concurrency (how much work overlaps in progress). You will also connect them with a small but powerful capacity-planning relationship.

---

Four different questions about performance

Imagine an online store’s product endpoint:

GET /products/42

A performance report might say:

• “A request takes 120 ms.”
• “The service completes 2,000 requests per second.”
• “The network interface is 1 Gbps.”
• “About 300 requests are in flight.”

These are not four ways of saying “the system is fast.” They answer four different questions.

Metric	The question it answers	Typical units	Example
Latency	How long does one request take?	ms, s	Product request completes in 120 ms
Throughput	How much work completes per unit time?	RPS, TPS, MB/s	Service completes 2,000 RPS
Bandwidth	What is the maximum data-transfer rate of this link?	Mbps, Gbps	Network link capacity is 1 Gbps
Concurrency	How many requests are being handled at once?	requests in flight	300 active requests

A useful first rule is this:

• Latency is about time per item.
• Throughput is about items per time.
• Bandwidth is about bits per time.
• Concurrency is about items currently in progress.

“Items” may mean HTTP requests, database queries, messages, video streams, or background jobs. In system design, always state what unit of work you are measuring.

---

Latency: the experience of one request

Latency is the elapsed time for a single operation. For a user, it is often end-to-end time: from clicking a button until seeing a useful response.

For the product endpoint, suppose the timing looks like this:

Part of the request	Time
Client-to-server network travel	25 ms
Load balancer and application processing	10 ms
Database query	60 ms
Response travel back to client	25 ms
Total observed latency	120 ms

The database is not necessarily “the latency”; it is one contributor. From the previous lesson’s request trace, you can see why latency is cumulative: DNS lookup or connection setup may add time on a cold request; network distance contributes time; busy servers and databases can force a request to wait.

Four common components are:

1. Propagation delay: physical travel time through a network. A user far from the data center cannot avoid the cost imposed by distance.
2. Transmission delay: time required to place the request or response bits onto a link. This becomes important for large payloads or slow links.
3. Processing delay: time spent executing application code, encrypting data, routing a request, or running a query.
4. Queuing delay: time spent waiting because a server, connection pool, CPU, or database is busy.

The last category often explains sudden slowness under load. A database query that normally executes in 10 ms might still yield 500 ms user-visible latency if it waits 490 ms for a free database connection.

Do not rely only on average latency

Averages conceal bad experiences. Consider 100 requests:

• 99 requests take 20 ms.
• 1 request takes 2,000 ms.

The average is about 40 ms, which sounds fine. Yet one user waited two seconds.

For this reason, production systems commonly use percentiles:

• p50: the median; half of requests are faster and half are slower.
• p95: 95% of requests are at or below this latency.
• p99: 99% of requests are at or below this latency.

An interview-ready statement is:

I would measure p50, p95, and p99 latency, not only the average, because queues, slow database queries, and retries can create a small but important slow tail.

Watch the following explanation before continuing. It formalizes latency, then contrasts it with throughput and shows why concurrency matters for capacity.

---

Throughput: completed work over time

Throughput is the rate at which a system successfully completes work.

For an HTTP service, we often use requests per second (RPS):

$$\text{throughput}=\frac{\text{completed requests}}{\text{elapsed time}}$$

If the product service completes 12,000 requests in one minute, its average throughput during that minute is:

$$\frac{12,000}{60}=200\text{ RPS}$$

For a payment service, the unit might be transactions per second (TPS). For an event-processing system, it might be messages per second. For a network link, it may be bytes per second or bits per second.

Throughput is not necessarily the same as the rate at which requests arrive:

• Arrival rate: how quickly clients send work.
• Throughput: how quickly the system completes work.

If clients send 3,000 requests per second but the service can complete only 2,000 requests per second, the remaining requests accumulate in queues or eventually fail. Initially, the completed throughput may remain near 2,000 RPS, but latency rises because each request waits longer. Once queues, timeouts, or downstream dependencies are exhausted, error rates rise too.

This is one reason a system can look “busy” while users report it as slow: high throughput alone does not prove good user experience.

The bottleneck sets sustainable throughput

A request path is limited by its slowest constrained resource. Consider an application tier that can process 5,000 RPS but calls a database that can safely handle only 1,500 queries per second for this endpoint. If every request requires one database query, the end-to-end service cannot sustainably complete 5,000 RPS.

Adding application servers will not remove a database bottleneck. It may simply cause more requests to reach the overloaded database, increasing queuing and tail latency.

---

Bandwidth: the capacity of a data path

Bandwidth is the maximum rate at which a particular medium can transfer data. For a network link, it is usually measured in bits per second:

• $100$ Mbps: 100 megabits per second
• $1$ Gbps: 1 gigabit per second
• $10$ Gbps: 10 gigabits per second

Be careful with uppercase and lowercase letters:

• b means bit.
• B means byte.
• One byte is eight bits.

Therefore, a 1 Gbps link has a theoretical raw capacity of roughly 125 MB/s:

$$\frac{1\text{ Gbps}}{8}=125\text{ MB/s}$$

Actual useful data transfer will be lower because network protocols add headers, acknowledgements, encryption work, congestion, packet loss, and application processing.

The distinction is:

Situation	Bandwidth	Actual throughput
Network interface can carry up to 1 Gbps	1 Gbps	Not yet known
A file transfer uses 650 Mbps in practice	1 Gbps	650 Mbps
A slow receiver can process only 200 Mbps	1 Gbps	At most about 200 Mbps

Bandwidth is a capacity ceiling for data transfer on that link. But it does not directly tell you application throughput in requests per second without knowing request and response sizes.

For example, an API returning 2 KB responses at 1,000 RPS sends roughly 2 MB/s of response payload. A 1 Gbps link is unlikely to be the bottleneck. The service could still be slow because every request waits on a database lock. Conversely, a media service may have low request throughput but consume enormous bandwidth because each request delivers a large video segment.

Read the following sections for a compact review of the metrics and their practical relationship.

---

Concurrency: work that overlaps in time

Concurrency is the number of independent work items a system is handling at the same time. In a web service, this is often described as requests in flight: requests that have begun but have not yet finished.

A request can count as concurrent even if it is waiting:

• for a database query,
• for a downstream HTTP service,
• for disk I/O,
• for a free thread or connection,
• or for CPU time.

Concurrency is not exactly the number of CPU cores or Java threads. A service may support many concurrent requests while only a few execute on the CPU at any one instant, especially when most are waiting on I/O. Parallelism refers more narrowly to work literally executing simultaneously, such as work on separate CPU cores.

The supplied Lambda timeline illustrates the basic capacity intuition.

The diagram separates an initialization phase from invocation work. In a real serverless system, cold-start initialization can add latency and temporarily reduce observed capacity. For the core calculation, focus on the steady-state assumption: ten available slots and 500 ms per invocation.

This relationship is captured by Little’s Law:

$$L=\lambda W$$

Where:

• $L$ is average concurrency, or average number of requests in the system.
• $\lambda$ is throughput, in completed requests per second.
• $W$ is average latency, in seconds per request.

In the diagram:

$$L=20\text{ requests/s}\times 0.5\text{ s}=10\text{ concurrent requests}$$

Rearranging the formula is especially useful in interviews:

$$\lambda =\frac{L}{W}$$

Suppose an API has average latency of 100 ms, or $0.1$ seconds, and can sustain 50 concurrent in-flight requests without resource saturation:

$$\lambda =\frac{50}{0.1}=500\text{ RPS}$$

This is an estimate, not a guarantee. It assumes stable demand and a system that can actually support that concurrency. If the database saturates at 100 concurrent queries, raising application concurrency to 500 can create queues, contention, timeouts, and worse latency rather than proportionally higher throughput.

High concurrency is not automatically good or bad

Concurrency has two very different meanings depending on context:

• Useful concurrency keeps resources productive. While one request waits for I/O, another can run.
• Unhealthy concurrency may mean requests are piling up behind a bottleneck.

Suppose a service normally has 100 concurrent requests and p95 latency of 150 ms. After a database problem, concurrency climbs to 2,000 while throughput remains flat and p95 latency rises to several seconds. The large concurrency number is evidence of an accumulating queue, not success.

---

Choosing the right metric in an interview

When a prompt says “make the system fast,” clarify the goal rather than treating all metrics as interchangeable.

Requirement or symptom	Metric to inspect first	Likely design direction
“A user must see a result within 200 ms.”	p95 or p99 latency	Reduce round trips, cache data, improve database access
“The system must handle 50,000 API calls per second.”	Sustained throughput	Find bottlenecks, distribute load, reduce per-request work
“We serve large images and video.”	Bandwidth and byte throughput	Use compression, CDN delivery, and sufficient network capacity
“There are too many requests waiting.”	Concurrency plus latency	Locate the constrained resource and prevent unbounded queues
“The average looks good, but some users complain.”	Tail latency	Investigate p95 and p99, retries, slow dependencies, and queues

A concise way to explain the four terms in an interview is:

Latency is the time for one request. Throughput is the number of completed requests per second. Bandwidth is the maximum data rate of a network link, usually in bits per second. Concurrency is the number of requests in progress. In a stable system, concurrency is approximately throughput multiplied by latency, so slower requests require more concurrent capacity to sustain the same throughput.

---

Key takeaways

• Latency measures time for one operation; measure its distribution with percentiles such as p50, p95, and p99.
• Throughput measures completed work per unit time, such as RPS or TPS.
• Bandwidth is a network link’s theoretical data-transfer capacity; actual byte throughput is usually lower.
• Concurrency is the amount of overlapping work in progress, including work waiting on I/O or a busy resource.
• In a stable system, Little’s Law connects the measures:

$$\text{concurrency}=\text{throughput}\times \text{latency}$$

• Raising concurrency can improve throughput only while the system has spare capacity. Beyond a bottleneck, it often increases queuing and tail latency.

Next, you will use these terms while comparing vertical scaling and horizontal scaling: two fundamental ways to increase a system’s capacity.