Stateless vs. Stateful Services: Scaling Implications

Welcome back. Last time, you compared vertical scaling—making a machine larger—with horizontal scaling—adding machines and distributing work. Horizontal scaling is most effective when every healthy application instance can handle any incoming request. The major obstacle is often state.

In this lesson, you will distinguish stateless and stateful services, identify where state can live, and explain why this design choice changes scaling, failure recovery, and load balancing. This is a foundational interview concept: “make the application tier stateless” is a common recommendation, but it only becomes meaningful once you can explain what it requires.

---

State is information that affects later work

A system has state when information from the past or present affects how it should process a later operation.

For an online store, relevant state might include:

• a customer’s authenticated identity;
• items in a shopping cart;
• the current inventory count for an item;
• an order’s status, such as paid or shipped;
• a live connection to a chat client.

State is not inherently a problem. Databases, message queues, caches, and file stores exist specifically to retain information. The important system-design question is:

Which component holds the state, and can another instance take over if one instance disappears?

At the application tier, distinguish these two designs:

• A stateful service retains interaction-specific context that later requests need. If that context lives only on one instance, future requests may need to return to that same instance.
• A stateless service does not treat its own local memory or disk as the durable source of interaction context. Any healthy, equivalent instance can process the next request using the request itself and shared dependencies.

“Stateless” does not mean “the system stores no data.” A stateless API can read and write a database on every request. More precisely, it means the application instances are interchangeable.

A useful operational test is this:

If application server A is terminated and the load balancer sends the next request to server B, can B handle it correctly?

If the answer is yes, the application tier is stateless with respect to that interaction. If the answer is no because crucial context existed only in A, it is stateful.

For a Java example, suppose each application instance stores logged-in sessions in its own:

ConcurrentHashMap<String, Session>

The map may be safe for concurrent threads within one JVM, but it is not shared with other JVMs. A request reaching a different server cannot see it. The service is therefore stateful across requests, even though the code is thread-safe.

---

Stateful application instances: local context creates affinity

Consider a shopping-cart service with three application instances behind a load balancer.

A user’s “add to cart” request reaches instance A. If A stores the cart only in its local memory, the user’s later “view cart” request must also reach A. Instance B has no knowledge of that user’s cart.

This is called client affinity, also known as session affinity or sticky sessions: the load balancer deliberately tries to route a client repeatedly to the same server.

Sticky sessions can make a stateful application appear to work, but they impose real costs:

1. Uneven load. One instance may accumulate many active users while another is relatively idle. Adding a new instance does not redistribute existing sessions automatically.
2. Fragile failure handling. If instance A fails, its local sessions may be unavailable. Users can lose carts or be forced to log in again.
3. Harder deployments. Draining or replacing a server means moving, replicating, or deliberately ending the sessions it owns.
4. Reduced flexibility. The load balancer cannot freely choose the least busy healthy server for each request.

A stateful design can be made reliable. Teams may replicate session data, assign users or connections to designated owners, or migrate state during maintenance. But those mechanisms add coordination and failure cases. Later modules will examine replication and partitioning, which are common ways to scale inherently stateful components.

---

Stateless application instances: externalize the state

A stateless application tier treats local memory as temporary working space, not as the only record of an interaction. It places durable or shared state somewhere reachable by all application replicas.

In the shopping-cart example:

1. The client sends a request containing a credential or session identifier and the item to add.
2. Whichever application instance receives the request validates it and reads or updates the cart in a shared state store.
3. A later request may land on a different instance, which retrieves the same cart from that shared store.

The request does not necessarily need to include the entire cart. A session identifier or authenticated user ID can be enough to locate it. What matters is that no particular application instance is the sole holder of the required context.

This distinction is often expressed too broadly as “stateful systems do not scale” and “stateless systems scale.” A more accurate statement is:

Stateless application replicas scale out easily because they are interchangeable. Stateful data still exists and must itself be designed for capacity, availability, and correctness.

For example, moving session state to a shared database or distributed cache removes affinity from the application servers, but it also makes that store part of the critical request path. If every request performs a session lookup, the state store must handle the resulting traffic and meet the required latency and reliability target.

Microsoft’s guidance below makes the practical connection: eliminating server-side session state and client affinity lets available replicas handle requests rather than preserving a particular server-to-client relationship.

---

Where state can live

When an interviewer says, “Make the service stateless,” ask yourself where the needed state should reside instead. There are three broad locations.

Location	Example	Scaling implication
Application instance	A local in-memory session map	Fast access, but requests need affinity; instance failure risks losing context.
Client	A signed token or request parameters	Application instances remain interchangeable, but the client must send sufficient context and sensitive data must be protected.
Shared external store	Session store, database, distributed cache	Any instance can access the data, but the store becomes a shared dependency that must scale and remain available.

The first location generally makes the service stateful. The latter two can support a stateless application tier.

Different kinds of data deserve different homes:

• Business data—users, orders, messages, inventory—normally belongs in durable storage such as a database.
• Session data—a login session, temporary preferences, a cart identifier—may live in a session service, distributed cache, or database, depending on how durable and consistent it must be.
• Temporary computation data—variables used while processing one request—can safely remain in local memory because no later request depends on it.

A local cache does not automatically make an application stateful. If an instance loses a cache entry and can retrieve or recompute the value from a shared source of truth, the cache is a performance optimization, not essential interaction state. In contrast, a local cart that exists nowhere else is essential state.

---

Scaling implications, compared precisely

The following table frames the comparison at the application-service tier. The database or cache behind either service may still be stateful.

Concern	Stateful service with local session context	Stateless service tier
Can any replica handle the next request?	Usually no; it needs the instance that owns the context or a state-transfer mechanism.	Yes, provided it can access the required shared dependencies.
Load balancing	Often requires sticky sessions or routing by an ownership key.	The load balancer can send requests to any healthy, available replica.
Adding instances	Helps with new sessions, but existing state may remain concentrated on older instances.	New identical instances can begin serving traffic immediately.
Instance failure	Local context may be lost or temporarily inaccessible without replication or recovery.	In-flight work may fail, but a later request can go to another instance if shared state is healthy.
Deployment and maintenance	Requires care around session draining, state migration, or replication.	Instances can generally be removed and replaced more freely.
Primary bottleneck after scaling	State ownership, synchronization, and uneven distribution.	Often the shared database, cache, network, or another downstream dependency.

The central scaling benefit is therefore fungibility: stateless replicas are replaceable units. If each application server can handle roughly $600$ requests per second, a load balancer can distribute independent requests across multiple replicas without caring which individual server previously handled that client.

But do not claim perfectly linear scaling. Suppose six stateless application servers all call one database. If the database is already saturated, adding a seventh application server may increase queueing and make user-visible latency worse. Statelessness removes a barrier at the application tier; it does not remove all bottlenecks from the system.

---

Stateless is a default for request-serving APIs, not a universal rule

For a conventional HTTP API—such as “get user profile,” “create order,” or “fetch product details”—stateless application instances are usually the preferred starting point. They support flexible load balancing, autoscaling, and recovery from ordinary instance failures.

Some workloads naturally maintain active state:

• a database owns durable records;
• a message broker tracks messages and consumer progress;
• a multiplayer game or real-time connection server may retain live connection or room context;
• a long-running workflow may need explicit progress tracking.

The correct design response is not to deny this state exists. It is to make its ownership, durability, routing, and recovery deliberate. For example, a real-time server can retain an active connection while a separate shared store records durable user data and message history.

In an interview, avoid saying “I will make the entire system stateless.” A stronger answer separates tiers:

I would run multiple stateless API instances behind a load balancer. Each request would carry authentication context or a session identifier, and the instances would retrieve required user and cart data from shared stores. This lets any healthy API instance process a request. The cart and session stores are stateful dependencies, so I would monitor their capacity and ensure they meet the availability requirements.

That explanation shows both the benefit and the remaining responsibility.

---

A quick architecture review method

When looking at any component, apply this short checklist:

1. Identify the state: What information from a previous interaction affects this request?
2. Locate it: Is it only in one process’s memory, on the client, or in shared storage?
3. Test replacement: Can a newly started replica serve the next request correctly?
4. Find the new bottleneck: If state is externalized, can the external store handle the load and tolerate failures?
5. State the trade-off: Stateless replicas simplify scale-out; the state layer still needs a deliberate design.

This method prevents two common interview mistakes: treating local memory as a reliable shared database, and assuming that a stateless API tier makes the whole architecture automatically scalable.

---

Key takeaways

• State is information whose current or past value affects later processing.
• A stateful service retains essential interaction context, often creating a dependency on a particular instance.
• A stateless service has interchangeable instances: any healthy replica can process a request using request-provided context and shared dependencies.
• Stateless does not mean data-free. Databases, session stores, caches, and file systems are stateful components that may support a stateless application tier.
• Local session state often forces sticky sessions, causing uneven load, fragile failover, and harder deployments.
• Externalizing session state allows free load balancing, but shifts capacity and availability requirements to the shared state store.
• In a design interview, explicitly distinguish the stateless application tier from the stateful data tier.

You have now completed the web and distributed-system foundations module. Next, you will begin a repeatable system-design interview method by learning how to ask scope-defining questions that turn an ambiguous prompt into a designable problem.