Configuring TCP Keepalives for Load Balancer Health Checks

Hello! Welcome back to our course on high-load distributed systems.

Introduction

In our previous lesson, we established the importance of connection reuse and pooling to optimize performance by avoiding the overhead of repeated TCP handshakes. However, this introduces a new challenge: what happens when a pooled, idle connection is silently terminated by a network intermediary? The application, unaware of the termination, might attempt to use this "dead" connection, leading to errors and long timeouts.

This lesson directly addresses that problem. Our learning outcome is to configure TCP keepalive parameters to detect dead connections in load balancers and other long-lived connection scenarios.

We will cover:

• The common problem of idle connection timeouts in cloud networking.
• The TCP keepalive mechanism as a solution.
• How to configure keepalive behavior at the OS level using key kernel parameters.
• The necessity of enabling keepalives at the application level.

This is a crucial operational skill for ensuring the resilience of any system that relies on long-lived TCP connections, from database clients to microservice communication through proxies.

---

1. The Problem: Silently Dropped Idle Connections

In modern distributed systems, especially in the cloud, traffic often passes through stateful network appliances like NAT gateways, firewalls, and load balancers. These devices maintain a state table of active connections. To conserve resources, they enforce an idle timeout: if no packets are seen on a connection for a certain period, the device removes the connection from its state table.

The problem is that this cleanup often happens without notifying the client or the server with a TCP FIN or RST packet. The connection is dropped silently by the intermediary.

To understand this scenario and its consequences, please read the first section of the following AWS blog post.

As the article describes, the client continues to wait for a response that will never arrive, and the server might complete a long computation only to find it cannot send the result. This is precisely the issue that TCP keepalive is designed to solve.

---

2. The Solution: TCP Keepalive

TCP keepalive is a mechanism built into the TCP protocol (specified in RFC 1122) that allows one side of a connection to check if the other is still alive, even when no application data is being exchanged.

It works by sending "probe" packets when a connection has been idle for a specified duration. These probes are carefully crafted TCP segments that elicit a response from the peer's TCP stack without being passed up to the application layer.

To get a visual and conceptual feel for what these packets are, please watch the following short video.

By sending these probes at an interval shorter than the network's idle timeout, you ensure the connection is never considered "idle" by the intermediary, preventing it from being silently dropped. Furthermore, if the probes go unacknowledged, the OS can confidently declare the connection dead and notify the application, allowing it to take corrective action (e.g., removing the connection from a pool and establishing a new one).

---

3. Configuring TCP Keepalive at the OS Level

To use keepalives effectively, you must configure their behavior at the operating system level. In Linux, this is controlled by three primary kernel parameters.

Now, please read the next section of the AWS blog post, which details these parameters.

To summarize, these are the key parameters:

• net.ipv4.tcp_keepalive_time: (seconds) The duration a connection must be idle before the first keepalive probe is sent. This is the most critical parameter. It must be set lower than the idle timeout of any network appliance in the path (e.g., lower than the 350s timeout of an AWS NLB).
• net.ipv4.tcp_keepalive_intvl: (seconds) The interval between subsequent probes if the previous one was not acknowledged.
• net.ipv4.tcp_keepalive_probes: (integer) The number of unacknowledged probes to send before the kernel considers the connection dead and notifies the application.

The total time to detect a dead connection after it has become idle is approximately:
tcp_keepalive_time + (tcp_keepalive_intvl * tcp_keepalive_probes)

Practical Configuration

You can view and modify these settings using either the /proc filesystem or the sysctl command. The sysctl command is generally preferred.

For further detail on the configuration commands, you can refer to this classic Linux HOWTO guide.

Let's walk through a practical example. Suppose we are using a load balancer with a 5-minute (300-second) idle timeout. We want to send a keepalive probe after 1 minute of inactivity and declare a connection dead if we don't get a response after 3 probes sent 15 seconds apart.

You would execute the following commands as root:

# Set keepalive to start after 60s of idle time
sysctl -w net.ipv4.tcp_keepalive_time=60

# Send subsequent probes every 15s
sysctl -w net.ipv4.tcp_keepalive_intvl=15

# Send 3 probes before giving up
sysctl -w net.ipv4.tcp_keepalive_probes=3

To make these changes permanent, you would add the following lines to /etc/sysctl.conf:

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 3

Then, run sudo sysctl -p to apply them.

---

4. Enabling TCP Keepalive in Your Application

This is a critical point: modifying the OS kernel parameters is necessary, but not sufficient. These settings define the default behavior for sockets that request keepalive, but applications must explicitly enable the SO_KEEPALIVE socket option on their connections.

Most modern HTTP clients, database drivers, and server frameworks provide a simple configuration flag to enable this. Your experience building low-latency systems in Java likely involved similar socket-level tuning.

The final section of the AWS article provides excellent, practical examples of how this is done in common tools and SDKs.

The key is that both layers must be configured: the OS defines how keepalives behave, and the application decides if a connection should use them.

---

Conclusion

In this lesson, we tackled the practical problem of maintaining the health of long-lived, idle TCP connections in complex network environments.

Key Takeaways:

• Stateful network intermediaries like load balancers and NAT gateways silently drop idle TCP connections, causing application-level errors.
• TCP keepalive is the standard mechanism to prevent this by sending periodic probes that both refresh idle timers and detect genuinely dead connections.
• Effective configuration requires a two-pronged approach:
  1. OS Tuning: Set tcp_keepalive_time to a value less than the network's idle timeout, and configure tcp_keepalive_intvl and tcp_keepalive_probes to control detection sensitivity.
  2. Application Enablement: The application code must explicitly enable the SO_KEEPALIVE socket option for the connections it creates.

Mastering this allows you to build more resilient systems that can reliably manage connection pools, a foundational element for the load balancers and proxies we are about to discuss.

Preview of the Next Lesson:

Now that we understand how to establish and maintain healthy TCP connections, we can explore how load balancers use them. In the next lesson, we will move up the OSI model to analyze the trade-offs between Layer 4 (TCP) and Layer 7 (HTTP) load balancing in terms of performance, observability, and routing capabilities.