Some of the references are quoted in this article. Please refer to the “Resources” section at the end of this article. Thank you for sharing the information.

1, the introduction

For IM developers, network survival is a familiar concept. For example, here is my recent article on network survival: Function, principle, implementation ideas, etc., and I share a lot of actual code code also must consider the implementation of this problem, such as the recent “follow the source IM(five) : the correct understanding of IM long connection, heartbeat and reconnection mechanism, and begin to achieve.”

For IM applications, the most direct way to keep the network alive at the application layer is the heartbeat mechanism. For example, there are wechat, QQ, Dingding. yixin and so on in the mainstream IM. (**PS: ** Yes, the “signaling crisis” between wechat and operators was related to this)

The so-called network heartbeat is usually when the client sends a packet (a heartbeat packet) to the server at short intervals to inform the server that it is still online (a heartbeat packet may also transmit some necessary data). Sending heartbeat packets is to maintain a long connection from the communication level. As for the contents of this packet, there is no special regulation, but in order to save traffic in mobile IM, they are generally small packets (for example, some third party IM clouds are called 1-byte heartbeat packets in order to show that heartbeat does not cost traffic).

It is often asked, however, why implement KeepAlive/heartbeat mechanisms at the application layer when TCP itself has a KeepAlive timer.

Yes, this is almost a must-ask question when you’re usually looking at instant messaging /IM programmers!

To answer this question, I usually recommend looking at “Why Does TCP-based mobile IM Still Need a Heartbeat?” This article. However, due to the limitation of space, this paper does not discuss the KeepAlive mechanism of TCP protocol itself, so this paper wants to sort out the KeepAlive mechanism of TCP protocol in detail, so that we can have a glimpse into it.

2. Series of articles

This is the 12th article in a series that Outlines the following:

  • “Unknown Network Programming (1) : A Simple Analysis of the DIFFICULT Problems in TCP Protocol (part 1)”
  • “Unknown Network Programming (II) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part II)”
  • TIME_WAIT and CLOSE_WAIT when closing TCP connections
  • “Network Programming under the radar (Part 4) : A Deep Dive into TCP’s Abnormal Shutdown.”
  • Hidden Network Programming part 5: UDP Connectivity and Load Balancing
  • Unknown Network Programming (6) : Understanding UDP in Depth and Using it Well
  • Unknown Network Programming (7) : How to Make unreliable UDP Reliable?
  • Network Programming under the Radar (Part 8) : Deep Decryption of HTTP from the Data Transport Layer
  • Unknown Network Programming (9) : Connecting Theory with Practice, Understanding DNS in all Directions
  • “Network Programming under the radar (Part 10) : Getting inside the Operating System and Understanding Network Package Reception from the Kernel (Part Linux)”
  • The Secret of Network Programming (11) : An In-depth Analysis of the Secret of TCP Connection Time
  • “Unknown Network Programming (12) : Thoroughly Understand the TCP Protocol Layer KeepAlive mechanism” (* article)

3. The purpose of TCP KeepAlive

In C/S mode, when both parties are idle, if either party crashes, goes down, disconnects the network cable, or is faulty, the other party cannot know that the TCP connection is invalid.

Then, the other party to the connection does not know about the peer and maintains the connection all the time. As a “server”, the accumulation of a long time will lead to a large number of half-open connections, resulting in the consumption and waste of end system resources, and may lead to the sending of business data at an invalid data link level, resulting in the sending failure.

Therefore, each end must quickly sense failures and reduce invalid link operations, which is the KeepAlive detection mechanism of TCP.

PS: Such a broad statement about the necessity of TCP’s KeepAlive mechanism is not very convincing. The next section will go into further analysis with specific examples.

4. Understand the necessity of TCP KeepAlive from the NAT perspective

When it comes to the necessity of TCP’s KeepAlive, most articles are written in general terms like the previous section, but this is not enough for inquisitive developers.

This section will analyze the necessity of TCP creators to design KeepAlive mechanism from the perspective of router NAT mechanism.

4.1 Start with the PRINCIPLE of NAT

In a narrow sense, NAT is divided into SNAT (original address translation) and DNAT (destination address translation). For DNAT, you can refer to the DNAT by yourself. Only SNAT is discussed here.

As we all know, the basic function of a router is to forward IP packets at layer 3 (network layer). In fact, one of the key features of the router is NAT. NAT is especially important for routers on ISP links to common users.

Why NAT?

** The reason is simple: **IPv4 addresses are scarce. The huge demand for Internet access makes it impossible for AN ISP to provide an independent public IP address for each user. Therefore, an ISP usually connects users to the LAN so that multiple users share the same public IP address and each user is assigned an Intranet IP address. The router that connects the public network and LAN is called a gateway. NAT takes place on the gateway router.

PS: “P2P technology in detail (A) : NAT in detail — detailed principle, P2P introduction” this article is helpful for a deeper understanding of NAT principle.

4.2 Layer-3 Address Translation

Network-layer IP packets sent by hosts on the LAN to the public network are forwarded to the public network through the gateway, and address translation occurs during the forwarding process. The gateway changes the source IP address of the IP packet from internal IP address of the host to public IP address of the gateway.

* * such as: ** The Intranet IP address obtained by the LAN host is 192.168.1.100, and the public IP address of the gateway is 210.177.63.2. In the IP packet sent by the LAN host to the destination host on the public network, the source IP address field is 192.168.1.100. The field data will be modified to 210.177.63.2.

I’m sure you’ve guessed why: After receiving the IP packet, the target host on the public network needs to know the source IP address of the packet and sends a response packet to the source IP address. However, without NAT, the target host obtains the source IP address 192.168.1.100, which is obviously a private address that cannot be accessed on the public network. The target host could not send the response to the correct source host. After NAT is enabled, the gateway changes the source IP address of an IP packet to 210.177.63.2, which is a public IP address. The target host sends response packets to this public IP address (that is, the public IP address of the gateway router).

** But please note: ** If the data segment of the IP packet does not contain a TRANSPORT layer packet, but is a pure network layer packet, the response packet from the destination host cannot be accurately forwarded by the gateway to one of the multiple LAN hosts.

**PS: ** The exception is ICMP packets, which have Identifier fields in their headers to identify different hosts or processes. Gateways handle identifiers like transport layer ports mentioned below.

4.3 Transport Layer Port conversion table

In three layer address translation, we can ensure the host in the LAN IP packet to the destination smoothly issued to the public host, but returned from the destination IP packet is not accurate, to the specified LAN host (we can’t let the gateway IP broadcast message to all the LAN host, because it brings security and performance issues).

To solve this problem, the gateway router relies on transport layer ports, typically TCP or UDP ports, to generate a port translation table.

Let’s use an example to illustrate how the port conversion table works:

Assume that LAN host A192.168.1.100 needs to communicate with target host B210.199.38.2:80 on the public network through TCP. The public IP address of C, the gateway of A’s LAN, is 210.177.63.2.

The steps are as follows:

**1) ** LAN host A192.168.1.100 sends A TCP connection request, and the TCP port on A assigns 53600 to the system. The TCP handshake packet contains the source IP address and port 192.168.1.100:53600, and the destination IP address and port 210.199.38.2:80.

**2) ** Gateway C changes the original address and port of the packet to 210.177.63.2:63000, where 63000 is the temporary port assigned by the gateway.

**3) ** Gateway C adds a record to the port conversion table:

**4) ** Gateway C sends the modified TCP packet to destination host B.

**5) ** After receiving the packet, destination host B sends a response TCP packet. The response TCP contains the following information: source address and port 210.199.38.2:80, destination address and port 210.177.63.2:63000.

**6) ** After receiving this response packet from B, gateway C looks for the record in the port translation table. The record must meet the following conditions: Destination host IP address ==210.199.38.2, destination host port ==80, gateway port ==63000.

**7) ** Gateway C searches for this record, which shows that the Intranet host IP address is 192.168.1.100 and the Intranet host port is 53600.

**8) ** Gateway C Changes the destination ADDRESS and port of the packet to 192.168.1.100:53600.

**9) ** Gateway C then forwards the modified TCP packet to 192.168.1.100:53600, namely LAN host A. At this point, an exchange of transport layer data has been completed.

4.4 Here’s the problem

On gateway C, there are a limited number of ports (0 to 65535), and the maintenance of the port conversion table consumes system resources. Therefore, records cannot be added endlessly to the port conversion table. The gateway needs to delete expired records.

How do I determine which records are out of date?

** The gateway considers: ** Connections that have been inactive for a period of time are expired and should periodically detect and discard inactive connections in the conversion table. The gateway does not notify either end of the connection in any way about this discarding process.

This process can be better understood in the following figure:

▲ The figure above is from TCP Keepalive.

So the question becomes: * * * * if a client application due to business needs, the need to maintain a long connection with the server (for example, based on the TCP IM chat application), and if in a very long time in this connection is not any data exchange, the gateway will think this connection expired and will be discarded from the connection port conversion table. When the connection is dropped, the client and server are completely unaware of it. After the connection is discarded, the client cannot receive the data push from the server, and the data packets sent by the client cannot reach the server.

A concrete example to get a sense of the seriousness of the problem:

In a financial application, the client needs to fill in a large number of form data, after the client and the server establish TCP connection, the client terminal user will spend several minutes or even dozens of minutes to fill in the form related information, the terminal user finally fill in the form required information, click “submit” button.

As a result, the intermediate device has deleted the TCP connection from the connection table, and directly discards the packet or sends RST packets to the client. As a result, the application fails. As a result, all the work of the client and terminal user needs to be redone, bringing great inconvenience and loss to the user.

4.5 Solutions

The solution to this problem is to use the KeepAlive mechanism to keep the TCP connection alive and prevent the gateway from “killing” the long connection.

By using NAT as a concrete example, you can understand the need for KeepAlive in TCP.

5. Working principle of TCP Keepalive

5.1 Technical Principles

When a TCP connection is established, the end with TCP Keepalive enabled starts a timer. When the timer reaches 0 (tcp_keep-alive_time, which will be discussed later), a TCP probe packet is sent. This TCP probe packet is a pure ACK packet (the RFC1122#TCP keep-Alives specification suggests that it should not contain any data, but can also contain a meaningless byte such as 0x0), and its Seq number is the same as the previous packet, so the probe keepalive packet is actually outside the scope of window control.

If a given connection does not take any action for two hours (the default duration), the server sends a probe segment to the client, which must be in one of the four states in the table below.

To explain in detail:

**1) ** The client host is still running properly and reachable from the server. The client’s TCP response is normal, and the server knows that the peer is normal. After two hours, the server resets the keepalive timer.

**2) ** The client host has crashed and is being shut down or restarted. In either case, the client’s TCP is not responding. The server will not receive a response to the probe and will time out after 75 seconds. The server sends a total of 10 such probes, each 75 seconds apart. If the server does not receive a response, it assumes that the client host is down and terminates the connection.

**3) ** The client host crashed and has been restarted. The server will receive a response to its keepalive probe, which is a reset that causes the server to terminate the connection.

**4) ** The client is running properly, but the server is unreachable, similar to 2. All TCP can detect is that there is no response to the probe.

Intuitively, the interaction process of TCP KeepAlive is as follows:

▲ The figure above is from TCP Keepalive.

5.2 Usage Examples

The Linux kernel is used as an example. To use TCP Keepalive, you need to set the SO_KEEPALIVE socket option to take effect.

Accordingly, there are three important parameters:

  • 1) tcp_keepalive_time: the interval between the last data exchange and TCP sending the first keepalive probe packet when TCP keepalive is enabled, that is, the allowed continuous idle time, or the normal heartbeat sending period, the default value is 7200s (2h);
  • 2) tcp_keepalive_probes Indicates the number of times that keepalive probe packets are sent after tcp_keepalive_time if no acknowledgement is received. The default value is 9.
  • 3) tcp_keepalive_intvl: after tcp_keepalive_time, if no acknowledgement is received, the sending frequency of keepalive probe packets is continued. The default value is 75s.

The above is about the configuration of Linux kernel parameters, in fact, other programming languages have corresponding Settings.

For example, Java’s Netty server framework also provides interfaces:

ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .option(ChannelOption.SO_BACKLOG, 100) // Heartbeat monitor. ChildOption (channeloption.so_keepalive, true) .handler(new LoggingHandler(LogLevel.INFO)) .childHandler(new ChannelInitializer() { @Override public void initChannel(SocketChannel ch) throwsException { ch.pipeline().addLast( new EchoServerHandler()); }}); // Start the server. ChannelFuture f = b.bind(port).sync(); // Wait until the server socket is closed.

f.channel().closeFuture().sync();

**PS: **Java program can only set the SO_KEEPALIVE option, as for TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL and other parameters configuration, the application layer is not configurable.

6. TCP KeepAlive may cause problems

Keepalive technology is only an optional TCP protocol. Because improper configuration can cause problems, it is turned off by default.

Specifically, it may lead to the following problems:

  • 1) During a short period of failure, a healthy TCP connection may be disconnected due to a short period of network fluctuation when Keepalive Settings are not properly set;
  • 2) extra bandwidth and data consumption (which seems to be a non-issue in this day and age);
  • 3) Increased cost in the Internet environment of charging by traffic.

7. Limitations of TCP KeepAlive in the era of mobile networks

There is no denying that TCP, as the most important part of the TCP/IP protocol family, has contributed greatly to the development of the Internet (see “Once Upon a Time in Technology: TCP/IP Changed the World”).

However, in the era of mobile network, wireless communication is becoming more and more popular. As the TCP protocol invented in the middle of the last century, objectively speaking, it does have congenital deficiencies in some scenarios (see: “5G era has arrived, TCP/IP is old, still can live? ).

So, back to the question at the beginning of this article — “Why implement network KeepAlive/heartbeat at the application layer when TCP itself has KeepAlive?” .

Take the MOBILE IM application as an example:

  • 1) on the one hand, ISP network resources are more scarce, and the default 2-hour KeepAlive TCP protocol is almost impossible to “KeepAlive” IM long connections (in order to improve the utilization of wireless network resources, operators may reclaim idle network connections in minutes or even tens of seconds).
  • 2) On the other hand, the wireless network itself has the problem of weak network, even if the TCP connection is “good”, but in fact in the “suspended” state, it can not play the role of the long connection.

Therefore, it is inevitable that the IM application layer does the network maintenance (heartbeat mechanism) itself.

For more information on this topic, you can read the following articles in depth:

Why Does TCP – based MOBILE IM Still Need heartbeat Keepalive mechanism?

Mobile IM Developers must Read (1) : Easy to Understand the “weak” and “Slow” mobile Web

Mobile IM Developers must Read (ii) : Summary of the Most Complete Mobile Weak Network Optimization Methods ever

Introduction to Zero-Base Communication technology for IM Developers (13) : Why Cell phone Signal is Bad?

How hard is wireless Internet access on high-speed Trains?

What is the difference between TCP Keepalive and HTTP keep-alive?

Many people confuse TCP Keepalive with HTTP keep-alive.

Here is a brief introduction to HTTP keep-alive.

In HTTP/1.0, short connections were used by default. That is, each time the browser and server perform an HTTP operation, a connection is established, but the connection is broken at the end of the task. If an HTML or other type of Web page accessed by the client browser contains other Web resources, such as JavaScript files, image files, CSS files, etc. Each time the browser encounters one of these Web resources, it establishes an HTTP session.

However, from HTTP/1.1 onwards, long connections are used by default to preserve the connection feature. If HTTP is used for long Connection, the Connection and keep-alive fields are added to the response header.

As shown below:

The differences in TCP connection usage between HTTP 1.0 and 1.1 are shown below:

To sum it up in a colloquial way:

  • 1) HTTP keep-alive is to make TCP connections live longer, so that the same connection can be reused when multiple HTTP requests are initiated to improve communication efficiency.
  • 2) The KeepAlive mechanism of TCP is intended to detect whether the peer end of a connection is alive. It is a preservation mechanism for detecting the status of TCP connections.

(This article has been simultaneously published at: www.52im.net/thread-3506…)

9. Reference materials

[1] TCP Keepalive

[2] TCP KeepAlive and HeartBeat packets

[3] HTTP keep-alive differs from TCP Keepalive.

[4] TCP KeepAlive is different from HTTP keep-alive

[5] THE TCP connection detects Keepalive and heartbeat packets

[6] The research of TCP Keepalive (1) : NAT and keepalive mechanism

[7] Understanding TCP Keepalive connections

[8] Why does TCP – based mobile IM still need heartbeat keepalive mechanism?

[9] Mobile IM Developer Must-read (ii) : Summary of the most comprehensive Mobile Weak Network Optimization Methods ever

[10] Introduction to Zero-Base Communication technology for IM Developers (13) : Why Cell Phone Reception is Poor? A text is understood!