1. Introductory words

TIME_WAIT is an important part of the reliability design of TCP protocol. Although the reliability is enhanced, a large number of TIME_WAIT will be generated in high concurrency scenarios. The port can be used during peak hours.

This article only does a simple learning test, does not guarantee the comprehensiveness and correctness of the content, do not easily modify the formal environment kernel configuration

The tcp_TW_reuse and tcp_TW_RECYCLE configurations of the Linux kernel are tested today

2. Set up the experimental environment

To simulate the network situation, we set the available port range to 81 only

sysctl -w "net.ipv4.ip_local_port_range=81 81"
Copy the code

3. Test the default configuration

Access the local Nginx service

Curl http://127.0.0.1 curl http://127.0.0.1 curl: (7) Failed to connect to 127.0.0.1: Cannot assign requested addressCopy the code

Viewing network Status

Netstat napo | grep 127.0.0.1Copy the code

We can see that the first time normal, in 2MSL time, the next access will be unable to assign the request address error. In Linux, the TIME_WAIT time is 60 seconds and cannot be changed

TIME_WAIT Expiration time macro definition

//include/net/tcp.h
/* how long to wait to destroy TIME-WAIT
 * state, about 60 seconds   */
#define TCP_TIMEWAIT_LEN (60*HZ) 
Copy the code

Verify the time

You can see that the red line is the countdown of time_wait, when the time is up, the port will be released automatically.

4. Configuration instructions for the TIME_WAIT case

Open tcp_tw_reuse

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
Copy the code

test

We can observe that ports in TIME_WAIT state can also continue to complete requests without changing the state and timing of TIME_WAIT itself.

This proves that tcp_TW_reuse can be enabled for port reuse when local ports are about to run out.

It’s not SO_REUSEADDR socket option. SO_REUSEADD is used for binding socket to LISTEN state even if it is in TIME_WAIT state.

Open tcp_tw_recycle

Open tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle 
Copy the code

test

The results were good and the request tests were completed as well. But how is it different from tcp_tw_reuse?

Compare the netstat detection with tcp_TW_reuse enabled to see what the difference is.

We found that there was no TIME_WAIT

Linux kernel judge code

When recycling is enabled, our timeout value is rto, which is a very short time, otherwise TCP_TIMEWAIT_LEN. Remember the time defined by the macro at the beginning of this article? Yes, this is the 60s.

5. Difference between tcp_TW_reuse and tcp_TW_recycle

Both parameters seem to work well, at least in the test results.

parameter function
tcp_tw_reuse Reuse does not change the TIMEWAIT state
tcp_tw_recycle Recycle, the fastest time to recycle

Net.ipv4. tcp_timestamps is enabled by default

How does tcp_TW_reuse work

If tcp_TW_reuse is enabled and the client sends a timestamp greater than the latest timestamp recorded for the previous connection to the kernel, Linux will re-use the existing connection in the state in time-wait for the new outbound connection. Outgoing connection time-wait in state can be reused after only one second.

How does tcp_TW_RECYCLE work

If tcp_TW_RECYCLE is enabled, the kernel remembers the timestamp of the last packet sent by the client. If the timestamp of the last packet sent by the client is smaller than the timestamp recorded by the kernel, the packet is discarded. In this case, in NAT mode, multiple machines delay or send packets at the same time. It’s dangerous. It creates anomalies that are hard to detect.

Net.ipv4. tcp_timestamps=0 net.ipv4.tcp_timestamps=0 net.ipv4.tcp_timestamps=0

6. Conclusion:

On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are absolutely certain that you will never have a NAT device in your service. Net.ipv4. tcp_tw_reuse is enabled for outgoing connection connections.

On the client side, enabling net.ipv4.tcp_tw_reuse is another nearly secure solution

Also, when designing the agreement, don’t let the client close it first. The client does not have to deal with the state that pushes the time-wait state to a server better suited to handle the problem.

Therefore, you are advised to enable tcp_TW_reuse and disable tcp_TW_RECYCLE.

7. Attach the TCP status diagram

Reference Documents:

Vincent. Bernat. Ch/en/blog / 201…

Linuxsyseng.blogspot.com/2017/03/the…

More exciting technology, please pay attention to the public number (dull Bear technology road):