I. Background of the problem

Applications have a high number of TimeWait connection requests under high concurrency

# count command: # count connected, Status to "established netstat na | grep established | wc - l # to check 80 port connections netstat ant | grep -i" 80 "| wc - l # if you need to statistics the number of TCP connection each state of the connection netstat -an | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'Copy the code

After a short time, all time_waits disappear and are reclaimed. All ports and services are normal. That is, in high concurrency scenarios, the existence of TIME_WAIT connections is normal. In online scenarios, continuous high concurrency scenarios

  • Some partial TIME_WAIT connections are recycled, but new TIME_WAIT connections are created;
  • In extreme cases, there are a lot of time_waits

In the case of a large number of time_waits, because Linux is a file-based operating system, each socket needs to create a file to maintain, and each process can expand the number of files that can be opened is 65535. If the number is larger than this, If a Web server such as Nginx is used as a reverse proxy, an exception such as addres already in Use :connect occurs.

Ii. Problem analysis

  • A large number of ## short links exist
  • The special change is that in HTTP requests, if the connection header value is set to close, the [server] initiates the initiative to close the connection
  • In TCP, time_wait is set to MSL (maximum packet survival time) twice to ensure ACK resending and discarding delayed data.
  • Hold for 2 MSL times, i.e. 4 minutes;

3. Solutions

  • Client, HTTP request header, connection set to keep-alive, kept alive for a while: modern browsers generally do this
  • Server side,
    • Allows sockets in time_WAIT state to be reused
    Echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse Echo "1" > /proc/sys/net/ipv4/tcp_tw_recycleCopy the code
    • See time_wait time, set to 1MSL

* THE time_wait state is necessary

  • Reliably implement termination of TCP full-duplex connections: If the ACK is lost, the peer party resends the FIN request. Therefore, you need to maintain a TIME_wait state to process the resending FIN request in the section in which the TCP connection is closed.
  • Processing delayed packets: Because the router may jitter, TCP packets may arrive late. To prevent delayed TCP packets from being mistaken for new TCP connections, you need to keep the state unavailable before allowing new TCP connections to be created and wait for all delayed packets to disappear. Generally, it is set to MSL (maximum lifetime of packets), which solves the problem of “delayed TCP packets”.

Addendum: TCP three waves and four handshakes

For details, please refer to:

  • TCP three-way handshake and four-way wave

Specific schematic diagram:

  1. Three handshakes establish the connection process
  2. Wave four times to release the connection process

A few core questions:

  1. Is time_wait the state of “server side”? Or client status?

RE: time_wait indicates the status of the “active closing TCP connection” party, either “client” or “server” 2. In general, it is the state of the “client”; Do not actively close the connection on the Server side