This paper is participating in theNetwork protocols must be known and must be known”Essay campaign

When we studied networks and the network transport layer in school, we probably always found it boring and hard to understand. But in fact, in production activities, this is a very common problem, if you do not understand, probably will be more confused.

Problems in the production environment

Talk about some of the TCP layer issues I encountered this year.

  • Q1: Short and long connection selection?
  • Question 2: The connection timed out. Why is the timeout time about 128s
  • Fault 3: The system is unreachable and port 80 is disconnected. However, port 80 can be checked locally. Why?
  • Problem 4: Many client connection pools are in close-wait?

The transport layer

To fully address those issues requires a very deep understanding of the transport layer protocols. The network layer may be a bit distant from software developers, but the transport layer, especially the widespread use of TCP, is relevant to our work.

Purpose of the transport layer

  • It includes the application layer, transport layer, network layer, link layer, and physical layer from top to bottom.

  • The application layer corresponds to different data
  • At the IP layer, packets can be correctly routed to the corresponding host

We need a mechanism to map different applications to the network layer of the same host, and we need to ensure accurate data arrival.

Distinguish between different applications -UDP

The transport layer initially introduced the design of ports to distinguish different applications.

  • Each application corresponds to a port, and the port information is encapsulated in a packet, so that the application to which the packet belongs can be identified.
  • Therefore, AS a simple protocol at the transport layer, UDP packets have a simple format.

  • UDP adds port information almost exclusively
  • A port is a 16-bit number

Ensure transmission quality -TCP

UDP does not ensure the accurate transmission of data, without quality assurance, thousands of packets lost, the network layer is not retransmission. Therefore, a new protocol TCP is also designed for transmission. In addition to port mapping, some processing mechanisms are added between the application layer and the network layer to ensure the quality of transmission.

The quality of transmission for an application is really two things:

  • All data sent by the peer end is received
  • All data sent by the peer end is received in order

  • C1 and C2 represent two clients that interact with APP1.

  • Assuming that APP1 sends data to C2, app1’s transport layer ensures that all data is sent to C2, ensuring that no packet loss event occurs

    • No packet loss is not really no packet loss, but if the lost packet can be retransmitted, to ensure that the data is eventually received.

  • Assuming C1 is sending data to APP1, the data needs to be sent in the correct order for App1 to receive

Key points: The specific working mechanism of TCP

What does TCP do

As described above, TCP does two main things:

  • To prevent the packet loss
    • Packet retransmission
      • Normally, an ACK signal is sent to the sender upon receipt of a packet
      • The timeout retransmission mechanism is used if no message is received
    • Reduce the packet loss
      • Tell me about your window: feel the processing power of the other side
      • Packet loss causes and optimization: Feel network congestion and control the transmission rate
  • Ensure the order in which packets arrive
    • Use serial numbers: Ensure that packets are delivered in the correct order

Basic probing – Establishing a connection

As mentioned above, TCP first tests the sending and receiving capabilities of the two peer ends as follows:

Mainly to test and confirm:

  • Verify A’s ability to send data
  • Verify B’s ability to receive data
  • Verify B’s ability to send data
  • Confirm A’s ability to receive data

This exploratory process, also known as a three-way handshake, establishes a TCP connection between two applications through a three-way handshake.

Specific processes and states

TCP connections are abstracted by the kernel for application use.

We can look at the process in more detail, before, during, and after the connection is made.

The state of three-way handshake changes over time as follows:

A three-way handshake is a three-way process:

  • 1.A sideSend SYNC packet: SYN, seq=x, sendThe system enters the sync-sent state

    It is worth noting that seq=x is described in more detail below

  • 2. After receiving the SYNC packet, the listener sends SYN,ACK, SEq = Y,ACK =x+1 to enter the sync-RCVD state
  • Seq =x+1, ACK =y+1Establishedstate
    • RTT indicates the time between sending data and receiving ACk
  • 4. The listener receives the ACK packet and enters the Established state

Accept and LISTEN

The server kernel uses the Accept system call to help establish the connection. After the server calls accep, it is in LISTEN state and can wait for the client to establish the connection. Netstat is used to check the connection in LISTEN state.

# netstat -alpntProto Recv -q Send -q Local Address Foreign Address State PID/Program name TCP 0 0 0.0.0.0:11110 0.0.0.0:* LISTEN 26523/javaCopy the code
  • *0.0.0.0:** indicates that all ports on the network are accepted
  • The kernel uses objects like sockets for real TCP connections, but the accept socket is special and does not establish a TCP connection.

ESTABLISHED

After three successful handshakes, the connection is saved in memory.

# netstat -alpntProto Recv -q send-q Local Address Foreign Address State PID/Program Name TCP 0 0 9.13.73.14:38604 9.13.39.104:3306 ESTABLISHED 26100/ Java TCP 00 9.13.73.14:32926 9.21.210.17:8080 ESTABLISHED 26100/ JavaCopy the code
  • ESTABLISHED: The connection is ESTABLISHED
  • The connection information includesLocal IP Indicates the port addressandRemote IP port

    • Local 9.13.73.14:38604 and remote 9.13.39.104:3306 are connected

    • A connection was established between local 9.13.73.14:32926 and remote 9.21.210.17:8080

Answer Question 3.

Fault 3: The system is unreachable and port 80 is disconnected. However, port 80 can be checked locally. Why?

Once our service was unreachable, that is, port 80 was disconnected. However, after logging in to the machine to send port 80, the listen state was still normal, but we found that many connections were in sync-RCvd state. After checking the log, we found that the system was Out of Memory. It is reasonable to suspect that there is a memory problem with the service connection setup.

TCP packet format and information exchange

All datagrams sent in the three-way handshake are TCP packets. The FORMAT of TCP packets is as follows:

It can be seen from the packet format and the three-way handshake process above:

  • The SYN bit is 1, indicating that the packet is of the SYN type and is used for handshake

    • The SYN, seq=x SYNC packet is sent by the initiator
    • After receiving the SYNC packet, the listener also sends its own SYN packet: SYN, seq=y
    • The start sequence x and y of the initiator and listener are 32 bits, which are randomly generated by the system
    • After the initial sequence number is exchanged in SYN packets, the sequence number increases monotonically
    • Is the initial sequence number ISN still used to close the connection?

  • The ACK bit is 1, indicating that the packet is an ACK packet

    • 32-bit confirmation number
    • The listener receives the SYNC package and also sends an ACK, ACK =x+1, all bytes less than x+1 have been received, ** expects to receive the next package seq=x+1
    • The initiator receives SYNC and sends ACK. ACK =y+1 and all bytes smaller than y+1 have been received. Expect to receive seq=y+1 packets ** next time

  • Data for packets used for SYN and ACK connections is null

In addition, the packet format also contains the following information:

  • The window size of the received SYN packet represents the size of the receiving window of the SYN packet
  • The RST bit is 1: it can be used to force disconnection
  • The PSH bit is 1: indicates that the packets should be sent to the upper-layer application immediately after they are received
  • MSS: indicates the maximum segment allowed by TCP to receive packets from the peer.

After the connection is established, data is transferred

Once the connection is established, data can be transferred.

Review the two main things and basic ideas that TCP guarantees:

  • The serial number SEQ is used to ensure that packets are delivered in the correct order
  • Ack signals and timeout retransmission mechanisms prevent packet loss
  • Use Windows to reduce packet loss

The process of establishing the connection has set the stage for this. Here’s how it works.

Timeout retransmission and ACK

  • If the sent data has not received ack, it will be retransmitted. Determine how long it will take to retransmit the data

  • If the receiver receives the same SEQ data more than once, it discards it
  • It needs to be bigger than RTT, but not too big
  • RTT is dynamically changing, kernel sampling, using RTO to calculate

Each time a timeout retransmission occurs, the next timeout interval is set to twice the previous value. If multiple times out, it indicates that the network environment is poor and it is not suitable for frequent and repeated sending. In this case, the maximum number of retransmission times can be set.

Question number two

Question 2: The connection timed out. Why is the timeout time about 128s

The initial 1S timed out, and then because the system set the number of retransmissions to 6, 6 retransmissions, 1+2+4+8+ 16+32+64 add up to about 128s.

Acks can be submitted in batches

The ACK of SEq =4 is not correctly received. However, if the ACK of SEQ =6 is received within the timeout period, seQ before 6 has correctly received the data of SEQ =4 and will not be retransmitted

A sliding window and a pointer to the right

The socket buffer at the sending end is shown as follows:

  • Data was sent and received for ACK confirmation
  • Data that was sent but did not receive an ACK acknowledgement
  • Not sent but the total size is within the receiver’s processing range
  • Not sent but the total size is out of the receiver’s processing range

Move the window to the right,

  • The sending window in the figure is related to the size of the window in the received ACK packet
  • When the ACK pointer moves right, the available window becomes larger
  • If the seQ pointer moves right, the window becomes smaller

The socket buffer at the receiving end is shown as follows:

  • ACK package, ACK =16, and (window=16)
  • Data that has been successfully received and acknowledged
    • If the ACK pointer moves right, the window can still move right
  • The receive window is the data that has not been received but can be received
    • The available window is related to the application’s ability to fetch data, and if the application keeps not fetching data from the socket buffer, the receive window becomes smaller and smaller
    • Sliding Windows are not static. When the receiver’s application reads data very quickly, the receive window can fill up quickly.
    • By using Windows, you can control flow, reduce unnecessary packet loss, and reduce network congestion.

Order guarantee

As shown in figure:

Suppose that the receiving end does not receive the packet 16 and 17, but receives the data from 18 to 27, it does not send an ACK to the sender. When the sender times out and retransmits the packet 16 and 17, and the receiving end receives the packet, the receiving end returns the packet with ACK =28 to the sender.

  • The ACK pointer can only move right, not back
  • This ensures sequential delivery

Summary of transmission

We mentioned earlier that the main function of TCP is to prevent packet loss and ensure the order of packet arrival. This section proves that TCP does achieve this goal by describing the specific working mechanism of TCP.

Design of high concurrency systems and closed connections

Finally finished the establishment of TCP connection and transmission of the basic principles and process, that can be relieved.

Closing a connection is part of TCP, but it’s not as important as what TCP is supposed to do.

But recently, in production activities, people are also paying attention to TCP’s four-wave process. The main reason is that there is a lot of concurrency in the system today.

The TCP connection is an important factor to measure the concurrency of the system. If the connection is closed and abnormal, the system will be affected.

Regarding the impact of connections on concurrency, you can analyze the question raised at the beginning of the analysis.

Q1: Short and long connection selection?

Long links VS short links

  • Short connectionGenerally, it is only passed between clients/serversAt a time pleaseFor operating
    • Either side can initiate close
    • Short connections are easy to manage, and all existing connections are useful connections without additional control.
    • Usually when the browser accesses the server, it’s usually a short connection.
  • A long connection
    • After a read/write operation, the connection between the Client and server is not closed. Subsequent read/write operations continue to use the connection.
    • So a connection can last for days, months, years, or even longer, as long as there are no anomalies or the user (application layer) actively closes it.
    • Long connections can save TCP establishment and closure operations and reduce the impact of network congestion.
    • Reduce CPU and memory usage because you don’t have to make and close connections as often.
    • Excessive connections affect server performance and concurrency.

So, how to choose long and short connection?

  • So forIf the number of concurrent requests is large and the request frequency is low, short connections are recommended.
    • For the server, a long connection consumes resources on the server
    • If you have hundreds of thousands of connections, millions of connections, the strain on the server side can be so great that it can crash
  • For small concurrency,High performance requirementsIs recommended to select the long connection
    • For example, mysql connection pools

TCP Closes the connection

The choice of long and short connections is so important that failing to close the connection properly can cause serious problems. TCP designs a complete connection closing mechanism. The interaction process for closing a connection is as follows:

  • Close the connection initiator initiates the first FIN at fin-wait1
  • Close the connection passiveKernel code replyACK, can also send data, at this pointClose-WAITstate
    • The passive closes the connection and waits for the application to send a FIN. If the upper application does not send a FIN, it can continue to send data
  • After receiving an ACK, the connection initiator is in fin-Wait2 state and can receive data and stop sending data
  • Close the connection to the passive until the application issues a FIN and is in last-ACK state
  • Disable the connection initiatorIf a FIN is received, an ACK will be sent, and the host will be inTIME-WAITAt this time
    • If the passive receives an ACK, the connection is closed
    • If the connection is closed and the passive party does not receive an ACK, the FIN is retransmitted

Time-wait How long to WAIT

MSL isMaximum TTL of packetsTCP’s TIME_WAIT state is also called the 2MSL wait state. TCP’s TIME_WAIT state is called the 2MSL wait state.

The main purpose of waiting for 2MSL is to prevent the other party from receiving the last ACK packet. If the ACK packet times out, the other party will send the FIN packet for the third handshake.

If the previous interaction was abnormal, the FIN that received the retransmission takes up to 2MSL, so the conclusion is to wait 2MSL

One last question

Problem 4: Many client connection pools are in close-wait?

Once, a data synchronization application synchronized a large amount of data from a server, the process was slow, so I thought how to speed up the synchronization speed.

  • First look at the network connections and see that there are many more connections in close_wait state in the pool of 20 connections

We know from the above analysis that close-wait means that the connection is closed because the connection is idle for too long, but the connection pool of our application has not sent a FIN release packet.

You can see that the number of connections is not a bottleneck, and to increase concurrency, you can increase the number of concurrent threads.

conclusion

This article provides a detailed description of the transport layer and TCP protocol to answer some of the TCP related questions encountered in production.