This is the 19th day of my participation in the Genwen Challenge

If ❤️ my article is helpful, welcome to like, follow. This is the biggest encouragement for me to continue my technical creation. More previous posts on my personal blog

What is network protocol

Web crawler, as its name implies, is a crawler that walks between the Internet and collects information.

The Internet is a network of computers covering all types of network devices (such as common mobile phones, computers, servers), so how do computers communicate and pass information?

The communication between computers depends on network protocol, and the network protocol that the crawler most often comes into contact with and will interview documents is TPC/IP protocol at the transport layer and HTTP protocol at the application layer.

Getting to know both of them, whether it’s later screening questions or beating the interviewer, will be of great benefit

Relationship between HTTP and TCP/IP

According to the TCP/IP five-tier model, two points should be made clear:

  • The TPC/IP protocol is a transport layer protocol that describes how data is transmitted over the network.

  • HTTP is an application-layer protocol that deals with how to package data.

The TCP/IP protocol and HTTP protocol are related to each other.

We only use TCP/IP, and the source computer can also transfer data to the destination computer. But the data is transmitted with a specific encoding, format… If the source is inconsistent with the target computer, the data cannot be parsed and used.

Therefore, protocols such as HTTP (application layer) stipulate the compilation and parsing methods of transmitted data to ensure the normal transmission and use of data

“We could just use [transport layer] TCP/IP when we’re transferring data, but then without the application layer, you can’t recognize the content of the data, and you have to use the application layer if you want to make sense of the data being transferred.

There are many application-layer protocols, such as HTTP, FTP, and TELNET. You can also define application-layer protocols. The WEB uses HTTP as an application-layer protocol to encapsulate HTTP text, and then uses TCP/IP as a transport-layer protocol to send it to the network.”

So you can think of IP as a kind of highway that allows other protocols to travel on and find exits to other computers. TCP and UDP are “trucks” on the highway, carrying the cargo of protocols like HTTP, FILE transfer protocol FTP, etc. – a qualified expression from an old driver

After talking about their relationship, move on to the most important (and frequently asked) points about both of them

TCP/IP protocol

TCP three-way handshake

The TCP three-way handshake is as follows. This is done to ensure that the sending and receiving capabilities of the sender and receiver are normal.

The sequence number of TCP packets that need to be confirmed by the peer end must be consumed.

SYN requires peer acknowledgment, so SYN consumes a sequence number. However, ACK does not need to be sent next time, the corresponding ACK sequence number should be added 1

Why not twice?

Root cause: The receiving capability of the client cannot be confirmed.

The steps of the problem are as follows:

  1. You send a SYN packet to shake hands, but the packet remains in the network.

  2. The TCP timeout retransmission mechanism retransmits packets as packets are lost. The second handshake establishes a connection. Close the link after you send the message

  3. The stranded packet in Step 1 arrives at this point, and the server establishes the connection by default, but now the client is disconnected.

This is the waste of connection resources caused by two handshakes.

Why not four times?

Root cause: There is no guarantee that the last handshake will arrive, so it is not very useful

The three-way handshake is used to confirm the ability of the sender and receiver to send and receive.

Four handshakes will do, a hundred. But you can never be sure that the last handshake will arrive.

So three times is enough to confirm the ability of the sender and receiver to send and receive, and the number of times you send doesn’t mean much.

Why doesn’t the first two handshakes carry data, but the third one does?

Root cause: If the first two handshakes carry data, the sender and sender are more vulnerable

The steps of the problem are as follows:

  1. You click on an unknown URL(winning SMS) and request a link to a toxic server

  2. When the server sends you a lot of data on the second handshake, you will consume more time and memory to process the data

Or vice versa

  1. The hacker wants to attack your server, using the proxy to put a lot of data in the SYN packet in the first handshake

  2. Your server is bound to consume more time and memory space to process this data

  3. Then black switch to a proxy to continue the operation above, your server pawn

On the third handshake, the client is in the ESTABLISHED state and can confirm that the server is receiving and sending data properly. In this case, the client can carry data.

TCP wave four times

At the beginning both sides are in the ESTABLISHED state:

First wave: To disconnect the client, the client sends a FIN release packet to its TCP connection and enters the FIN-WaIT-1 state. The client cannot send packets but can only receive them. Wait for confirmation from the server.

Second wave: The server sends an ACK packet after receiving the FIN release packet from the client.

  • The server enters close-wait to disable the WAIT state. In this case, TCP is half-closed. To complete the transmission of unsent data.

  • The client receives an ACK packet from the server and changes to the FIN-Wait2 state. Procedure

Third wave: After data transmission is complete, the server sends a FIN release packet to the client and enters the last-ACK state.

Fourth wave: After receiving the FIN release packet from the server, the client sends an ACK packet to the server. When the client enters the time-wait state, the client waits for two MSLS (Maximum Segment Lifetime). Otherwise, the client resends the ACK packet

Why four waves?

Root cause: After receiving the FIN from the client, the server sends an ACK to indicate that it has received the FIN from the client and then sends the rest packets. The server FIN then sends it to the client, which takes only four waves

Why can’t it be three waves?

Root cause: The data of packets to be sent by the server is variable, causing the client to repeatedly resend the client FIN

If the three-way handshake is equivalent to merging the second and third packets (steps 3 and 4 above) and waiting for the server to finish sending the remaining packets (with more or less data), the client may mistakenly think that the client FIN did not reach the server during this period and continuously resend the client FIN

Wait for 2MSL meaning

Root cause: The last ACK packet sent by the client can reach the server.

What is MSL?

Each TCP implementation must select a Maximum Segment Lifetime (MSL), which is the Maximum time that a packet Segment is allowed to live in the network before being discarded. This time is limited because TCP segments travel across the network as IP datagrams, and IP datagrams have TTL fields that limit their lifetime.

To ensure that the last ACK packet segment sent by the client can reach the server. The ACK may be lost. As a result, the server in the last-ACK state cannot receive the fin-ACK acknowledgement packet. The server will timeout to retransmit the FIN-ACK, and the client will retransmit again for confirmation, with the restart time waiting for a timer (2MSL).

So wait for 2MSL to be used respectively:

  • An MSL ensures that the last ACK message from the active closing party in the four waves will finally reach the peer end

  • One MSL ensures that the peer end can receive FIN packets that have not received ACK retransmission

Semi-connected queue and SYN Flood attacks

Semi-connected queue

When a client sends a SYN to a server and the server replies with a SYN and ACK, the client enters the SYN_RCVD state and the connection is pushed into the SYN queue, also known as the half-connection queue.

Full connection queue

When the three-way handshake is complete, the connection is pushed to another TCP-maintained Queue, known as the Accept Queue, to be taken by the specific application.

Mechanism of SYN Flood attacks

The client forges a large number of non-existent IP addresses in a short period of time and frantically sends SYN to the target server. Cause:

  • A large number of connections are in the SYN_RCVD state, filling the entire half-connection queue and unable to process normal requests.

  • If a large number of connections use non-existent IP addresses, the server cannot receive ACK packets from the client for a long time. As a result, the server constantly resends packets until the server resources are exhausted.

How do you find yourself under attack

  • The access to the server is denied or times out, and the TCP service cannot be provided

  • Check the system by running the netstat -an command. A large number of SYN_RECV connections are found

How do I cope with SYN Flood Attacks?

  • After receiving repeated SYN packets from an IP address within a short period of time, the device considers the attack as an attack and records the IP address information to seal the IP address. This may affect the customer’s access.

  • Shortening the SYN Timeout time and the half-connection queue time can double the load of the server. However, an excessively low SYN Timeout setting may affect the normal access of customers.

  • The SYN Cookie technology is used to reply to the client with the Cookie in the second handshake. When the client replies with ACK, the Cookie value is added. The server verifies that the Cookie is legitimate before allocating connection resources.

  • SYN Proxy firewall is used to authenticate SYN requests that attempt to pass through.

Function of timestamp in TCP packets

Timestamp is an optional header of a TCP packet, consisting of 10 bytes in the following format:

Kind (1 byte) + length(1 byte) + info(8 bytes)

Where kind = 8, length = 10, info consists of two parts: timestamp and timestamp echo, each containing 4 bytes.

So what are these fields for? What problems are they used to solve?

TCP timestamps address two major problems:

  • Calculate round-trip Time (RTT)

  • Prevent serial number wrapping problems

Calculate the round-trip delay RTT

In the absence of a timestamp, the problems encountered in calculating RTT are shown in the following figure:

If the first packet is sent as the start time, the problem in the left picture will appear. RTT is obviously too large, so the start time should be the second one.

If the second packet is sent as the start time, it will cause the problem in the right picture. RTT is obviously too small, so the start time of the first packet should be adopted.

In fact, no matter whether the start time is based on the first or second packet, it is not accurate.

The introduction of time stamps at this time is a good solution to this problem.

For example, a sends a packet S1 to B, and B replies a packet S2 containing ACK to A:

  • Step 1: When A sends to B, the content stored in timestamp is the kernel moment TA1 when host A sends.

  • Step 2: When B replies s2 packet to A, TB is stored in timestamp of host B, and the timestamp echo field is ta1 resolved from S1 packet.

  • Step 3: After A receives S2 message of B, the kernel time of host A is TA2. In the timestamp Echo option of S2 message, ta1 can be obtained, which is the initial sending time of s2 corresponding message. And then you just take ta2-TA1 and you get RTT

Prevent serial number winding problems

The seQ serial number has a range and the range of serial numbers is actually between 0 and 2 ^ 32-1,

If the last packet with the same sequence number is stuck in the network, it will cause confusion when the next packet with the same sequence number is sent

Timestamp is a good solution to this problem, because every time the packet is sent, the kernel time of the sending machine is recorded in the packet, so even if the serial numbers of the two packets are the same, the timestamp cannot be the same, so that the two packets can be separated.

Confused window syndrome

If, due to heavy load, the receiver cannot process so many bytes that it is too late to fetch the data in the receive window, then the sender’s send window will become smaller and smaller.

At the end of the day, if the receiver frees up a few bytes and tells the sender that there is now a few bytes of window, the sender will send those bytes anyway, which is called confused window syndrome.

Remember, our TCP + IP header has 40 bytes, and it’s uneconomical to transfer those bytes of data.

Thus, the phenomenon of confused window syndrome can occur on both sender and receiver:

  • Recipients can advertise in a small window

  • And the sender can send small data

Therefore, to solve the confused window syndrome, it is ok to solve the above two problems

  • Let the receiver not notify the sender of the widget

  • Use the Nagle algorithm to let the sender avoid sending small data

TCP sliding window with flow control

TCP sliding window

TCP sliding Windows are divided into receiving Windows and sending Windows, which are composed as follows:

Here’s my understanding of sliding Windows:

  1. A window is a sequence of bytes that can be sent by a sender. Its contiguous range is called “window”.

  2. Sliding means that the range of allowed messages can be changed as the message is sent, by “sliding” in order.

The sliding window represents the data transmission capability of the sender and the receiver. In essence, it is a traffic control measure at the transport layer. The receiver informs the sender of its window size, so as to control the sending speed of the sender and prevent the sender from being overwhelmed by the sending speed

Let’s take an example to describe the implementation process of TCP sliding window

TCP flow control implementation process

First, the TCP three-way handshake is completed, and the sender and receiver establish a connection.

  1. Receiver notifies sender, ownExpect to receive packet serial number ACK= 0,Receive window SIZE SIZE= 10. The sender builds its own sending window according to ACK and SIZE

  1. The sender sends data packets with sequence numbers 1 to 10. Assume that the receiver receives data packets with sequence numbers 1 to 3, 5 to 10, and 4 not

  2. Receiving an ACK packet indicates that it has received packets with serial numbers 1 to 3, and caches packets with serial numbers 5 to 10 (ensure sequence, and generate a hole to store packets with serial numbers 4).

  3. After receiving an ACK, the sender cuts serial number 1 to 3 packets from sent and to be acknowledged to sent and acknowledged. Assuming that the SIZE of the notification remains the same, the window moves to the right, creating some new Spaces that the receiver is allowed to send

  1. If the lost data packet with serial number 4 exceeds a certain period of time, TCP retransmits the data packet (retransmission mechanism), and 4, 5, and 10 packets are confirmed as successful retransmission. If not, 5~10 will also be discarded

The above five steps are repeated continuously. As the window keeps sliding, the whole data stream will be sent to the receiver. In fact, the SIZE notification of the receiver’s window will also change. The schematic diagram is as follows:

Congestion window and congestion control

Here is another point of knowledge, the above mentioned sliding window and flow control is to send and receive two days of device traffic problems.

But the reality is that the sender and the receiver are separated by an indefinite number of routes, switches, and network devices.

Traffic control can prevent the sender’s data from filling the receiver’s cache, but does not take into account the network’s situation. Once network congestion occurs, if a large number of packets continue to be sent, delay, loss, and TCP timeout retransmission will further aggravate network congestion into a vicious cycle.

To avoid this, TCP has congestion Windows and congestion control, which takes into account the transmission capacity of intermediate devices, network conditions,

Relationship between congested Windows and sliding Windows

Congestion window CWND is a state variable maintained by the sender, which changes dynamically according to the congestion degree of the network.

Congestion window CWND change rules are as follows:

  • No congestion CWND increase occurred in the network

  • Network congestion CWND decreased

Therefore, the actual relationship between the sending (sliding) window SWND and the receiving (sliding) window RWND mentioned above is as follows:

SWND = min(CWND, RWND) # The minimum value of the congestion window and the receive (sliding) window

When timeout retransmission occurs on the sender, the network congestion occurs by default.

Since congestion occurs, it has to be maintained. Otherwise, it’s not an option to keep plugging away. Here is an example of the control algorithm for congestion control

Congestion control algorithm

There are four congestion control algorithms:

  • Slow start

  • Congestion avoidance

  • Congestion occurs

  • Fast recovery

Slow start

After the TCP three-way handshake is established, the sender increases the number of packets to check whether the network is congested.

Slow start has a very simple rule:

After the sender and sender initialize their own CWND value of the congestion window. The congestion window CWND value increases by 1 each time the sender receives an ACK.

However, it will not continue indefinitely (or congestion will still occur). After the congestion window CWND reaches the slow start threshold ssthRESH (slow Start threshold), the following congestion avoidance algorithm will control the size of CWND

  • When CWND < SSTHRESH, the slow start algorithm is used.

  • When CWND >= SSTHRESH, the congestion avoidance algorithm is used.

Ssthresh is typically 65535 bytes in size.

Congestion avoidance

When the congestion window CWND reaches the slow start threshold SSthRESH, the rules for automatically entering the congestion avoidance algorithm are as follows:

CWND increases by 1/ CWND each time an ACK is received.

That is, a round trip time RTT down, received CWND ACK, the final size of the congestion window only increased by 1.

The fast retransmission

During TCP transmission, if packet loss occurs, that is, when the receiving end finds that the data segment does not arrive in sequence, the receiving end repeatedly sends ACK packets of the previous packet.

When the receiver finds that an intermediate packet is lost, it sends an ACK of the previous packet. Then the sender realizes that the lost packet will be retransmitted quickly without waiting for a Retransmission TimeOut (RTO).

TCP considers this situation not serious, because most of them are not lost, only a small part is lost, then ssTHRESH and CWND change as follows:

  • CWND = CWND /2, that is, set to half of the original

  • ssthresh = cwnd

  • The fast recovery algorithm is displayed

Selective retransmission

Then you may ask, if retransmission is required, do you retransmit only “lost packets” or “all from the lost packets”?

When packets are lost, the TCP design allows the receiving end to add the SACK attribute to the next ACK packet to record which packets are received and which are not, and then retransmits them accordingly.

After receiving the ACK packet from the server, the sender will resolve the Left Edge (the first serial number of the received discontinuous block) and Right Edge (the last serial number of the received discontinuous block +1) in the SACK attribute to selectively retransmit the lost packet

This is Selective Acknowledgment (SACK)

SACK structure is as follows:

Fast recovery

When the sender retransmits timeout, the network is considered congested.

This will enter the quick reply state, the sender will produce the following changes:

  • cwnd = cwnd/2

  • CWND began to increase linearly

The four main algorithms for congestion control are all finished here

Nagle algorithm and delayed validation

Nagle algorithm

Imagine a scenario where the sender keeps sending tiny packets of 1 byte to the receiver, and it takes 1,000 times to send a thousand bytes. Such frequent sending has problems. Not only RTT consumption of transmission delay, but also time consumption of sending and confirmation, frequent sending and receiving brings huge delay.

Nagle algorithm is to avoid the frequent sending of packets, and its rules are as follows:

  • When sending data for the first time, there is no wait, even a 1byte packet is sent immediately

  • If one of the following conditions is met, you can send:

  • Packet Size reaches maximum Segment Size (MSS)

  • Acks for all previous packets have been received

Delay to confirm

Imagine a scenario where I receive a packet from the sender and then receive a second packet in a very short period of time. Should I reply one by one, or wait a little and combine the ACKS of the two packets and reply together?

Delayed ack this will be involved, the ACK will be merged with a slight delay, and it will be returned to the sender.

TCP requires that the delay be less than 500ms. Common operating systems implement this delay less than 200ms.

However, some scenarios can not be delayed confirmation, received immediately reply:

  • A packet larger than one frame is received and the window size needs to be adjusted. Procedure

  • TCP in Quickack mode (set by tcp_in_quickack_mode)

  • Out-of-order packages found

Note that:

The Nagle algorithm means delayed sending, and delayed acknowledgement means delayed receiving. Using both together can cause greater latency, which can cause performance problems.