GIF diagram! Is the data sent after the code successfully sends?

Today is another day of overwhelming demand.

Does anyone know where to sign up for the chicken soup class where “I got the product to cut 18 needs for me in 3 sentences”?

Forget the “Understand applause” class. It takes too much work.

Anyway, back to our main topic today, let’s get to the table of contents of this article.

Is the data sent after the code successfully sends?

Before answering this question, you need to understand what a Socket buffer is.

The Socket buffer

What is a socket buffer

When programming, to establish a connection to an IP, we need to call the socket API provided by the operating system.

A socket can be understood as a file at the operating system level.

We can do some methods on this file.

With the LISTEN method, you can have your program act as a server to listen for connections from other clients.

With connect, you can connect to the server as a client.

Use send or write to send data, and recv or read to receive data.

After the connection is established, the socket file acts as an “agent” to the remote machine. For example, if we want to send something to a remote server, all we need to do is write to the file.

After writing this file, the operating system kernel will do the rest of the sending.

Since it is written to the operating system, the operating system needs to provide a place for the user to write. The same goes for receiving messages.

This is the socket buffer.

The user sends the message to the send buffer.

The user writes to the RECv buffer when he receives the message.

That is, a socket has two buffers, one for sending and one for receiving. Because these are first-in, first-out structures, they are sometimes called send and receive queues.

How to observe the socket buffer

To view the socket buffer, run the netstat -nt command on Linux.

# netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0     60 172.22.66.69:22         122.14.220.252:59889    ESTABLISHED
Copy the code

As indicated above, there is a TCP Proto connection, Local Address and Foreign Address IP information, and the State is connected.

Send -q is the Send buffer. The following number 60 indicates that 60 bytes are not sent in the Send buffer. Recv -q represents the receive buffer, which is empty and the data has been received by the application process.

The TCP part

[GIF buffer sending and receiving process, TCP execution sending and receiving process]

After we set up a connection using TCP, we usually use SEND to send data.

int main(int argc, char *argv[])
{
    / / create a socket
    sockfd=socket(AF_INET,SOCK_STREAM, 0))
     
    // Establish the connectionConnect (sockfd, server IP information,sizeof(server))  
   
    // Execute send to send a message
    send(sockfd,str,sizeof(str),0))  

    / / close the socket
    close(sockfd);
  
    return 0;
}
Copy the code

The above is a piece of pseudocode, just to show the general logic, in which we typically execute the send method after establishing a connection. So at this point, the message is immediately sent to the peer machine?

Will the bytes sent by send be sent immediately?

The answer is no! After sending, the data is simply copied to the socket buffer. To what time will send data, send how much data, all listen to the operating system arrangement.

In the user process, the program enters the kernel mode from the user mode by operating the socket, and the send method sends data all the way to the transport layer. The tcp_sendMSG method is called when the TCP protocol is recognized.

// net/ipv4/tcp.c
// A lot of logic has been omitted
int tcp_sendmsg(a)
{  
  // If there is room for the data
  if (skb_availroom(skb) > 0) {
    // Try to copy the data to be sent to the send buffer
    err = skb_add_data_nocache(sk, skb, from, copy);
  }  
  // The following is the logic code to try to send, omitted first
}
Copy the code

In tcp_sendMSG, the core task is to organize the data to be sent into the send buffer in order, and then determine whether to send the data based on the actual situation (such as congestion window, etc.). If no data is sent, it is returned directly at this point.

What happens if the buffer is full

In the case mentioned earlier, there is enough space in the send buffer to copy the data to be sent.

What happens if the send buffer runs out of space, or is full, and the send is executed?

There are two cases.

First, when a socket is created, it can be set to be either blocking or non-blocking.

int s = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, IPPROTO_TCP);
Copy the code

For example, with the above code, you can set the socket to non-blocking (SOCK_NONBLOCK).

When the send buffer is full, if the socket is still sent

If the socket is blocked, the program will wait until new cache space is released, then continue to copy data into it, and then return.

If the socket is nonblocking, the program willImmediately returnaEAGAINError message, meaningTry againNow the buffer is full. Don’t wait. Try again later.

We can take a brief look at how the source code is implemented. Let’s go back to the tcp_sendMSG sending method.

int tcp_sendmsg(a)
{  
  if (skb_availroom(skb) > 0) {
    / /.. Balabla is executed if there are enough buffers
  } else {
    // If there is no space in the send buffer, wait until there is space
    if((err = sk_stream_wait_memory(sk, &timeo)) ! =0)
        gotodo_error; }}Copy the code

Sk_stream_wait_memory will be returned depending on whether the socket is blocked or not.

int sk_stream_wait_memory(struct sock *sk, long *timeo_p)
{
	while (1) {
    // In non-blocking mode, EAGAIN is returned until timeout
		if(Wait timeout)return -EAGAIN;     
     // Block and wait until there is enough space in the send buffer
		if(sk_stream_memory_free(sk) && ! vm_wait)break;
	}
	return err;
}
Copy the code

What happens when recv is performed if the receive buffer is empty?

The same is true for the receive buffer.

If recV is also executed to the socket when the receive buffer is empty

If the socket is blocked, the program waits until the receive buffer has data, copies the data from the receive buffer to the user buffer, and returns.

If the socket is nonblocking, the program willImmediately returnaEAGAINError message.

Here is a picture for you to save for your interview.

What if the socket buffer still has data and executes close?

The first thing to know is that in general, both the send and receive buffers should be empty.

If the buffer area for sending and receiving is not empty for a long time, data is accumulated. This is usually due to network problems or problems at the user application layer. As a result, data is not processed properly.

Normally, if the socket buffer is empty, close is executed. It triggers four waves.

This is also the interview of the old eight-part text content, here we only need to pay attention to the first wave, hair is FIN enough.

What happens if close is executed when the receive buffer has data?

For socket close, the main logic is implemented in tcp_close().

First of all, there are two main situations in the closing process:

If there is still data in the receive buffer, the system emptying the buffer and then sends an RST to the peer end.
If the receive buffer is empty, then calltcp_send_fin()Start the first wave of the four-wave process.

void tcp_close(struct sock *sk, long timeout)
{
  // If the receive buffer has data, then empty the data
	while((skb = __skb_dequeue(&sk->sk_receive_queue)) ! =NULL) {
		u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq -
			  tcp_hdr(skb)->fin;
		data_was_unread += len;
		__kfree_skb(skb);
	}

   if (data_was_unread) {
    // If the buffer is empty, send RST
		tcp_send_active_reset(sk, sk->sk_allocation);
	 } else if (tcp_close_state(sk)) {
    // Wave four times normally and send FIN
		tcp_send_fin(sk);
	}
	// Wait to close
	sk_stream_wait_close(sk, timeout);
}
Copy the code

What happens if close is executed when the send buffer has data?

It was thought that in this case, the kernel would empty the send buffer and then wave four times.

But this is not the case for finding source code.

void tcp_send_fin(struct sock *sk)
{
  // Get the last block of data in the send buffer
	struct sk_buff *skb, *tskb = tcp_write_queue_tail(sk);
	struct tcp_sock *tp = tcp_sk(sk);

  // If the send buffer still has data
	if (tskb && (tcp_send_head(sk) || sk_under_memory_pressure(sk))) {
		TCP_SKB_CB(tskb)->tcp_flags |= TCPHDR_FIN; // Set the last piece of data to FIN
		TCP_SKB_CB(tskb)->end_seq++;
		tp->write_seq++;
	}  else {
    // Create a FIN packet if the send buffer has no data
  }
  // Send data
	__tcp_push_pending_frames(sk, tcp_current_mss(sk), TCP_NAGLE_OFF);
}
Copy the code

At this point, if there is still data to be sent, the kernel will pull out the last block of data from the send buffer. And I’m going to set it to FIN.

The socket buffer is a first-in, first-out queue, which means that the kernel waits until the TCP layer is quiet enough to finish sending the buffer, and then executes the first of four waves (the FIN packet).

It is important to note that it is only possible to get to tcp_send_fin() if the receive buffer is empty. It is only after entering this method that it is possible to consider sending a scenario where the buffer is empty.

UDP part

Does UDP also have a buffer

With TCP out of the way, let’s talk about UDP. These two gay friends are both important protocols in the transport layer. Since TCP has send and receive buffers, what about UDP?

I used to think that.

“Each UDP socket has a receive buffer. There is no send buffer. The concept is to send as long as the data is available, whether or not the other party can receive it correctly.

I found out later that I was wrong.

A UDP socket is also a socket. A socket has both the receive and send buffers. It has little to do with the protocol.

Whether or not is one thing, whether or not is another.

UDP does not send buffer?

In fact, UDP not only has a send buffer, but also uses a send buffer.

Normally, data will be copied directly to the send buffer and sent directly.

In another case, when the data is sent, a flag of MSG_MORE is set.

ssize_t send(int sock, const void *buf, size_t len, int flags); // Set flag to MSG_MORE
Copy the code

Tell the kernel that there will be more messages to be sent along with it. At this point, the kernel will cache the data in the send buffer, and the application layer will say OK, and then send the data together.

We can look at the source code.

int udp_sendmsg(a)
{
	// If corkreq is true, it indicates the MSG_MORE mode, and only organizes packets but does not send them.
	intCorkreq = up - > corkflag | | MSG - > msg_flags & MSG_MORE;// The data to be sent is divided according to the MTU size, with one SKB for each segment; And these
	// The SKB is placed in the send buffer of the socket; This function simply organizes packets and does not perform the send action.
	err = ip_append_data(sk, fl4, getfrag, msg->msg_iov, ulen,
			     sizeof(struct udphdr), &ipc, &rt,
			     corkreq ? msg->msg_flags|MSG_MORE : msg->msg_flags);

	// If the MSG_MORE feature is not enabled, the data in the send queue is sent directly to the IP.
    if(! corkreq) err = udp_push_pending_frames(sk); }Copy the code

So, whether it’s MSG_MORE or not, the IP will put the data in the send queue first, and then decide whether to send it right away depending on the actual situation.

And most of the time, we don’t use MSG_MORE, which is sending a packet as soon as it comes in. In terms of this behavior, although UDP uses a send buffer, it does not actually “buffer” it.

The last

I’ve only been writing this article for 20 hours. Painting is also vomit, every morning at 7 o ‘clock up to write for more than an hour before going to work.

The brothers are all their own family. It doesn’t matter whether they like it or not. It doesn’t matter whether they are watching it or not.

Watching, liking, I don’t really care, really, really, don’t believe it.

No, it really doesn’t matter.

Don’t worry about it guys.

I don’t mean what I say. See you next time!

Recently, I want to build a group. Considering that everyone has added a lot of technology group, so I decided to build a chat fish paddling bucket group.

I may not be as skilled as the bosses, but I’m a lot of emojis!

Everyone’s taking a lunch break, or getting off work.

Can occasionally brush group, chat with each other, blow boast, very decompress.

Make more friends in the group. Your next interview with an interviewer is likely to be in the group.

Add me. Add you to the group.

Don’t say, together in the ocean of knowledge to choke water

Pay attention to the public number: [small white debug]