Preface:

I just talked with the colleague in charge of recruitment. In network socket programming, candidates often have no experience in asynchronous communication, and the client will block and wait after sending data, or start a process or thread to deal with communication problems. Few candidates could write an elegant flow of asynchronous communication that would accommodate the current requirements of large-scale parallel computing. I’d like to share this with you using FTP as an example.

Question:

FTP is a standard protocol for file transfer between clients and servers based on TCP/IP. To implement a complete client file download process, there are generally three steps:

1. Establish a TCP control channel through a three-way handshake, and then send the user name and password for user authentication. You also need to specify the mode for downloading files and the parameters necessary to receive and establish data channel transfers.
2. Establish a data channel between client and server. Then, file operation commands (upload or download) and file names are sent through the control channel, and file data is transmitted through the data channel.
3. After the file transfer is complete, close the data channel. And notify the client of the end of file transfer through the control channel. In this case, the client can close the control channel and end FTP.

The first of these three steps is sufficient to cover all the mechanisms of asynchronous communication. Based on the length, this paper will focus on this step. This step involves many TCP interactions. Because TCP is based on flow, a system call may not complete the data or instruction, so it needs to wait for the other party to be able to receive data, and then send or receive again. In addition, even after the client sends a command, it must wait for an answer from the server before it can proceed. According to the interview results, most interviewees use the blocking waiting method. Call the socket’s send or recv command to send or receive data. Use the SELECT system call until time out while waiting for the server to reply. Such an implementation causes the following problems:

1. The process cannot do any other work during a blocking wait, especially for timer updates. Once the socket transmission ends, a timeout exception may be raised. Even if there is no timeout, the CPU will be rescheduled due to blocking, and CPU utilization will be greatly reduced.
2. To avoid blocking the original process, some candidates will start a special communication process to handle communication. However, the process cost is high, if the client has to connect to tens of thousands of servers at the same time, so that the client will quickly expand the machine. If you use threads, you can reduce the cost, but you can’t avoid it. In addition, if a new thread needs to read and write a data structure of the original process, it may need to be locked, resulting in poor program performance and increased maintenance difficulty.

In both cases, an interviewer is generally considered a potential candidate if he or she can use process to avoid congestion. If someone can thread the process and use locks or other mechanisms to synchronize data, consider hiring. But we’d still like to see candidates write a truly asynchronous newsletter.

Asynchronous solutions:

Asynchronous solutions are event-based drivers in nature. In order to explain this mechanism in detail, this paper will first describe the specific problem with sequence diagrams, then use state machines for functional design, and finally give pseudocode.

Sequence diagram:

The sequence diagram is needed to describe the communication process of the first FTP step, which is the basis of the following two steps.

The state machine:

As can be seen from the sequence diagram, the FTP client and server interact multiple times. Asynchronous communication is when an instruction is sent and the process is allowed to do other work before a response is received. When the response is received, it continues from the code point where it was sent. To implement this mechanism, we can use states to remember code points, which naturally leads to design with finite state machines. How do you define the FTP state machine? Strictly speaking, each system call needs to define a state if the next action needs to be determined based on the result returned. The FTP client state machine established according to the sequence diagram is shown in the following three figures:

In order to support the three-way handshake, state machine 1 of the FTP control channel defines three states: Connect, Connecting, and Connected. Connecting is the state after connect(). Connected is the status of the socket channel when it is available. These three states generally change in order, and if there is an error or a timeout, they are returned to the end state for restarting. Then each state is divided into three situations, namely success, enter the next state; If the system fails or times out, the system enters the end state. The message has not been sent or received, leaving the status unchanged. It is important to point out that while this state remains unchanged, the process can still do other tasks. When the underlying socket is ready to continue sending or receiving, the state will continue to run from the current state.

Pseudo code:

To illustrate the implementation of the state machine, here is the pseudo-code implemented with epoll under Linux. On other platforms, the system call may change, but the idea remains the same.

1. Data structure

• Session: Records each SessionFTPSession, in the middle of all required running data and statistics, can also have related configuration information, such as user name, password, etc. The code is as follows:

  struct session {
  struct channel channel;
  struct cfg {
      char servername[64];
      char username[32];
      char password[32];
      charmode; . }uint32_t resolved_server_ip;
  uint64_t num_errors;
  uint32_t error_code;
  }
  Copy the code

• Channel: data cache required to control the Channel, state of the state machine, epoll and socket descriptors, etc. The code is as follows:

  struct channel {
      struct session *session;
      int rd_off, rd_len;
      char rd_buf[BUFF_SZ];
      int wr_off, wr_len;
      char wr_buf[BUFF_SZ];
      int state;
      struct epoll_handle {
          int fd;
          uint32_t event;
          int (*read)(struct epoll_handle *);
          int (*write)(struct epool_handle *);
          void*data; }}Copy the code

2. The state machine

2.1 framework

```c static void ftp_channel_stmch_do_update(struct channel *channel, void *msg) { int old_state = channel->state; int new_state = old_state; switch(old_state) { case CONNECT: new_state = process_tcp_connect(channel, msg); break; case statex.. new_state = process_statex(channel, msg); break; . } channel->state = new_state; return (new_state ! = old_state); } static void ftp_channel_stmch_update(struct channel *channel, void *msg) { while (ftp_channel_stmch_do_update(session, msg)); } ` ` `Copy the code

Ftp_channel_stmch_do_update is the execution component of the state machine. Each time the old state is saved, the process_statex function is executed to get the new state. If the new and old states are equal, 0 is returned and the state machine exits. Otherwise, the while loop ftp_channel_stmCH_update continues until the new and old states are the same, indicating that the state machine is stable and can exit.

2.2 Sending and receiving

Data is sent and received here. To avoid process blocking, you need to set the socket to non-blocking mode first.

```c fcntl(fd, F_SETFL, old_flags | | O_NONBLOCK); ` ` `Copy the code

Then the channel’s epoll is associated with the socket and the epoll event is set to EPOLLOUT. When the TCP three-way handshake succeeds, the EPOLLOUT event is triggered. In the EPOLLOUT callback function, the channel state can be set to Connected and instructions can be sent. If the state changes, you need to execute the state machine. The pseudocode for the EPOLLOUT callback function is as follows:

```c static int ftp_ctrl_epollout(struct epoll_handle *handle) { struct channel *channel = (void *)handle->data; if (ftp_epollout_error_check(handle) < 0) { channel->state = FTP_STATE_ERROR; ftp_channel_stmch_update(channel); return -1; } if (ftp_ctrl_do_epollout(channel)) lnkmt_ftp_stmch_update(channel); return 1; } static int ftp_ctrl_do_epollout(struct channel *channel) { int old_state = channel->state; int new_state = old_state; if (! ftp_msg_send(channel)) return 0; switch(old_state) { case statex: new_state = ... ; break; case ... } channel->state = new_state; return (new_state ! = old_state); } static int ftp_msg_send(lnkmt_ftp_epoll_channel_t *channel) { int ret; int fd = channel->ep.fd; if (channel->wr_off == channle->wr_len) return 1; ret = write(fd, channel->wr_data + channel->wr_off, channel->wr_len - channel->wr_off); if (ret < 0) { if (errno == EAGAIN || errno == EINTR) return 0; else { channel->sate = FTP_STATE_ERROR; return 1; } } else if (ret == 0) { channel->state = FTP_STATE_CLOSE; return 1; } channel->wr_off += ret; if (channel->wr_off < channel->wr_len) return 0; return 1; ` ` `Copy the code

The ftp_MSg_send function returns 0 if there is more data in the channel to send or if the socket returns EINTR or EAGAIN. Indicates that the state is unchanged and the state machine does not need to be started. The system will issue the EPOLLOUT Event again next time to execute the callback function.

When the socket receives data from the server, an EPOLLIN event will be triggered. We can design a similar callback function, which will not be described here.

2.3 Timeout and Errors:

When sending and receiving, errors may occur at the bottom of the operating system. In this case, you need to add judgment in EPOLLOUT and EPOLLIN. If the socket fails, the state machine can be triggered immediately to process the error state. The error function is as follows:

```c
getsockopt(fd, SOL_SOCKET, SO_ERROR, &ret, &len)
```
Copy the code

When TCP communication, if the intermediate device is broken, the two parties can not receive data, but also can not receive the CLOSE signal, so that the two-method TCP connection will remain for quite a long time, which is a great waste of resources. In this case, the Keep Alive or Timer mechanism can be used. Once the timeout occurs, the corresponding callback function will be triggered immediately to call the state machine and deal with the error state. The pseudocode for the callback function is as follows:

```c static void ftp_channel_timeout(unsigned long arg) { struct channel *channel = (void *)arg; channel->state = FTP_STATE_ERROR; lnkmt_ftp_stmch_update(channel->ppf); } ` ` `Copy the code

Performance analysis

Through asynchronous communication, all data of an FTP session is contained in one session. When there are thousands of FTP sessions, you can leverage a session array. Each sesion is distinguished by an ID, and a hash is used for quick location. There is no public data between sessions and no need to lock them. Because this is all in a process, there is no process switching, can adapt to large-scale network requirements. From a programming point of view, although there are many states, it seems a bit complicated. In fact, many states are handled the same way, are easier to implement and debug, and state expansion is easy. This is the framework for an elegant program.

conclusion

In the other two steps of FTP, asynchronous communication requires the definition of two state machines: the control channel state machine and the data channel state machine, and the synchronization of these two state machines needs to be considered. I won’t go into it for space. If you are interested, consider writing a sequel on synchronization of complex state machines. In short, the development of the network to the present stage, large-scale, distributed, parallel and so on are the basic requirements of communication. If only two entities on the network communicate, this can be done with a few simple socket calls. But if you have tens of thousands or hundreds of thousands of network entities communicating, you’re going from a quantitative change to a qualitative change, and you’re going to be thinking a lot differently. I hope you have a good interview.