This article was first published in 51CTO technology stack public number author Chen Caihua article reprint exchange please contact [email protected]Copy the code

With the development of the Internet, the traditional blocking server architecture has become powerless in the face of massive user and high-concurrency services. Therefore, this paper aims to provide a useful overview and comparison of network service models to uncover the mystery of designing and implementing high-performance network architectures

1 The server processes network requests

Let’s start with a typical server process for network requests:

As you can see, the main processing steps include:

• 1. Obtain request data The client establishes a connection with the server and sends a request, and the server accepts the request (1-3)
• When the server receives the request, it processes the client’s request in user space until the build response is complete (4)
• 3. Return data The server sends the constructed response back to the client via network I/O in kernel space (5-7)

When designing the server-side concurrency model, there are two key points:

• How does the server manage connections and retrieve input data
• How does the server handle requests

Both of these key points ultimately relate to the OPERATING system’s I/O model and the threading (process) model, which are described in detail below

2 I/O model

2.1 Conceptual theory

Before introducing the OPERATING system I/O model, let’s understand a few concepts:

• Blocking and non-blocking calls
  • A blocking call means that the current thread is suspended until the result of the call is returned. The calling thread does not return until it gets the result
  • A non-blocking call does not block the current thread until the result is not immediately available

The biggest difference between the two is whether the caller waits until the request is received and the result is returned. Blocking is when the caller waits and does nothing else. Non-blocking means that the caller does something else first

• Synchronous and asynchronous processing

  • Synchronous processing means that the final result is returned to the caller after the called party gets it
  • Asynchronous processing means that the called party first returns the reply, then calculates the result of the call, and then notifies and returns the final result to the caller

• The difference between blocking and non-blocking and synchronous and asynchronous is that blocking and non-blocking are different from synchronous and asynchronous: the blocking and non-blocking discussion object is the caller; synchronous and asynchronous discussion object is the called

• Recvfrom function The recvfrom function (which receives data through the socket) is treated as a system call

An input operation usually consists of two distinct stages

• Wait for the data to be ready
• Copy data from the kernel to the process

For an input operation on a socket, the first step usually involves waiting for data to arrive from the network. When the waiting group arrives, it is copied to a buffer in the kernel. The second step is to copy the data from the kernel buffer to the application process buffer

The actual application program in the system call to complete the above two steps of operation, call mode blocking, non-blocking, operating system in the processing of application requests in the way of synchronous and asynchronous processing, refer to ** “UNIX network programming volume 1” **, can be divided into five I/O models

2.2 Blocking I/O Model

Introduction In the blocking I/O model, an application blocks from the time it calls recvFROM until it returns a datagram ready. After recvFROM returns successfully, the application process begins to process the datagram

This is a metaphor for a man who is fishing. When no fish is hooked, he sits on the bank and waits

Advantages simple procedure, in the blocking waiting for data during the process/thread hang, basically does not occupy CPU resources

Disadvantages Each connection needs to be handled by an independent process/thread. In order to maintain the program when the number of concurrent requests is large, the overhead of memory and thread switching is large. This model is rarely used in actual production

2.3 Non-blocking I/O Model

Introduction to the In a non-blocking I/O model, application set a set of interfaces to non-blocking is to tell the kernel, when the requested I/O operation cannot be completed, don’t sleep will process, but returns an error, the application based on I/O operations function will keep polling data are ready, if not well prepared, continue to polling, Until the data is ready

The analogy of fishing while playing mobile phone, will see if there is a fish hook, if there is quickly pull rod

Advantages It will not block the process of waiting for data in the kernel, and each I/O request can be immediately returned without blocking waiting, which has good real-time performance

Disadvantages Polling constantly asks the kernel, which takes up a lot of CPU time and system resource utilization is low, so the typical Web server does not use this I/O model

2.4 I/O multiplexing model

In the I/O multiplexing model, select or poll functions or epoll functions are used. These two functions also block the process, but unlike blocking I/O, these two functions can block multiple I/O operations at the same time, and can block multiple read operations at the same time. The I/O function of multiple write operations is checked until data is readable or writable, and the I/O operation function is actually called

Put a pile of fishing rods, in the shore has been watching the pile of fishing rods, until a fish bite

Instead of using multiple threads (one thread per file descriptor), the advantage can be based on a blocking object and waiting ready on multiple descriptors at the same time, which can greatly save system resources

Disadvantages When the number of connections is small, the efficiency is lower than that of multi-thread + blocking I/O model, and the delay may be larger, because the processing of a single connection requires two system calls, which will increase the elapsed time

2.5 Signal-driven I/O Model

In the signal-driven I/O model, an application uses a socket for signal-driven I/O, installs a signal handler function, and the process continues to run without blocking. When the data is ready, the process receives a SIGIO signal and can process the data by calling the I/O operation function in the signal handler function

The metaphor is that the fishing rod is tied with a bell, when the bell rings, you know the fish is hooked, and then you can concentrate on your mobile phone

The advantage is that threads are not blocked while waiting for data, which increases resource utilization

disadvantages

• Signal I/O may fail to be notified when a large number of I/O operations occur due to signal queue overflow
• Signal-driven I/O, though, is useful for handling UDP sockets, where such signal-driven notifications mean arrival of a datagram, or return of an asynchronous error. For TCP, however, signal-driven I/O is almost useless because there are so many conditions that lead to such notifications, each one of which consumes a lot of resources and loses its advantage over the previous approaches

2.6 Asynchronous I/O Model (Asynchronous I/O)

Defined by the POSIX specification, an application tells the kernel to start an operation and lets the kernel notify the application when the entire operation, including copying data from the kernel to the application’s buffer, is complete. The main difference between this model and the signal-driven model is that the signal-driven I/O model lets the kernel tell the application when to start an I/O operation, whereas the asynchronous I/O model lets the kernel tell the application when the I/O operation is complete

Advantages Asynchronous I/O can take full advantage of DMA features, allowing I/O operations to overlap with calculations

Disadvantages To implement true asynchronous I/O, the operating system needs to do a lot of work. At present, under Windows, real asynchronous I/O is realized through IOCP, while under Linux, Linux2.6 was introduced. Currently, AIO is not perfect, so in Linux, high concurrency network programming is mainly based on IO multiplexing model

2.5 Summary of five I/O models

As you can see from the figure above, as you can see, the further you go, the less blocking, the more efficient it is in theory. Of the five I/O models, the first four are synchronous I/O because the true I/O operation (RECvFROM) blocks the process/thread, and only the asynchronous I/O model matches posiX-defined asynchronous I/O

3 Thread Model

After explaining how the server manages connections and gets input data based on the I/O model, how does the server handle requests based on the process/thread model

It is worth noting that the specific choice of thread or process depends more on the platform and programming language. For example, C language can use both threads and processes (for example, Nginx uses processes, Memcached uses threads). Java language generally uses threads (for example, Netty)

3.1 Traditional blocking I/O service model

The characteristics of

• The blocking I/O model is used to obtain input data
• Each connection requires a separate thread to complete data entry, business processing, and data return

There is a problem

• When the number of concurrent connections is large, a large number of threads need to be created to process connections, occupying large system resources
• After a connection is established, if the current thread has no data to read temporarily, the thread blocks on the read operation, resulting in a waste of thread resources

3.2 Reactor model

The common solutions to the two disadvantages of the traditional traditional blocking I/O service model are as follows:

• Based on the I/O multiplexing model, multiple connections share a blocking object, and applications only need to wait on one blocking object, instead of blocking for all connections. When a connection has new data to process, the operating system notifies the application, and the thread returns from the blocked state to begin processing the business
• Reuse thread resources based on thread pool. Instead of creating threads for each connection, the business processing tasks after the connection is completed are assigned to threads for processing. One thread can process the services of multiple connections

I/O reuse combined with thread pooling is the basic design concept of Reactor schema

The Reactor pattern refers to the event-driven processing pattern of service requests that are delivered simultaneously to the service processor through one or more inputs. The server program processes the incoming multiple requests and synchronously dispatches them to the corresponding processing thread. The Reactor model is also called the Dispatcher model, which means that I/O is multiplexed with unified monitoring events and dispatched to a process after receiving the events. It is one of the necessary techniques for programming high performance network servers

There are two key components to the Reactor model:

• A Reactor runs in a separate thread, listening for and distributing events to the appropriate handlers to react to IO events. It acts like a corporate telephone operator, answering calls from customers and redirecting the line to the appropriate contact

• Handlers perform the actual event that the I/O event is to accomplish, similar to the actual official in the company that the customer wants to talk to. Reactor responds to I/O events by scheduling appropriate handlers that perform non-blocking actions

Depending on the number of reactors and the number of resource pool threads that are processed, there are three typical implementations:

• Single Reactor Single thread
• Single Reactor multithreading
• Reactor is multithreaded

These three implementations are described in detail below

3.2.1 Single Reactor Single thread

Select is the standard network programming API introduced in the I/O multiplexing model above, which enables applications to listen for multiple connection requests through a blocking object, similar to other schemes

Solutions that

• The Reactor object monitors client request events through SELECT and distributes events through Dispatch
• In the case of a connection request event, the Acceptor processes the connection request through Accept, and then creates a Handler object to handle subsequent business processing after the connection is completed
• If the connection is not established, the Reactor dispatches the Handler that calls the connection in response
• Handler completes the complete business process of read-> business processing -> Send

Advantages of the model is simple, there is no multi-threading, process communication, competition, all in a thread to complete

disadvantages

• Performance problem: Only one thread can not fully use the performance of a multi-core CPU. When a Handler processes services on a connection, the entire process cannot process other connection events, resulting in performance bottlenecks
• Reliability problem: Threads run off unexpectedly or enter an infinite loop. As a result, the communication module of the entire system is unavailable and cannot receive and process external messages, resulting in node faults

Usage scenario The number of clients is limited, and the business processing is very fast, such as Redis, and the time complexity of business processing is O(1).

3.2.2 Single Reactor Multithreading

Solutions that

• The Reactor object monitors client request events through SELECT and distributes events through Dispatch
• In the case of a connection request event, an Acceptor accepts the connection request and creates a Handler object to handle subsequent events after the connection is completed
• If the connection is not established, the Reactor dispatches the Handler that calls the connection in response
• Handler is only responsible for responding to events and does not handle specific services. After reading data through read, it is distributed to subsequent Worker thread pools for service processing
• The Worker thread pool allocates a separate thread to complete the actual business processing. How do I send the response results to the Handler for processing
• After receiving the response result, the Handler sends the response result to the client

Advantages Can make full use of multi-core CPU processing power

disadvantages

• Multithreaded data sharing and access are complicated
• Reactor monitors and responds to all events and runs in a single thread, which can easily become a performance bottleneck in high concurrency scenarios

3.2.3 Principal/Slave Reactor Multithreading

In the single-reactor multithreading model, Reactor runs in a single thread, which is easy to become a performance bottleneck in high concurrency scenarios. Therefore, THE Reactor can be run in multithreading

Solutions that

• Reactor Mainline The MainReactor uses select to monitor connection establishment events. After receiving the events, the MainReactor receives the events through Acceptor and processes the connection establishment events
• After an Accepto process a connection, the MainReactor assigns the connection to the Reactor subthread for processing
• SubReactor adds connections to the connection queue for listening and creates a Handler to handle various connection events
• When a new event occurs, the SubReactor calls the Handler corresponding to the connection to respond
• After the Handler reads the data through read, it distributes it to the following Worker thread pool for business processing
• The Worker thread pool allocates a separate thread to complete the actual business processing. How do I send the response results to the Handler for processing
• After receiving the response result, the Handler sends the response result to the client

advantages

• The data interaction between the parent thread and its child thread is simple and has clear responsibilities. The parent thread only needs to receive new connections, and the child thread completes the subsequent business processing
• The data interaction between the parent thread and the child thread is simple, the Reactor main thread only needs to pass the new connection to the child thread, the child thread does not need to return data

This model is widely used in many projects, including support for the Nginx Master-slave Reactor model, Memcached master-slave model, and Netty master-slave model

3.2.4 summary

The three models can be understood metaphorically: restaurants often employ receptionists to greet customers, and when customers are seated, waiters serve the table

• Single Reactor Single Thread The receptionist and the waitress are the same person who serves the customer the entire time
• Single Reactor Multithreading 1 receptionist, multiple waiters, the receptionist is only responsible for reception
• Reactor Multiple threads Multiple hosts, multiple waiters

The Reactor model has the following advantages:

• Fast response and not blocked by a single synchronization time, although the Reactor itself is still synchronous
• Programming is relatively simple, can maximize avoid complex multithreading and synchronization problems, and avoid multithreading/process switching overhead;
• Scalability, which makes full use of CPU resources by increasing the number of Reactor instances
• Reusability: The Reactor model itself is independent of the specific event processing logic and has high reusability

3.3 Proactor model

In the Reactor model, the Reactor waits for an event or an applicable operation state (such as file descriptor read/write, or socket read/write) to occur, and then passes the event to a pre-registered Handler (event Handler or callback) that does the actual read/write. All read and write operations require application synchronization, so Reactor is a non-blocking synchronous network model. If the I/O operation is changed to asynchronous, that is, to the operating system to complete the performance can be further improved, this is the asynchronous network model Proactor

Proactor is related to asynchronous I/O. The detailed scheme is as follows:

• ProactorInitiator creates Proactor and Handler objects and registers them with the kernel through AsyOptProcessor (Asynchronous Operation Processor)
• AsyOptProcessor processes registration requests and I/O operations
• AsyOptProcessor notifies Proactor when the I/O operation is complete
• Proactor calls back different handlers based on different event types for business processing
• Handler Completes service processing

The difference between a Proactor and a Reactor is that a Reactor is a pre-registered event that is notified when an event occurs (read and write are processed in the application thread). Proactor is based on asynchronous I/O to complete the read and write operations (completed by the kernel) when the event occurs. After the I/O operation is completed, the processor of the application program is called back to process the business

Theoretically, Proactor is more efficient than Reactor, and asynchronous I/O gives full play to the advantages of DMA(Direct Memory Access), but it has the following disadvantages:

• Programming complexity Developing asynchronous applications is more complicated because the initialization and completion of the events of the asynchronous operation process are separated in time and space. Applications can also become more difficult to Debug because of reverse flow control
• Memory using the buffer must be held for the duration of a read or write operation, which can cause continuous uncertainty, and each concurrent operation requires a separate cache. In contrast to the Reactor model, caching is not required until the socket is ready to read or write
• Operating systems support true asynchronous I/O through IOCP in Windows, while Linux2.6 was introduced in Linux, so asynchronous I/O is still not perfect

Therefore, Reactor model is the main method to realize high concurrency network programming under Linux

More exciting, welcome to pay attention to the author’s public account [Distributed System Architecture]

reference

Learning architecture from scratch — Li Yunhua, Alibaba technology expert

Netty introduction and Actual combat

Technology: Linux network IO model

Multithreaded Network service model

Blocking, non-blocking, synchronous, and asynchronous in IO

UNIX Networking Programming Volume 1: Socket Networking apis (version 3)

Asynchronous network model

Ali cloud recently began to issue vouchers, new and old users can get free, new registered users can get 1000 yuan vouchers, old users can get 270 yuan vouchers, we suggest that everyone get a copy, anyway is free to get, maybe later need? Ali cloud vouchers to receive promotion.aliyun.com/ntms/yunpar…

Popular activities High performance cloud server preferential level power companies on the performance of cloud host 2-5 fold promotion.aliyun.com/ntms/act/en…