While reading this article, did you ever wonder how the server sent this article to you?

Simply put, isn’t it a user request? On request, the server pulls the article out of the database and sends it back over the network.

Complicated as it is, how does the server process thousands of user requests in parallel? What technology is involved?

This article will answer that question.

Multiple processes

One of the earliest and easiest ways to handle multiple requests is to use multiple processes.

For example, in the Linux world, we can create multiple processes using fork, exec, etc. We can receive link requests from the user in the parent process, and then create child processes to handle user requests, like this:

The advantages of this approach are that:

  1. Programming is simple and very easy to understand
  2. Because the address Spaces of processes are isolated from each other, the crash of one process does not affect other processes
  3. Make full use of multi-core resources

The advantages of multi-process parallel processing are obvious, but the disadvantages are also obvious:

  1. The advantage of the isolation of the process address space becomes the disadvantage, which is that it becomes more difficult for processes to communicate with each other. You need to use IPC (interprocess Communications) mechanisms. Think about what inter-process communication mechanisms you know now and let you implement them in code. Obviously, interprocess communication programming is relatively complex, and performance is also an issue
  2. We know that process creation is more expensive than thread creation, and frequent process creation and destruction will undoubtedly add to the system burden.

Fortunately, in addition to processes, we have threads.

multithreading

Isn’t it expensive to create a process? Isn’t interprocess communication difficult? None of this is a problem for threads.

What? If you don’t know anything about threads, take a look at this article titled “Threads and Thread pools in High Concurrency”, which explains in detail where the concept of threads came from.

Since threads share the process address space, communication between threads naturally does not require any communication mechanism, just reading memory directly.

The overhead of thread creation and destruction is also reduced. Remember that threads are like hermit crabs in that the house (address space) is a process and they are only a tenant, so they are very lightweight and the overhead of creation and destruction is very small.

​

We can create one thread per request, and even if one thread is blocked for performing I/O operations such as reading a database, the rest of the thread will not be affected, like this:

But are threads perfect and cure-all? Obviously, the computer world is never that simple.

Since threads share process address space, this brings convenience for communication between threads, but also brings endless trouble.

It is Shared between threads address space, thus a thread collapse causing collapse the entire process, at the same time communication between threads is too simple, easy to communication between threads need to directly read only memory, also easy to appear problem is extremely easy, deadlock, thread synchronization between incompatible, and so on, these very prone to bugs, A good portion of countless programmers’ precious time is devoted to solving endless problems caused by multithreading.

Although threads have disadvantages, they have advantages over multi-processes, but it is impractical to solve the problem of high concurrency by using multi-threading alone.

Although thread creation is less expensive than process creation, it is still expensive. For a high-concurrency server with tens of thousands of links, creating tens of thousands of threads can have performance issues, including memory footprint, switching between threads, and scheduling overhead.

So we need to think further.

Event Loop: indicates the Event driver

So far, when we say “parallel,” we think of processes and threads. But should parallel programming only rely on these two techniques? Not really.

Another concurrency technique that is widely used in GUI programming as well as server programming is event-based Concurrency, which has become very popular in recent years.

Don’t think of this as a difficult technique, event-driven programming is actually quite simple in principle.

This technique requires two ingredients:

  1. event
  2. A function that handles events, often referred to as an Event Handler

The rest is simple:

You just need to wait quietly for the event to arrive. When the event arrives, check the type of the event and find the corresponding event handling function according to the type, that is, event Handler, and then call the event Handler directly.

That’s it !

That’s all for event-driven programming, isn’t it easy?

As we can see from the discussion above, we need to continuously receive events and then process them, so we need a loop (either a while loop or a for loop), which is called an event loop.

This is what happens when you use pseudocode:

while(true) {
    event = getEvent();
    handler(event);
}
Copy the code

What you need to do in the Event loop is actually very simple. You just need to wait for the Event and then call the corresponding Event handler function.

Note that this code only needs to run in one thread or process, and only one Event loop is needed to process multiple user requests simultaneously.

Why can an event loop process multiple requests at the same time?

The reason is simple: for web servers, most of the time when processing a user request is actually spent on I/O operations, such as database read and write, file read and write, network read and write, etc. When a request comes in, simple processing may require database query and other I/O operations, we know that I/O is very slow, after initiating I/O we can continue to process the following user requests without waiting for the COMPLETION of the I/O operation.

So now you can see, even though the last user request hasn’t been processed yet we can actually process the next user request, and that’s called parallelism, and that parallelism can be handled with event-driven programming.

This is just like a waiter in a restaurant. A waiter cannot wait for the last customer to place an order, serve food, eat and pay the bill before serving the next customer. What does the waiter do? When a customer orders directly after the next customer, when the customer after the meal will come back to pay the bill.

As you can see, the same server can also process multiple customers at the same time. This server is equivalent to the Event loop here, which can process multiple user requests at the same time even if the Event loop only runs in a thread (process).

Now that you have a clear understanding of event-driven programming, the next question is event-driven, event-driven, and how do you get the event?

Source: I/O multiplexing

In the Linux/Unix world, everything is a file, and our programs use file descriptors for I/O operations, including sockets. How do we handle multiple file descriptors at the same time?

IO multiplexing is used to solve this problem. Through IO multiplexing, we can monitor multiple file descriptions at once and be notified when a socket is readable or writable.

In this way, IO multiplexing technology becomes the engine of event loop, providing us with a steady stream of events, so that the source of events can be solved.

Of course, for a detailed explanation of I/O multiplexing, please refer to “I/O Multiplexing at Last.”

At this point, are all the questions about using event driven concurrent programming solved? The origin of the event is resolved, and when the event is obtained, the corresponding handler is called, which looks like it’s done. Think about any other questions?

Problem: blocking IO

Now we can use a single thread (process) to do event-driven parallel programming without the annoying locks, synchronous mutexes, deadlocks, and so on. But there has never been a single technology in computer science that solves all problems, not now, and not in the foreseeable future.

So what’s wrong with that?

Remember that our Event loop runs in a thread (process), which solves the multithreading problem, but what if we need to do IO while processing an event?

Read in the file, the program has experienced what the article, we explain the most commonly used file read at the bottom is how to implement, the programmer is called the most commonly used means of this kind of IO blocking IO, that is to say, when we go on IO operations, such as read a file, if the file does not read the complete, so our program (thread) will be blocked and suspended, This is not a problem in multithreading because the operating system can also schedule other threads.

However, there is a problem in the single-threaded Event loop, because when we execute blocking IO operations in the event loop, the whole thread (Event loop) will be suspended, and then the operating system will have no other threads to call. Since there is only one Event loop in the system handling user requests, all user requests cannot be processed when the Event Loop thread is blocked and suspended. Can you imagine your request being suspended while the server is processing other user requests reading from the database?

Therefore, one caveat in event-driven programming is that blocking IO is not allowed.

Some students may ask, if you can’t initiate blocking IO, then how do you do I/O? Where there is blocking IO, there is non-blocking IO.

Non-blocking IO

To overcome in the process of blocking type IO, modern operating systems started IO request to provide a new method, this method is asynchronous I/o, for the block type IO is synchronous IO, concept of synchronous and asynchronous both pay attention to the public, farmers “code island” and reply the number “5” and you know the answer.

For asynchronous I/O, if we call the aio_read function (please refer to the specific operating system platform for the asynchronous I/O API), which is called asynchronously, we can immediately return and continue with other things, even though the file may not have been read yet, so that the calling thread is not blocked. In addition, the operating system provides other methods for the calling thread to detect that the IO operation has completed.

In this way, with the help of the operating system, the blocking call problem of IO has been solved.

Difficulties in event-based programming

While there are asynchronous IO solutions to the problem of event loop blocking, event-based programming is still difficult.

First of all, we mentioned that the Event loop is run in a thread. Obviously, a thread cannot make full use of multi-core resources. Some students might say that it is ok to create multiple Event Loop instances, so that there will be multiple Event loop threads.

Another point is that programming, in the from white to master, you need to understand the synchronous and asynchronous, in this article, we described the asynchronous programming need to combine the callback function, this style of programming required the processing logic is divided into two parts, part of the caller yourself, the other part of the process in the callback function, the programming changes increased the burden on the programmer to understand, Event-based programming can be difficult to extend and maintain later in a project.

So is there a better way?

To find a better way, we need to address the essence of the problem, and what is the essence of the problem?

A better way

Why do we program asynchronously in a way that’s hard to understand?

This is because blocking programming, while easy to understand, causes a thread to block and suspend execution.

So you’re smart enough to ask, is there a way to combine the simple understanding of synchronous IO without blocking the thread because of synchronous calls?

The answer is yes, follow the public account “mynong’s desert island survival” and reply “high performance” to find out.

Despite the disadvantages of event-based programming, event-based programming is still very popular in today’s high performance and high concurrency servers. However, it is no longer purely event-driven based on a single thread, but event loop + Multi Thread + User level Thread. It’s also worth writing an article about this combination, which we’ll discuss in more detail in a future article.

conclusion

High concurrency has evolved from multi-process technology to event-driven technology, and computer technology has evolved much like biology, but understanding history helps us understand the present better. Hopefully this article has helped you understand high-concurrency servers.