High performance gateway design practices

preface

The previous high performance short chain design article is quite well received by everyone, was reprinted 47 times, flattered, thank you for your recognition! I mentioned OpenResty at the end of this article, which is of interest to some readers. It happens that we also use OpenResty to access the gateway layer, so I hope to summarize the knowledge about OpenResty by introducing the gateway design. To give you an overview of OpenResty, a high-performance Web platform. This article will explain from the following aspects.

• Role of gateway
• Design and implementation of access layer gateway architecture
• Technology selection
• OpenResty principle analysis

Role of gateway

As the traffic entrance of all requests, gateways mainly undertake security, traffic limiting, fusing degradation, monitoring, logging, risk control, authentication and other functions. There are two types of gateways

• One is the Access gateway, which is mainly responsible for routing, WAF (preventing SQL Injection, XSS, path traversal, stealing sensitive data,CC attack, etc.), traffic limiting, log, cache, etc. The gateway of this layer mainly carries the function of routing requests to the gateways of each application layer

• Another kind is the application layer gateway, such as popular now micro services, each service may be written in different languages, such as PHP, Java, etc., then the access layer is routes the request to the appropriate application cluster, then by the corresponding application layer gateway authentication processing, such as micro processed before calling the appropriate service for processing, the application layer gateway also plays a routing, Timeout, retry, fuse and other functions.

The most popular system architectures in the market are as follows

It can be seen that the access layer gateway carries all the traffic of the company and has high requirements on performance. Its design determines the upper limit of the whole system. So today we are going to focus on the design of the access layer gateway.

Design and implementation of access layer gateway architecture

First of all, we should understand that the core function of the access layer gateway is to distribute requests to the corresponding back-end cluster according to routing rules, so the following functional models should be implemented.

1. Routing: forwarding to the specified upstream (corresponding back-end cluster) according to the requested host, URL and other rules. This is the soul of the gateway. There are rules such as identity authentication, traffic limiting and speed limiting, security protection (such as IP blacklist, refer exception, UA exception, which need to be rejected in the first time) and other rules in the route. These rules are combined together in the form of plug-ins so as to only take effect for a certain type of request. These plugins should be dynamically configurable and effective (no need to restart the service). Why should they be dynamically configurable because the routing logic, traffic limiting rules, back-end cluster rules of the final request are different for each request

As shown in the diagram, the routing rules corresponding to two requests are different, and their corresponding routing rules (traffic limiting, rewrite, etc.) are combined together through various rule plug-ins. As you can see, there are quite a lot of routing rules for two request urls alone. If a system is large enough, there will be many urls and many rules. In this way, the rules for each request must be configurable and dynamic. It is better to centrally control and deliver the rules at the management end.

3. Dynamic change of back-end cluster

Routing rules of application is to determine which one kind of request after these rules to a cluster, and we know we do is to make the request to a cluster of IP, and scalability of the machine capacity is more common, so you have to support dynamic change, can’t I every up and down line machines to restart the system let it take effect.

4, monitoring statistics, request volume, error rate statistics, etc

This is easier to understand, in the access layer for all traffic requests, error statistics, easy to dot, alarm, analysis.

Implementing these requirements requires a detailed understanding of the technology we are using: OpenResty, so this article will briefly cover OpenResty.

Technology selection

Nginx may be the first thing that comes to mind. Yes, thanks to Nginx’s epoll model (non-blocking IO model), it does meet the needs of most scenarios (100 W + concurrency is not a problem when optimized), but Nginx is better suited for static Web servers. Because Nginx does not provide an API to control runtime behavior if any changes are made that require modification of the configuration on disk and reloading to take effect, dynamics, as mentioned above, is an important feature of access layer gateways. So after some research, we chose OpenResty. What is OpenResty?

OpenResty® is a high-performance Web platform based on Nginx and Lua that integrates a large number of excellent Lua libraries, third-party modules, and most of the dependencies. It is used to easily build dynamic Web applications, Web services and dynamic gateways that can handle ultra-high concurrency and scalability. The goal of OpenResty® is to have your Web services run directly inside Nginx services, taking full advantage of Nginx’s non-blocking I/O model, Provide consistent, high-performance responses not only to HTTP client requests, but also to remote backends such as MySQL, PostgreSQL, Memcached, and Redis.

OpenResty = Nginx + Lua, a scalable Web platform that extends Nginx through Lua. It takes advantage of Nginx’s high performance and adds the Scripting language of Lua to make Nginx dynamic. With the Lua API provided in the Lua-Nginx-Module in OpenResty, we can dynamically control routing, upstream, SSL certificates, requests, responses, and so on. You can even change the business processing logic without restarting OpenResty, not limited to the Lua API provided by OpenResty.

There’s a good analogy between static and dynamic: if you think of a Web server as a car speeding down the highway, Nginx has to stop to change the tire, change the paint color, and OpenResty can change the tire, change the paint, and even change the engine on the run, turning an ordinary car into a supercar!

In addition to this dynamism, there are two other features that make OpenResty unique.

1. Detailed documentation and test cases

As an open source project, documentation and testing are undoubtedly the key to making it sound. The documentation is very detailed, and the authors have documented every point of attention, most of the time just looking at the documentation. Every test case includes complete Nginx configuration and Lua code. And test input data and expected output data.

2. Synchronization is non-blocking

OpenResty has supported coroutines since its inception and implemented a synchronous non-blocking programming model based on this. We can think of a coroutine as a user-mode thread, but this thread is scheduled by ourselves, and the switching between different coroutines does not need to fall into kernel mode, which is more efficient. (Generally speaking, when we say threads, we mean kernel-mode threads, which are scheduled by the kernel, need to fall into the kernel space from user space, compared to the coroutine, which has a large impact on performance.)

What is synchronous non-blocking. Suppose you have the following two lines of code:

local res, err  = query-mysql(sql)
local value, err = query-redis(key)
Copy the code

Synchronous: the following redis query must be executed before the mysql query is completed, or asynchronously before the mysql query is executed.

Block: If it takes 1s to execute the SQL statement, it is blocked if the CPU waits and does nothing else during that time. If it can do something else during the SQL execution (note that the following redis query cannot be executed because it is synchronous), it is not blocked.

Synchronization of statement execution order successively, if on a statement must be performed to executing the next statement is synchronous, if not, it is asynchronous, obstruction of thread is whether you need the CPU during IO wait for, if during the period of IO (or other) during lengthy operations can do other things, it is blocked, can’t move, Is blocking.

So how does OpenResty work, and how does it achieve synchronous non-blocking?

OpenResty principle analysis

Analysis of working principle

Since OpenResty is implemented based on Nginx, let’s take a look at how Nginx works

After Nginx starts, there will be a master process and multiple worker processes. The master process accepts the semaphore of the administrator (such as Nginx -s reload, -s stop) to manage the worker process. The master itself does not receive requests from the client, but the worker process receives requests. Different from Apache, each request occupies a thread and is synchronous IO, Nginx is asynchronous and non-blocking. The number of requests that each worker can process at the same time is only limited by memory size. Here’s a quick look at the epoll model used by Nginx:

Epoll adopts multiplex model, that is, at the same time, although there may be multiple requests to come in, but will only use a thread to monitor, which request data are ready, and then call the corresponding thread to deal with, as shown in the figure, as well as dial switch, at the same time there is only one thread in processing, epoll is based on the event-driven model, Each request registers the event and registers the callback function. When the data is ready, the callback function is called for processing. It is asynchronous and non-blocking, so the performance is high.

Make a simple example, we all have the experience of the booking, when we entrust the hotel reservation, the receptionist will put down our phone number and related information (registration), hang up the phone receptionist during operation after we can do other things (non-blocking), after the receptionist to do well formalities will take the initiative to call us to inform our tickets booked well (the callback).

Worker processes come from the master fork, which means that worker processes are independent of each other. In this way, there is almost no synchronization lock between different worker processes to handle concurrent requests. The advantage is that the suspension of one worker process will not affect other processes. We generally set the number of workers to the number of cpus, so as to reduce unnecessary CPU switching and improve performance. Each worker is executed by a single thread. So where does LuaJIT fit into the OpenResty architecture?

The master process with LuaJIT virtual machine is started first, while the worker process is fork from the master process. The work of processes in the worker is mainly completed by Lua coroutines, that is to say, all coroutines in the same worker, All workers share the LuaJIT VIRTUAL machine, and the execution of Lua in each worker process is also completed in this virtual machine.

Worker processes can only handle one user request at a time, meaning only one Lua coroutine is running. Why OpenResty supports millions of concurrent requests requires understanding how Lua coroutines work with Nginx events.

As shown, when a Lua call is made to query MySQL or network IO, the vm suspends itself by invoking the yield of the Lua coroutine and registering the callback in Nginx. At this point, the worker can process additional requests (non-blocking). Nginx calls Resume to wake up the Lua coroutine.

In fact, all of the apis provided by OpenResty are non-blocking, and the interactions with MySQL, Redis, and others mentioned below are non-blocking, so performance is high.

OpenResty request life cycle

Nginx has 11 phases per request, and OpenResty has 11 *_by_lua instructions, as shown below:

The phases *_by_lua are explained below

Set_by_lua: Sets variables; Rewrite_by_lua: forwarding and redirection. Access_by_lua: access and permission. Content_by_lua: Generates returned content; Header_filter_by_lua: response header filtering. Body_filter_by_lua: response body filtering; Log_by_lua: logs are recorded.Copy the code

What’s the advantage of this phasing, assuming your original API requests are in clear text

# Plaintext protocol version
location /request {
    content_by_lua '... ';       # handle requests
}
Copy the code

Now it needs to add encryption and decryption mechanism, which only needs to be decrypted in the Access stage and encrypted in the Body filter stage. There is no need to change the original content logic, which effectively realizes the code decoupling.

# Encryption protocol version
location /request {
    access_by_lua '... ';        Request body decryption
    content_by_lua '... ';       # Handle requests without concern for communication protocols
    body_filter_by_lua '... ';   # Response body encryption
}
Copy the code

For example, one of the core functions of the gateway mentioned above is to monitor logs, so that logs can be reported in log_by_lua without affecting the logic of other phases.

Sharing data between workers: shared dict

Since workers are independent processes, they need to consider sharing data. OpenResty provides an efficient data structure: Shared dict can share data among workers. Shared dict provides more than 20 Lua apis, all of which are atomic operations, avoiding competition problems under high concurrency.

Plug-in implementation of routing policies

With the OpenResty points mentioned above, let’s see how the core gateway function “plug-in routing policy” and “dynamic change of back-end cluster” can be implemented

The first routing policy for a request looks something like this

The steps of the plug-in are as follows

1. Each policy is composed of URL, Action, cluster, etc., which represents the routing rules that the request URL finally experienced in the process of calling the back-end cluster. These rules are uniformly configured in our routing management platform and stored in db.

2. When OpenResty is started, during the init phase of the request, the worker process will pull these rules and compile them into executable Lua functions, which are corresponding to each rule.

It should be noted that in order to avoid repeatedly pulling data from MySQL, after a worker finishes pulling rules from MySQL (this step requires locking to avoid all workers pulling) or back-end cluster configuration information, it should be saved in shared dict. In this way, all worker requests need to get these rules from the shared dict and map them into functions of the corresponding module. If the configuration rules change, the configuration background can notify OpenResty through the interface to reload them

Once the routing rules have determined that each request corresponds to the back-end cluster to call, we need to use upstream to determine which machine in which cluster to call. Let’s see how to manage the cluster dynamically.

Dynamic configuration of back-end clusters

Upstream is configured in the following format in Nginx

upstream backend {
    server backend1.example.com weight=5;
    server backend2.example.com;
    server 192.0. 01. backup;
}
Copy the code

The example above is divided by weight, 6 requests coming in, 5 requests going to backend1.example.com, 1 request going to backend2.example.com, if both machines are unavailable, 192.0.0.1, This static configuration is feasible, but we know that the machine’s expansion and contraction capacity is sometimes quite frequent. If the machine needs to be manually changed every time it goes up and down, and needs to be reload after modification, it is undoubtedly not feasible, and the probability of error is very high. To solve this problem, OpenResty provides a Dyups module to solve this problem. It provides a DyUPS API that can dynamically add, delete, and create upsteam. Therefore, in the init stage, we will first pull the cluster information and build upstream. Later, if the cluster information changes, we will call dyups API to update upstream in the following form

Upstream dynamically configure the upstream interface site
server {
	listen 127.0. 01.:81; location / { dyups_interface; }}- increase upstream: user_backend
curl -d "server 10.53.10.191;" 127.0. 01.:81/upstream/user_backend

- delete the upstream: user_backend
curl -i -X DELETE 127.0. 01.:81/upstream/user_backend
Copy the code

Dyups solves the problem of dynamically configuring upstreams

Gateway final architecture design drawing

Through this design, finally realize the configuration of the gateway, dynamic.

conclusion

As the gateway to carry all the company’s traffic, it has very high requirements on performance, so we should be careful in the selection of technology. The reason why WE choose OpenResty is that it has high performance, and the other reason is that xiaomi, Ali, Tencent and other big companies are using it, which has been tested by the market for a long time. This article provides a brief introduction to OpenResty by summarizing the main functions of OpenResty, but there is more to OpenResty than that. If you are interested, please refer to the tutorial at the end of this article to learn more about it.

digression

“Code sea” history articles have been made into e-books, if necessary, welcome to add my personal wechat, in addition to sending your ebook, you can also join my reader group, discuss problems together, common progress! There are many technical directors and other big coffee, I believe that whether it is career advancement or personal confusion or, can help you oh

Welcome to pay attention to the public number and my personal wechat communication oh

Shoulders of giants

• Talk about API gateway in micro service www.cnblogs.com/savorboard/…
• Openresty dynamic update (not reload) principle and implementation of the TCP Upstream developer.aliyun.com/article/745…
• www.ttlsa.com/Nginx/Nginx…
• Geek time OpenResty from getting started to getting real