Authors: A Shang, Hu Ming

Pain points and requirements

– High cost: Advertising campaign platform has always been a thing with the most changes and the most manpower consumption, and every time we have to develop a new page, and each page is only used for two or three days.

– Services have the following basic requirements for the platform: c-terminal stability and B-terminal flexibility.

– Technical requirements for the platform: service stability should be 99.999%

Project introduction

The main function of the configuration platform is that it can generate activity and other related pages flexibly, conveniently and quickly, reducing the development cost. The generation of new activities does not require additional development work, but can be quickly online with simple configuration. This introduction covers the background configuration and the client side, but before we get started, let’s take a quick look at Openresty and Lua.

Openresty

OpenResty (also called: Ngx_openresty (ngx_OpenResty) is a high-performance Web platform based on Nginx and Lua, which integrates a large number of excellent Lua libraries, third-party modules and most of the dependencies. It is used to easily build dynamic Web applications, Web services and dynamic gateways that can handle ultra-high concurrency and scalability.

The goal of OpenResty is to have your Web services run directly inside Nginx services, taking full advantage of Nginx’s non-blocking I/O model, not just for HTTP client requests, Even remote backends such as MySQL, PostgreSQL, Memcached, and Redis provide consistent high performance responses. Therefore, for some high performance services, OpenResty can directly access Mysql or Redis, instead of accessing the database and returning through a third party language (PHP, Python, Ruby), which greatly improves the performance of the application.

Lua

Lua is a lightweight scripting language written in standard C and open as source code. It is designed to be embedded in applications to provide flexible extension and customization capabilities.

Lua features

– Lightweight: it is written in standard C language and open in the form of source code, compiled only more than one hundred K, can be easily embedded in other programs.

– Extensible: Lua provides very easy to use extension interfaces and mechanisms that are provided by the host language (usually C or C++). Lua can use them as if they were already built in.

– Other features:

  • Support for procedure-oriented programming and functional programming;

  • Automatic memory management. Only provides a general type of table (table), with it can achieve array, hash table, collection, object;

  • Language built-in pattern matching; Closure; A function can also be viewed as a value; Multithreading (collaborative processes, not threads supported by the operating system) support;

  • Closures and tables make it easy to support some of the key mechanisms needed for object-oriented programming, such as data abstraction, virtual functions, inheritance, and overloading.

Application scenarios

  • The game development

  • Standalone application scripts

  • Web Application Scripts

  • Extensions and database plug-ins such as MySQL Proxy and MySQL WorkBench

  • Security systems, such as intrusion detection systems

The background configuration

Background configuration processes involve Openresty, Nginx, Lua scripts, ElasticSearch, and Redis.

Nginx configuration

Some nginx.config configurations are as follows:

1. Server configuration

The locations configuration is also imported into the server configuration as follows:

When the request URL matches a string containing /example/, the request data is processed, which contains:

  1. A single JSON string is limited to 5M

  2. Lua handles the request access phase

  3. Forward to Es service

Single-json limits and request-style interceptions are clearly understood in the configuration, but what does Lua do?

2. Lua

In this configuration platform, the main function of Lua is to synchronize data to Redis, so that when the client accesses the data, if it is already in Redis and not expired, the data will be directly read in Redis, instead of obtaining the data through ES service.

Let’s look at the implementation. First, lua-dependent configuration is introduced in location, and lua-dependent processing is embedded by access_by_lua_file.

(1) Link Redis

(2) PUT and POST request data cache

If it is a PUT or POST request, the data will be cached and the cache expiration time will be set. The code is as follows:

(3) DELETE request to clear cache

When the request is DELETE, the data is cleared from the Redis data and the cache expiration time is set to 0, with the following code:

ElasticSearch

ElasticSearch, ES for short. ES is an open source and highly extensible distributed full text search engine. It can store and retrieve data in near real time. Its scalability is very good, can be extended to hundreds of servers, processing PB level of data. ES is also developed in Java and uses Lucene as its core to implement all indexing and searching functions, but it aims to hide the complexity of Lucene through a simple RESTful API to make full-text searching simple.

ElasticSearch vs. traditional relational database:

Elasticsearch – head plug-in

You can use the elasticSearch-head plugin to check the index, type, shard, document, etc., as shown in the following figure:

– ES Indicates background configuration

For example, the section configuration contains the fragment Settings in Settings, mappDings for menu, and the partial index information in elasticSearch-head is as follows:

client

Previously, we configured some activity-related data through the background of configuration platform and stored them in ES service and Redis. On the client side, when a user accesses our generated active page, the content_by_lua_file configuration will first use Lua in Nginx to handle requests related to the active configuration. For non-GET requests, 403 will be returned. Otherwise the data will be queried in Redis and returned to the client.

– Server configuration in client Nginx

– Lua requests access processing on the client

Lua link Redis has been mentioned in the background configuration as well as the interception handling for different request modes. Next, let’s look at the related processing on the client side, the code is as follows:

As you can see, intercept requests other than Get and return 403, otherwise Redis will be linked as follows: