Nginx dynamic discovery scheme and practice

2. Module architecture

Previous research has found that some companies are using etCD/Consul + Nginx-upsync-module to implement nginx zero restart updates to upstream. Instead of using ETCD or Consul internally to store back-end instance configurations, we use zK services extensively to store back-end configurations. Most businesses register instance IP with ZK, so our nginx needs to pull back-end instance IP and port from ZK. Some students in our company also developed a module for nginx to connect zK, but this module connects ZK through each worker process. An Nginx may have multiple or even dozens of worker processes, which will cause a sudden increase in the number of ZK connections and bring great pressure to THE ZK cluster. We found the dyUPS module through research, and then connected to ZK through our own coding, pulled the configuration from ZK, and updated it to upstream’s shared memory through the interface of the DyUPS module. We can also update the list of Nginx upstreams with zero restart. At the same time, by coding the logic to interact with ZK, you can also control the logic to be executed when ZK is not available.

The dyups module is a nginx module that can directly update the list of upstreams for the running nginx without needing to reload the nginx configuration. The module does this by opening an interface through which an external post or GET request is made to update or retrieve the corresponding upstream’s back-end list. You can visit the website for more detailed usage

Check out the Github description for this module. However, since the Dyups module can only modify the shared memory of nginx and cannot persist the current upstream configuration into a file, another core task of our module is to persist the upstream configuration into a configuration file.

This module combines the application scenarios of our company’s common business, problems encountered in daily use and shortcomings of DyUPS, and mainly realizes the following functions:

1) Obtain the backend list registered in ZK, format the obtained list data, and save it to the corresponding Nginx configuration file for persistence

2) Write the back-end server list saved to the file to the shared memory of the nginx upstream through the interface of the Dyups module and dynamically update the back-end list in the upstream

3) In the event of a ZK failure, this module will not update the nginx shared memory and local Nginx configuration file, while keeping the nginx upstream configuration at the same state as before the ZK failure

4) Support to read multiple ZK node configurations of multiple ZK clusters

Base dependencies:

Support dyups module nginx

Python 2.6/2.7

Here we put the module code in the /home/work directory

cd /home/work
git clone http://v9.git.n.xiaomi.com/liuliqiu/nginx-upstream-reloader.gitCopy the code

cd nginx-upstream-reloader
bash install_venv.shCopy the code

After venv is installed, modify the upstream_zk_nodes.conf configuration file in the nginx-upflow-reloader /conf directory. This configuration file is used to define the destination ZK cluster and ZK node where the back-end instance resides and the name of the corresponding Nginx upstream. This can be modified as follows:

1) If multiple back-end services are registered in a ZK cluster, perform the following operations

upstream_zk_nodes.conf
zk_servers:  zk-hadoop-test01:11000,zk-hadoop-test02:11000
zk_nodes:
    bonus-api: /web_services/com.miui.bonus.api.resin-webCopy the code

Zk_servers: zK cluster address and port registered by the back-end service

Zk_nodes: upstream_name: zK node path registered by the back-end service

When we start up, the module will automatically generate upstream_name. Upstream by pulling the backend list information from the specified ZK node path. This directory is controlled by the files_output_path option in the nginx-upstream-reloader/conf/main.conf configuration file. “/home/work/nginx/site-enable” : “/home/work/nginx/site-enable” : “/home/work/nginx/site-enable” : “bonus-api.upstream” : “bonus-api.upstream” : “bonus-api.upstream” : “bonus-api.upstream” : “bonus-api.upstream” : “bonus-api.upstream” : “bonus-api.upstream” :

upstream bonus-api { server .... ; server .... ; }Copy the code

2) Multiple backend servers are registered in different ZK clusters

upstream_zk_nodes.conf - zk_servers: tjwqstaging.zk.hadoop.srv:11000 zk_nodes: ocean-helloworld-upstream1: /ocean/services/job.ocean-helloworld-nginx-upstream_service.ocean-helloworld-nginx-upstream_cluster.staging_pdl.oceantes t_owt.inf_cop.xiaomi ocean-helloworld-upstream2: /ocean/services/job.ocean-helloworld-nginx-upstream_service.ocean-helloworld-nginx-upstream_cluster.staging_pdl.oceantes t_owt.inf_cop.xiaomi - zk_servers: tjwqstaging.zk.hadoop.srv:11000 zk_nodes: ocean-helloworld-upstream3: /ocean/services/job.ocean-helloworld-nginx-upstream_service.ocean-helloworld-nginx-upstream_cluster.staging_pdl.oceantes t_owt.inf_cop.xiaomiCopy the code

Zk_servers: zK cluster address and port registered by the back-end service

Zk_nodes upstream_name: zK node path registered by the back-end service

Some students told me why they use configuration files in YAML format instead of JSON format. Json has less strict format requirements than YAML, but yamL configuration files look more hierarchical.

When we start the module, the module automatically generates the upstream_name. Upstream file by pulling the backend list information from the specified ZK node path. The module generates one at /home/work/nginx/site-enable Upstream, ocean-helloworld-upstream1.upstream, ocean-helloworld-upstream2.upstream, ocean-helloworld-upstream3.upstream, ocean-helloworld-upstream3.upstream The contents of the file will be as follows:

upstream ocean-helloworld-upstream1 { server ... ; server ... ; } upstream ocean-helloworld-upstream2 { server ... ; server ... ; } upstream ocean-helloworld-upstream3 { server ... ; server ... ; }Copy the code

The upstream configuration file is automatically generated after connecting to the ZK node to obtain the back-end configuration. We need to include upstreams in nginx to use them in the server block. Currently, due to the limitations of the dyups module, you need to set upstream_name as a variable, and then use this variable in the proxy_pass directive to configure forwarding. For details, see the following configuration:

include /home/work/nginx/site-enable/ocean-helloword-upstream1.upstream;
include /home/work/nginx/site-enable/ocean-helloword-upstream2.upstream;
include /home/work/nginx/site-enable/ocean-helloword-upstream3.upstream;

server {
        listen   80;
        location /helloworld1 {
                set $ups1 ocean-helloword-upstream1;
                proxy_pass http://$ups;
        }
        location /helloworld2 {
                set $ups2 ocean-helloword-upstream2;
                proxy_pass http://$ups2;
        }
        location /helloworld3 {
                set $ups3 ocean-helloword-upstream3;
                proxy_pass http://$ups3; }}Copy the code

Add a separate server here that listens on port 14443 of the local address

Nginx-upstream-reloader /start.sh: nginx-upstream-reloader: nginx-upstream-reloader: nginx-upstream-reloader: nginx-upstream-reloader Upstream has not yet been generated, but we include these upstream profiles in our nginx configuration file, and an error will be reported when starting nginx

bash nginx-upstream-reloader/start.sh
/home/work/nginx/sbin/nginx Nginx is installed in /home/work/nginx/Copy the code

The module has been tested on centos6 and Centos7 and is suitable for containers and physical machines.