In the recent project upgrade, the core code was reconstructed, and the interface address was switched from API to NewAPI. Although a comprehensive test was done before the launch, from the perspective of operation and maintenance, in case of any abnormality after the upgrade, the interface should be quickly switched back, so a layer of reverse proxy was made through Nginx, and newAPI was first proxy by default. You can also switch to the API manually, which involves dynamic upstreams.

Nginx Plus provides key Value storage to enable dynamic upstreams with zero downtime. Unfortunately, we don’t have the money to buy it. However, there are already very mature open source solutions, such as:

  • Weibo nginx – upsync – module
  • Taobao’s ngx_http_dyups_module
  • And slardar of clouds

All of these modules are awesome and can be used to update upstream without reload, but it’s a little bit more difficult to use. Some of them have to compile Nginx manually, and some use Lua scripts. Later found below the Consul and Consul template dynamically update Nginx configuration files, automatically reload plan, implementation and deployment up to the most simple, the records about its installation, use and deployment process.

The installation

First, it’s easy to install. Go to your official address, download consul, extract the package, and use the environment variable PATH to access the consul file. Brew can be installed on Mac:

$ brew install consul
==> Downloading https://homebrew.bintray.com/bottles/consul-1.8.4.catalina.bottle.tar.gz
To have launchd start consul now and restart at login:
  brew services start consul
Or, if you don't want/need a background service you can just run: Consul agent - dev - bind 127.0.0.1 = = > the Summary 🍺 / usr/local/Cellar/consul / 1.8.4:8 files, 103 MBCopy the code

After the installation is complete, type:

$ consul 
Usage: consul [--version] [--help] <command> [<args>]

Available commands are:
    acl            Interact with Consul's ACLs agent Runs a Consul agent catalog Interact with the catalog config Interact with Consul's Centralized Configurations
    connect        Interact with Consul Connect
    debug          Records a debugging archive for operators
    event          Fire a new event
    exec           Executes a command on Consul nodes
    force-leave    Forces a member of the cluster to enter the "left" state
    info           Provides debugging information for operators.
    intention      Interact with Connect service intentions
    join           Tell Consul agent to join cluster
    keygen         Generates a new encryption key
    keyring        Manages gossip layer encryption keys
    kv             Interact with the key-value store
    leave          Gracefully leaves the Consul cluster and shuts down
    lock           Execute a command holding a lock
    login          Login to Consul using an auth method
    logoutDestroy a Consul token created with login maint Controls node or service maintenance mode members Lists the members of a  Consul cluster monitor Stream logs from a Consul agent operator Provides cluster-level toolsfor Consul operators
    reload         Triggers the agent to reload configuration files
    rtt            Estimates network round trip time between nodes
    services       Interact with services
    snapshot       Saves, restores and inspects snapshots of Consul server state
    tls            Builtin helpers for creating CAs and certificates
    validate       Validate config files/directories
    version        Prints the Consul version
    watch          Watch for changes in Consul
Copy the code

You can see that Consul provides many CLI commands, such as start agent, key-value store, and join cluster.

Start the

The following command starts a single-node Consul agent in development mode:

$ consul agent -dev 
consul agent -dev
==> Starting Consul agent...
           Version: '1.8.4'
           Node ID: '3b99c8c0-2bf9-cfb9-fa6b-7b6ac47ed6bf'
         Node name: 'lx-4.local'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: falseClient Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600) Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302) Encrypt: Gossip:false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

==> Log data will now stream in as it occurs:
Copy the code

Note: -dev cannot be used in production because it does not persist any state. If you want to persist it, you need to start it like this:

$ consul agent -server -bootstrap-expect=1 -data-dir=/tmp/consul -bind=192.168.31.134 ==> Starting Consul... Version:'1.8.4'
           Node ID: 'a55e6040-cdeb-42b5-c12f-48e0e85c18b6'
         Node name: 'promote.cache-dns.local'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: true)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
      Cluster Addr: 192.168.31.134 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Copy the code

Note: If the -server option is not added, the client is started by default, and subsequent operations will fail:

$ consul kv put abc 123
Error! Failed writing data: Unexpected response code: 500 (No known Consul servers)
Copy the code

Note: If -bootstrap-expect=1 is not added, the cluster does not have a leader and subsequent operations will fail:

$ consul kv put abc 123
Error! Failed writing data: Unexpected response code: 500 (No cluster leader)
Copy the code

Note: An error may occur if -bind is not added:

==> Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.
Copy the code

Because Consul by default will bind to all addresses on the local machine, and advertise the first available private IPv4 address to the rest of the cluster. If multiple private IPv4 addresses are available, Consul will exit on startup and display the error above.

If data-dir is specified, data will be persisted to/TMP /consul, even if the server is restarted:

$ consul kv put abc 123
Success! Data written to: abc
Data can still be retrieved after server restart
$ consul kv get abc
123
Copy the code

There are other boot parameters that can be added at boot time, such as:

$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=keliq -bind= 192.168.31.134 - config dir/etc/consul. D - the UICopy the code

You can use Consul Agent –help to view all configuration items. Here are some common configurations:

  • -server: Starts the proxy in server mode
  • -data-dir: Configure consul data store path
  • -bootstrap-expectConsul does not boot the entire cluster until the number of sever nodes is specified
  • -bind: Bound IP address. The default value is 0.0.0.0
  • -node: Node name in a cluster. It must be unique in a cluster. The default name is the host name of the node
  • -ui: Opens the Web management page
  • -config-dir: configuration file directory, all to.jsonThe files at the end are loaded, either for the service or Consul’s own configuration
  • -client: Provides HTTP, DNS, and RPC services. The default value is 127.0.0.1 but does not provide external services. If required, change the value to 0.0.0.0
  • -join: Specifies the IP address of an agent that has been started. Multiple IP addresses can be specified
  • -retry-join: Allows you to try after the first time you fail
  • -retry-interval: Indicates the interval between two joins. The default value is 30s
  • -retry-max: The number of repeated join attempts. The default value is 0, that is, unlimited attempts
  • -log-level: Indicates the log level. The default value is INFO. Other options are Trace, DEBUG, INFO, WARN, and ERR.

In addition to being able to start from the command line, you can also write parameters to a configuration file and then use -config-dir to specify a configuration file directory to start. If started this way, all files with the.json or.hcl suffix in the directory are loaded alphabetically and merged together. Note: The -config-dir parameter can be used multiple times to specify multiple configuration file directories.

$ consul agent -config-dir $PWD/config
Copy the code

An example configuration file is as follows:

{
  "bind_addr": "192.168.31.134"."datacenter": "keliq"."data_dir": "/tmp/consul"."encrypt": "txrd7Y61t8Mjim14tNaYB7cNpr8nvcAWwNifaSG3qY8="."log_level": "INFO"."enable_debug": true."node_name": "ConsulServerKeliq"."server": true."ui": true."bootstrap_expect": 1."leave_on_terminate": false."skip_leave_on_interrupt": true."rejoin_after_leave": true
}
Copy the code

Consul Agent is running in server mode. The Consul Agent has only one node and is the leader. If a node fails, data is lost. Therefore, it is not recommended to deploy a single server node. In the case of multiple nodes, you need to run the Consul join command on any node to construct a cluster:

$ consul join <Node A Address> <Node B Address> <Node C Address>
Successfully joined cluster by contacting 3 nodes.
Copy the code

After the Server node is started, client nodes can start joining. A client is a very lightweight process that registers services, runs health checks, and forwards queries to servers, and must run on every host in the cluster. Client nodes can be added by joining any existing node. All nodes are discovered through the Gossip protocol. Therefore, once a node is connected to any cluster member, the new node will automatically find the server node and register itself with the server node.

The command

Consul provides a wealth of commands, such as the following command to view members of the Consul cluster:

$consul members Node Address Status Type Build Protocol DC Segment lx-4.local 127.0.0.1:8301 alive server 1.8.4 2dc1 <all>Copy the code

In addition to shell commands, Consul also provides an HTTP API operation interface, such as the following request to view all nodes:

$ curl localhost:8500/v1/catalog/nodes
[
    {
        "ID": "2ee111ce-3599-7651-6686-9103a4e5975b"."Node": "lx-4.local"."Address": "127.0.0.1"."Datacenter": "dc1"."TaggedAddresses": {
            "lan": "127.0.0.1"."lan_ipv4": "127.0.0.1"."wan": "127.0.0.1"."wan_ipv4": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 7,
        "ModifyIndex": 7}]Copy the code

Consul one of the more useful features in Consul is the key/value store, which can be obtained using the following command: Consul

$ consul kv get nginx/upstream/device  
Error! No key exists at: nginx/upstream/device
Copy the code

Data does not exist, we try to write data:

$ consul kv put nginx/upstream/device/localhost:8080 'weight=1 max_fails=1 fail_timeout=20s; '
Success! Data written to: nginx/upstream/device/localhost:8080
Copy the code

Read data again:

$ consul kv get nginx/upstream/device/localhost:8080
weight=1 max_fails=1 fail_timeout=20s;
Copy the code

Note: read nginx/upstream/device does not exist, because the Key is nginx/upstream/device/localhost: 8080, these are the two different Key, set the value to them will not cover another: :

$ consul kv put nginx/upstream/device localhost:8080
Success! Data written to: nginx/upstream/device
Copy the code

However, we can recursively query all subkeys/values with the -recurse option:

$ consul kv get -recurse nginx/upstream/device
nginx/upstream/device:localhost:8080
nginx/upstream/device/localhost:8080:weight=1 max_fails=1 fail_timeout=20s;
Copy the code

You can also recursively delete:

$ consul kv delete -recurse nginx/upstream/device
Success! Deleted keys with prefix: nginx/upstream/device
Copy the code

In addition to key/value, Consul has a powerful feature: It can monitor data via Watch and trigger the execution of specified handlers if changes occur. For example, echo ‘hello change’ is executed whenever the hello key changes:

$ consul watch -type key -key hello echo 'hello change'
Copy the code

There are two types of watch

  • Key: listens for key-value pair changes
  • Keyprefix: monitors the change of the specified prefix key pair
  • Services: Listens to the list of services
  • Nodes: list of listening nodes
  • Service: listens on service instances
  • Checks: monitors health checks
  • Event: Monitors user events

Consul also provides a Web management interface, which can be accessed at http://localhost:8500 by specifying the -ui parameter at startup. The following figure shows how to add key-value pairs on the Web interface:

The Web management interface provides comprehensive functions, such as service registration, key/value pair, and account permission management.

Consul template

Consul Template is a lightweight tool that reads Consul content and dynamically generates profiles. You can not only generate profiles from templates in real time, but also execute commands after they are generated. It is a very powerful automation tool. Consul Template is also easy to install, and can be downloaded directly into the path directory:

$$tar XZF wget HTTP: / / https://releases.hashicorp.com/consul-template/0.25.1/consul-template_0.25.1_linux_amd64.tgz Consul-template_0.25.1_linux_amd64. TGZ $cp consul-template /usr/local/bin
$ ls /usr/local/bin consul-template docker-compose hexo pm2 pm2-dev pem2-docker pm2-Runtime $consul-template -v consul-template v0.25.1  (171d54d)Copy the code

Brew installation can also be used under Mac:

$ brew install consul-template
Copy the code

View options:

$ consul-template -h
Copy the code

Generate a configuration file from a template:

$ consul-template \
  -consul-addr demo.consul.io \ # Consul service address
  -template "/tmp/template.ctmpl:/tmp/result" The path to the template and build file
Copy the code

}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}} The template. CTMLP template file is in the following format:

upstream device_upstream {
{{range ls "upstreams/device" -}}
{{""}}server {{.Key}} {{.Value}}
{{end -}} 
}
Copy the code

Where {{and}} represent left and right delimiters, and the middle part is the variable read from Consul. However, delimiters can be configured:

left_delimiter  = "{{"
right_delimiter = "}}"
Copy the code

You can also add commands that are automatically executed when the configuration file is updated:

$ consul-template \
  -consul-addr demo.consul.io \
  -template "/tmp/template.ctmpl:/tmp/result:service nginx restart"
Copy the code

It also supports simultaneous rendering of multiple templates and execution of multiple commands:

$ consul-template \
  -consul-addr my.consul.internal:6124 \
  -template "/tmp/nginx.ctmpl:/var/nginx/nginx.conf:service nginx restart" \
  -template "/tmp/redis.ctmpl:/var/redis/redis.conf:service redis restart" \
  -template "/tmp/haproxy.ctmpl:/var/haproxy/haproxy.conf"
Copy the code

These parameters can be provided to Consul Template in the form of a configuration file:

$ consul-template -config /root/config.hcl
Copy the code

The config. HCL file format is as follows:

consul {
  address = "10.154.156.24:8500"
}

template {
  source = "ctmpl/nginx.conf.ctmpl"
  destination = "conf/test-consul-template.conf"
  command = "echo done"
}
Copy the code

At this point, a dynamic upstream solution is complete, summarized as follows:

The Nginx configuration file is dynamically generated using Consul template. If the Consul value changes, the local Nginx configuration file is updated and the Nginx -s reload command is launched.

Advanced content

The above only introduces the basic usage of Consul. In practice, there are also some issues related to Docker deployment, ACL permission, and data backup. The details are not described here, but recorded as follows:

The container is changed

Consul official also provides Docker containers for easy use:

$ docker pull consul # Download image
$ docker run --name consul -p 8500:8500 -itd consul # start container
Copy the code

Note The port number must be mapped to the host; otherwise, it cannot be accessed externally.

Accounts and Permissions

By default, Consul does not enable account and permission (ACL) on Consul. To enable this function, run the following command on the console:

$ consul acl bootstrap
Failed ACL bootstrapping: Unexpected response code: 401 (ACL support disabled)
Copy the code

However, when Consul starts, you need to add the following configuration information to the configuration file. Otherwise, an error message is displayed:

{
  "acl": {
    "enabled": true."default_policy": "deny"."down_policy": "extend-cache"}}Copy the code

Then you can see the administrator account and password:

$ consul acl bootstrap
AccessorID:       6e15f2d3-7213-2f80-c8af-986e9733830b
SecretID:         1b352e5a-1ff1-ffa9-a56c-00efd7472cd6
Description:      Bootstrap Token (Global Management)
Local:            falseCreate Time: 2020-10-23 14:34:13.590304 +0800 CST Policies: 00000000-0000-0000-0000-000000000001-global-managementCopy the code

SecretID = password = AccessorID = password = SecretID = password = password

$ consul members -token 1b352e5a-1ff1-ffa9-a56c-00efd7472cd6
$ curl --location --request GET 'http://localhost:8500/v1/catalog/nodes' \
--header 'X-Consul-Token: 1b352e5a-1ff1-ffa9-a56c-00efd7472cd6'
Copy the code

If you’re having trouble, you can set environment variables and Consul will automatically include:

$ export CONSUL_HTTP_TOKEN=1b352e5a-1ff1-ffa9-a56c-00efd7472cd6
Copy the code

To view all tokens, run the following command:

$ consul acl token list
AccessorID:       6e15f2d3-7213-2f80-c8af-986e9733830b
Description:      Bootstrap Token (Global Management)
Local:            false
Create Time:      2020-10-23 14:34:13.590304 +0800 CST
Legacy:           false
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

AccessorID:       00000000-0000-0000-0000-000000000002
Description:      Anonymous Token
Local:            false
Create Time:      2020-10-23 14:34:06.170368 +0800 CST
Legacy:           false
Copy the code

Account 002 was automatically created by Consul, but it does not have any permissions and represents an anonymous account. The first bootstrap token is a management token with unlimited permissions because it has global-management policies. Check the list of policies:

$ consul acl policy list
global-management:
   ID:           00000000-0000-0000-0000-000000000001
   Description:  Builtin Policy that grants unlimited access
   Datacenters:
Copy the code

If we want to be able to see all nodes in most environments without token. We can configure a Consul policy that allows anonymous token access to the node list when no token is needed.

Create a permission policy for viewing all nodes
$ consul acl policy create -name 'list-all-nodes' -rules 'node_prefix "" { policy = "read" }'
ID:           72574f4a-1c7b-8b67-44c4-1331da37b5e4
Name:         list-all-nodes
Description:  
Datacenters:  
Rules:
node_prefix "" { policy = "read" }
Copy the code

Then update the access to the anonymous token:

$ consul acl token update -id 00000000-0000-0000-0000-000000000002 -policy-name list-all-nodes -description "Anonymous Token - Can List Nodes"
AccessorID:       00000000-0000-0000-0000-000000000002
SecretID:         anonymous
Description:      Anonymous Token - Can List Nodes
Local:            falseCreate Time: 2020-10-23 14:34:06.170368 +0800 CST Policies: 72574f4A-1C7b-8b67-44C4-1331da37b5e4 -list-all-nodesCopy the code

The token is not required to access the node later. In the same way, you can add other permissions to it:

Create a policy for accessing Consul
$ consul acl policy create -name 'service-consul-read' \
                           -rules 'service "consul" { policy = "read" }'
Update the anonymous token
$ consul acl token update -id 00000000-0000-0000-0000-000000000002 \
--merge-policies \
-description "Anonymous Token - Can List Nodes" \
-policy-name service-consul-read
Copy the code

Create a policy that reads only the specified key or key prefix:

# specify that only nginx key can be read
$ consul acl policy create -name "kv-nginx-read" \
                           -description "read kv" \
                           -rules 'key "nginx" { policy = "read" }'
ID:           75ead007-71bd-92b1-ea23-621546aee435
Name:         kv-nginx-read
Description:  read kv
Datacenters:  
Rules:
key "nginx" { policy = "read" }
# read all keys starting with nginx
$ consul acl policy create -name "kv-nginx-prefix-read" \
                           -description "read kv with nginx prefix" \
                           -rules 'key_prefix "nginx" { policy = "read" }'
ID:           97b8f62d-2f87-0ea8-61d4-7927dc7d0eac
Name:         kv-nginx-prefix-read
Description:  read kv with nginx prefix
Datacenters:  
Rules:
key_prefix "nginx" { policy = "read" }

# give anonymous token access to nginx KV
$ consul acl token update -id 00000000-0000-0000-0000-000000000002 \
--merge-policies \
-description "Anonymous Token - Can read nginx kv" \
-policy-name kv-nginx-read

Create a new account and give it access to the nginx key
$ consul acl token create -description "Nginx Token" -policy-name "kv-nginx-read"
AccessorID:       532ae236-81a6-18fe-fce1-c5817482bb34
SecretID:         9f671c12-175f-2782-a7e0-2733e2bf94d4
Description:      Nginx Token
Local:            falseCreate Time: 2020-10-23 15:50:07.458503 +0800 CST Policies: 75EAD007-71BD-92B1-eA23-621546aEE435-kv-nginx-readAfter the account is created, you can also update the permissions
$ consul acl token update -id 532ae236-81a6-18fe-fce1-c5817482bb34 \
--merge-policies \
-description "Anonymous Token - Can List Prefix Nginx" \
-policy-name kv-nginx-prefix-read
AccessorID:       532ae236-81a6-18fe-fce1-c5817482bb34
SecretID:         9f671c12-175f-2782-a7e0-2733e2bf94d4
Description:      Anonymous Token - Can List Prefix Nginx
Local:            false
Create Time:      2020-10-23 15:50:07.458503 +0800 CST
Policies:
   75ead007-71bd-92b1-ea23-621546aee435 - kv-nginx-read
   97b8f62d-2f87-0ea8-61d4-7927dc7d0eac - kv-nginx-prefix-read
Copy the code

The data backup

Consul will persist data on Consul, but you may need to manually create a snapshot or save the data using the export and snapshot commands, for example, to export all key values to local:

$ consul kv export  ' ' > consul.d/consul_kv.json
Copy the code

You can import into Consul later:

$ consul kv import consul_kv.json
Copy the code

Generate a snapshot:

$ consul snapshot save consul_state.snap
Copy the code

Restoring a snapshot:

$ consul snapshot restore consul_state.snap
Copy the code