Author: HelloGitHub- Lao Xun

Hello, here is HelloGitHub’s HelloZooKeeper series, free, open source, fun, entry-level ZooKeeper tutorials for beginners with basic programming skills.

Project address: github.com/HelloGitHub…

Today I’m going to introduce some hidden features of ZK, so let’s get started

A, JMX

JMX (Java Management Extensions) is a framework for implementing Management capabilities for applications, devices, systems, and so on. JMX allows flexible development of seamlessly integrated system, network and service management applications across a range of heterogeneous operating system platforms, system architectures and network transport protocols.

Don’t you understand? If you’ve never developed JMX or an application before, I’ll give you a brief introduction: JMX is a standard provided by Java that exposes the runtime state of Java objects (called MBeans) that are needed to monitor or modify configuration information at run time.

Now that I have a simple ZK cluster running, I want to know which node is the Leader. What should I do?

First of all, ZK itself will actively register some objects as MBeans when starting, and we can directly use the Java built-in tool JConsole to check, below I demonstrate, I have a simple ZK cluster:

$ jps
5266 QuorumPeerMain
2964
10438 Jps
4550 Launcher
5286 QuorumPeerMain
5767 JConsole
5388 QuorumPeerMain
9215 Launcher
Copy the code

You can see that there are three QuorumPeerMain processes representing the three nodes of the cluster.

Let’s use JConsole to open the tool (the tool is automatically available when you install the JDK)

$ jconsole
Copy the code

Select any ZK process and select Connect.

JMX can look at objects’ properties and perform methods such as:

All of the operations shown in the figure can be called by a button on the right (and can also be passed as an argument). I won’t disclose more details about JMX in ZK, but I will explain them separately later. All you need to know is that JMX exposes ordinary Java objects. A standard for viewing properties or methods that call the object through tools.

Admin Server

The JMX query is a bit of a hassle, because now we are testing to access my local process. If it is a remote JVM process, it will be even more troublesome to access with JConsole. Is there an easier way to access it? There must be! ZK itself supports four word commands (4lW) for interacting with the server.

The client ports of my local cluster are 2181, 2182, and 2183 respectively. I connect to a node randomly through Telnet command:

$ telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Copy the code

Enter interactive mode, type SRVR and press Enter to get the following output

$ telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'. srvr Zookeeper version: 3.6.2-803 c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 on GMT Latency min/avg/Max: 0/3.8261/44 Received: 30 Sent: 29 Connections: 1 Outstanding: 0 Zxid: 0x100000009 Mode: leader Node Count: 6 Proposal sizes last/min/max: 48/48/94 Connection closed by foreign host.Copy the code

The SRVR command is used to check the status of the service node. It can be seen from the Mode field in the output that the node listening to port 2181 is the Leader. Let’s change another node 2182 to see that Mode is the Follower.

$ telnet localhost 2182
Trying ::1...
Connected to localhost.
Escape character is '^]'. srvr Zookeeper version: 3.6.2-803 c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 on GMT Latency min/avg/Max: 0/0.0/0 Received: 3 Sent: 3 Connections: 1 Outstanding: 0 Zxid: 0x100000009 Mode: follower Node count: 6 Connection closed by foreign host.Copy the code

Here to mention the default four word command not fully open, if you want to enable all four word order needs to be specified in the environment variable zookeeper.4lw.com mands. Whitelist = *, can also by listing the specific command (comma-separated) to enable a specified some words of command. Let’s try another command, such as envi, which prints the environment parameters of the current node

java.io.tmpdir=/var/folders/19/bx8xsqgd1c78g5j1mt_zq5v80000gp/T/ java.compiler=<NA> os.name=Mac OS X os.arch=x86_64 OS, version = 10.16 user name = junjiexun user. Home = / Users/junjiexun user. Dir = / Users/junjiexun/develop/zk os.memory.free=101MB os.memory.max=889MB os.memory.total=123MBCopy the code

Here I list all the functions of the four word command, I will not demonstrate the specific, leave the reader to try, the official website of the four word command list, the four word command is actually the alias of its right command, the function is exactly the same. (* still means TODO.

The command The role of returning data
conf / configuration Configuration information (commondataDir,clientPortEtc.)
cons / connections The connection information
crst / connection_stat_reset Reset connection statistics
dump Session information and temporary nodes
envi / environment Environment variable information (os.name,user.homeEtc.)
ruok Service is normal
srst / stat_reset Resetting statistics
srvr / server_stats Server information Overview
stat Statistics of server information
wchs / watch_summary Call back watcher’s summary
wchc / watches Register the session information summary for callbacks
dirs Log and SNAP file size in bytes
wchp / watches_by_path Register the path information for the callback
mntr / monitor All monitoring information
isro / is_read_only Whether the current node is read-only
hash Numbers in this paper,
gtmk / get_trace_mask* Gets the trace mask
stmk / set_trace_mask* Set the trace mask
lsnp / last_snapshot Information about the last snapshot
icfg / initial_configuration The server starts the initial configuration
orst / observer_connection_stat_reset Reset Observer connection statistics
obsr / observers Get Observer information
sysp / system_properties Environment variable information andenviThe difference is that it returnszookeeperCustom configuration at the beginning
lead / leader Whether the current node is the Leader
voting_view Cluster ballot information
zabstate Summary of ZAB status

I have also seen ruOK, which is used to check whether the server node is started (successful return does not mean external service can be provided).

In addition to Telnet to the four-word commands provided by ZK, ZK also provides a more friendly way to call the Admin Server.

ZK starts by default listening on port 8080 of the host and starts a Jetty container as a Web server. Accessing the/Commands path on this port yields:

Accessing these hyperlinks directly has the same effect as the previous four-word commands. You can also access IP directly from the URL :port/commands/

.

MNTR, for example, you can directly visit http://localhost:8080/commands/mntr or http://localhost:8080/commands/monitor are the same.

Because the Admin Server is enabled by default and accept requests from any IP, for the sake of safety of words can be configured environment variables to zookeeper. Admin. ServerAddress = 10.3.54.12, Such increase request IP requirements or directly through the zookeeper. Admin. EnableServer = false to disable the admin Server.

ZK officially recommends using Admin Server directly, instead of the command line four word command, based on these official interface is possible to do some ZK monitoring platform.

Dynamic configuration

The ZK cluster configuration is usually read from the configuration file at startup and is not changed after that, and if I want to add a new node to the cluster, I need to modify the configuration file and restart to take effect.

However, after 3.5.0, ZK has updated the function of dynamic configuration, cluster configuration no longer need to stop the reconfiguration, can be directly modified at runtime, can directly add and remove nodes to the cluster, modify their role, and even modify the cluster vote counting rules! Isn’t it a bunker?

3.1 Vote counting Rules

Before we do that, LET me introduce the two voting rules that ZK supports.

3.1.1 Half mechanism

This is the ZK default counting rule for various server clusters that require ACK, assuming the current configuration is like this:

server.1=zoo1:2888:3888:participant
server.2=zoo2:2888:3888:participant
server.3=zoo3:2888:3888:participant
server.4=zoo4:2888:3888:observer
server.5=zoo5:2888:3888:observer
Copy the code

Since observers do not count votes, the actual machines that participate are the first three nodes: 1, 2, and 3

Regardless of who the Leader is, the default vote counting rule requires that at least two of these three nodes successfully submit acks (or other information that needs to be counted) before the election (or proposal) can be submitted again. This is the half-majority mechanism.

3.1.2 Group Weight

ZK also provides a new counting rule, which supports dividing each node into different groups (of course, only one group can be used). Different nodes in the same group can also be assigned different weights. Here is an example:

group.1=1:2:3
group.2=4:5:6:7:8
group.3=9

weight.1=1
weight.2=1
weight.3=1
weight.4=1
weight.5=1
weight.6=1
weight.7=1
weight.8=1
weight.9=1
Copy the code

Group and weight correspond to group and weight configurations as follows:

  • The format of group isgroup.<groupId>=<serverId>:<serverId>...
  • The format of weight isweight.<serverId>=<weight>
  • The serverId is the number configured in myID for each service node
  • Each node can belong to only one group

If I continue with my current configuration, there are three groups in total, and their weights are calculated as follows:

Total weight of group1 = Server1 + Server2 + Server3 = 1 + 1 + 1 = 3 Total weight of group2 = Server4 + Server5 + Weight of Server6 + Weight of Server7 + Weight of Server8 = 1 + 1 + 1 + 1 = 5 Total weight of Group3 = Weight of Server9 = 1Copy the code

If there are 1, 4, 5, 8, 9 service nodes for successful ACK and the count is carried out in this configuration

  • For group1, only server1 successfully replies to ACK. The weight value is 1, which does not exceed half of the total weight of group1 (3)
  • Group2 has three nodes (4, 5, and 8) successfully replied to ACK, with the weight value of 3, which is more than half of the total weight of 5 of Group2 (3 > 5/2)
  • Then look at group3, because there is only one node 9, and successfully reply to ACK, so it also meets more than half of the weight sum 1 of group3 (1 > 1/2), so group3 ACK succeeds
  • Finally, check whether the number of successful ACK groups exceeds more than half of the total number of groups. Now, there are two successful ACK groups (2 > 3/2), so the ACK finally passes

It looks pretty much like the default half mechanic, No, No, No. And the reason why it looks the same here is because I set the weight to 1, what if I set it to some other number?

The ACK for group1 failed because server1 was the only node to reply successfully. However, if I change the weight of group1 to group1 (the other two groups are omitted), the ACK for group1 fails.

group.1=1:2:3

weight.1=3
weight.2=1
weight.3=1
Copy the code

Now the total weight of group1 becomes 3 + 1 + 1 = 5, and the weight of server1 is 3. Even if only it replies, it is more than half of group1 (3 > 5/2). If the weight is configured at this time, Group1 also counts as a successful ACK.

It should also be noted that when ZK reads these configurations, it will calculate the sum of the weights of each group. If the sum of the weights of a certain group is calculated to be 0, the group will be removed from the vote counting rule.

Unlike the default half-rule mechanism, it is possible to involve the Observer with reconfiguration.

3.2 Recommended Configuration

After all this talk, how to use this weight counting rule?

  • (Recommended) inzoo.cfgIn the configurationdynamicConfigFileThe server, group, and weight option is used to specify a dynamically configured path address to move all configurations starting with server, group, and weight to the configuration file for that path.
  • Configure all server, group, and weight configurations directly in thezoo.cfgIn the file

As long as ZK finds a configuration starting with group or weight in the configuration file, it indicates that the weight counting rule is enabled. Otherwise, the default half-rule mechanism is used.

Our previous server prefix configuration looked like this:

server.1=zoo1:2888:3888:participant
Copy the code

The actual server configuration should look like this, where the client port can be configured in the Server configuration (after the last semicolon)

server.1=zoo1:2888:3888:participant; 2181Copy the code

If configured this way, zoo.cfg does not require the clientPort option.

So zoo.cfg is configured in the recommended way (the path should be adjusted according to the reader’s computer).

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/junjiexun/develop/zk/zk01/data
dynamicConfigFile=/Users/junjiexun/develop/zk/zk01/conf/zoo.dyn.cfg
...
Copy the code

And I am/Users/junjiexun/develop/zk/zk01 / conf/zoo. The dyn. CFG file can be configured

Server. = 127.0.0.1 10000000000:2888-3888: the participant; 2181 server. 2 = 127.0.0.1:2887-3887. 2182 server. 3 = 127.0.0.1:2886-3886; 2183 group.1=10000000000:2:3 weight.10000000000=1 weight.2=1 weight.3=1Copy the code

See the official documentation for more details

3.3 Dynamic Change

So the question is, how can I dynamically add or delete nodes to the cluster after I have configured them?

The Java client provides a getConfig method

ZooKeeper client = new ZooKeeper("127.0.0.1:2181".3000.null);
byte[] config = client.getConfig(false.null);
System.out.println(new String(config));
client.close();
Copy the code

The printed result is

Server. 2 = 127.0.0.1:2887-3887: the participant; 0.0.0.0:2182 server. 3 = 127.0.0.1:2886-3886: the participant; 0.0.0.0:2183 server. = 127.0.0.1 10000000000:2888-3888: the participant; 0.0.0.0:2181 group.1= 2:3:100000000000 weight.2=1 weight.3=1 weight.10000000000=1 version=0Copy the code

The information looks a little similar to but a little different from zoo.dyn. CFG configured by us. The difference is actually the format that ZK helps us to automatically complete, and where does ZK store the returned data? ZK creates the following nodes in the root path by default when it starts

/
|--zookeeper
     |--config
     |--quota
Copy the code

GetConfig returns the same data as the/zooKeeper /config node, which has only Read permissions

ZooKeeper client = new ZooKeeper("127.0.0.1:2181".3000.null);
byte[] config = client.getData("/zookeeper/config".false.null);
System.out.println(new String(config));
client.close();
Copy the code

Now that the configuration is available, how do I modify it? ZK officially provides two ways: command line, Java API.

If you use Java’s built-in command-line tools, among the supported commands is the reconfig command with the following arguments:

reconfig [-s] [-v version] [[-file path] | [-members serverID=host:port1:port2;port3[,...]*]] | [-add serverId=host:port1:port2;port3[,...]]* [-remove serverId[,...]*]
Copy the code

ZooKeeper has a subclass called ZooKeeperAdmin. This subclass has a reconfigure method that can modify the configuration. Let me show you. But before I do that, I have to explain the dynamic configuration modification feature

  • Dynamic configuration modification can be performed in either incremental or non-incremental modes
  • Because what you’re actually modifying is/zookeeper/configData for the node, which by default has only Read permissions, so you can either modify it directly with administrator permissions or configure it in environment variableszookeeper.skipACL=yesThe ACL verification is skipped
  • When using the incremental mode to modify the configuration, the cluster vote rule must be half mechanism!
  • When the configuration is modified in non-incremental mode, both mechanisms can be used.
  • Deleting followers from the cluster configuration merely relegates it to Observer status, allowing it to still serve as a server and receive messages from the Leader
  • If the Leader is deleted from the cluster configuration, the cluster cannot provide services until a new Leader is selected. Therefore, do not do this
  • Adding nodes is much easier. The newly added nodes automatically synchronize data with the Leader

3.3.1 Deleting a Node incrementally

Let’s say I have 3 nodes in total, using the half mechanism (mandatory), and the three ids are 10000000000, 2, and 3. We try to delete the node with ID 3, and I use skipACL to skip permission verification.

ZooKeeperAdmin client = new ZooKeeperAdmin("127.0.0.1:2181".3000.null);
List<String> leavingServers = new ArrayList<>();
leavingServers.add("3");
byte[] reconfigure = client.reconfigure(null, leavingServers, null, -1.null);
System.out.println(new String(reconfigure));
client.close();
Copy the code

The interface returns the configuration information after the modification of /zookeeper/config. You can see that the data related to 3 in the new configuration disappears

Server. 2 = 127.0.0.1:2887-3887: the participant; 0.0.0.0:2182 server. = 127.0.0.1 10000000000:2888-3888: the participant; 0.0.0.0:2181 version = 400000004Copy the code

What does version=400000004 do? ZK to modify the configuration of the default also provides version control, will start after the success in your configuration automatically generated under the dynamicConfigFile path of a file, I here is the zoo. CFG. Dynamic. 300000000 readers probably is not the same with me. The 300000000 is the version number, and when I put my ID to 3 after deleting nodes, ZK and automatically generate a file zoo. CFG. Dynamic. 400000004 of the 400000004 is the new version number, If the version number of the current cluster configuration is required, the fourth parameter of the reconfigure method can be used to specify the desired version number. In my example, -1 means to ignore the version number, and the version field of delete and setData is used to specify the desired version number.

3.3.2 Adding a Node

Let’s add node 3 back

ZooKeeperAdmin client = new ZooKeeperAdmin("127.0.0.1:2181".3000.null);
List<String> joiningServers = new ArrayList<>();
joiningServers.add("Server. 3 = 127.0.0.1:2886:3886. 2183");
byte[] reconfigure = client.reconfigure(joiningServers, null.null, -1.null);
System.out.println(new String(reconfigure));
client.close();
Copy the code

The resulting new configuration is

Server. = 127.0.0.1 10000000000:2888-3888: the participant; 0.0.0.0:2181 server. 2 = 127.0.0.1:2887-3887: the participant; 0.0.0.0:2182 server. 3 = 127.0.0.1:2886-3886: the participant; 0.0.0.0:2183 version = 400000013Copy the code

Node 3 and back, and the version number changed again, one more zoo. The CFG. Dynamic. 400000013 file

3.3.3 incremental

ZooKeeperAdmin client = new ZooKeeperAdmin("127.0.0.1:2181".3000.null);
List<String> newMembers = new ArrayList<>();
newMembers.add("Server. = 127.0.0.1 10000000000:2888-3888: the participant; 2181");
newMembers.add("Server. 3 = 127.0.0.1:2886:3886. 2183");
byte[] reconfigure = client.reconfigure(null.null, newMembers, -1.null);
System.out.println(new String(reconfigure));
client.close();
Copy the code

This is equivalent to deleting the node whose ID is 2 (it can also be added, but I won’t show it here).

Section 3.4

The three different methods correspond to the three Parameters of the Java API: joiningServers, leavingServers, and newMembers. And Java API arguments can use strings (comma-separated) as well as lists to achieve the same effect. Dynamic addition and removal of service nodes allows us to adjust the service capacity of the entire ZK cluster without downtime (I find dynamic addition useful).

The grouping weight counting rules provide a new return strategy, especially with dynamic configuration, can modify the weight during operation, but in general, the grouping weight counting rules compare chicken, I also do not know in what kind of scenarios can be used (high-performance machines can weight a bit more important? The problem is that now all cloud services, containerized and virtualized, the configuration of the machine can be dynamically adjusted, and the general machine configuration is the same, really can not think of any use)

If readers have ideas about the use of group weight can be shared with you oh ~ for more dynamic configuration can refer to the official documentation

Iv. ZK monitoring

Metrics added after ZK 3.6 is a searchable ZK metric that can be used in conjunction with Prometheus or Grafana, according to the ZK website. What? You haven’t used it! You haven’t even heard of it! It happens that I have never played these two things, so I take this opportunity to learn with you and create a Hello World, but because this series is based on ZK, so everything is simple configuration, Let’s GO!

4.1 Prometheus

Mac installation is especially easy, and other platforms can download the package from the official website

$ brew install prometheus
Copy the code

I here the default installation path is/usr/local/Cellar/Prometheus / 2.23.0

To do this, add two lines to the zoo.cfg configuration of the ZK node

metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000
Copy the code

I am the local three nodes, the other two nodes need to be changed to 7001 and 7002, because the ports cannot be repeated

After modify the default configuration of Prometheus, on my computer path is/usr/local/etc/Prometheus. Yml is modified to

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "test-zk"
    static_configs:
      - targets: ["localhost:7000"."localhost:7001"."localhost:7002"]
Copy the code

The targets target address and scrape_interval can be used to start Prometheus

$CD/usr/local/Cellar/Prometheus 2.23.0 $. / bin/Prometheus -- config. The file = / usr/local/etc/Prometheus. YmlCopy the code

Then go to localhost:9090 and you will see the following interface:

Just check Enable AutoComplete and you can enter it in the input box, and you’ll get a prompt, so LET me just type in a few parameters

The simple demo here, the rest to the reader ~

4.2 Grafana

Installing Grafana for Mac is also very simple

$ brew install grafana
Copy the code

After the installation is complete, you can start it by running commands

$ grafana-server --config=/usr/local/etc/grafana/grafana.ini --homepath /usr/local/share/grafana --packaging=brew cfg:default.paths.logs=/usr/local/var/log/grafana cfg:default.paths.data=/usr/local/var/lib/grafana cfg:default.paths.plugins=/usr/local/var/lib/grafana/plugins
Copy the code

Grafana’s default port is 3000, and the default user name and password for accessing localhost:3000 are admin

Grafana naturally supports data sources for Prometheus and can be added directly

The default configuration only needs to change the URL (localhost:9090 by default)

Then we need to add a dashboard template, ZK official provided us with a template, sweet

Where did 10465 come from? Templates for official documents

And you’re done! Much better looking than Prometheus

So much for monitoring ZK. Traditional kung fu, so far

5. Introduction of ZK visual open source project

Using the command line to operate ZK is too troublesome, so visualization is necessary. Here are some good visualization clients, some are local clients, some are Web services, you can get it on demand

  • PrettyZoo:github.com/vran-dev/Pr… Visual GUI client, which has installation files for each platform, is handy when you need to connect to the ZK server
  • Zkdash:github.com/ireaderlab/… The JavaScript + Python visual Web client is a Web service that can be run directly. The shortcoming is Python2.7. If you don’t need to develop it again, there is no problem
  • Zoonavigator-web:github.com/elkozmon/zo… Typescript-written visual Web client that runs directly as a Web service
  • Visual-zookeeper:github.com/ghostg00/vi… , Electron + React written by the client

Six, the last

As always, if you have any questions about this article, it can also be suggestions or questions about the principles of ZK, please come to the warehouse to make an issue to us, or come to the topic of language sparrow discussion. This article was first published on the public account HelloGitHub

Address: www.yuque.com/kaixin1002/…