They started.

Introduction to ZooKeeper

ZooKeeper is a high-performance, highly available distributed coordination service with strict sequential access control. It was created by Yahoo!, an open source implementation of Google’s Chubby, and a key component of Hadoop and Hbase.

1. The ZooKeeper

  1. Simple data structure: shared tree structure, similar to file system, stored in memory.
  2. Can build a cluster: avoid single point of failure, 3 to 5 machines can form a cluster, more than half of the normal work can provide external services.
  3. Sequential access: For each read request, ZooKeeper assigns a globally unique ascending number, enabling advanced coordination services.
  4. High performance: performs memory operations and serves non-transaction requests. It is applicable to read operations. Three ZooKeeper clusters can achieve 13w QPS.

2. Those scenarios can be used

  • Data Publish and subscribe
  • Load balancing
  • The naming service
  • Master the election
  • Cluster management
  • Configuration management
  • Distributed queue
  • A distributed lock

2. Install ZooKeeper

Installing and configuring ZooKeeper is simple, whether it’s a stand-alone installation or a cluster (or cluster pseudo-distribution).

First preparations

  1. For Linux, install and configure the JDK.
  2. fromThe ZooKeeper’s official websiteDownload the ZooKeeper installation package and upload it to Linux/usr/localDirectory.

The following content is based on installing ZooKeeper(3.4.12) based on CentOS7.

1. Install a single device

1. Decompress the package.

The tar - ZXVF zookeeper - 3.4.12. Tar. GzCopy the code

2. Go to the decompressed directory and copy configuration file zoo_sample. CFG to zoo.cfg. The ZooKeeper configuration file is zoo.cfg by default. ./zkServer.sh start .. / conf/zoo_sample. CFG).

cdZookeeper - 3.4.12 / conf cp zoo_sample. CFG zoo. The CFGCopy the code

Modify the dataDir attribute of the zoo_sample. CFG configuration file. The ZooKeeper snapshot file is stored here.

vim zoo.cfg

dataDir=/usr/local/ zookeeper - 3.4.12 / dataCopy the code

4. Go to the bin directory in the decompressed directory zookeeper-3.4.12 and start zooKeeper. Run the following commands:

Sh stop Restart command:./ zkserver. sh restart Command:./ zkserver. sh status Check the statusCopy the code

5. After the startup, check whether the status is successfully started, and use the cli to log in. Then, you can use the ZooKeeper CLI.

. / zkCli. Sh - server 192.168.182.130:2181Copy the code

2. Cluster installation

A cluster pseudo-distribution is used to identify multiple ZooKeeperers by the port number in the configuration file. An odd number of ZooKeeperers is preferred.

The process is as follows

1, do not use cluster pseudo distribution, then first prepare three Linux servers, I use three virtual machines CentOS7, IP: 192.168.182.130, 192.168.182.131, and 192.168.182.132 enable port 2888 and port 3888 at the same time. Otherwise, the clusters cannot communicate with each other and the check status is always ERROR.

2, and then the same as the single installation, decompress the installation file, copy the configuration file, modify the configuration file, but the configuration file also need to add a cluster configuration, paste my configuration, where: 2888: the cluster machine communication use (the Leader listens to this port); 3888: Used to elect the leader.

# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, / TMP here is just # example sakes. dataDir=/usr/local/zookeeper-3.4.12/data # The port at which the clients will connect  clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge.  # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in  dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature # autopurge. PurgeInterval = 1 # increase the cluster configuration server. 0 = 192.168.182.130:2888:3888 server. 1 = 192.168.182.131:2888-3888 Server. 2 = 192.168.182.132:2888-3888Copy the code

/usr/local/zookeeper-3.4.12/data/dataDir=/usr/local/zookeeper-3.4.12/data/dataDir=/usr/local/zookeeper-3.4.12/data 0, server.1, and server.2 in the preceding configuration. For example, 192.168.182.130 corresponds to 0.

echo 0 > myid
Copy the code

Run the SCP command to copy the zookeeper-3.4.12 directory to the /usr/local directory of the servers 192.168.182.131 and 192.168.182.132. Run the SCP command to copy the directory to the /usr/local directory of the servers 192.168.182.131 and 192.168.182.132. You can also copy using other tools.

scp -r /usr/local/ zookeeper - 3.4.12 [email protected]: / usr /local
scp -r /usr/local/ zookeeper - 3.4.12 [email protected]: / usr /local
Copy the code

5. As mentioned earlier, 192.168.182.130 corresponds to 0 in the configuration file, so 192.168.182.131 corresponds to 1,192.168 and 182.132 corresponds to 2, so you need to change the contents of the myID file on the two servers respectively.

echo 1 > myid

echo 2 > myid
Copy the code

6. Start ZooKeeper on each server and check the status of the three server nodes. If Mode: follower or Mode: leader is displayed, the cluster is successful.

3. Structure of the ZooKeeper installation directory

  • Bin: stores the system script
  • Conf: stores the configuration file
  • Contrib: Supports additional functions
  • Dist -maven: Maven repository file
  • Docs: document
  • Lib: third-party library that you depend on
  • Recipes: Sample code for classic scenarios
  • SRC: source

Bin and conf are two very important directories, which are often used.

Bin directory

Zkserver. sh is the server and zkCli is the command line client.

The conf directory

The main parameters are configuration file parameters. Some of the important ones are as follows:

  • ClientPort: access port, that is, the port used by the application for external services.
  • DataDir: The folder where the in-memory database snapshots are stored, as well as the myID file for the cluster (note: A profile can only contain one dataDir, even if it is commented out), the new installation has no files in the folder, and the snapCount parameter can be used to configure the timing of the snapshot.
  • SnapCount: After each snapCount transaction log is generated, a snapshot is triggered. ZooKeeper generates a snapshotsnapshot.*File, and create a new transaction log filelog.*. Default is 100000 (the real code implementation, will be a random number processing, in order to avoid all servers at the same time snapshot and impact performance) (Java system property: zookeeper. SnapCount).
  • DataLogDir: A directory for setting transaction logs separately. Transaction logs are separated to avoid competing with normal logs and snapshots.
  • TickTime: indicates the heartbeat time in milliseconds to ensure that the connection exists. The minimum timeout period is two heartbeat times.
  • InitLimit: How many heartbeat times to allow other servers to connect and initialize data. If ZooKeeper manages large amounts of data, increase this value accordingly.
  • SyncLimit: How many ticktimes to allow the followers to synchronize. If the followers fall too far behind, they will be dropped.

3. ZooKeeper feature

1. Session

A session connection between the client and the server is a TCP persistent connection. The session can be used for heartbeat detection and data transmission.

Session state

After the ZooKeeper client and server are successfully connected, a session is created. The ZooKeeper session switches between different session states during the entire running life cycle. These states include: CONNECTING, CONNECTING, RECONNECTING, RECONNECTED, and CLOSE.

Once the client starts to create the Zookeeper object, the client state changes to CONNECTING state. At the same time, the client tries to connect to the server. After the connection succeeds, the client state changes to CONNECTED. If the client is disconnected from the server, the Zookeeper client automatically reconnects to the server and the client becomes CONNCTING again. Normally, when the client is CONNECTED again, the Zookeeper client becomes CONNECTED again. The client state is always between CONNECTING and CONNECTED. However, if there is a situation such as a session timeout, a permission check, or the client actively exits the program, the client state changes directly to the CLOSE state.

2. Data model

ZooKeeper’s view structure is similar to a Unix file system, in which each node is called a “data node” or ZNode. Each ZNode can store data and can also mount child nodes, so it is called a “tree”. The node cannot be created successfully.

  • In Zookeeper, a Znode is a node with a similar path to a Unix file system. Data can be stored or obtained from this node.
  • You can add, delete, modify, and check ZNodes on the client, and register a Watcher to monitor zNode changes.

3. Node type

Znodes have two types: persistent and ephemeral.

Persistent node creation: No other parameters are required. Value can be any value. The persistent node will not be deleted after the client is disconnected.

create /SunnyBear value
Copy the code

Second: Temporary node creation: If the -e parameter is added, a temporary node is created. After the client is disconnected, it will be deleted.

create -e /testSunnyBear ephemeralNodeTest
Copy the code

ZNode nodes have four types of directory nodes: holding nodes, persistent sequential nodes, temporary nodes, and temporary sequential nodes

Persistent and temporary nodes have already been mentioned, so let’s look at persistent and temporary order nodes. The -s parameter indicates that the order is added. For example, create persistent and temporary order nodes, respectively.

create -s /SunnyBear2 tests
# returns Created /SunnyBear20000000008

create -e -s /testSunnyBear ephemeralNodeTest
# returns Created /testSunnyBear0000000009
Copy the code

If the name of a Node already exists and the Node name is not in the order in which the Node name is created, the Node already exists will not be created. If the Node name already exists, the Node name already exists. /SunnyBear, and temporary nodes are not allowed to have children.

4. ZooKeeper status attribute

attribute The data structure describe
cZxid long The Zxid value of a node when it is created
ctime long Time when the node was created
mZxid long The modified Zxid value of a node
mtime long Time when a node was modified
pZxid long The Zxid value of the child node when it is last modified
cversion long Version number of the child node
dataVersion long Modified version number of the node
aclVersion long Version number of the modified ACL of a node
ephemeralOwner long If the node is temporary, its value is the session ID of the node’s holder; otherwise, its value is 0
dataLength int Length of the node data domain
numChildren int The length of the children that the node has
  • Where Zxid is the transaction ID, which can identify the global order of requests.
  • The distributed data atomicity operation is guaranteed based on CAS theory.

5. ACL ensures data security

The ACL mechanism is in the scheme: ID :permissions format. The first field indicates which mechanism to use, the second ID indicates the user, and the permissions represent related permissions (such as read-only, read and write, management, etc.). The detailed command usage is described later.

4. Command line

1. Common server commands

Sh stop Restart command:./ zkserver. sh restart Command:./ zkserver. sh status Check the statusCopy the code

2. Common client commands

Use the./ zkcli. sh IP: port command to connect to ZooKeeper. After the connection is successful, you can use the following commands, as well as some of the previous commands.

Query information about child nodes in the root directory of ZooKeeper
ls /

# Check the current subnode information of the root directory in ZooKeeper, and see the number of updates
ls2 /

-e for temporary nodes and -s for sequential nodes
create [-e] [-s] zNodeName zNodeValue

Get information about creating a node
get /zNodeName

If the key does not exist, then an error will be reported
set /zNodeName zNodeValue

# Delete node, if there is a child node deletion failure
delte /zNodeName

# Recursive delete, child nodes are deleted at the same time
rmr /zNodeName

Exit the client
quit

# Help command
help
Copy the code

3. Common ACL commands

I mentioned earlier that the ACL command is in scheme: ID: Permissions format, so take a look at what each of these three paragraphs represents.

  • Schema: indicates the authorization policy
  • Id: indicates a user
  • Permission: indicates the permission

scheme

Scheme has four ways:

  • World: The default, accessible to all
  • Auth: indicates an authenticated user
  • Digest: User name and password is the most commonly used authentication mode in service systems
  • IP: IP address authentication is used

id

Corresponding to Scheme, there are four types: auth for plaintext, and Digest for ciphertext

  • world -> anyone
  • auth -> username:password
  • digest -> username:BASE64(SHA1(password))
  • IP -> IP address of the client

permission

C (CREATE), D (DELETE), R (READ), w (WRITE), a (ADMIN) these several permissions, corresponding to add, DELETE, search, change, management permissions, referred to as CDRWA

  • C: Permission to create a child node
  • D: Delete the permission of a child node
  • R: permission to read node data
  • W: Permission to modify node data
  • A: The management permission is granted to the child node

Specific commands

# Obtain the ACL information of the child node
getAcl /zNodeName

Set subinterface ACL information, for example:
Set the testDir node to be accessible to all, but not to delete
setAcl /testDir world:anyone:crwa

/testDir /testAcl1 /testDir /testDir /testDir /testAcl1 /testDir /testDir
create /testDir/testAcl1 testAcl1
addauth digest user1:123456
setAcl /testDir/testAcl1 auth:user1:123456:crwa

The difference between auth and plaintext passwords is the difference between plaintext passwords and ciphertext passwords. If ciphertext passwords are not applicable, the setting fails. Therefore, you need to obtain ciphertext passwords
Obtain ciphertext passwords using shell commands
java -Djava.ext.dirs=/usr/local/ zookeeper - 3.4.12 / lib - cp/usr /local/ zookeeper - 3.4.12 / zookeeper - 3.4.12. Jar org. Apache. Zookeeper. Server auth. DigestAuthenticationProvider user2:123456# Get results
user2:123456->user2:hZG2W+NR7DCvADzOkGR6JGLqoTY=
# Next is the basic operation, using the zkCli client login
create /testDir/testAcl2 testAcl2
addauth digest user2:123456
setAcl /testDir/testAcl2 digest:user2:hZG2W+NR7DCvADzOkGR6JGLqoTY=:crw

# Set IP mode
create /testDir/testAcl3 testAcl3
set/ testDir/testAcl3 IP: 192.168.31.6: cdrwaCopy the code

After setting, you can use the command to test whether the following permissions are set successfully. If auth and Digest permissions are set, you need to log in to obtain the permissions. Quit quit the connection and reconnect to ZooKeeper, and then run the following command to test.

addauth digest user1:123456
Copy the code

5. ZooKeeper logs

The previous conf configuration submitted to dataDir and dataLogDir

  • DataDir: The data directory of ZooKeeper is used to store serialized snapshots of memory databases. You can set the snapshot creation time by snapCount parameter. The default value is 100000. However, in the actual source code, 50000 + Random (50000) will be used as the actual number of operations to generate snapshots, to avoid multiple nodes to generate snapshots at the same time, which will affect performance. If the path of the Transaction log (that is, the dataLogDir configuration item) is not configured, the ZooKeeper transaction log is also stored in the data directory.
  • DataLogDir: specifies the directory for storing transaction logs. Transaction logs have a great impact on ZooKeeper. It is strongly recommended that the transaction log directory and the data directory be separated. Do not record transaction logs in the data directory (mainly used to store in-memory database snapshots).

A snapshot is similar to the RDB in Redis. Every 50000 + random(50000) record operation generates a snapshot. The transaction log is similar to the AOF in Redis, recording every operation command.

Command to view snapshot and transaction log file contents

Note that the following command’s delimiter is on window; On Linux, it is:, and you also need to pay attention to the JAR version number and the file directory location, change it to your own.

Viewing a snapshot:

java -cp .. /.. / zookeeper - 3.4.12. Jar:.. /.. / lib/slf4j - API - 1.7.25. Jar org. Apache. The zookeeper. Server SnapshotFormatter snapshot. XXXCopy the code

View the transaction log:

java -cp .. /.. / zookeeper - 3.4.12. Jar:.. /.. / lib/slf4j - API - 1.7.25. Jar org. Apache. The zookeeper. Server LogFormatter log. XXXCopy the code

Now that you’ve read this, like, comment, follow, or collect it!

Author: IT wang2 xiao3 er4 starting address: www.itwxe.com/posts/8c686… Copyright notice: The content of the article is subject to the authorship-non-commercial-no deduction 4.0 international license. If you want to reprint, please link to the author in a prominent position on the article page.