Wechat official account: Xiaolei

When you work hard enough, luck will happen.

What is Zookeeper

1.1 an overview of the

In my impression, ZooKeeper can be used as a registry. The previous microservice architecture is more used with Dubbo+ ZooKeeper. Therefore, ZooKeeper mainly serves distributed systems.

Distributed system is characterized by the existence of multiple nodes, real-time perception of each node state, management of each node becomes particularly important. Zookeeper solves this problem. From the perspective of design patterns: It is a distributed service management framework based on the observer mode design. It is responsible for storing and managing the data that everyone cares about, and then accepting the observer registration. Once the status of the data changes, ZooKeeper is responsible for notifying the observers who have registered with ZooKeeper and asking them to respond accordingly.

Zookeeper can be summarized as follows: file system + notification mechanism.

Working mechanism:

  • 1. Deregister information when the server starts (temporary nodes are created)
  • 2. Obtain the list of current online servers and register the listener
  • 3. The server node is offline
  • 4. Notification of server node offline events
  • 5. Process retrieves the server list again and registers the listener

1.2 the characteristics of

Zookeeper is a design pattern that is a cluster of one leader and many followers

2. Half rule: The ZooKeeper cluster can run normally as long as more than half of the nodes in the cluster are alive

3. Global data consistency. Each server keeps a copy of the same data, and the data is consistent regardless of which server the client connects to

4. Atomicity of data update: a data update either succeeds or fails.

5, real-time: in a certain time range, the client can read the latest data.

1.3 Data Structure

The data model of ZooKeeper is similar to that of the Linux file system. It can be regarded as a tree as a whole. Each node is called zNode and can store 1Mb of data by default.

These nodes fall into two categories: temporary and persistent. So an interview question arose: What are the node types of ZooKeeper?

  • Temporary ordinary node
  • Temporary ordered node
  • Persistent common node
  • Persistently ordered node

1.4 What can ZooKeeper Do?

Zookeeper provides the following services: unified naming service, unified configuration management, unified cluster management, soft load balancing, and distributed lock.

1.4.1 Unified Naming Service

When accessing the domain name: www.baidu.com, there are many servers under this domain name: 192.168.1.100, 192.168.1.101, 192.168.1.102, etc. When someone visits the domain name, ZooKeeper forwards the request to one of the registered servers. Users don’t need to remember which IP address to visit, just the domain name.

1.4.2 Unified Configuration Management

Extract common configuration files and distribute them to other systems.

Configuration file synchronization is very common. The configuration information of all nodes in a cluster must be consistent. For example, in a Kafka cluster, you need to quickly synchronize the configuration information to all nodes after modifying the configuration file.

Blog.csdn.net/u011320740/…

1.4.3 Unified Cluster Management

1) In distributed environment, it is necessary to grasp the status of each node in real time, and adjustment can be made according to the real-time status of the node.

  1. Zookeeper monitors node status changes in real time.
  • You can write node information to a Znode on ZooKeeper
  • Listen on the ZNode for real-time status changes.
1.4.4 Soft load Balancing

Zookeeper records the number of accesses to each server so that the server with the least access requests processes the latest client requests.

1.4.5 Distributed Lock

If multiple clients A, B, and C attempt to seize a Distributed lock of ZooKeeper, the distributed lock must be unlocked. It works like this:

  • When you first access the /lock lock node, everyone comes up and creates one ordered node after another. For example, system A creates node 001. Node 002 is created in system B, and node 003 is created in system C.
  • After the system gets all the nodes, it will compare its own node is the smallest, if so, then get the lock, if not, on the last node listener, monitoring it.
  • As long as the last node to release the lock, they will row to the front, equivalent to a queuing mechanism.

The first purpose of using temporary nodes is that it does not matter if a client breaks down accidentally after creating a temporary order node. Zookeeper detects that the client breaks down and automatically deletes the corresponding temporary order node, which is equivalent to automatically releasing the lock or canceling its own queuing.

2. Install ZooKeeper

2.1 Local Mode Installation and Deployment

1. Steps:

  • Configuration modification: Change the zoo_sample. CFG file in conf to zoo.cfg
  • Create a dataDir folder in the ZooKeeper directory to store the data storage directory.
  • CFG file in the Zookeeper conf.
DataDir = / opt/module/zookeeper - 3.5.7 / zkDataCopy the code
  • Start the
[root @ hadoop101 zookeeper - 3.5.7] # bin/zkServer. Sh startCopy the code
  • Check the status
[root @ hadoop101 zookeeper - 3.5.7] # bin/zkServer. Sh statusCopy the code
  • Starts the zK service for the client to connect to the specified host
[root@hadoop101 zookeeper-3.5.7]# bin/ zkcli. sh -server host:portCopy the code
  • stop
[root @ hadoop101 zookeeper - 3.5.7] # bin/zkServer. Sh stopCopy the code

2.2 Interpretation of Configuration Parameters

1) tickTime =2000: Communication heartbeats, heartbeat duration between the Zookeeper server and client (unit: ms) Basic time used by Zookeeper, interval between servers or clients to maintain heartbeats, that is, each tickTime sends a heartbeat, in milliseconds. It is used for the heartbeat mechanism and sets the minimum session timeout to twice the heartbeat time. (The minimum session timeout is 2*tickTime)

InitLimit =10: LF initial communication duration Maximum heartbeat beats (number of Ticktimes) that can be tolerated between the Follower server in the cluster and the Leader server during the initial connection. This parameter is used to limit the connection duration between the Zookeeper server in the cluster and the Leader server.

3) syncLimit =5: Indicates the maximum response time between the Leader and Follower in the LF synchronization communication. If the response time exceeds syncLimit * tickTime, the Leader considers that Follwer is dead and deletes Follwer from the server list.

4) dataDir: Data file directory + persistent path is used to save data in Zookeeper.

5) clientPort =2181: Client connection port The port that listens to the client connection

Internal principles of Zookeeper

3.1 Node Type (Znode)

Persistent: The node created after the client and server are disconnected is not deleted.

1) Ordinary persistent nodes

2) Persistent node with serial number (serial number zooKeeper maintains)

Brief: After the client and server are disconnected, the created node is deleted.

1) Common transient node

2) Transient node with serial number (serial number zooKeeper maintains)

3.2 Stat structure

Describes the status of each ZNode

[zk: localhost:2181(CONNECTED) 12] stat /zookeeper
cZxid = 0x0
ctime = Thu Jan 01 08:00:00 CST 1970
mZxid = 0x0
mtime = Thu Jan 01 08:00:00 CST 1970
pZxid = 0x0
cversion = -2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2
Copy the code

Each time the ZooKeeper status is changed, it will receive a timestamp in the form of an ZXID, that is, the ZooKeeper transaction ID. The transaction ID is the total order of all changes in ZooKeeper. Each modification has a unique zxID, and if ZXID1 is less than zxid2, then zxid1 occurs before zxid2.

(2) The number of milliseconds ctime-zNode was created (since 1970)

(3) mzxid – znode last updated transaction zxID

(4) mtime-znode last modified milliseconds (since 1970)

(5) pZxid-znode indicates the zxID of the last updated child node

(6) cversion-znode child node change number, znode child node change times

(7) DatAversion – Znode data change number

(8) aclversion-znode ACL change number

(9) ephemeralOwner- If it is a temporary node, this is the session ID of the ZNode owner. 0 if not a temporary node.

(10) datalength-znode dataLength

(11) Numchildren-znode Number of child nodes

3.3 Principle of listeners

1, the principle of

  • First, there is a main () thread
  • When the ZooKeeper client is created in the main thread, two threads will be created, one for network connection communication (CONNECT) and one for listener.
  • Registered listening events are sent to ZooKeeper via the Connect thread
  • Add the registered listening events to the Registered listening list of ZooKeeper
  • Zookeeper sends this message when it detects data or path changes
  • The listener thread calls the process () method internally

2, common monitoring

  • Listen for changes in node data: get path
  • Listen for changes in the increase or decrease of children: ls path

3.4 Election Mechanism

3.4.1 ZAB Protocol:

ZAB protocol is a crash recovery atomic broadcast protocol specially designed for distributed coordination service Zookeeper. The ZAB protocol includes two basic modes: crash recovery and message broadcast. When a cluster is first started, it elects a leader. The followers server in the cluster then starts to synchronize data with the new leader server. When more than common machines in the cluster complete data synchronization with the leader server, the recovery mode is pushed to broadcast mode. Start receiving client message processing at this point.

An algorithm based on messaging and guaranteeing data consistency (protocol)

Agreement Objectives:

1. Elect the leader when there is no leader

2. In the case of leader, ensure data consistency as far as possible.

3.4.2 what zj had cluster

(1) Half mechanism: more than half of the machines in the cluster are alive and the cluster is available. Therefore, Zookeeper can be installed on an odd number of servers.

(2) Zookeeper does not specify Master and Slave in the configuration file. However, when Zookeeper works, one node serves as the Leader and the other nodes serve as followers. The Leader is temporarily elected through an internal election mechanism.

3.4.3 Mechanism Concept

Serverid: indicates the Serverid. For example, there are three servers whose ids are 1, 2, and 3

2. Zxid: indicates the data ID

Maximum data ID stored in the server: The larger the value is, the newer the data is, and the more weight the data has in the election algorithm.

Epoch: logical clock

Or the number of votes, the logical clock value is the same for each round of voting. This number increases after each vote, and then makes different judgments based on different values compared to the number of votes received back.

4. Server status: Election status

  • & : campaign
  • FOLLOWING
  • OBSERVING: the state of OBSERVING,
  • LESDING: Leader status.
3.4.4 Election process

Suppose there is a Zookeeper cluster consisting of five servers, whose ids range from 1 to 5, and they are all newly started, that is, there is no historical data. In terms of the amount of data stored, they are all the same.

Voting principles:

  • The selfish principle (the server votes for itself when it first registers)
  • The rule of “follow the wind” :(if you find that other servers are better than you, vote for other servers) the comparison can be the zxid timestamp, which server has the latest data vote it.

Process:

(1) Server 1 starts and initiates an election. Server 1 votes for itself. At this point, server 1 has one vote, less than half (3 votes), the election cannot be completed, and the state of server 1 remains LOOKING.

(2) Server 2 is started to initiate another election. Server 1 and server 2 vote for each other and exchange vote information: server 1 finds that server 2’s ID is larger than the one it currently votes for (server 1) and changes the vote to server 2. At this time, server 1 has 0 votes, and server 2 has 2 votes. If there is no more than half of the results, the election cannot be completed. The state of server 1 and 2 remains LOOKING

(3) Server 3 starts and initiates an election. Both servers 1 and 2 change the vote to server 3. The result of this vote: 0 votes for server 1, 0 votes for server 2, 3 votes for server 3. At this point, the number of votes of server 3 exceeds half, and server 3 is elected as the Leader. Change the status of server 1,2 to FOLLOWING, and server 3 to LEADING.

(4) Server 4 starts and initiates an election. At this point, servers 1, 2 and 3 are no longer LOOKING and will not change the ballot information. Result: 3 votes for server 3, 1 vote for server 4. At this point, server 4 follows the majority, changes the vote information to server 3, and changes the status to FOLLOWING;

(5) Server 5 starts, just like server 4.

3.4.5 Election after the Leader Fails

If the leader fails, the cluster can work normally as long as the number of remaining machines is greater than half. However, a new leader needs to be elected. The process of voting is still voting because the cluster is working, so the ID of each machine may be different. Therefore, among the remaining machines in the cluster, the one with the largest ZXID is elected the leader. If the zxids are all the same, myID will win the most in theory.

Zxid time stamp, latest data. In a sense, it can represent the integrity of the data stored in the current machine.

3.5 Data Writing Process

1) The client connects to any machine in the ZK cluster and sends a write request

2) If the ZK cluster to which the client is connected deploys the Leader, the current machine forwards the client’s write requests to Leadet

3) When the leader receives the write request, it constructs the current write operation into a transaction, corresponding to an ZXID

4) After each follower receives a write operation, it first stores the write operation in the queue and reports back to the leader

5) When the leader receives feedback from more than half of the followers in the cluster, it indicates that the write operation can be carried out normally.

The leader broadcasts again to each follower, telling the follower to commit the write operation.

Four, the last

Zookeeper is more than that, this article is more to introduce some zooKeeper concepts, a few client command operations every put up, today we know zooKeeper storage nodes and listening mechanism, can achieve a lot of functions. This is enough for the present stage. If you have the opportunity to go further, I will write a follow-up article to introduce it.