Click “like” to see, form a habit, wechat search [dime small assistant] pay attention to more original technical articles. This article has been included in GitHub org_Hejianhui /JavaStudy.
preface
For Java development, distributed systems will probably come up during the interview, and after the interview has confirmed that you have some knowledge of distributed services /RPC frameworks, it is likely that the interviewer will want to discuss other distributed issues with you. Distributed locks, for example, are very common. If you do Java system development, distributed systems may have some scenarios to use. The most common distributed lock implementation is based on ZooKeeper.
Because ZooKeeper is a common basic system in distributed systems, and the interview often asks: what are the usage scenarios of ZooKeeper? See if you know some basic usage scenarios. But in fact, Zookeeper can ask deep deep deep. Based on this, we will learn more about ZooKeeper and start to write zooKeeper series articles. This article focuses on the features and node description of ZooKeeper.
Zookeeper background
distributed
The development of Internet technology leads to higher and higher computing power and storage capacity required by large systems/websites. The system/site architecture is moving from centralized to distributed.
What is distributed
A calculation task is divided into several units and assigned to several different computers for execution, and then the calculation results are summarized.
The distributed way of working is somewhat similar to teamwork. When a task is assigned to a team, members of the team begin to perform their own duties, and then summarize the work results to the team leader, who then organizes the team’s work results and reports them to the company.
Problems with distributed existence
Although distributed systems have many advantages over centralized systems, such as providing greater computing and storage capacity and avoiding single points of failure. However, because of the distributed deployment mode, network failure often occurs, and how to ensure the consistency and availability of data in distributed system is also a relatively key problem.
In a centralized system, for example, there are some key configuration information, can be directly stored in the server’s memory, but in a distributed system, how to save the configuration information, and how to ensure that all the machine configuration information are consistent, and how to ensure that modify a configuration can fix the synchronization to the all the machines?
For example, in a centralized system, when writing the same data for a synchronous operation, you can directly use transaction + lock to manage the guaranteed data ACID. But in distributed systems, how do you ensure that multiple machines don’t write the same piece of data at the same time?
Background generated by Zookeeper
When a project changes from a single entity to a distributed one, the problem of coordination between multiple nodes arises. Such as:
- Who and which node performs scheduled tasks every day?
- Service discovery during RPC calls?
- How can concurrent requests be idempotent
- .
These problems can be uniformly summarized as multi-node coordination problems, which are very unreliable and undesirable in terms of performance if coordinated by nodes themselves. The coordination must be done by a separate service, which must be reliable and ensure performance.
They are introduced
They are the origin of the
In 2006, Google launched Chubby to solve the distributed consensus problem. All the servers in the cluster used Chubby to select a Master Server. Finally, the Master Server coordinates the work. In a nutshell, the principle is: In a distributed system, there is a group of servers running the same program, they need to determine a Value, the information provided by the server is the main/subject, when the server is selected by n/2+1, all the processes on the machine will be notified to the server is the Master server. Everyone depends on the information he provides. I’d love to know about Google Chubby, but it’s not open source.
But three years later in 2009 Yahoo launched a similar product on Apache called ZooKeeper, which improved on Google’s Chubby design because ZooKeeper didn’t fully comply with the Paxos protocol. It is based on a 2 Phase Commit protocol designed and optimized by itself.
From Paxos to Zookeeper, chapter 4, Section 1.
Zookeeper was originally developed by a research group at Yahoo Research. At that time, researchers found that many large systems in Yahoo basically rely on a similar system for distributed coordination, but these systems often have distributed single point problems. So the developers at ** Yahoo! Tried to develop a universal distributed coordination framework with no single point of problem to allow developers to focus on handling business logic. ** There’s actually an interesting story about the name of the ZooKeeper project. In the early days of the project, yahoo engineers wanted to name the project after an animal, considering that many internal projects had been named after animals (the famous Pig project, for example). Raghura Marishnan, the institute’s chief scientist at the time, joked: “At this rate, we’ll be a zoo!” Yet a, you have said just call zoo keeper because each 111 distributed components together, named for the animal > yahoo! The whole distributed system looks like a big zoo, and they are just used for the distributed environment of coordination so they name it was born.
ZooKeeper overview
ZooKeeper is an open source distributed coordination service. The ZooKeeper framework was first developed in Yahoo! Built to access their applications in a simple and robust manner.
Later, Apache ZooKeeper became the standard for organized services used by Hadoop, HBase, and other distributed frameworks.
For example, Apache HBase uses ZooKeeper to track the status of distributed data. ZooKeeper is designed to encapsulate complex and error-prone distributed consistency services into an efficient and reliable set of primitives that can be delivered to users in a series of easy-to-use interfaces.
Primitives: Operating system or computer network language category. It is composed of several instructions, used to accomplish a certain function of a process. Indivisible, that is, the execution of primitives must be continuous and cannot be interrupted during execution.
ZooKeeper is a typical distributed data consistency solution. Distributed applications can implement functions such as data publishing/subscription, load balancing, naming service, distributed coordination/notification, cluster management, Master election, distributed lock, and distributed queue based on ZooKeeper.
One of the most common use scenarios for ZooKeeper is to act as a registry for service producers and service consumers.
Service producers register their services with the ZooKeeper center. Service consumers first search for services in ZooKeeper during service invocation, and then invoke the content and data of service producers after obtaining the detailed information of service producers.
As shown in the figure below, ZooKeeper acts as the registry in the Dubbo architecture.
- Provider: Exposes the Provider of the service. You can start the service through a JAR or container
- Consumer: Service Consumer that invokes the remote service.
- Registry: Service Registry and discovery center.
- Monitor: collects statistics on services and invocation times, and calls time monitoring center. (Dubbo can be seen on the console page, only a simple version is available)
- Container: The Container in which the service runs.
The znode node
The basic unit of data in Zookeeper is called node. Under the node, there may be child nodes. Finally, the data is displayed in tree-level mode. Each node has a unique path, path. The client uploads node data based on the PATH, and Zookeeper notifies the client that monitors the PATH in real time.
Deployment and general configuration
Zookeeper is developed based on JAVA and can be downloaded and run as long as the corresponding JVM environment is available. The default port number is 2181 and you have to make sure it doesn’t conflict before you run it.
Release notes
Version 3.6.2, released on September 9, 2020, is the latest stable release.
In addition, 3.5.5, released on September 9, 2020, is the first stable release of the 3.5 branch. This release is considered a successor to the 3.4 Stable branch and can be used in production. Based on 3.4, it includes the following new features
- Dynamic reconfiguration
- Local meeting
- New node type: container, TTL
- SSL support for atomic broadcast protocols
- The ability to delete observers
- Multithreaded submission processor
- Upgrade to Netty 4.1
- Maven build
Also note: The recommended minimum JDK version is 1.8 files.
- Apache-zookeeper-xxx-tar. gz represents the source code
- Apache-zookeeper-xxx-bin.tar. gz Running version
Download address: zookeeper.apache.org/releases.ht…
Specific deployment process:
# download wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz # extract tar ZXVF. - Gz # Copy the default configuration CD {zookeeper_home}/conf cp zoo_sample. CFG zoo. CFG # Start./ zkserver. sh start Sh stop # Start client./ zkcli. sh -server 127.0.0.1:2181Copy the codeGeneral configuration file description
TickTime =2000 # Maximum length of time allowed for followers to initiate connections to the leader, It stands for tickTime multiple :initLimit*tickTime initLimit=10 # Maximum length of time allowed for data synchronization between the follower and the leader. It stands for tickTime multiple syncLimit=5 #zookeper ClientPort =2181 # Maximum number of concurrent connections between a single client and ZooKeeper maxClientCnxns=60 Outside will be cleared autopurge. SnapRetainCount = 3 # trigger automatically remove task time interval, hour as the unit. The default value is 0, indicating that the alarm is not automatically cleared. autopurge.purgeInterval=1Copy the codeClient command
System commands
Configuration commands
Node command
The quota command
Permission to command
Monitor command
Add, delete, change and check node data
Create/HJH "HJH is good man" get/HJH # create/HJH /sex "man" # delete/HJH /sex # Deleteall nodes including child nodes deleteall/HJHCopy the codeThis section describes the Zookeeper node
The Zookeeper node is named ZNode and is stored in a tree structure similar to a file system. The difference is that ZNode has no concept of directories and cannot execute commands like CD. The znode structure is as follows:
- Path: indicates a unique path
- ChildNode: indicates a childNode
- Stat: indicates the status attribute
- Type: indicates the node type
The node type
- PERSISTENT nodes
Persisted nodes are also created by default
Create /test by defaultCopy the code- PERSISTENT_SEQUENTIAL node
Zookeeper adds sequence numbers to paths as suffixes during path creation. It is ideal for distributed locks, distributed elections, and other scenarios. Add the -s parameter.
Create -s /test # create /test0000000001 Create -s /test # create /test0000000002 2 create /test0000000002Copy the code- EPHEMERAL
Temporary nodes are automatically deleted when the client session is disconnected. Applicable to heartbeat and service discovery scenarios. Add the -e parameter during creation.
Create -e /temp will be automatically deleted when the session is disconnectedCopy the code- EPHEMERAL_SEQUENTIAL node
Like persistent sequential nodes, the difference is that EPHEMERAL_SEQUENTIAL is temporary and deleted when the session is disconnected. Add -e -s during creation
create -e -s /seq
Copy the codeNode properties
Check node attributes stat /testCopy the codeIts attributes are described in the following table:
Ctime = Sat Oct 03 22:20:27 CST 2020 mZxid = 0x8 mtime = Sat Oct 03 PZxid = 0x8 # This indicates the number of changes made to the children of this znode (excluding the children) cversion = 0 # DataVersion = 0 Number of changes aclVersion = 0 # ephemeralOwner = 0x0 # dataLength dataLength = 0 # number of children (excluding children) numChildren = 0Copy the codeNode listening
By adding the -w parameter, customers can monitor the changes of nodes and sub-nodes in real time and receive notifications in real time. It is ideal for ensuring data consistency in distributed situations. Its use is as follows:
Acl Permission Setting
Access Control Lists (ACLs) are used to Control the Access permissions of resources. ZooKeeper uses acLs to control zNode query prevention. Permission control is implemented based on Scheme :id:permission. Scheme indicates the authorization mode, the corresponding value of the ID mode, and Permission indicates the increment, deletion, and modification of the permission bit.
Scheme: authentication model
Permission to access a
Acl-related Commands
World permission Example
Syntax: setAcl world:anyone:< permission bit >
Note: Anyone is the only value in World mode, indicating everyone
- View default node permissions:
# create a node create -e /testAcl # check node permissions getAcl /testAcl # 'world,'anyone: cdrwaCopy the code- Change the default permission to read and write
SetAcl /testAcl world:anyone:rw # Read properly get /testAcl # Failed to create child node create -e /testAcl/t "hi" # Return no permission exception Authentication is not valid : /testAcl/tCopy the codeIP Permission Example
Grammar: setAcl IP: | address < IP address > : < permission bit >
Example of Auth mode
Grammar:
- SetAcl auth:< username >:< password >:< permission bit >
- Addauth digest < user name >:
Example digest Permission
Grammar:
- SetAcl digest :< username >:< key >:< permission bit >
- Addauth digest < user name >:
Note 1: The key is generated by combining SHA1 and Base64 with a password. You can run the following command to generate the key
Echo -n < user name >, < code > | openssl DGST binary -- sha1 | openssl base64Copy the codeNote 2: After setting the digest permission for a node, you must run the addauth command before accessing the node.
- Setting the Digest Permission
# encrypt sha1 first, Then base64 encryption echo -n HJH: 123456 | openssl DGST binary -- sha1 | openssl base64 # Td6f6IYc3Kjci9RQP1A6WzjkZv8 = return key # set permissions digest setAcl/HJH digest: HJH: Td6f6IYc3Kjci9RQP1A6WzjkZv8 = : CDRWCopy the code- No permission is displayed when you view the node
# check node get HJH # showed no permission to access/org. Apache. Zookeeper. KeeperException $NoAuthException: KeeperErrorCode = NoAuth for/HJHCopy the code- Add authentication to the current session and check again
Addauth digest HJH :123456 get/HJH digest HJH is good manCopy the codeSpecial note of ACL: Permission is only valid for the current node and cannot be inherited by child nodes. For example, if the IP address is prohibited from asking node A, it does not prevent the IP address from asking child nodes /A/B of NODE A.
Zookeeper series articles, please pay attention, continue next!
Some pictures from the network, copyright to the original author, delete.Copy the code