1. A brief introduction to zookeeper

1.1 introduction

Zookeeper is a distributed coordination service, in other words, a coordination service for a user’s distributed applications

  • Zookeeper serves other distributed applications
  • Zookeeper itself is a distributed program (zK works as long as more than half of the nodes are alive)
  • Zookeeper provides the following services: primary/secondary coordination, dynamic server node login and logout, unified configuration management, distributed shared lock, and unified name service
  • Although it offers a variety of services, ZooKeeper really only provides two functions (managing data and listening to data) at the bottom: managing (storing and reading) data submitted by user programs; And provide data node listening service for user program;

1.2 Roles of the Zookeeper cluster: Leader and follower

Zookeeper does not specify a master or slave in the configuration file. After Zookeeper is started, a leader and followers are elected through the internal election mechanism. There is only one leader and the others are followers. A ZooKeeper cluster can provide services as long as more than half of the nodes are alive. 2. Zookeeper cluster mechanism Half mechanism: More than half of the machines in the cluster survive and the cluster is available. Zookeeper fits on an odd number of machines!!

2. Install and configure ZooKeeper

2.1 they are installed

  • Install the ZooKeeper package on three VMS (JDK must be installed in advance). Upload the ZooKeeper compressed package to the /apps/package directory and decompress it
The tar - ZXVF zookeeper - 3.4.5. Tar. GzCopy the code
  • rename
Mv Zookeeper-3.4.5 ZooKeeper (rename the folder zookeeper-3.4.5 to ZooKeeper)Copy the code
  • Modifying environment variables
Vi /etc/profile Add the following content:export ZOOKEEPER_HOME=/apps/package/zookeeper
export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin
Copy the code
  • Recompile files: source /etc/profile all three machines need to be modified

2.2 Modifying a Configuration File

  • Run CD zookeeper/conf cp zoo_sample.cfg zoo.cfg
  • Edit vi zoo. CFG
Add: dataDir = / apps/package/zookeeper/data dataLogDir = / apps/package/zookeeperlog
server.1=mini1:2888:3888
server.2=mini2:2888:3888
server.3=mini3:2888:3888
Copy the code
  • Create a folder:
cd /apps/package/zookeeper
mkdir -m 755 data
mkdir -m 755 log
Copy the code
  • Create a new file myid in the data folder. The contents of the file myid are:
cdData vi myID Add 1Copy the code

For mini2 and mini3 servers, please change to 2,3. In the future, the leader and follow will be selected according to this myid.

  • Copy the cluster to other machines
scp -r /apps/package/zookeeper root@mini2:/apps/package/
scp -r /apps/package/zookeeper root@mini3:/apps/package/
Copy the code

If mini2 and mini3 cannot be pinged from MINI1, configure the IP addresses of mini2 and mini3 in the hosts file

  • Mini2: change myID to: 2 to mini3: change myID to: 3 and don’t forget to change the /etc/profile path
  • To start (each machine).
Sh start zkserver. sh start-foreground(you can see the start log)Copy the code
  • Viewing Cluster Status
JPS (check the process) zkserver. sh status (Check the cluster status, primary and secondary information)Copy the code

If press mouth, links refer to the following address: http://blog.csdn.net/u014686180/article/details/51767863

3. Zookeeper data structure and common operations

3.1 a zookeeper features

  • Zookeeper: a cluster consisting of one leader and multiple followers
  • Global data consistency: Each server keeps the same data copy, and the data is consistent no matter which server the client connects to
  • Distributed read and write, update request forwarding, by the leader implementation
  • Update requests are processed in order. Update requests from the same client are processed in the order in which they are sent
  • Atomicity of data updates, a data update either succeeds or fails
  • Real-time, the client can read the latest data within a certain time range

3.2 ZooKeeper Data Structure

  • Hierarchical directory structure;
  • Each node is called zNode in ZooKeeper and has a unique path identifier;
  • The node Znode can contain data and child nodes (but EPHEMERAL nodes cannot have children);
  • Client applications can set up monitors on the node.

3.3 Node Types

  • Znodes have two types: ephemeral (ephemeral) (ephemeral) (ephemeral) (ephemeral) (ephemeral) (ephemeral)
  • Znode has four forms of directory nodes (persistent by default) persistent PERSISTENT_SEQUENTIAL (persistent sequence /test0000000019) EPHEMERAL EPHEMERAL_SEQUENTIAL
  • The sequence id is set when creating a ZNode, and a value is appented to the zNode name. The sequence id is a monotonically increasing counter maintained by the parent node
  • In a distributed system, sequence numbers can be used to order all events globally so that clients can infer the order of events from the sequence number

Using clients to operate on nodes

  • Run the following command to connect to the ZooKeeper server:
Sh -host name (IP):2181 For example, run zkcli. sh -mini2:2181Copy the code
  • Run the ls command to view the current contents in ZooKeeper:
 ls /
Copy the code
  • To create a new ZNode, use create /zk myData. This command creates a new Znode node “zk” and the string associated with it:
create /zk "MyData"Copy the code
  • We run the get command to verify that the ZNode contains the string we created:
get /zk
Copy the code

– Listens for changes to this node and outputs the changes when another client changes /zk

get /zk watch
Copy the code
  • Use the set command to set the string associated with zk:
set /zk "ZSL"Copy the code
  • Run delete to delete a Znode:
delete /zk
Copy the code
  • Delete nodes (the difference is that this can delete directories) : RMR
rmr /zk
Copy the code

Reference: http://www.cnblogs.com/likehua/tag/zookeeper/

4. Use the Java API to operate ZooKeeper

  • First, you need to introduce the ZooKeeper JAR package. This jar package depends on other jars and can be downloaded directly from maven repository.
<dependency> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <version>3.4.10</version> <groupId> <groupId>org.apache.zookeeper</groupId> <artifactId>type>pom</type>
</dependency>

Copy the code
  • Common operations for adding, deleting, checking, and changing APIS are as follows: Delete Deletes a node exists Exists Test whether the target node exists Get /set Data Reads/updates data from the target node get/set ACL Obtains/sets the ACL information about the target node GET Sync waits for the data to be transferred using the following Java action code:
public class SimpleZkClient {
    private static final String CONNECT_URL = "mini1:2181,mini2:2181,mini3:2181";

    private static final int SESSION_TIME_OUT = 2000;

    ZooKeeper zkCli = null;
    @Before
    public void init() throws Exception{
        zkCli = new ZooKeeper(CONNECT_URL, SESSION_TIME_OUT, new Watcher() {
            @Override
            public void process(WatchedEvent event) {
                System.out.println(event.getType()+"-- -- -- -- -- -- -- -- -- -- -"+event.getPath());
                try{
                    zkCli.getChildren("/".true); }catch (Exception e){ } } }); Public void create() throws Exception{// Parameter 1: throws Exception (); Parameter 3: node permission parameter 4: node type Byte [] String s = zkcli.create ()"/zk"."test".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); Public void isExist() throws Exception{Stat exists = zkcli.exists ()"/zk".false);
        System.out.println(null==exists?"Nonexistent":"There"); Public void getData() throws Exception{byte[] data = zkcli.getData ()"/zk".false, null);
        System.out.println("Node data:"+new String(data)); Public void getChildren() throws Exception{List<String> children = zkCli.getChildren("/".false);
        for(String s : children){
            System.out.println("Node name:"+s); } Thread.sleep(Long.MAX_VALUE); Public void delete() throws Exception{// parameters 2: Specifies the version to be deleted. -1 indicates that all versions are to be deleted."/zk", 1); this.isExist(); Public void update() throws Exception{Stat.} /** * @update () throws Exception{Statstat = zkCli.setData("/zk"."newtest".getBytes(), -1); this.getData(); }}Copy the code

Thread.sleep(Long.MAX_VALUE); Zkcli.getchildren (“/”, true); zkcli.getChildren (“/”, true); If we operate on the ZooKeeper operation node from the Linux command line, the listener event will be triggered here.

5. Usage scenarios of ZooKeeper

5.1 Scenario 1: The Client dynamically senses node changes on the service end to Achieve high availability

Now suppose there is such a requirement: the server node has multiple, can dynamically up and down the line; Any client needs to be able to sense the changes of nodes on the service end in real time, and then connect to the nodes that currently provide services. Implementation idea: We can use the third-party middleware zooKeeper to register the server node information with ZooKeeper when each server is started (for example: /servers/server01; / servers/server02); Before each invocation, the client obtains the latest server node information through the getChildren method. Meanwhile, the client registers in ZooKeeper to listen to the changes of the server node. If server01 goes offline, ZooKeeper notifies the client of the node change and calls the process method to retrieve the latest server node information. The server code is as follows:

public class DistributeServer {

    private static final String CONNECT_URL = "mini1:2181,mini2:2181,mini3:2181";

    private static final int SESSION_TIME_OUT = 2000;

    private static final String PARENT_NODE = "/servers"; private ZooKeeper zkCli = null; @author @date 2017/12/13 */ public void getConnect() throws Exception{zkCli = new ZooKeeper(CONNECT_URL, SESSION_TIME_OUT, newWatcher() {
            @Override
            public void process(WatchedEvent event) {
                System.out.println(event.getType()+"-- -- -- -- -- -- -- -- -- -- -"+event.getPath());
                try{
                    zkCli.getChildren("/".true); }catch (Exception e){ } } }); } /** * @description * @author * @date 2017/12/13 */ public void registerServer(String) hostName) throws Exception { String s = zkCli.create(PARENT_NODE +"/", hostName.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
        System.out.println("Server:"+hostName+"Registration completed."); } /** * @description throws @author @date 2017/12/13 */ public void handelBusiness(String Hostname) throws Exception { System.out.println("Server:"+hostname+"Taking care of business..."); Thread.sleep(Long.MAX_VALUE); } public static void main(String[] args) throws Exception { DistributeServer server = new DistributeServer(); server.getConnect(); server.registerServer(args[0]); server.handelBusiness(args[0]); }}Copy the code

The client code is as follows:

/** * @author * @description Pull up the latest server node list and set listening to ZooKeeper */ public class DistributeClient {private static final String CONNECT_URL ="mini1:2181,mini2:2181,mini3:2181";

    private static final int SESSION_TIME_OUT = 2000;

    private static final String PARENT_NODE = "/servers"; private ZooKeeper zkCli = null; private volatile List<String> serverList = null; @author @date 2017/12/13 */ public void getConnect() throws Exception{zkCli = new ZooKeeper(CONNECT_URL, SESSION_TIME_OUT, newWatcher() {@override public void process(WatchedEvent event) { System.out.println(event.getType()+"-- -- -- -- -- -- -- -- -- -- -"+event.getPath()); Try {// update the server list again, and register to listen getServerList(); }catch (Exception e){ } } }); } /** * @description Public void getServerList() throws Exception {List<String> children = zkCli.getChildren(PARENT_NODE,true);
        List<String> servers = new ArrayList<String>();
        forByte [] data = zkcli.getData (PARENT_NODE +){// Child is only the node name of the child node"/" + child, false, null); servers.add(new String(data)); } // Assign servers to the member variable serverList so that each business thread can use serverList = servers; System.out.println("Node data:"+serverList); } @author @date 2017/12/13 */ public void handelBusiness() throws Exception { System.out.println("The client starts working..."); Thread.sleep(Long.MAX_VALUE); } public static void main(String[] args) throws Exception { DistributeClient client = new DistributeClient(); client.getConnect(); client.getServerList(); client.handelBusiness(); }}Copy the code

5.2 Scenario 2: Distributed Lock Implementation

If you have 50 machines in the cluster making changes to the same file on one machine, how can you ensure that the file will not be writable? You can’t use Java’s synchronized locks because they apply to a program, and we’re not even on a server. Distributed locking with ZooKeeper can be implemented. Design idea: When starting a server, register a ZNode (such as /lock/1; /lock/2) and set to listen for parent changes; Obtain all child nodes under the parent node, and compare the serial number size; Conventions such as obtaining the lock with the smallest serial number, operating on a certain file, deleting its own node after the operation (equivalent to releasing the lock), and registering a new znode with “short + serial number”; When other programs receive notifications from ZooKeeper about node changes, they compare the number sizes to see who gets the new lock.

Public class DistributedClientLock {// Session timeout private static final int SESSION_TIMEOUT = 2000; // Address of the ZooKeeper cluster Private String hosts ="mini1:2181,mini2:2181,mini3:2181";
    private String groupNode = "locks";
    private String subNode = "sub";
    private boolean haveLock = false; private ZooKeeper zk; Private volatile String thisPath; /** * Connecting zooKeeper */ public void connectZookeeper() throws Exception {zk = new ZooKeeper(hosts, SESSION_TIMEOUT, newWatcher() {@override public void process(WatchedEvent event) {try {// Determine the event typeif (event.getType() == EventType.NodeChildrenChanged && event.getPath().equals("/"+ groupNode)) {List<String> childrenNodes = zk.getChildren("/" + groupNode, true);
                        String thisNode = thisPath.substring(("/" + groupNode + "/").length()); // Compare if you are the minimum ID collections.sort (childrenNodes);if(childrenNodes.indexof (thisNode) == 0) {// Access the shared resource to process business and remove locks after processingdoSomething(); ThisPath = zk.create(thisPath = zk.create)"/" + groupNode + "/"+ subNode, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); } } } catch (Exception e) { e.printStackTrace(); }}}); ThisPath = zk.create(); thisPath = zk.create();"/" + groupNode + "/" + subNode, null, Ids.OPEN_ACL_UNSAFE,
                CreateMode.EPHEMERAL_SEQUENTIAL);

        // waitThread.sleep(new Random().nextint (1000)); List<String> childrenNodes = zk.getChildren(List<String> childrenNodes = zk.getChildren("/" + groupNode, true); // If the only program competing for resources is itself, it can directly access the shared resourcesif (childrenNodes.size() == 1) {
            doSomething();
            thisPath = zk.create("/" + groupNode + "/"+ subNode, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); }} /** * Handle the business logic and release the lock at the end */ private voiddoSomething() throws Exception {
        try {
            System.out.println("gain lock: " + thisPath);
            Thread.sleep(2000);
        } finally {
            System.out.println("finished: "+ thisPath); Zk.delete (this.thispath, -1); } } public static void main(String[] args) throws Exception { DistributedClientLock dl = new DistributedClientLock(); dl.connectZookeeper(); Thread.sleep(Long.MAX_VALUE); }}Copy the code

Reference: http://www.cnblogs.com/likehua/tag/zookeeper/

6 Election mechanism of ZooKeeper

6.1 New PaXOS Cluster

Take a simple example to illustrate the whole electoral process. Suppose you have a ZooKeeper cluster consisting of five servers, whose IDS range from 1 to 5. They are all newly started, that is, they have no historical data, and they are all the same in terms of the amount of data stored. Let’s see what happens if these servers start up in sequence.

  1. Server 1 is started, and it is the only server that is started, and it sends out a message without any response, so its election status is always LOOKING
  2. Server 2 started, it communicate with the most to start server 1, exchange their election results, since neither history data, so the id value larger server 2 to win, but since there is no agreed with election reached more than half of the server it (in this case, more than half is 3), so the server 1, 2, or continue to LOOK ING state.
  3. Server 3 is started. According to the previous theoretical analysis, server 3 becomes the leader of servers 1,2, and 3. Unlike the above, it is elected by three servers, so it becomes the leader of this election.
  4. Server 4 starts up. Based on the previous analysis, server 4 should theoretically be the largest of the servers 1,2,3, and 4, but since half of the servers voted for server 3, it will have to accept its younger brother’s life.
  5. Server 5 starts, just like server 4, as the little brother.

6.2 Election Mechanism for Non-New Clusters (Data Recovery)

When zooKeeper is running for a period of time, the election is conducted according to the above instructions. However, when zooKeeper is running for a period of time, the machine goes down and the election is held again, the election process becomes more complicated. The data ID, leader ID, and logical clock need to be added. Data ID: The new id of the data is large, and the id is updated every time the data is updated. Leader ID: this is the value of myID that we configured, one per machine. Logical clock: This value increases from 0 and corresponds to one value for each election. That is, if the election is the same, then this value should be the same. A larger value indicates that the process of the elected leader is updated this time. If the logical clock is small, the result of the election will be ignored. If the logical clock is unified, the leader with the largest id will win. If the logical clock is the same, the leader with the largest ID will win.