I am participating in the Mid-Autumn Festival Creative Submission contest, please see: Mid-Autumn Festival Creative Submission Contest for details.

In the previous chapter, we looked at how to build kafka on a standalone basis. Now we’ll look at how to build Kafka on a cluster basis. You need to set up a ZooKeeper cluster and a Kafka cluster.

1. Prepare the server

Here I have prepared three VMS: 192.168.184.134 192.168.184.135 192.168.184.136 ZK and Kafka are deployed on each machine.

2. Set up the ZooKeeper cluster

In order to ensure high availability, we need to cluster ZooKeeper. We can use Kafka’s own ZK, or we can deploy a cluster by ourselves. We can completely reduce the coupling with Kafka, and it is easy to maintain, which is conducive to the common use of other applications, so we choose to build our own ZK cluster.

2.1 Download and Upload

The apache’s official website to download the zookeeper, address is as follows: www.apache.org/dyn/closer….

There is a pit here, please pay attention to download the tar package with the word bin. Do not bring the source package, in theory should be the price SRC.

Upload the files to three servers /opt and decompress the files:

[root@bogon /]# CD /opt/ [root@bogon opt]# tar-xvf apache-zookeeper-3.6.2-bin.tar.gzCopy the code

2.2 Modifying the Configuration File

[root@bogon opt]# cd apache-zookeeper-3.6.2-bin
[root@bogon apache-zookeeper-3.6.2-bin]# cd conf/
Copy the code
2.2.1 Renaming zoo_example.cfg
[root@bogon conf]# cp zoo_sample.cfg zoo. CFG [root@bogon conf]# ll total amount 16-rw-rw-r --. 1 1000 1000 535 9月 4 20:43 Configuration. XSL -rw-rw-r--. 1 1000 1000 3435 9月 4 20:43 log4j.properties -rw-r--r--. 1 root root 1148 2月 2 10:51 Zoo.cfg -rw-rw-r--. 1 1000 1000 1148 9月 4 20:43 zoo_samp.cfgCopy the code
2.2.2 Editing a Configuration File
[root@bogon conf]# vi zoo.cfg
Copy the code

Add the following configuration:

Server. 0 = 192.168.184.134:2888:3888 server. 1 = 192.168.184.135:2888:3888 server. 2 = 192.168.184.136:2888-3888Copy the code

As above configuration 192.168.184.134:2888:3888-2888 is used for zk elections; 3888 is the communication port between ZK servers.

Modify the default data path of ZK, change the original path as follows:

# dataDir = / TMP/zookeeper dataDir = / opt/apache - they are - 3.6.2 - bin/dataCopy the code
2.2.3 Creating the myID file

The main function of this file is to mark the id of the current ZK node. Above we added server.0 in the zoo configuration file, where 0 represents the current node ID, we only need to record a 0 in myID.

[root@bogon conf]# cd .. / [root@bogon apache-zookeeper-3.6.2-bin]# mkdir data [root@bogon apache-zookeeper-3.6.2-bin]# CD data/ [root@bogon data]# vi myidCopy the code

The myID file looks like this:

Accordingly, the myids of servers 135 and 136 are 1 and 2, respectively.

So far all the configurations have been modified.

2.3 Starting the Service

We go to the bin folder of ZK:

[root@bogon data]# cd .. / [root@bogon apache-zookeeper-3.6.2-bin]# CD bin/Copy the code

Run zkserver. sh on the three machines respectively.

[root@bogon bin]# ./zkServer.sh start ZooKeeper JMX enabled by default Using config: / opt/apache - they are - 3.6.2 - bin/bin /.. /conf/zoo.cfg Starting zookeeper ... STARTEDCopy the code

To view the status, the follower is the primary node and the leader is the primary node:

[root@bogon bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: / opt/apache - they are - 3.6.2 - bin/bin /.. /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: followerCopy the code
[root@bogon bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: / opt/apache - they are - 3.6.2 - bin/bin /.. /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leaderCopy the code

If you are interested in stopping the primary node, other secondary nodes will be elected as the primary node.

If the service fails to start, the following output is displayed:

[root@bogon bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: / opt/apache - they are - 3.6.2 - bin/bin /.. /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Error contacting service. It is probably not running.Copy the code

Logs can be used to view problems. The current version of the log is in the home directory logs:

[root@bogon bin]# CD /opt/apache-zookeeper-3.6.2-bin/logs/ [root@bogon logs]# ll total usage 24 -rw-r--r-- 1 root root 0 2月 2 12:22 zookeeper_audit.log -rw-r--r--. 1 root root 24352 February 2 12:29 Zookeeper-root-server-bogon.outCopy the code

To view logs regularly, run the following command:

cat zookeeper-root-server-bogon.out
Copy the code

If the startup fails due to port connectivity: For Aliyun, enable the security group port. Configure firewalls for the VM or Intranet environment

There’s a lot to be said for a ZooKeeper cluster, so we’ll continue building kafka clusters in the next section.