The reason for doing it

The company’s business features require a replacement of the architecture’s downstream data queues with a Kafka messaging engine to replace rabbitMQ, which cannot stack large amounts of data. In case you forget it, but also to give you a reference (there’s not a lot on the web about this). Some of the advantages of Kafka will not be generalized, interested students can go to understand, this article is only the implementation of the framework.

Environment to prepare

1. Prepare three cloud hosts and modify their /etc/hosts to ensure that the three hosts can communicate with each other through hostname.

Put the IP and hostname of the three machines in /etc/hosts
192.168.2.163  dev-kafka-01  
192.168.2.123  dev-kafka-02  
192.168.2.9    dev-kafka-03  Copy the code

2. Install the JDK

Install JDk1.8 with apt or yumCopy the code

3. Disable the firewall and check the security group on the cloud host. Also, if it’s centos7, check selinux and turn it off.

Start the detailed setup procedure

Start by building a ZooKeeper cluster. Kafka relies on ZooKeeper

1. Download the zookeeper package from zookeeper.apache.org/ and decompress it. The version I use here is ZooKeeper-3.4.14

2. Modify the ZK configuration file

[web@dev-kafka-01 zookeeper-3.4.14]$ mv conf/zoo_sample.cfg zoo.cfg

[web@dev-kafka-01 zookeeper-3.4.14]$ vi conf/zoo.cfg  
# set a custom data data directory
dataDir=/data/server-data/zookeeper
#zookeeper nodes address
# Address and port of the ZooKeeper nodeServer. 1 = 192.168.2.163:2888-3888 for server 2 = 192.168.2.123:2888:3888 server. 3 = 192.168.2.9:2888-3888Copy the code

3. Run the command to start the system

[web @ dev - kafka - 01 zookeeper - 3.4.14] $. / bin/zkServer. Sh startCopy the code

4. Check whether the program is started

sudo jps
# Check whether the QuorumPeerMain process exists. If so, zK process has been started successfully.Copy the code

5. Perform the preceding steps to install ZooKeeper on the remaining two servers

6. Check the zooKeeper cluster status

[web@dev-kafka-01 zookeeper-3.4.14]$./bin/ zkserver. sh status Zookeeper JMX enabled by default Using config: / data/server - side/zookeeper - 3.4.14 / bin /.. /conf/zoo.cfg Mode: follower# # # # # # # # # # # # # # # # # # # # # # # # # #[web@dev-kafka-02 zookeeper-3.4.14]$./bin/ zkserver. sh status Zookeeper JMX enabled by default Using config: / data/server - side/zookeeper - 3.4.14 / bin /.. /conf/zoo.cfg Mode: leader# # # # # # # # # # # # # # # # # # # # # # # # # #[web@dev-kafka-03 zookeeper-3.4.14]$./bin/ zkserver. sh status Zookeeper JMX enabled by default Using config: / data/server - side/zookeeper - 3.4.14 / bin /.. /conf/zoo.cfg Mode: followerCopy the code

Then build the Kafka clusterCopy the code

1. Download kafka 2.2.1 from kafka.apache.org/.

[web@dev-kafka-01 ~]$tar-xvf kafka_2.12-2.2.1. TGZ [web@dev-kafka-03 ~]$mv kafka_2.12-2.2.1 kafkaCopy the code

2. Modify the configuration file

[web@dev-kafka-03 kafka]$ vi config/server.properties 

# Change the broker ID. Here I correspond to the machine number
#dev-kafka-01 broker.id=0
#dev-kafka-02 broker.id=1
#dev-kafka-03 broker.id=2
broker.id=2

# Data directory
log.dirs=/data/server-data/kafka

# ZooKeeper connection address
zookeeper.connect=dev-kafka-01:2181,dev-kafka-02:2181,dev-kafka-03:2181

Copy the code

3. Start as a daemon process

[web@dev-kafka-03 kafka]$ ./bin/kafka-server-start.sh -daemon /data/server-side/kafka/config/server.properties 

Copy the code

4. Follow the preceding steps to start the other two machines and run the JPS command to check them

sudo jps
# Check whether the Kafka process exists. If so, the Kafka process is started successfully.Copy the code

Manager plug-in selection and installation

The company has been using rabbitMQ’s built-in manager page for a long time. The benefits of visualization are ease of operation and visibility of message queue congestion.

So before kafka, manager plug-in selection, comprehensive consideration, decided to use Yahoo kafka-Manager, kafka2.2.0 has been supported.

By the way, for a small team of dozens of people, it is common to go to a manager or something like that, but a regular company may require this not to be used, mainly for the sake of security. They may use only native commands and Java calls, with the monitoring aspect combining the Kafka-API and the popular Prometheus+Grafana

Kafka-manager installation steps refer to the following link:

www.cnblogs.com/frankdeng/p…

In addition, the compilation process is really very long. I compiled a package and uploaded it to Baidu Cloud disk:

https://pan.baidu.com/s/19APT4Gykf5oD2O4zRUJHGQ password: x3rsCopy the code

Kafka-manager I use the latest 2.0.0.2 kafka-Manager, because the Kafka-Manager community is not very active in updating, so I use the newer version to be compatible with the newer versions of Kafka. Kafka manager currently supports kafka versions up to 2.2.0. After testing the kafka2.2.1 installed by me, it also works. At this point, the Kafka cluster is installed.

An aside:

Kafka has made a change in version 2.4 to avoid the decompression operation that brokers need to perform verification

Specific issues address: https://issues.apache.org/jira/browse/KAFKA-8106