Kafka cluster + Zookeeper + Kafka-Manager

Kafka build:

Create a Zookeeper container:

Here we use the simplest way to create a single Zookeeper node. If you want to consider the high availability of Zookeeper, you can make it into a cluster, preferably with multiple machines.

$ docker run -d –name zookeeper -p 2181:2181 -t wurstmeister/zookeeper

By default, the container configuration files are in /conf/zoo. CFG, and the data and log directories are in /data and /datalog by default, which can be mapped to the host’s reliable file directory if necessary.

Create kafka cluster nodes:

Use the Docker command to quickly set up multiple KafKas on the same machine with only brokerId and port changes

Node 1:

$docker run -d –name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT= 192.168.0.102:2181-e KAFKA_ADVERTISED_LISTENERS = PLAINTEXT: / / 192.168.0.102-9092 – e KAFKA_LISTENERS = PLAINTEXT: / / 0.0.0.0:9092 – t wurstmeister/kafka

Node 2:

$docker run -d –name kafka1 -p 9093:9093 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT= 192.168.0.102:2181-e KAFKA_ADVERTISED_LISTENERS = PLAINTEXT: / / 192.168.0.102-9093 – e KAFKA_LISTENERS = PLAINTEXT: / / 0.0.0.0:9093 – t wurstmeister/kafka

Node 3:

$docker run -d –name kafka2 -p 9094:9094 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT= 192.168.0.102:2181-e KAFKA_ADVERTISED_LISTENERS = PLAINTEXT: / / 192.168.0.102-9094 – e KAFKA_LISTENERS = PLAINTEXT: / / 0.0.0.0:9094 – t wurstmeister/kafka

There are four main parameters set

KAFKA_BROKER_ID = 0 KAFKA_ZOOKEEPER_CONNECT = 192.168.0.102:2181 KAFKA_ADVERTISED_LISTENERS = PLAINTEXT: / / 192.168.0.102:9092 KAFKA_LISTENERS = PLAINTEXT: / / 0.0.0.0:9092Copy the code

The middle two parameters 192.168.0.102 are changed to the IP address of the host machine. If you do not set this, kafka may not be accessible on other machines.

How to check the local IP address:

$ ifconfig en0

Test kafka:

Enter the kafka container command line

$ docker exec -ti kafka /bin/bash

Go to the directory where Kafka resides

$CD opt/kafka_2. 12-1.1.0 /

Create topic where Replication = 2 and Partition = 2

$bin/kafka-topics. Sh –create –zookeeper 192.168.0.102:2181 –replication-factor 2 — Partitions 2 –topic partopic

Or it can be created through kafka-manager

Check the status of the topic:

Enter the opt/kafka_2.12-1.1.0/ directory in the kafka container

$bin/kafka-topics. Sh –describe — Zookeeper 192.168.0.102:2181 –topic partopic

The Leader machine for each partition is displayed as Broker0, with backups on Broker0 and 1, and Isr stands for those that are alive on living backup machines. When kafka1 is disabled,

$ docker stop kafka1

Query topic status and output result:

Topic:partopic PartitionCount:2 ReplicationFactor:2 Configs: Topic:partopic Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0 Topic: partopic Partition: 1 Leader: 0 Replicas: 1,0 Isr: 0Copy the code

Create Kafka management node:

Kafka-manager has a graphical UI that makes it easy to monitor cluster status and adjust queue configuration

$docker run -itd –restart=always –name=kafka-manager -p 9000:9000 -e ZK_HOSTS=”192.168.0.102:2181″ sheepkiller/kafka-manager

After the container is started, it accesses port 9000 of the host, 0.0.0:9000

You need to add a cluster id for the first time, as shown in the figure below

Once configured, the Zookeeper management node can read information about the entire Kafka cluster, and we can see the health status of the cluster directly through JMX

You can also see the read/write speed, offset and other information of the topic, as shown below:

Brokers,Partitions, and Replicas are visible

And the group:

Reference article:

Blog.csdn.net/lblblblblzd…

Kafka cluster + Zookeeper + Kafka-manager