Kafka relies on ZooKeeper. Install the ZooKeeper cluster before installing the Kafka cluster. Zookeeper cluster after the zooKeeper cluster is installed, run the zkserver. sh status command to check whether the ZOOKeeper cluster is started successfully

Zkserver. sh status ZooKeeper JMX enabled by default Using config: /opt/mashibing/zookeeper-3.6.1/bin/.. /conf/zoo.cfg Client port found: 2181. Client address: localhost. Mode: followerCopy the code

In the kafka_2.12-2.1.0/config directory, modify the server.properties configuration file:

node01:
broker.id=0
listeners=PLAINTEXT://node01:9092
log.dirs=/var/kafka-logs
zookeeper.connect=node01:2181,node02:2181,node03:2181/kafka

node02:
broker.id=1
listeners=PLAINTEXT://node02:9092
log.dirs=/var/kafka-logs
zookeeper.connect=node01:2181,node02:2181,node03:2181/kafka

node03:
broker.id=2
listeners=PLAINTEXT://node03:9092
log.dirs=/var/kafka-logs
zookeeper.connect=node01:2181,node02:2181,node03:2181/kafka
Copy the code

Add kafka’s bin directory to /etc/profile for ease of use.

Export KAFKA_HOME = / opt/ningcg/kafka_2 12-2.1.0 export PATH = $PATH: $JAVA_HOME/bin: $ZOOKEEPER_HOME/bin: $KAFKA_HOME/binCopy the code

Run the kafka-server-start.sh server.properties command to start Kafka on the three machines. After the startup is complete, run the ls /kafka command to view the information kafka stores on the ZK

ls /kafka
[admin, brokers, cluster, config, consumers, controller, controller_epoch, isr_change_notification, latest_producer_id_block, log_dir_event_notification]

ls /kafka/brokers/ids
[0, 1, 2]
Copy the code

Kafka commands:

Create a topic:  kafka-topics.sh --zookeeper node01:2181,node02:2181,node03:2181/kafka --create --topic test007 --partitions 2 --replication-factor 2 query topic: Kafka - switchable viewer. Sh -- -- zookeeper node01:2181, node02:2181, node03:2181 / kafka - the list view topic description information:  kafka-topics.sh --zookeeper node01:2181,node02:2181,node03:2181/kafka --describe --topic test007 Topic:test007 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic: Test007 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 consumption message: Kafka - the console - consumer. Sh -- -- the bootstrap - server node01:9092, node02:9092, node02:9092 - topic test007 - group ningcg production news: Kafka-console-producer.sh --broker-list node03:9092 --topic test007 Kafka-consumer-groups. sh --bootstrap-server node01:9092 --list Consumer group information:  kafka-consumer-groups.sh --bootstrap-server node01:9092 --describe --group testCopy the code

Consuming messages can specify different consuming groups, and consuming messages between different consuming groups do not affect each other.

Within a consumer group, the same message can only be consumed by one member of the consumer group.

Consumer Group Information:

Messages end up on the corresponding partition of the topic, and can be scaled horizontally (dynamically adding partitions) if the messages are not ordered. If there are different categories of messages, and the same kind of messages need to be ordered, you can set the key of the message (K, V type messages), Kafka will be the same key type of messages into the same partition.