Kafka relies on the JVM environment and requires JDK installation
Use CentOS7 system, first delete the JDK, built-in is not easy to use (such as JPS terminal command can not use). Download the full JDK from the official website and set the environment variable JAVA_HOME. See:
Blog.csdn.net/libaineu200…
[root@localhost ~]# Java -version openJDK version "1.8.0_141" OpenJDK Runtime Environment (build 1.8.0_141-b16 64-bit Server VM (build 25.141-b16, mixed mode)Copy the codeSecond, the download
Kafka (kafka.apache.org/) depend on the ZooKeeper mirrors.shuosc.org/apache/zook… – stable version of http://mirrors.shuosc.org/apache/kafka/1.1.0/kafka-1.1.0-src.tgz mirror.bit.edu.cn/apache/kafk… / 1.1.0 kafka_2. 12-1.1.0. TGZ
Since kafka_2.12-1.1.0.tgz is an executable program that integrates with zooKeeper lite, just download it for standalone testing. But the simple version has limited functionality. In the production environment, you are advised to download the official ZooKeeper software. \
\
Run and test, the following are terminal commands
CD kafka_2.12-1.1.0 2. Start Zookeeper. If no error message is displayed, the Zookeeper is successfully started. Daemons and Nohup & are implemented in the background, i.e. daemons.
Method 1 (recommended). / bin/zookeeper – server – start. Sh – daemon config/zookeeper. Properties
Methods 2 nohup. / bin/zookeeper – server – start. Sh config/zookeeper properties & 3, start, Kafka, if no error occurs then started successfully. Daemons and Nohup & are implemented in the background, i.e. daemons. Method 1 (recommended)./bin/kafka-server-start.sh -daemon config/server.properties\
Nohup./bin/kafka-server-start.sh config/server.properties& \
4. Check whether the process is running properly. Check whether port 2181 is zooKeeper and port 9092. 9092 is kafka netstat tunlp | egrep “(2181 | 9092)” TCP 0 0: : : 2181: : : * LISTEN/Java TCP 19787 0 0: : : 9092: : : * LISTEN 28094/java \
\
We can also use the JPS terminal command to check whether it is started:
[root@bogon kafka_2.12-1.1.0]# JPS 4384 Kafka 3921 QuorumPeerMain 4412 JPS \
\
5. Create a theme, such as test
./bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test
The. / bin/kafka – switchable viewer. Sh — — create — zookeeper 172.16.6.170:2181172.16. 6.170:2182172.16. 6.170:2183 – replication – factor 3 –partitions 3 –topic emqtest\
-- ZooKeeper: Indicates the list of ZooKeeper clusters, separated by commas (,). You can specify one or several ZooKeeper nodes instead of the entire zooKeeper cluster. Replication-factor: specifies the number of replication nodes and provides the failover mechanism. 1 indicates that data is recorded on only one broker. Generally, the value is greater than 1. It means that one piece of data is automatically synchronized to multiple brokers to prevent data loss if one broker fails. Partitions: A topic can be partitioned into multiple partitions. One consumer can consume multiple partitions, but only one consumer can consume one partitions, so increasing partitions increases consumer throughput. Kafka only guarantees that messages within one partitions are ordered and data between multiple partitions is unordered.Copy the code\
Pay attention to,
(1) When using a cluster, you must manually create a topic and specify the ZooKeeper node list. Standalone should also be created. Otherwise, the client will report an error.
(2) Do not use “.” or “_” characters in the subject name.
[root@localhost kafka_2.12-1.0.0copy1]#./bin/kafka-topics. Sh --create --zookeeper 172.16.6.170:2181172.16. 6.170:2182172.16. 6.170:2183 - replication - factor 3-3 partitions - topic emq_broker_message1 WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, But not both. Created topic "emq_broker_message1". [root@localhost kafka_2.12-1.0.0copy1]#./bin/kafka-topics - create - zookeeper 172.16.6.170:2181172.16. 6.170:2182172.16. 6.170:2183 - replication - factor 3-3 partitions --topic emqtest Created topic "emqtest".Copy the code(3) It is recommended that you specify partitions as an integer multiple of the number of Kafka Broker nodes when creating a topic. For example, if there are 3 brokers, then partitions are set to 3 or 6 or 9 –partitions are 3. Once the theme is created, you can see the folder for the theme and partition in the path log.dirs. For details, see section 3.6.1 of Kafka Getting Started and Doing. Mou Da ‘en
(4) –replication-factor 3, section 6.3.1 of the Authoritative Kafka Guide recommends a replication factor of 3. However, the value must not be greater than the number of nodes of the broker.
If the number of broker nodes is 1 and the replication factor is set to 3, an error will be reported:
Error while executing topic command : Replication factor: 3 larger than available brokers: 1.\
(5) In single-node mode, the number of ZooKeeper is 1. In a cluster, you are advised to configure at least three ZooKeepers in an odd number.
(6) Set the default value for the server.properties file
num.partitions=3
default.replication.factor=1\
\
6. View themes
./bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic test\
./bin/kafka-topics.sh –list –zookeeper localhost:2181
For example, a broker cluster with three nodes. Three partitions are also assigned to topic mynewt. Each partition has a leader and follow.
/bin/kafka-topics. Sh --describe -- Zookeeper localhost:2181 --topic mynewt topic :mynewt PartitionCount:3 ReplicationFactor:3 Configs: Topic: mynewt Partition: 0 Replicas: 2,1,0 Isr: 2,1,0 Topic: Leader: 0 Replicas: 0,2,1 Isr: 0,2,1 Topic: mynewt Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2Copy the code\
Kafka./bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test /bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic test –from-beginning /bin/kafka-console-consumer.sh –bootstrap-server localhost:2181 –topic test –from-beginning ./bin/kafka-server-stop.sh ./bin/zookeeper-server-stop.sh\
When Kafka starts, start ZooKeeper first and then Kafka. On the contrary, disable Kafka and then ZooKeeper. \
\
Four,
1. What does Kafka have? Kafka operates as a cluster, consisting of one or more services. A service is a Broker ZooKeeper that coordinates forwarding
2. Configuration files and message persistence paths
/config/server.properties
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
Copy the codeFor example, the client sends the test as the theme of the message, will persist in this file: / TMP/kafka – logs/test – 0/00000000000000000000. The log \
The list of topics is logged at/TMP /zookeeper
\
/config/zookeeper.properties
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
Copy the codeYou are advised to change the kafka log directory and ZooKeeper data directory. These two directories are stored in the TMP directory by default. The TMP directory will be lost after the restart.
Server. The properties: Log. dirs=/ TMP /kafka-logs changed to log.dirs=/usr/local/kafka/logs zookeeper.properties dataDir=/ TMP /zookeeper changed to dataDir=/usr/local/zookeeper/data\
\
The bootstrap.servers configuration file is different from the broker.list configuration file. The bootstrap.servers configuration file is different from the broker.list configuration file. The bootstrap.servers configuration file is different from the broker.list configuration file
\
Kafka. Sh — this script has a problem and is unstable
#! /bin/bash kafka_home=/root/Downloads/kafka_2.12-1.0.0/ case $1 in start) $kafka_home/bin/zookeeper-server-start.sh -daemon $kafka_home/config/zookeeper.properties #sleep 1 echo "kafka start" $kafka_home/bin/kafka-server-start.sh -daemon $kafka_home/config/server.properties #sleep 1 ;; Echo "kafka stop" $kafka_home/bin/kafka-server-stop.sh #sleep 1 echo "zookeeper stop" $kafka_home/bin/kafka-server-stop.sh $kafka_home/bin/zookeeper-server-stop.sh #sleep 1 ;; The following steps are required to restart the service: ;; # check the status. ;; *) echo "$0 {start|stop|restart|status}" exit 4 ;; esacCopy the code\
5. Zookeeper cluster
Kafka uses ZooKeeper to manage clusters. The Kafka package includes a simplified version of ZooKeeper, but feels limited. In the production environment, you are advised to download the official ZooKeeper software. Download the latest version of the ZooKeeper software: \
Zookeeper zookeeper.apache.org/ http://mirror.bit.edu.cn/apache/zookeeper/ – 3.4.11. Tar. Gz \
\
Welcome to my own Kafka Broker and ZooKeeper cluster practice.
\
—
reference
Internet of Things Architecture Growth path (8)-EMQ-Hook Understand and connect Kafka to send messages
11-1.0.0 ZooKeeper installation and deployment Kafka Installation and deployment CentOS ZooKeeper single-machine deployment mode, cluster deployment mode Kafka single-machine deployment mode, cluster deployment mode (I)\
KafkaOffsetMonitor: KafkaOffsetMonitor: KafkaOffsetMonitor github.com/quantifind/ \