“This is the 23rd day of my participation in the Gwen Challenge.
This article is the second part of Kafka, mainly introduces the Kafka cluster construction and use. First introduction to Kafka.
Environment to prepare
The JDK installation
Since Kafka is developed in Scala and runs on the JVM, you need to install the JDK before installing Kafka.
Yum install Java - 1.8.0 comes with - its * - yCopy the codeThe ZooKeeper cluster is created
Previous versions of Kafka relied on ZooKeeper, so zooKeeper had to be installed first. Now kafka does not rely on ZK.
Install the latest version of ZK, everyone pay attention to the download file. Download xx.bin.tar.gz!! Otherwise there will be unexpected problems!! , such as the installation of zK, automatic ZK when the report can not load classes!!
Zookeeper.apache.org/releases.ht…
Download the zk
Zk address: mirrors.bfsu.edu.cn/apache/zook…
Wget HTTP: / / https://mirrors.bfsu.edu.cn/apache/zookeeper/stable/apache-zookeeper-3.6.3-bin.tar.gzCopy the codeUnpack the zk
Tar ZXVF - apache - they are - 3.6.3 - bin. Tar. GzCopy the codeCreating a Soft connection
This step is optional, just a personal habit.
Ln -s/soft/apache - zookeeper 3.6.3 - bin/soft/zookeeperCopy the codeCreating a Configuration File
Go to the zoo_sample. CFG directory and copy the zoo_sample. CFG configuration file.
cp zoo_sample.cfg zoo.cfg
Copy the codeCFG file will be loaded when the zK service is started.
Modifying a Configuration File
Create the data and logs directories ahead of time
dataDir=/soft/zookeeper/data
dataLogDir=/soft/zookeeper/logs
server.1=s201:8888:6666
server.2=s202:8888:6666
server.3=s203:8888:6666
Copy the codeserver.x=A:B:C
- X: where x is a number indicating the server number, and its value corresponds to the value in the myID file.
- A: Indicates the domain name or IP address of the server.
- B: port used by the server to exchange messages with the leader in the cluster.
- C: Configures the port used to elect the leader.
Specific zK related knowledge will be introduced in the CHAPTER of ZK, here is not more.
Create myid
Go to the data directory and create a myID file that corresponds to x on the server.
echo 1 > myid
Copy the codeData is synchronized to the other two nodes
Distribute ZK to the other two nodes
Scp-r apache-zookeeper-3.6.3-bin root@s202:/soft/ scp-r apache-zookeeper-3.6.3-bin root@s203:/soft/Copy the codeCreate soft connections on the other two servers
SSH s202 ln -s /soft/apache-zookeeper-3.6.3-bin /soft/zookeeper SSH S203 ln -s /soft/apache-zookeeper-3.6.3-bin /soft/zookeeperCopy the codeNote: Do not forget to change the myID on the other two servers.
Start the ZK cluster
If conditions are available, build a ZK cluster. If resources are tight, build a ZK pseudo cluster. It is only necessary to load the corresponding configuration file during startup. If there are conflicts in the configuration file, such as ports, data, and logs, please separate them.
Bin/zkserver. sh press Enter to see why the configuration file is zoo.cfg.
Start ZK on three machines separately
/soft/zookeeper/bin/zkServer.sh start &
Copy the codeCheck zK cluster status
/soft/zookeeper/bin/zkServer.sh status
Copy the code
Kafka cluster setup and use
download
Kafka.apache.org/downloads.h…
Wget HTTP: / / https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgzCopy the codeUnpack the
The tar - ZXVF kafka_2. 13-2.8.0. TGZCopy the codeCreating a Soft connection
May or may not, personal habits.
Ln -s/soft/kafka_2. 13-2.8.0 / soft/kafkaCopy the codeModify the server.properties file
conf/server.properties
Create the /soft/kafka/kafka-logs directory
broker.id=1
listeners=PLAINTEXT://s201:9092
log.dirs=/soft/kafka/kafka-logs
zookeeper.connect=s201:2181,s202:2181,s203:2181
Copy the codeSynchronize data to the other two nodes
Scp-r kafka_2.13-2.8.0 root@s202:/soft/ scp-r kafka_2.13-2.8.0 root@s203:/soft/ SSH s202 ln-s /soft/kafka_2.13-2.8.0 /soft/kafka SSH s203 ln -s /soft/kafka_2.13-2.8.0 /soft/kafka SSH s203 ln -s /soft/ kafka_2.8.0 /soft/kafkaCopy the codeExample Modify information about Sever2 and Server3
# sever2
broker.id=2
listeners=PLAINTEXT://s202:9092
log.dirs=/soft/kafka/kafka-logs
zookeeper.connect=s201:2181,s202:2181,s203:2181
# sever3
broker.id=3
listeners=PLAINTEXT://s203:9092
log.dirs=/soft/kafka/kafka-logs
zookeeper.connect=s201:2181,s202:2181,s203:2181
Copy the codeStart the
Start it on three servers
/soft/kafka/bin/kafka-server-start.sh -daemon /soft/kafka/config/server.properties
Copy the codeCreate a topic
Create a Topic named MOE with only one partition and one replica
./kafka-topics. Sh Press enter to see the details to see which parameters are available and which are required.
./kafka-topics.sh --bootstrap-server s201:9092 --create --topic moe --partitions 1 --replication-factor 1
Copy the code
View topic information
View information for TOPIC MOE
./kafka-topics.sh --bootstrap-server s201:9092 --topic moe --describe
Copy the code
- The Leader node is responsible for all read and write requests to a given partition.
- Replicas indicates which brokers have backups for a partition. Whether or not the leaders are listed, even if the node fails.
- Isr is a subset of REPLICas, which lists only the nodes that are currently alive and have been synchronously backed up to different partitions.
Send a message
Send a message to the topic for the MOE.
./kafka-console-producer.sh --bootstrap-server s201:9092 --topic moe
Copy the codeIt enters the console and waits for user input.
News consumption
Consumption topic is MOE news.
/soft/kafka/bin/kafka-console-consumer.sh --bootstrap-server s201:9092 --topic moe --from-beginning
Copy the code
conclusion
This article introduces kafka cluster construction and ZK cluster construction, you can also install the pseudo cluster, load a different configuration file when starting, configuration file port and other attention can be different to build pseudo cluster.
Welcome everyone to pay attention to the public account (MarkZoe) to learn from each other and communicate with each other.