1. Scenario description
Because kafka used not much before, only to write and read data in topic, this time just to use, record kafka cluster construction and combined with springboot use.
2. Solutions
2.1 Introduction
Kafka is often used in big data projects. In our project, kafka is used to receive log flow data.
(ii) There are four main aspects of message middleware:
(1) ActiveMQ: it has a long history and was frequently used in previous projects, but now it is updated slowly and its performance is relatively low. Kafka: distributed, high performance, cross-language, super high performance, millions per second, simple mode. (4) RocketMQ: Alibaba open source messaging middleware, pure Java implementation, commercial version, charging, leading to general promotion.
(c) Kafka’s advantages over the other three are:
(1) High performance, millions per second level;
(2) Distributed, high availability, horizontal expansion.
(4) Kafka official website
There is a Chinese official website, you can have a detailed look.
Address: kafka.apachecn.org/intro.html
2.2 Software Download
2.2.1 kakfa download
Address: kafka.apache.org/downloads, download the latest against 2.4.1.
2.2.2 zookeeper download
(1) Kafka relies on ZooKeeper for scheduling. Kafka is actually built with Kafka, but it is generally recommended to use an independent ZooKeeper to facilitate subsequent upgrades and public use.
(2) Download address:
Zookeeper.apache.org/, the latest version is 3.6.0, but it is not long ago. It is recommended to use the 3.5.7 version in accordance with kafka built-in ZooKeeper.
2.2.3 Download Instructions
The files are small. Zk is over 9m, kafka is over 50 megabytes
2.3 Kafka Single-node deployment and Cluster Deployment
** Software Lao Wang locally made three virtual machines, IP respectively:
192.168.85.158
192.168.85.168
192.168.85.178
Copy the code2.3.1 Single-machine Deployment
(1) Upload the JAR package to the /root/tools directory. You do not need to create a new user. Run the jar package under the root account.
(2) Decompress
[root@ruanjianlaowang158 tools]# tar -zxvf kafka_2.12-2.4.1. TGZ [root@ruanjianlaowang158 Tools]# tar -zxvf Apache - they are - 3.5.7 - bin. Tar. GzCopy the code(3) Configure zooKeeper and start it
[root@ruanjianlaowang158 apache-zookeeper-3.5.7-bin]# CD /root/tools/apache-zookeeper-3.5.7-bin Configure [root@ruanjianlaowang158 apache-zookeeper-3.5.7-bin]# mkdir data [root@ruanjianlaowang158 conf]# CD in the following configuration file /root/tools/apache-zookeeper-3.5.7-bin/conf [root@ruanjianlaowang158 conf]# cp zoo_sample.cfg zoo.cfg [root@ruanjianlaowang158 conf]# vi zoo. CFG DataDir =/ TMP /zookeeper dataDir=/root/tools/apache-zookeeper-3.5.7-bin/data # start zookeeper [root@ruanjianlaowang158 bin]# CD /root/tools/apache-zookeeper-3.5.7-bin/bin [root@ruanjianlaowang158 bin]#./ zkserver. sh startCopy the code(4) Configure kafka and start it
[root@ruanjianlaowang158 kafka_2.12-2.4.1]# CD /root/tools/kafka_2.12-2.4.1 Create an empty folder [root@ruanjianlaowang158 kafka_2.12-2.4.1]# mkdir data Change the configuration file [root@ruanjianlaowang158 config]# CD /root/tools/kafka_2.12-2.4.1/config [root@ruanjianlaowang158 config]# vi Server.properties #listeners =/ TMP /kafka-logs log.dirs=/root/tools/kafka_2.12-2.4.1/data # process = PLAINTEXT: / / your. Host. Name: 9092 listeners = PLAINTEXT: / / 192.168.85.158:9092 # zookeeper. Connect = localhost: 2181 Zookeeper. connect=192.168.85.158:2181 # kafka [root@ruanjianlaowang158 bin]# CD /root/tools/kafka_2.12-2.4.1/bin [root@ruanjianlaowang158 bin]# ./zookeeper-server-start.sh .. /config/server.properties &Copy the codeAfter the startup, the single-machine verification is not verified, and the verification is carried out directly in the cluster.
2.3.2 Cluster Deployment
(1) Cluster mode: Decompress the preceding single-machine mode and configure it on servers 192.168.85.168 and 192.168.85.178.
(2) Zookeeper is changed to zoo.cfg
The three servers are the same:
[root@ruanjianlaowang158 conf]# CD /root/tools/apache-zookeeper-3.5.7-bin/conf [root@ruanjianlaowang158 conf]# vi Zoo. CFG # other unchanged, add the last three lines, three server configuration is the same, Software Laowang server.1=192.168.85.158:2888:3888 server.2=192.168.85.168:2888:3888 server.3=192.168.85.178:2888:3888 158 Echo "1" > /root/tools/apache-zookeeper-3.5.7-bin/data/ myID 168 Run the following command on the server: Echo "2" > /root/tools/apache-zookeeper-3.5.7-bin/data/myid 178 Echo "3" > / root/tools/apache - they are - 3.5.7 - bin/data/myidCopy the code(3) Kafka cluster configuration
If you think the article is helpful to you, welcome to wechat search "software Lao Wang" for the first time to read or communicate!Copy the code[root@ruanjianlaowang158 config]# CD /root/tools/kafka_2.12-2.4.1/config [root@ruanjianlaowang158 config]# vi Server.properties #broker.id 158 Server set to 1,168 Server set to 2,178 Server set to 3 broker.id=1 # All three servers are configured the same Zookeeper. Connect = 192.168.85.158:2181192168 85.168:2181192168 85.178:2181Copy the codeKafka Broker Broker
| Configuration items | Default value/Example value | instructions |
|---|---|---|
| broker.id | 0 | Broker unique identification |
| listeners | PLAINTEXT: / / 192.168.85.158:9092 | To listen for information, PLAINTEXT means PLAINTEXT transmission |
| log.dirs | / root/tools/apache – they are – 3.5.7 – bin/data | Kafka data storage address. You can enter multiple addresses. With “, “interval |
| message.max.bytes | message.max.bytes | Limit on the length of a single message, in bytes |
| num.partitions | 1 | Default number of partitions |
| log.flush.interval.messages | Long.MaxValue | The maximum number of messages accumulated before data is written to the hard disk and available to consumers |
| log.flush.interval.ms | Long.MaxValue | The maximum time before data is written to the hard disk |
| log.flush.scheduler.interval.ms | Long.MaxValue | The interval at which data is to be written to the hard disk. |
| log.retention.hours | 24 | Controls the retention time of a log, in hours |
| zookeeper.connect | 192.168.85.158:2181, 192.168.85.168:2181, 192.168.85.178:2181 |
IP address of the ZooKeeper server. Multiple servers are separated by commas (,) |
(4) The cluster starts
The startup mode is the same as that of a single machine:
[root@ruanjianlaowang158 bin]# CD /root/tools/apache-zookeeper-3.5.7-bin/bin [root@ruanjianlaowang158 bin]# Sh start # kafka [root@ruanjianlaowang158 bin]# CD /root/tools/kafka_2.12-2.4.1/bin [root@ruanjianlaowang158 bin]# ./zookeeper-server-start.sh .. /config/server.properties &Copy the code(5) Pay attention
Kafka: Configured broker. Id 2 doesn't match stored broker. Id 0 in meta.properties. Solution: There is a file in server 158 data: meta.properties. The broker.id in the file also needs to be changed to be the same as the broker.id in server.Copy the code(6) Create a topic for later springboot project test.
[root@ruanjianlaowang158 bin]# CD /root/tools/kafka_2.12-2.4.1/bin [root@ruanjianlaowang158 bin]#./kafka-topics - create - zookeeper 192.168.85.158:2181192168 85.168:2181192168 85.178:2181 - replication - factor 3 - partitions --topic aaaaCopy the code2.4 Combine with springboot project
Against 2.4.1 pom file
<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion > 4.0.0 < / modelVersion > < the parent > < groupId > org. Springframework. Boot < / groupId > The < artifactId > spring - the boot - starter - parent < / artifactId > < version > 2.2.0. RELEASE < / version > < relativePath / > <! -- lookup parent from repository --> </parent> <groupId>com.itany</groupId> <artifactId>kafka</artifactId> <version>0.0.1-SNAPSHOT</version> <name>kafka</name> <description>Demo project for Spring Boot</description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>Copy the codeDescription:
There are mainly two GAVs, one is spring-boot-starter-Web, which starts the use of Web services; One is spring-Kafka, which is the SpringBoot integrated kafka core package.
Application 2.4.2. Yml
Bootstrap-servers: bootstrap-servers: 192.168.85.158:9092192168 85.168:9092192168:85.178 9092 producer: key - serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.apache.kafka.common.serialization.StringSerializer consumer: group-id: test key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.apache.kafka.common.serialization.StringDeserializerCopy the code2.4.3 Producers
@RestController public class KafkaProducer { @Autowired private KafkaTemplate template; // Software Lao Wang, Aaaa @requestMapping ("/sendMsg") public String sendMsg(String topic, String message){ template.send(topic,message); return "success"; }}Copy the code2.3.4 Consumer
@component public class KafkaConsumer {// @kafkalistener (topics = {"aaaa"}) public void listen(ConsumerRecord record){ System.out.println(record.topic()+":"+record.value()); }}Copy the code2.4.5 Verifying results
(1) Enter in the browser
http://localhost:8080/sendMsg?topic=aaaa&message=bbbb
Copy the code(2) Software Lao Wang’s IDEA console printing information
More knowledge please pay attention to the public number: “software Lao Wang”, IT technology and related dry goods to share, reply keywords to get corresponding dry goods!