The author The date of The weather
Yuan childe 2019-12-19 (Thursday) Cloudy, light rain

The less you know, the less you don’t know

You can’t level up without your friends’ likes

A, gossip

Last night, I found out that a new version was released.

Kafka is a brand of coffee. Not right. It was originally developed by linkedin as a distributed messaging and streaming platform. (This must also be registered in the delay queue for reading the source code…)

Also, compared to RocketMQ, her deployment is very simple, there is no understanding of who is boss, just care about where to start a few nodes, and we vote the rest internally.

Er… By the way, Kafka also needs a male bestie buddy to use it, and he’s called ZooKeeper. ZooKeeper comes with the official package and you can choose to install it as a package or a la carte option.

Download the installation package

The official address: kafka.apache.org/downloads

Download the latest version of the software: kafka_2.13-2.4.0.tgz (using a higher version of Scala)

Iii. System environment check/preparation

  • Java JDK1.8 or above
  • The Centos7 operating system is used as an example
  • ZooKeeper middleware

4. Install Zookeeper

If the guest officer chooses the official package, please refer to the official document. The second step is described as follows:

bin/zookeeper-server-start.sh config/zookeeper.properties
Copy the code

If you have a personality and want to order your own, please refer to the Zookeeper deployment menu.

And if you are full, please ignore the above nonsense…

Five, the single

This section describes the single-node deployment of Kafka.

1. Create a dedicated user (you can skip this step and use root or you prefer)

A reference article for creating Linux users

2. Select the installation directory (place the tools in the /soft folder, or as you wish)

unzip

[root@hadoop-master /soft]# tar-xvzf kafka_2.13-2.1.0.tgz [root@hadoop-master /soft]# chown -r hadoop: Hadoop Kafka_2. 13-2.4.0Copy the code

Create a soft connection to facilitate software upgrade or multi-version switchover

[root@hadoop-master /soft]# ln -s kafka_2.13-2.4.0 kafka
Copy the code

3. Set environment variables

Append the following content to the bottom of the file

[root@hadoop-master /soft]# vi /etc/profile export KAFKA_HOME=/soft/kafka export PATH=$PATH:$KAFKA_HOME/bin  Esc :wq [root@hadoop-master /soft]# source /etc/profileCopy the code

4. Change the configuration – Initial stage

  1. Properties configuration file config/server.properties Find the following contents in the file, browse and modify.
#High energy alerts hadoop users
[root@hadoop-master /soft]# su - hadoop
[hadoop@hadoop-master ~]$ mkdir -p ~/kafka/config
[hadoop@hadoop-master ~]$ cp /soft/kafka/config/server.properties ~/kafka/config/server.properties
[hadoop@hadoop-master ~]$ vi ~/kafka/config/server.properties
#The unique identifier of the broker, a natural number
broker.id=0
#Listen for messages, default port 9092. PLAINTEXT:// Local IP address or domain name :9092
listeners=PLAINTEXT://hadoop-master:9092
#Message log path
log.dirs=/home/hadoop/kafka/data
#Zookeeper connection information. Multiple ZooKeepers are separated by commas (,). You can enter hostname or IP address + port number.
zookeeper.connect=hadoop-master:2181,hadoop-master2:2181,hadoop-dn1:2181
#Whether to automatically create a topictrue
# auto.create.topics.enable=falseRun the Esc :wq command to save the configurationCopy the code

  1. The JVM parameter

    In bin/kafka-server-start.sh, the default value is 1 GB.

[hadoop@hadoop-master ~]$ cp /soft/kafka/bin/kafka-server-start.sh /soft/kafka/bin/kafka-server-start.sh.bak
[hadoop@hadoop-master ~]$ vi /soft/kafka/bin/kafka-server-start.sh
#Find this line to modify:export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"Export KAFKA_HEAP_OPTS=" -xmx256m -xMS256m "Run the Esc :wq command to save the configurationCopy the code

  1. Log path

    In bin/kafka-run-class.sh, define the LOG_DIR environment variable.

[hadoop@hadoop-master ~]$ cp /soft/kafka/bin/kafka-run-class.sh /soft/kafka/bin/kafka-run-class.sh.bak
[hadoop@hadoop-master ~]$ vi /soft/kafka/bin/kafka-run-class.sh
#Reference line limitations under the License.
#Add the variable below, which can be replaced later by defining the environment variable in the self-start service.LOG_DIR="/home/hadoop/kafka/logs" Run Esc :wqCopy the code

5. Start the test

Hadoop user. Since the instruction has been added to the environment variable, it can be executed in a random directory.

[hadoop@hadoop-master /soft/kafka]$ kafka-server-start.sh /home/hadoop/kafka/config/server.properties
#Start the background: kafka - server - start. Sh - daemon/home/hadoop/kafka/config/server properties

[hadoop@hadoop-master /soft/kafka]$ jps
58301 Kafka
Copy the code

Other related instructions:

Stop the service: kafka-server-stop.sh

Sixth, the cluster

There are two main aspects to Kafka clustering. One is the configuration file (server.properties) for each node, and the other is the path where the log files are stored. But if you don’t care about logging, you can ignore this aspect. This time we will increase the difficulty by deploying three service nodes on one machine. Come on, slut…

Why often lane single stage, come when practicing, much stage is too troublesome. Second, because this kind of information on the Internet is still relatively few. If the guest officer you want more than one, after reading this article, more than one of you to play wisdom, their own fix. That has nothing to do with me, you are not very fierce ~

  1. Add three server.properties files and place them in the cluster folder, as shown below:
#High energy alerts hadoop users
[hadoop@hadoop-master /soft/kafka]$ mkdir -p ~/kafka/cluster
[hadoop@hadoop-master /soft/kafka]$ cp /soft/kafka/config/server.properties ~/kafka/cluster/server-1.properties
[hadoop@hadoop-master /soft/kafka]$ cp /soft/kafka/config/server.properties ~/kafka/cluster/server-2.properties
[hadoop@hadoop-master /soft/kafka]$ cp /soft/kafka/config/server.properties ~/kafka/cluster/server-3.properties
Copy the code

  1. Edit three configuration files, including broker. Id, Listeners, and log.dirs

    server-1.properties

[hadoop@hadoop-master /soft/kafka]$ vi ~/kafka/cluster/server-1.properties broker.id=1 listeners=PLAINTEXT://hadoop-master:9093 log.dirs=/home/hadoop/kafka/cluster/server-1/data Zookeeper. connect=hadoop-master:2181,hadoop-master2:2181,hadoop-dn1:2181 Run Esc :wq to save the configurationCopy the code

​ server-2.properties

[hadoop@hadoop-master /soft/kafka]$ vi ~/kafka/cluster/server-2.properties broker.id=2 listeners=PLAINTEXT://hadoop-master:9094 log.dirs=/home/hadoop/kafka/cluster/server-2/data Zookeeper. connect=hadoop-master:2181,hadoop-master2:2181,hadoop-dn1:2181 Run Esc :wq to save the configurationCopy the code

​ server-3.properties

[hadoop@hadoop-master /soft/kafka]$ vi ~/kafka/cluster/server-3.properties broker.id=3 listeners=PLAINTEXT://hadoop-master:9095 log.dirs=/home/hadoop/kafka/cluster/server-3/data Zookeeper. connect=hadoop-master:2181,hadoop-master2:2181,hadoop-dn1:2181 Run Esc :wq to save the configurationCopy the code
  1. Added kafka-server-stop-single.sh script to support stopping a single service. The key is to replace grep Java with $1.properties
[hadoop@hadoop-master /soft/kafka]$ cp /soft/kafka/bin/kafka-server-stop.sh ~/kafka/kafka-server-stop-single.sh
[hadoop@hadoop-master /soft/kafka]$ vi ~/kafka/kafka-server-stop-single.sh
#Overwrite the original commandPIDS = $(ps ax | grep -i 'kafka \. Kafka' | grep $1. The properties | grep -v grep | awk '} {print $1) perform save: Esc: wqCopy the code
  1. Three startup services have been added, focusing on logging environment variables and the corresponding server.properties file

​ kafka-1.service

#High energy reminder to use root user[root@hadoop-master /soft/kafka]# vi /etc/systemd/system/kafka-1.service [Unit] Description=kafka After=syslog.target network.target [Service] Type=forking User=hadoop Group=hadoop Environment=JAVA_HOME=/soft/jdk Environment=LOG_DIR=/home/hadoop/kafka/cluster/server-1/logs ExecStart=/soft/kafka/bin/kafka-server-start.sh -daemon /home/hadoop/kafka/cluster/server-1.properties ExecStop=/home/hadoop/kafka/kafka-server-stop-single.sh server-1 [Install] WantedBy=multi-user. Target  Esc :wq [root@hadoop-master /soft/kafka]# chmod 755 /etc/systemd/system/kafka-1.service [root@hadoop-master /soft/kafka]# systemctl enable kafka-1Copy the code
kafka-2.service
Copy the code
#Pick only different configurationsEnvironment=JAVA_HOME=/soft/jdk Environment=LOG_DIR=/home/hadoop/kafka/cluster/server-2/logs ExecStart=/soft/kafka/bin/kafka-server-start.sh -daemon /home/hadoop/kafka/cluster/server-2.properties ExecStop=/home/ hadoop-kafka /kafka-server-stop-single.sh server-2 ExecStop=/home/ hadoop-kafka /kafka-server-stop-single.sh server-2  Esc :wq [root@hadoop-master /soft/kafka]# chmod 755 /etc/systemd/system/kafka-2.service [root@hadoop-master /soft/kafka]# systemctl enable kafka-2Copy the code
kafka-3.service
Copy the code
Environment=JAVA_HOME=/soft/jdk Environment=LOG_DIR=/home/hadoop/kafka/cluster/server-3/logs ExecStart=/soft/kafka/bin/kafka-server-start.sh -daemon /home/hadoop/kafka/cluster/server-3.properties ExecStop=/home/hadoop/kafka/kafka-server-stop-single.sh server-3 ExecStop=/home/hadoop/kafka/kafka-server-stop-single.sh server-3  Esc :wq [root@hadoop-master /soft/kafka]# chmod 755 /etc/systemd/system/kafka-3.service [root@hadoop-master /soft/kafka]# systemctl enable kafka-3Copy the code
  1. Try starting and stopping these three services. If you don’t want to play, try disabling service again.
  2. Sleep…

7. The service starts automatically

Using the root user

[root@hadoop-master /soft/kafka]# vi /etc/systemd/system/kafka.service
#Or vi/lib/systemd/system/kafka. Service
#File to
[Unit]
Description=kafka  
After=syslog.target network.target

[Service]
Type=forking
User=hadoop
Group=hadoop
#Add environment variables, or add them in bin/kafka-run-class.sh. If not, the experiment will show an error
Environment=JAVA_HOME=/soft/jdk
Environment=LOG_DIR=/home/hadoop/kafka/logs
ExecStart=/soft/kafka/bin/kafka-server-start.sh -daemon /home/hadoop/kafka/config/server.properties
ExecStop=/soft/kafka/bin/kafka-server-stop.sh
#This will cause the process to stop, and then automatically start the service, very overbearing
#Restart=on-failure[Install] WantedBy=multi-user. Target  Esc :wq [root@hadoop-master /soft/kafka]# chmod 755 /etc/systemd/system/kafka.service [root@hadoop-master /soft/kafka]#  systemctl enable kafka [root@hadoop-master /soft/kafka]# service kafka startCopy the code

Other related instructions:

  • Stop: service kafka stop
  • Check the service status: service kafka status
  • After the service configuration is modified, the command systemctl daemon-reload takes effect
  • Disable the startup service (whether to delete service files after the startup service is disabled is optional) : systemctl disable kafka
  • Check whether the system starts the service: systemctl is-enabled kafka
  • View has launched the list of services: systemctl list – unit – files – type = service | grep enabled

Eight, encounter pit

1. Kafka process cannot be stopped

I did not encounter this problem in the deployment, here is the record of the online leaders of the solution, in case any partners encountered.

Run the bin/kafka-server-stop.sh script to find the kafka process ID from the JPS

#Reference line PIDS = $(ps ax | grep -i'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')
#Comment out the reference line and add the following instead,'kafka.Kafka'Filter characters can be adjusted based on actual conditions
PIDS=$(jps -lm | grep -i 'kafka.Kafka' | awk '{print $1}')
#Or the following command
#PIDS=$(ps ax | grep -i 'Kafka' | grep java | grep -v grep | awk '{print $1}') Run the Esc :wq command to save the configurationCopy the code

Appendix:

  • Official documents:
    • kafka.apache.org/quickstart
    • Kafka.apache.org/documentati…
  • Information Retrieval:
    • www.jianshu.com/p/49f23183a…
    • www.jianshu.com/p/6e5c89822…