Welcome to pay attention to my wechat public number [old Week chat architecture], The principle of Java backend mainstream technology stack, source code analysis, architecture and a variety of Internet high concurrency, high performance, high availability solutions.
First, environmental preparation
3dCH kafka zookeeper3.4.14 ZooKeeper 3dCH kafka kafka2.12 zookeeper3.4.14 ZooKeeper 3dCH kafka
1.1 Java environment is the prerequisite
1.1.1 uploadjdk-8u261-linux-x64.rpmGo to the server and install
Install command
rpm -ivh jdk-8u261-linux-x64.rpm
Copy the code
1.1.2 Configuring Environment Variables
# Edit the configuration file, JDK bin directory to /etc/profile file, valid for all users' shells
vim /etc/profile
# take effect
source /etc/profile
Copy the code
exportJAVA_HOME = / usr/Java/jdk1.8.0 _261 - amd64export PATH=$PATH:$JAVA_HOME/bin
Copy the code
# validation
java -version
Copy the code
1.2 Zookeeper Installation and Configuration
1.2.1 uploadZookeeper - 3.4.14. Tar. GzGo to the server, unzip to/opt
Unzip zk to the specified directoryTar -zxf zookeeper-3.4.14.tar.gz -c /optCopy the code
ZookeeperThe directory where the data is stored,dataDir
Go to the conf configuration directory
cdThe/opt/zookeeper - 3.4.14 / confCFG: copy zoo_sample. CFG to zoo.cfg
cp zoo_sample.cfg zoo.cfg
# edit zoo. CFG file
vim zoo.cfg
dataDir=/var/riemann/zookeeper/data
Copy the code
/etc/profileTo make the configuration take effect
Set the environment variable ZOO_LOG_DIR to specify the location where Zookeeper saves logs. ZOOKEEPER_PREFIX points to the decompression directory of Zookeeper. Add Zookeeper’s bin directory to PATH:
exportZOOKEEPER_PREFIX = / opt/zookeeper - 3.4.14export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/riemann/zookeeper/log
Copy the codeThe configuration takes effect only after the configuration:
source /etc/profile
Copy the code1. Start theZookeeperAnd confirm thatZookeeperThe state of the
zkServer.sh start
Copy the code
Zookeeper is successfully installed.
1.3 Installation and configuration of Kafka
1.3.1 uploadKafka_2. 12-1.0.2. TGZGo to the server and unzip
Tar -zxf kafka_2.12-1.0.2.tgz -c /optCopy the code
vim /etc/profile
Copy the codeexportKAFKA = / opt/kafka_2. 12-1.0.2export PATH=$PATH:$KAFKA/bin
Copy the codesource /etc/profile
Copy the code1.3.3 configuration/ opt/kafka_2. 12-1.0.2 / configIn theserver.propertiesfile
Vi/opt/kafka_2. 12-1.0.2 / config/server propertiesCopy the code
Kafka IP address for connecting to Zookeeper. The local Zookeeper instance is used
The connection address is' localhost:2181 '. MyKafka is the root path of 'Kafka' in ZookeeperCopy the code
log.dirs=/var/riemann/kafka/kafka-logs
Copy the codeCreate the persistent data directory above
mkdir -p /var/riemann/kafka/kafka-logs
Copy the code1.4 start the Kafka
Go to the Kafka installation root directory and run the following command:
kafka-server-start.sh .. /config/server.propertiesCopy the code
1.5 Open a new window to view the Zookeeper node
1.6 Kafka starts in foreground mode. To stop Kafka, press Ctrl+C
To start in the background, use the following command:
kafka-server-start.sh -daemon config/server.properties
Copy the codeKafka background processes
ps -ef | grep kafka
Copy the code
Stop Kafka running in the background:
kafka-server-stop.sh
Copy the codeIi. Production and consumption
Check the ZooKeeper status. Zookeeper starts successfully. Then start Kafka.
View the help information about the command
[root@master1 bin]# kafka-topics.sh
Create, delete, describe, or change a topic.
Option Description
------ -----------
--alter Alter the number of partitions,
replica assignment, and/or
configuration for the topic.
--config <String: name=value> A topic configuration override for the
topic being created or altered.The
following is a list of valid
configurations:
cleanup.policy
compression.type
delete.retention.ms
file.delete.delay.ms
flush.messages
flush.ms
follower.replication.throttled.
replicas
index.interval.bytes
leader.replication.throttled.replicas
max.message.bytes
message.format.version
message.timestamp.difference.max.ms
message.timestamp.type
min.cleanable.dirty.ratio
min.compaction.lag.ms
min.insync.replicas
preallocate
retention.bytes
retention.ms
segment.bytes
segment.index.bytes
segment.jitter.ms
segment.ms
unclean.leader.election.enable
See the Kafka documentation for full
details on the topic configs.
--create Create a new topic.
--delete Delete a topic
--delete-config <String: name> A topic configuration override to be
removed for an existing topic (see
the list of configurations under the
--config option).
--describe List details for the given topics.
--disable-rack-aware Disable rack aware replica assignment
--force Suppress console prompts
--help Print usage information.
--if-exists if set when altering or deleting
topics, the action will only execute
if the topic exists
--if-not-exists if set when creating topics, the
action will only execute if the
topic does not already exist
--list List all available topics.
--partitions <Integer: # of partitions> The number of partitions for the topic
being created or altered (WARNING:
If partitions are increased for a
topic that has a key, the partition
logic or ordering of the messages
will be affected
--replica-assignment <String: A list of manual partition-to-broker
broker_id_for_part1_replica1 : assignments forthe topic being broker_id_for_part1_replica2 , created or altered. broker_id_for_part2_replica1 : broker_id_for_part2_replica2 , ... > --replication-factor <Integer: The replication factorfor each
replication factor> partition in the topic being created.
--topic <String: topic> The topic to be create, alter or
describe. Can also accept a regular
expression except for --create option
--topics-with-overrides if set when describing topics, only
show topics that have overridden
configs
--unavailable-partitions if set when describing topics, only
show partitions whose leader is not
available
--under-replicated-partitions if set when describing topics, only
show under replicated partitions
--zookeeper <String: urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
[root@master1 bin]#
Copy the code# List existing topics
[root@master1 ~]# kafka-topics.sh --list --zookeeper localhost:2181/myKafka
This topic contains a partition called the Leader partition, which has no copy of the Follower partition.
[root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --create --topic topic_test --partitions 1 --replication-factor 1
Query partition information
[root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --list
View details about the specified topic
[root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --describe --topic topic_test
# delete the specified theme
[root@master1 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --delete --topic topic_test
Copy the codeCreate a topic that contains a partition, the Leader partition, that does not have a copy of the Follower partition.
View details about the specified topic
Multiple partitions: scale horizontally
Multiple copies: highly availableCopy the code
2.2 kafka-console-consumer.sh Is used to consume messages
# Turn on consumers
[root@node1 ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test
The second way to open consumers is to consume from the beginning, not according to the offset consumption
[root@node1 ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test --from-beginning
Copy the code2.3 kafka-console-producer.sh Is used to produce messages
# turn on producers
[root@node1 ~]# kafka-console-producer.sh --topic topic_test --broker-list localhost:9020
Copy the code2.4 Specific Operations
Unlock consumers and producers, producing and consuming messages.
