Today we are going to practice Kafka security configuration. Through this tutorial, we will configure a Kafka that requires user name and password access.

This article uses SASL/SCRAM authentication mechanism, which is introduced in Kafka 0.10.2.

Test environment: MacOs Mojave, KafKA_2.12-2.4.0

The first step is to create a user

Description:

Admin is used for communication between brokers, Writer is used for producers, and Reader is used for consumers.

bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=admin]' --entity-type users --entity-name admin


bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=writer]' --entity-type users --entity-name writer


bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=reader]' --entity-type users --entity-name reader
Copy the code

Step 2, create the JAAS file

Description:

This file is used to start the broker. The file can be named kafka-broker.jaas and needs to be specified when the broker starts.


KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin";
};
Copy the code

Step 3, configure the server.properties file

Description:

Since authentication is required, the server.properties file is configured, as shown below.

sasl.enabled.mechanisms=SCRAM-SHA-256

sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256

security.inter.broker.protocol=SASL_PLAINTEXT

listeners=SASL_PLAINTEXT://localhost:9092
Copy the code

Step 4: Start the broker

Description:

When starting the broker, specify the JAAS file, along with the modified server.properties file.

KAFKA_OPTS=-Djava.security.auth.login.config=<yourPath>/kafka-broker.jaas bin/kafka-server-start.sh config/server.properties
Copy the code

Step 5: Send a message

Description:

After authentication is enabled, our producer needs to provide the user name and password. Therefore, we need to configure a file, which is specified at startup. The content can be named as producer.conf

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="writer" password="writer";
Copy the code
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test  --producer.config <yourPath>/kafka_212.-2.4. 0/producer.conf
Copy the code

Step 6, receive the message

Description:

A consumer and a producer need to specify a configuration file during startup. The content of the configuration file is similar to that of a producer except that the user name and password are different. It can be named consumer.conf


security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="reader" password="reader";
Copy the code
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config <yourPath>/kafka_212.-2.4. 0/consumer.conf
Copy the code

conclusion

We configured a simple Kafka authentication using SASL/SCRAM authentication mechanism. Users are created, passwords are set, server configuration files are modified, and user names and passwords are provided when starting producers and consumers.

This completes the configuration of a simple secure access Kafka. The next part will continue to explain how to configure authorization.

Reference documentation and recommended reading

Kafka core technology and combat

Apache Kafka