demand

  1. Build a 3-node Kafka cluster to test functionality and performance
  2. Implement Spring Kafka under kafka cluster operation

Build a compact Kafka cluster

[Wurstmeister/Kafka] [Wurstmeister/Kafka] [Wurstmeister/Kafka]

  1. Create docker-comemess. yml file with the following contents:
version: '1'

services:
  zookeeper:
    image: zookeeper
    container_name: bitkylin-kafka-zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    environment:
      kafka_ADVERTISED_HOST_NAME: 172.17. 01.
      kafka_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
Copy the code

Docker-compose: docker-compose does not specify the kafka container name and does not expose kafka-related ports, which is convenient for docker-compose to scale the container.

  1. Cluster operations

In the docker-comemess. yml directory, you can perform the following operations:

Start a single kafka cluster
docker-compose up -d

Start a cluster of three Kafka nodes
 docker-compose up --scale kafka=3

# Destroy cluster
docker-compose stop
Copy the code

The full kafka cluster

Creating a compact cluster is easy, but not practical. Here’s how to create a usable cluster.

Wurstmeister/Kafka mirror for a long time, out of the cluster can not be accessed outside the container, which brings inconvenience for debugging, development, now replace the mirror again.

Replace the image with bitnami-docker-kafka

  1. Create a single-node Kafka cluster:
version: '1'

services:
  zookeeper:
    image: zookeeper
    container_name: bitkylin-kafka-zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: 'bitnami/kafka'
    ports:
      - '9092:9092'
    environment:
      - kafka_BROKER_ID=1
      - kafka_LISTENERS=PLAINTEXT://:9092
      - Kafka_ADVERTISED_LISTENERS = PLAINTEXT: / / 127.0.0.1:9092
      - kafka_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
Copy the code
  1. Create a 3 node Kafka cluster:
version: '2'

services:
  zookeeper:
    image: zookeeper
    container_name: bitkylin-kafka-zookeeper
    ports:
      - "2181:2181"

  kafka-1:
    image: 'bitnami/kafka'
    container_name: bitkylin-kafka-1
    ports:
      - '9081:9081'
    environment:
      - kafka_BROKER_ID=11
      - kafka_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - kafka_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - kafka_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9081
      - kafka_CFG_ADVERTISED_LISTENERS=CLIENT://:9092,EXTERNAL://localhost:9081
      - kafka_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper

  kafka-2:
    image: 'bitnami/kafka'
    container_name: bitkylin-kafka-2
    ports:
      - '9082:9082'
    environment:
      - kafka_BROKER_ID=12
      - kafka_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - kafka_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - kafka_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9082
      - kafka_CFG_ADVERTISED_LISTENERS=CLIENT://:9092,EXTERNAL://localhost:9082
      - kafka_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper

  kafka-3:
    image: 'bitnami/kafka'
    container_name: bitkylin-kafka-3
    ports:
      - '9083:9083'
    environment:
      - kafka_BROKER_ID=13
      - kafka_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - kafka_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - kafka_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9083
      - kafka_CFG_ADVERTISED_LISTENERS=CLIENT://:9092,EXTERNAL://localhost:9083
      - kafka_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper
Copy the code
  1. Start the three-node Kafka cluster
docker-compose up -d
Copy the code
  1. operation
#-- Within the container cluster -- sends messages based on node 1 --
#Enter one of the Kafka containers
docker exec -it bitkylin-kafka-1 /bin/bash

#Sends messages from node 1
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-bitkylin

#-- Outside the container cluster -- Receives messages based on node 3 --
#Go to the Kafka installation directoryCD/opt/kafka_2. 13-2.7.0 / bin
#Listen for the specified topic from node 3
 ./kafka-console-consumer.bat --bootstrap-server localhost:9083 --topic test-bitkylin --from-beginning
Copy the code

After the preceding operations, you can verify that sending and receiving messages are normal. In particular, the verification of external access kafka container is normal, this is difficult!!

Spring Boot + Spring Kafka to send and receive messages to kafka cluster example, see the following code: bitkylin-kafka

After the Spring Boot application is started, the message producer in the Bitkilin-kafka-1 container is still used to send messages. You can see that both the Spring Boot application and the external consumers connected to Bitkilin-kafka-3 consume the messages in full.

cheat

  1. Kafka must be configured with a complete proxy IP address :PORT. If you want to access the Kafka container outside the container cluster, you must use the exposed IP address :PORT. This is supposedly designed to prevent man-in-the-middle attacks.

  2. The bitnami-Docker-kafka image file is well documented and can be read.

Refer to the link

  1. Wurstmeister – Kafka Docker image usage instructions (Wurstmeister – Kafka)
  2. The docker image: wurstmeister/kafka
  3. Docker image: bitnami docker — kafka
  4. Example code: bitkilin-kafka