This is the 16th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

Kafka consumers

Kafka consumer related concepts

  • Consumers and consumer groups
    • A partition can only be consumed by one consumer within a group. Multiple consumers cannot consume a partition together.
  • Consumer groups and partitions are rebalanced
    • When a new consumer joins a consumer group, it will consume one or more partitions, which were previously responsible for by other consumers and need to be transferred to share.
    • When a consumer leaves the consumer group, the partition it consumes is assigned to another partition.

Consumers add and subtract, Kafka scales horizontally, but the more consumers the better, if consumers > partition number. That will lead to some consumers can not be divided into regions. (Therefore, the number of consumers must be smaller than the number of partitions)

For general messaging middleware, there are generally two modes of message delivery

  • Point-to-point (P2P) mode
    • The point-to-point pattern is queue-based
  • Publish/subscribe (Pub/Sub) mode
    • The publish subscribe pattern defines how to publish and subscribe messages to a content node. The publish/subscribe pattern is used for one-to-many broadcast of messages.

Kafka supports both message delivery modes.

  • A consumer group: equivalent to point-to-point
  • Different consumer groups: equivalent to publish subscriptions

Consumer code

  1. Configure the properties to create the consumer object
  2. Subscription topics (you can subscribe to multiple topics) can also specify specific partitions.
  3. Loop over poll to pull data
Properties props = new Properties();
props.put("bootstrap.servers"."node1:9092,node2:9092");
props.put("group.id"."CountryCounter");
props.put("key.deserializer"."org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer"."org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String,String>(props);
// loop pull data
try {
   while (true) {
       ConsumerRecords<String, String> records = consumer.poll(100);
       for (ConsumerRecord<String, String> record : records)
       {
           System.out.println(JSONUtil.toJsonString(record))
       }
   }
} finally {
      consumer.close();
}
Copy the code

Consumption in Kafka is based on a pull pattern. Messages are generally consumed in two modes: push mode and pull mode.

  • In push mode, the server actively pushes the message to the consumer
  • In pull mode, the consumer initiates a request to the server to pull the message

An important concept for consumers -offset: represents a message shift that records the position of a consumer within the consumer.

This location needs to be persisted!

  • In the old consumer client: the consumption shift is stored in ZooKeeper
  • New consumer client: stored in the theme consumer_offsets inside Kafka

The consumer consumes one message at a time and needs to commit this shift. The submitted offset value is the current offset+1.