Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

Kafka is a distributed streaming platform, high throughput messaging system. Very classic for peak clipping, decoupling, asynchrony. For more explanation, see Getting Started with Message queues, starting with Kafka

In this article, the terms Kafka Leader election, consumer group, offset and their functions are simply illustrated with diagrams.

Leader election

  • Leader Election Process

Election of Kakfa Broker Leader: The Kakfa Broker cluster is managed by ZooKeeper. All Kafka Broker nodes go to ZooKeeper to register a temporary node, because only one Kafka Broker will register successfully and all others will fail. So the Kafka Broker that successfully registers temporary nodes on ZooKeeper becomes the Kafka Broker Controller, and the other Kafka brokers are called Kafka Broker follower. (This process is called a Controller registering a Watch in ZooKeeper).

thiscontrollerThey’ll be listening for the othersKafka BrokerAll of the information if thiskafka broker controllerIt’s down. It’s downzookeeperThe temporary node above will disappear.

  • The broker breaks down and the replica becomes the leader

Such as: When a broker fails, the Kafka Broker Controller reads the status of all partitions on the ZooKeeper. One replica in the ISR list is selected as the partition leader. (If all replicas in the ISR list are faulty, one surviving replica is selected as the leader.

If all replicas of the partition are down, set the new leader to -1 and wait for recovery. Wait for any replica in the ISR to “live” and choose it as the leader. Or choose the first replica (not necessarily the Leader in the ISR) that is “alive”. The Kafka Controller also notifies ZooKeeper of the breakdown of this broker, and ZooKeeper notifies other Kafka Brokers.

Consumer group

  • Consumer rules

A group in Kafka consists of multiple consumers. Each message in a partition under a topic can be consumed by only one consumer in the consumer group. So these consumers have to be in different groups. Kafka does not allow messages in a partition to be processed by two or more consumer threads in the same consumer group. Unless a new consumer group is started, the code often sets different group names for different services. Therefore, if you want to consume on a topic at the same time, you can start multiple consumer groups.

Note, however, that multiple consumers must read messages from the partition in sequence. By default, a newly started consumer reads messages from the very top of the partition queue. Kafka allows only one consumer thread in the same consumer group to access a partition to ensure throughput.

If the efficiency is not high, we can add the number of partitions to horizontally expand, and then add new consumer threads to consume. If multiple businesses need the data of this topic, it is better to have multiple consumer groups. They all read messages sequentially, and the values of offset do not affect each other. In this way, there is no lock competition, giving full play to the horizontal scalability and high throughput. This has led to the concept of distributed consumption.

  • Optimal consumption allocation

When a consumer group is started to consume a topic, no matter how many partitions there are in the topic or how many consumer threads are configured in the consumer group, All consumer threads in the consumer group must consume all partitions. Even if there is only one consumer thread in the consumer group, the consumer thread will consume all partitions. Therefore, the optimal design is that the number of consumer threads in the consumer group is equal to the number of partitions, thus achieving the highest efficiency.

Under a consumer group, no matter how many consumers there are, the consumer group must consume all partitions under this topic. When the number of consumers in the consumer group is smaller than the number of partitions in the topic, a conusmer thread consumes multiple partitions. In short, all partitions in the topic will be consumed.

If the number of consumers in a consumer group is equal to the number of partitions in this topic, the efficiency will be the highest, and each partition has one consumer thread to consume. When the number of consumers in the consumer group is greater than the number of partitions in the topic, one consumer thread will be free. Therefore, when you set a consumer group, you just need to specify how many consumers there are in it. You don’t need to specify the corresponding consumer partition number. Consumer rebalance automatically.

Offset update policy

  • The renewal of the offset

The consumer reads the messages in order O(1) when consuming the partition. So you have to maintain offset information about where you last read. The offset is generally stored in ZooKeeper. The low level API offset is maintained by the zooKeeper. The default value for consumer is “commmit”, and the default value for autoCOMMIT is “true”. The default value for autocommit is “commit”, and the default value for autocommit is “offset+1”. It can also be configured to finish the message processing and then commit, in which case the consumer will be slow to respond until the processing is complete.

  • Consume submit offset process

Kafka Producer sends messages without maintaining the message’s offset, because the offset is an increment ID and the producer sends the message. Kafka is different from ActiveMQ. ActiveMQ is mainly used to process business logic, while Kafka is mainly used to process logs. Therefore, Kafka producer generally sends messages in large quantities and sends a large number of messages to this topic at a time. Add the load balance to a partition and insert the offset as the id of the partition.

However, the consumer side needs to maintain information about which offset the partition is currently consuming, and Kakfa processes messages without locking them. So if the message fails to be processed, the commit offset+1 has not yet been committed, and the consumer Thread will re-consume the message when it restarts. However, as a real-time processing system with high throughput and high concurrency, it can be tolerated that the system will be processed at least once. If this is not tolerated, you have to use your own program to maintain the offset information, so you can control when you want to commit offset+1.

reference

  • The Definitive Guide to Kafka
  • kafka.apache.org/
  • Summary of Kafka’s most detailed principles ever