A sow's nest does not fold a sow's ear. Abandon and not give up, stone can be engraved. -- Warring States: Xun Zi's "Persuasion"Copy the code

In this paper, from the author’s CSDN (blog.csdn.net/zhanshenzhi…). Original, since the reprint function has been removed, so we have to re-upload, the following pictures still maintain the original published watermark (such as CSDN watermark). (The original will be shared and published on multiple platforms in a newly created state)

preface

The second issue of Kafka series [(2) message queue -Kafka core concept (know each other)] has talked about the concept of partition, here to share the next partition rebalancing, this in Kafka core is a ruthless move that

What is partition rebalancing

A Kafka partition can be consumed by consumers from different consumer groups, and if a consumer is down or crashes or a new consumer joins, then the partition coordinator redistributes the partition, transferring ownership of the partition from one consumer to another. This is partition rebalancing. If a new partition is added, the partition rebalancing is triggered.

Zone rebalancing policy

RangeAssignor

This single-topic topic, which only allows consumers to subscribe to topic topics, is also Kafka’s default allocation strategy. Partition. The assignment. The strategy of corresponding strategy is org. Apache. Kafka. Clients. Consumer. RangeAssignor

  1. All partitions of the same topic are sorted according to the sequence number and stored in TP (Topic Partition for short).
  2. All consumers in the same consumer group are sorted according to the data dictionary (letters), stored in CG (Consumer Group for short).
  3. Divide the number of TP partitions by the number of CG consumers, if not all, the excess will be put into the first consumer consumptionAs can be seen from the above figure, if the more topics, the more C1 consumers, this is obviously a disadvantage of RangeAssignor volume rate.

RoundRobinAssignor

This only applies to all topics and all partitions, but two conditions must be met:

  • At instantiation time, each consumer gives each topic the same number of streams, stream.nums
  • Each consumer is assigned the same topic
Map<String, Integer> topicCountMap = newHashMap<String, Integer>(); Topiccountmap. put(Topic, number of streams of type Integer); Map<String, List<KafkaStream<byte[].byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
Copy the code

Partition. The assignment. Corresponding strategy is the strategy ` ` org. Apache. Kafka. Clients. Consumer. RoundRobinAssignor

  1. Hash all topic+partition to string, then hash sort
  2. All consumers are sorted by data dictionary (letters)
  3. Partitions are then assigned to consumers in a polling manner

StickAssignor

This allocation policy is available from kafka0.11.x. Stick is a stick.

  1. The partition allocation should be balanced as far as possible, and the difference in the number of subject partitions is at most 1
  2. Try to keep the partition allocation the same as the last partition
  3. If the first point and the second point conflict, the first point takes precedence

Partition rebalancing process

  1. The member sends a heartbeat check to the coordinator to determine survival
  2. Coordinator ACK
  3. Member request to Join consumer Group (joinGroup Request)
  4. Coordinator ACK joins successfully or not
  5. The member requests a partition rebalancing SyncGroup
  6. Coordinator ACK reblance is successful or not (SyncGroup response)

After the order

Kafka Broker message queue Kafka Broker message queue