The interviewer is likely to ask you a puzzling question as they test your knowledge of Kafka: What about elections in Kafka? Unless you are asked about a specific election, the interviewer who asks this question probably doesn’t know anything about Kafka, and that’s the time to kill him. Of course, if you don’t have a certain amount of knowledge, that’s when you get “killed”.

Generally ask this question, then he must know one of them, such as the election of the partition leader. The so-called leader election of a zone is the process of re-electing a leader copy when the leader copy in the ISR stops running. For this question is your chance to turn the tables, because Kafka has many elections, not only the partition leader election, even if the partition leader election is specified, then also need to be divided into four cases. The details of this are unknown to most people.

Elections in Kafka can be broken down into three broad categories: controller elections, partition leader elections, and consumer-related elections, which can be broken down into seven sub-categories. Let’s go through it one by one. This article is just a brief list of the general content, as for the internal details of the logic needs to rely on readers to explore. The difference between being abused and being abused is your drive.

Controller election

There are one or more brokers in a Kafka cluster. One broker is elected as a Kafka Controller, which manages the state of all partitions and replicas in the cluster. For example, when the leader replica of a partition fails, the controller elects a new leader replica for the partition. The controller is responsible for notifying all brokers to update their metadata information when changes to the ISR collection of a partition are detected.

The Kafka Controller is elected through Zookeeper, and becomes the Kafka Controller if the broker successfully creates the EPHEMERAL node/Controller in the Kafka cluster.

It is important to note that the implementation of Kafka Controller is quite complex and involves many aspects. If you master Kafka Controller, you have mastered half of Kafka. I don’t have enough space to go through it here, but if you’re interested, check out Chapter 6 of Understanding Kafka.

Election of the zone leader

I’m not going to talk about PacificA here, I’m just going to talk about election specifics.

The election of the leader replica of a partition is implemented by the Kafka Controller. When a zone is created (creating a theme or adding a zone involves creating a zone) or when a zone goes online (for example, the original leader copy in the zone goes offline, and a new leader needs to be elected to provide services externally), the leader election action needs to be performed.

The basic idea is to find the first surviving copy in the order of the copies in the AR collection, and that copy is in the ISR collection. The AR collection of a partition is specified at allocation time, and the order of the copies within the collection remains the same as long as no redistribution occurs, while the order of the copies in the ISR collection of a partition may change. Note that the election is done in order of AR, not in order of ISR. This is a bit abstract, but interested readers can manually turn off/on brokers in a cluster to see what happens.

In other cases, the leader election of a partition may also occur. For example, the leader election action needs to be performed when a partition is reassigned. The idea is simple: find the first surviving copy from the reassigned AR list that is in the current ISR list.

For example, when the preferred Replica partition leader election occurs, the leader can be directly set as the preferred replica, and the first replica in the AR set is the preferred replica.

Kafka has a lot of XX copy, if is not very understanding, can focus on the next article in the series of Kafka science | how many kind of copy in Kafka?”

In another case, when a node is gracefully shut down (i.e., ControlledShutdown), all leader replicas on that node go offline, so the corresponding partition needs to perform leader elections. The idea here is to find the first surviving replica from the AR list that is currently in the ISR list, while ensuring that the replica is not on the node that is being closed.

Consumer related elections

For this part of the understanding, well.. If you don’t understand consumers, consumer groups, consumer coordinators, and group coordinators, then… Inside Kafka, heh heh.

A GroupCoordinator needs to elect a consumer group leader for consumers ina consumer group. The election algorithm is also simple and can be analyzed in two cases. If there is no leader in the consumer group, the first consumer to join the consumer group is the leader of the consumer group. If the leader consumer quits the consumer group at some point due to some reasons, a new leader will be elected, and the process of re-electing the leader is more “arbitrary”, the relevant code is as follows:

//scala code.
private val members = new mutable.HashMap[String, MemberMetadata]
var leaderId = members.keys.head
Copy the code

To explain these two lines of code: In a GroupCoordinator, consumer information is stored as a HashMap, where the key is the member_id of the consumer and the value is the metadata information related to the consumer. The leaderId represents the member_id of the leader consumer and takes the value of the key of the first key-value pair in the HashMap, which is basically the same as a random election. Generally speaking, the leader election process of the consumer group is very random.

Insert: recently found that the article was stolen fierce, issued a few hours the article appeared on each major portal website, all is the original target. Although there is no ability to stop, but I found that most do not carefully copy directly (there is even my following public two-dimensional code also copied to go, of course, also useful PS to P off my picture watermark chicken thief operation), so I am witty. Why don’t we put my book in the article and let them steal it to promote me. I am very ~~ skin

Is this the end of it? There’s also a zoning strategy election.

Xu you are a bit strange, but perhaps Kafka classmates used to partition. The assignment. The strategy (values for RangeAssignor RoundRobinAssignor StickyAssignor, etc.) this parameter is not strange. Each consumer can set its own partition allocation strategy. For the consumer group, it is necessary to elect a mutually “convincing” strategy from each distribution strategy submitted by each consumer to carry out the overall partition allocation. Elections for this partition allocation are not determined by the Leader consumers, but by the votes of individual consumers within the consumer group. As for the details… Hey hey.

Welcome to support the author’s book: “Illustrated Kafka Guide” and “Illustrated Kafka’s Core Principles”

Welcome to support my new books: In-depth Understanding of Kafka: Core Design and Practice Principles and RabbitMQ Field Guide. Also welcome to follow my wechat official account: Zhu Xiaosi’s blog.