Abstract
In this article, I’ll start with the physical way messages are stored in Kafka and introduce the layers of partition-journal-logging.
And then I’m going to pick up where I left off, and I’m going to talk a little bit about the consumer, the distinction between the consumer and the consumer group, and how this works.
Based on the problems that consumer consumption can cause, I will introduce the shift theme in Kafka and how consumers can submit shifts to this shift theme.
Finally, I’ll talk about the reasons for consumer Rebalance, and what’s missing.
1. log
In the last article, we mentioned the concept of “partition.”
What we meant at that time was that messages were produced and consumed in the partition dimension of the topic, not in the topic dimension.
That is, the way we understood Kafka at the time was that every parititon in a topic had a data structure called a queue, and all messages sent to that topic were allocated to one of the parititons.
This design avoids message queue performance bottlenecks on IO.
In this section, we’ll take a closer look at how Kafka stores messages.
Messages, as we understand them, are called logs in Kafka.
Within each broker, there are multiple folders named {Topic}-{Parititon}, such as test-1 and test-2.
This broker can handle messages with topic Test and partitions 1 and 2.
Note, however, that the term “parititon” is also a logical concept that corresponds to a folder within the broker. What is a physical concept? Let’s move on.
The {Topic}-{Parititon} folder contains many, many files whose names are long integers of 64 bits.
Such as:
In this figure, a partition contains multiple Log segments. Note that Log segments are logical concepts, and only specific Log files are physical concepts.
If we look at the far right part of the image, the file names are all 20-digit integers. This number is called the “base offset” of the message. For example, if our second Log Segment starts at 121, then the offset of the first message representing the Log Segment starts at 121, which means that there were 121 Log entries before that.
Note that since our offset starts at 0, there are 121 data items before the offset 121, not 120.
Then we talk about file formats, and we see that there are three types of files: *.log, *.index, and *.timeIndex.
The log file records the message, index is the offset index, and timeIndex is the timestamp index. But we won’t go into that, and the aim of this article is to learn more about the components.
As a result, when the broker receives a message from the producer, it needs to write the message in the last Log Segment. As an added benefit, messages are written in sequential IO. Because of this, the last Log Segment is called an “active Log Segment”.
2. Consumers and consumer groups
In the last article, we only mentioned the concept of “consumer.”
Also in this article, we’ll take a closer look at the “consumer” in Kafka.
In Kafka, consumers consume as a group of consumers.
Let’s make an assumption that there is no such concept as consumer group. We now have 10 consumers subscribing to the same topic. When there is new news about this topic, should these 10 consumers “steal news” to consume?
This is a waste of resources. Therefore, the consumer group can also be considered as a more reasonable allocation of resources, load balancing design.
If five consumers belong to the same consumer group, and this consumer group subscribles to a topic with 10 partitions, then each consumer in the group is responsible for processing messages for two partitions.
This ensures that when a message is sent to a topic, it will be consumed by only one consumer and will not result in duplicate consumption.
In addition, the design of the consumer group allows us to easily extend the consumption power of the system horizontally. Imagine in but we found that the accumulation of information in the system is becoming more and more consumer speed to keep up with production speed, only need to new customers, and will be the consumers into the original group, Kafka will automatically adjust the distribution of the consumer to the partition in the group, this process is called weight balance, we will be mentioned at the back.
Note, however, that the number of consumers in the group cannot exceed the number of partitions for the topic. Otherwise, the extra consumers will be idle. For example, if a topic has 10 partitions and a group has 11 consumers, the extra consumer will be free.
Kafka is designed so that a partition can only be consumed by one consumer, which has to do with displacement management, which we’ll talk about later.
In addition, Kafka allows multiple consumer groups to subscribe to the same topic, so that the same message is sent to all consumer groups that subscribe to the topic.
Note: We said that the same partition can only be consumed by the same consumer, but only if those consumers are in the same consumer group. In other words, consumers in different consumer groups can consume the same topic partition.
So we can also think of Kafka as a consumer group for both point-to-point and broadcast messaging.
3. Shift the subject
As mentioned in the previous section, consumers in the consumer group consume the information in the partition, and there is the situation of consumers joining and leaving.
So in this section we’ll talk about how Kafka is able to keep messages from being lost or re-consumed when consumers change.
It is easy to imagine that we could achieve this goal simply by recording the displacement of consumption.
Let’s go straight to the topic of displacement, regardless of the previous implementation of saving displacement in ZK.
There is a special kind of theme in Kafka called the shift theme. The theme name in Kafka is __consumer_offsets.
Because a shift theme is a theme, it fits the characteristics of a theme in Kafka. You can send messages, pull messages, and delete themes at will. However, because the topic data is designed by Kafka, messages cannot be sent randomly, or they will fail to be parsed at the broker side and cause a crash.
Then we discuss the format of the message sent to the displacement topic. Because we want to preserve displacement, it’s easy to think that this is a KV structure. So what messages should be stored in the Key?
The Key contains the subject name, partition name, and consumer group name.
There is no need to store information such as consumer IDS, that is, it is sufficient to specify which consumer group consumes how much data in which topic and which partition. Why is that? As we mentioned above, consumers are likely to change, and our purpose is to let consumers know where to continue consumption after the change. Thus, the accuracy of displacement information to the consumer group level is sufficient.
And, in Value, it’s enough to save the consumption shift.
Having said how displacement information is stored, let’s talk about the topic of displacement itself. Because the displacement topic is a topic, there will inevitably be partitions, and there will be replicas. So where should the consumer send the displacement after consuming the information?
The shift theme in Kafka is created when the first consumer is created, and there are 50 partitions by default. When the consumer submits the displacement, it will obtain the partition ID of the submitted displacement information according to the hash value of its group ID, and then find the leader node of this partition ID and submit the displacement information to the broker where the leader node resides.
4. Submission of displacement
Talking about displacement, I think you get the idea that Kafka saves displacement states, so in this section, we’ll talk about how displacement is committed.
Before we talk about the submission of displacement, it is important to be clear that having a displacement theme does not mean that messages will not be re-consumed or lost.
In addition, Kafka strictly enforces the information submitted in the displacement theme. For example, if you have already consumed 0-20 messages, if you commit a shift of 100, then the next pull message must start at 100 and 20-99 messages will be lost. For example, if you submit a displacement of 10, then 10-20 messages will be consumed repeatedly.
In Kafka, there are two ways to commit a shift, either automatically or manually.
4.1 Automatic Submission
The automatic commit of the shift occurs at the time of the POLL operation.
Before consumer POLL pulls the latest message, it will judge whether it has reached the Deadline for submitting the displacement at present. If it has reached this time, it will submit the displacement first and then pull the information.
Note that the following could happen:
At some point you commit a shift of 100, then you pull a message of 100-150, but before the next shift is committed, the consumer goes down. Maybe only 100-120 messages are consumed at this time, then after the consumer restarts, because the 120 shift is not committed, this part of the message will be consumed again.
Now imagine a situation where you pull messages from 100 to 150, and it’s time for automatic submission, and you commit this shift of 150, and the consumer is down, and you restart and you start pulling messages from 150, and the information before that will be lost.
4.2 Manual Submission
You can avoid information loss and repeated consumption caused by automatic submission by manual submission.
Manual submission is divided into synchronous submission and asynchronous submission.
Synchronous lift does not return until the message is written to the shift topic, which is safe, but can cause TPS degradation.
Asynchronous commit is the operation that triggers the commit and returns. This is fast, but can cause a commit to fail.
5. Rebalance
We mentioned one case above:
Changes in the number of members in a consumer group may cause a member of the group to realign the area of consumption for which he is responsible.
This is what we call ‘Rebalance’, or ‘Rebalance’.
To put it more technically, it is the process by which consumers within a consumer group reach a consensus on how to consume all segments of a topic.
But this process has a huge impact on Kafka throughput because it’s a bit like STW in GC, where all consumers have to do is Rebalance and consume no messages.
Here’s what can cause Rebalance:
- The number of members in the group changed
- The number of subscribed topics has changed
- The number of partitions subscribed to the topic has changed
And when the Rebalance is made, Kafka doesn’t assign the extra partitions to the same consumers. Instead, all the consumers reassign the partitions together.
When there are new consumers to join, it is not the original each consumer to divide some regions to new consumers, but all consumers to participate in the redistribution of all regions.
This allocation strategy sounds strange and inefficient, but there’s no way around it.
A new strategy called StickyAssignor does just that, but there are some bugs.
Write in the last
First of all, thank you for being here!
The first two articles on Kafka, I think, are popular science. I hope to give you a simple overview of Kafka’s features and how they work.
In later articles, I hope to be able to explain some of the principles of Kafka to you.
Since the author has just started to study Kafka, his understanding of many aspects may not be in place. So during this period, if you find any problems, or what I have explained badly, please leave a message and let me know. Thank you!
Thanks again for being here!
PS: If you have other questions, you can also find me in the public account, welcome to find me to play ~
