Kafka is a distributed messaging engine system that provides a complete set of message publishing and subscription solutions.

In Kafka, the publish-subscribe object is a Topic, which can be created for each business, each application, and even each type of data.

1 Producer & Consumer

A client application that publishes messages to a topic is known as a Producer. Producer programs usually send messages continuously to one or more topics. A client application that subscribs to messages on these topics is known as a Consumer. Like producers, consumers can subscribe to messages on multiple topics simultaneously. We refer to producers and consumers collectively as Clients. You can run multiple producer and consumer instances at the same time, which continuously produce and consume messages to multiple topics in the Kafka cluster.

2 Broker

The server side of Kafka consists of service processes called brokers. A Kafka cluster consists of brokers that receive and process requests from clients and persist messages.

Although multiple Broker processes can run on the same machine, it is more common to spread brokers across different machines so that if one machine fails in a cluster, brokers on other machines can still serve even if all Broker processes running on it die. This is one of the ways that Kafka is highly available

3 Partitioning – The Scalability savior

Each partition is an ordered, immutable sequence of records. A record in a partition is assigned a self-growing ID-offset

While the replica mechanism ensures that data is persisted or messages are not lost, it does not address scalability issues

In the case of replicas, there are now leader and follower replicas, but what if the leader replicas accumulate too much data to fit into a single Broker machine? A natural idea – could you split the data into multiple pieces and store them in different brokers? That’s how Kafka is designed!

This mechanism is Partitioning. Other distributed systems, you might hear the term slice, zone, etc., for example

  • Sharding in MongoDB and Elasticsearch
  • The Region in HBase

Actually, it’s the same principle. Partitioning is the most standard name.

Kafka’s partitioning mechanism involves partitioning each topic into multiple partitions, each of which is an ordered set of message logs. Each message produced by the producer is sent to only one partition, that is, to the subject of a double partition, either in partition 0 or in partition 1 (Kafka’s partition number starts at 0).

How does the copy I just mentioned relate to the partition here? In effect, replicas are defined at the partition level. Each partition can be configured with multiple replicas, including only one leader replica and n-1 follower replica. The producer writes messages to the partition, and the position of each message in the partition is represented by a number called Offset. The partition displacement always starts at 0. Suppose a producer writes 10 messages to an empty partition, then the displacement of the 10 messages is 0, 1, 2… , 9.

4 Topic

A data topic is an abstraction used in Kafka to represent a data flow. Available topics categorize data when it is published, and also serve as topics when it is subscribed to. A Topic can have multiple producers and consumers simultaneously.

A Topic can be understood as a queue, with producers and consumers oriented towards a Topic.

5 3 Replication – Duplicate

Another way to achieve high availability.

To ensure distributed reliability, kafka0.8 starts backing up data for each partition (on different brokers) in case one of the brokers fails and the partition data becomes unavailable

  • Each partition is replicated to another server as replication, which is a redundant backup strategy

  • Multiple replicas of the same partition are not allowed to be on the same broker
  • For each partition in replication, there is a leader with zero or more followers
    • The leader handles all read and write requests for this partition
    • Followers passively copy data
  • If the leader is down, a new leader is elected from the followers

The number of replicas is configurable. Replicas hold the same data but are different

3.1 Classification of copies

Kafka defines two types of replicas

  • Leader Replica

Provides services externally and interacts with client programs

  • Follower Replica

Passively follow the leader copy without any interaction with the outside world

Follower replicas are available on many other systems, such as MySQL’s slave library that handles reads, but follower replicas are not available in Kafka.

It is no longer recommended to use master-slave to refer to this kind of master-follower relationship. After all, Slave means Slave. In the United States, where racial discrimination is strictly prohibited, this expression is somewhat politically incorrect, so most systems are changed to leader-follower.

3.2 How copies work

  • Producers always write messages to leader replicas
  • The consumer always reads the message from the leader copy

As for the follower replica, it only does one thing: it sends a request to the leader replica asking the leader to send it news of the latest production so that it can stay in sync with the leader.

6 Record

Each record contains key, value, and TIMESTAMP information

Partition ID +offset can determine the location of the data partition is in order!

Kafka’s three-tier messaging architecture

  • The first layer is the theme layer, where each theme can be configured with M partitions, and each partition can be configured with N replicas
  • The second layer is the partition layer. Only one of the N copies of each partition can act as the leader and provide services externally. The other N-1 replicas are follower replicas for data redundancy purposes
  • The third layer is the message layer, which contains several messages in the partition, and the displacement of each message starts from 0 and increases successively

Finally, the client program can only interact with the leader copy of the partition.

Leaving the message hierarchy behind, let’s talk about how Kafka Broker persists data. In general, Kafka uses a message Log to store data. A Log is a physical file on disk that can Append only messages. Since only appending writes are available, slow random I/O operations are avoided in favor of sequential I/O writes that perform better. This is also an important way to achieve Kafka’s high throughput characteristics. However, if you keep writing messages to a log, eventually you will run out of disk space, so Kafka must periodically delete messages to reclaim the disk. How do I delete it? In simple terms, Log Segment mechanism. Underneath Kafka, a log is further subdivided into multiple log segments. Messages are appended to the current log segment. When a log segment is full, Kafka automatically splits a new log segment and stores the old one. Kafka also has a scheduled task in the background that periodically checks to see if old log segments can be deleted in order to reclaim disk space.

Let’s talk about consumers again. Peer to Peer (P2P) model and publish and subscribe model. Peer-to-peer means that the same message can only be consumed by one downstream consumer, and no other consumer can touch it. One way to implement this P2P model in Kafka is to introduce Consumer groups. The so-called consumer group refers to multiple consumer instances that together form a group to consume a set of topics. Each partition in this set of topics is consumed by only one consumer instance within the group, and no other consumer instance can consume it. Why introduce consumer groups? The main purpose is to improve throughput on the consumer side. Multiple consumer instances consume simultaneously, accelerating throughput (TPS) across the consumer side. I’ll cover the consumer group mechanism in more detail later in the column, so for now you just need to understand what a consumer group does. In addition, the Consumer Instance can be a process running the Consumer application or a thread, both of which are called a Consumer Instance.

Not only do all the consumer instances in the consumer group “split” the data for the subscription topic, but even cooler, they can help each other. If an instance in the group fails, Kafka automatically detects it and transfers the partition that Failed instance was responsible for to another living consumer. This process is known in Kafka as Rebalance. Well, it’s both famous and notorious for the consumer problems caused by rebalancing. In fact, many of the current rebalanced bugs are beyond the reach of the community.

Each Consumer in the process of consuming a message must have a field that records where it is currently consuming in the partition. This field is the Consumer Offset. Notice that this is not at all the same as displacement. The above “displacement” represents the position of the message within the partition. It is immutable, that is, once the message is successfully written to a partition, its displacement value is fixed. The consumer shift, on the other hand, can change over time because it is, after all, an indicator of the pace of consumer spending. In addition, each consumer has its own consumer displacement, so it is important to distinguish the difference between these two kinds of displacement. I personally refer to the message shift in the partition as the partition shift and the consumer shift.

8 summarizes

Message: Record. Kafka is a message engine. Messages are the main objects that Kafka processes. Topic: Topics are logical containers that carry messages and are used to distinguish specific businesses in practice. Partition: Partition. An ordered invariable sequence of messages. There can be multiple partitions under each topic. Message displacement: Offset. Represents the location information of each message in the partition, and is a monotonically increasing and constant value. Replica: Replica. In Kafka, the same message can be copied to multiple places to provide data redundancy. These places are called replicas. Replicas are also divided into leader replicas and follower replicas, each with a different role division. Replicas are at the partition level, that is, each partition can be configured with multiple replicas for high availability. Producer: Producer. An application that publishes new messages to a topic. Consumer: Consumer. An application that subscribes to new messages from topics. Consumer Offset. Characterizing consumer consumption progress, each consumer has its own consumer displacement. Consumer Group: Consumer Group. A group of multiple consumer instances consuming multiple partitions simultaneously to achieve high throughput. Rebalance. The process by which other consumer instances automatically reassign subscribed topic partitions when a consumer instance in a consumer group fails. Rebalance is an important way to achieve high availability on the Kafka consumer side.

Why doesn’t Kafka allow follower copies to be read like MySQL does?

There are several reasons for not reading from followers. 1. Kafka’s partition has been configured to read from multiple brokers, thus balancing the load. Kafka is a message queue, so consumption needs to be shifted. The database is an entity data. There is no such concept. 3. For producers, kafka can be configured to control whether or not to wait for messages to be acknowledged by followers. If kafka reads from above, it also needs all the followers to confirm the message before replying to the producer

First of all, let’s be clear: there is no absolute superiority or inferiority of master-slave separation, it is only an architectural design, each has its own applicable scenarios. Second, as you mentioned, both Redis and MySQL support master-slave read-write separation. I personally think this has something to do with their usage scenarios. For load types with lots of reads and relatively few writes, using read/write separation is a good solution – we can add a lot of followers to improve read performance. Kafka, on the other hand, is a message engine that provides read services to the outside world rather than as a data store. It usually involves frequent production and consumption of messages. This is not typical of read and write scenarios, so read/write separation is not suitable for this scenario. Third, the Kafka replica mechanism uses asynchronous message pull, so there is inconsistency between the leader and follower. If read/write separation is to be adopted, it is necessary to deal with the consistency problems introduced by copy lag, such as how to implement read-your-writes, how to ensure monotonic reads, and how to deal with the reversal of the causal order of messages. On the other hand, if you don’t use read/write separation, all client read/write requests are processed only by the Leader, and you don’t have these problems. Of course, the global message ordering problem still exists in Kafka. A common solution is to use a single partition. But Kafka does not currently offer one. Finally, the community is considering introducing moderate read-write separation, such as allowing certain follower replicas (mainly for geographical proximity) to be read from outside. Of course, this plan is still under discussion

reference

  • Apache Kafka in Action
  • Why does Kafka not support master-slave separation?
  • Kafka official documentation