The most detailed principle summary of Kafka ever

2.1 Some of the terms Kafka are explained as follows:

• Broker: A Kafka node is a Kafka node. Multiple brokers can form a Kafka cluster.
• Topic: A category of messages, such as page View logs, click logs, etc., can exist in the form of a Topic. The Kafka cluster can distribute multiple topics at the same time.
• Partition: A physical grouping of topics. A topic can be divided into multiple partitions, each of which is an ordered queue
• Segment: Partition physically consists of multiple segments, each containing message information
• Producer: Messages are sent to topics
• Consumer: Subscribes to a topic that consumes messages. A Consumer acts as a thread to consume
• Consumer Group: A Consumer Group contains multiple consumers, which are pre-configured in the configuration file. Each consumer thread can form a consumer group. Each message in a partition can be consumed by only one consumer in the consumer group. If a message can be consumed by multiple consumers (consumer threads), the consumers must be in different groups. Kafka does not allow messages in a partition to be processed by two or more consumer threads, even from different consumer groups. It can’t handle messages with multiple BETS as consumers, as AMQ does. This is because when multiple BETS consume data in a Queue, they want to ensure that no more than one thread can hold the same message. Therefore, row-level pessimism is required, which results in consume performance degradation and insufficient throughput. Kafka allows only one consumer thread to access a partition to ensure throughput. If the efficiency is not high, we can add the number of partitions to horizontally expand, and then add new consumer threads to consume. In this way, there is no lock competition, giving full play to the horizontal scalability and high throughput. This has led to the concept of distributed consumption.
• 2.2 Some principles and concepts of Kafka
  
  

• 2.3 Kafka Topology
  
  
  
  

The analysis process is divided into the following four steps:

• Partition stores distribution in topic
• File storage in Partiton (partition is a directory (folder) on Linux server)
• Partiton segment file storage structure
• How do I find messages in a partition by offset

Through the above 4 process detailed analysis, we can clearly recognize the mystery of Kafka file storage mechanism.

Assume that the Kafka cluster has only one broker, XXX /message-folder as the root directory for storing data files. Properties file configuration in Kafka Broker (parameter log.dirs= XXX /message-folder), For example, create two topics named Report_push and launch_info with the number of partitions=4

The storage path and directory rules are as follows:

xxx/message-folder

We can set the expiration time of messages so that only expired data is automatically cleared to free disk space.















A partition can only be consumed by one consumer (a consumer can consume multiple partitions simultaneously)

• Each partion is equivalent to a huge file divided equally among multiple equally-sized segment data files. However, the number of messages in each segment file may not be the same. This feature allows old segment files to be deleted quickly.
• Each partiton only needs to support sequential reads and writes. The segment file life cycle is determined by server configuration parameters.

In this way, unnecessary files can be deleted quickly and disk utilization can be improved.

• Segment file: Consists of two main files: index file and data file. These two files correspond to each other in pairs. The suffix “.index” and “.log “denote the segment index file and data file respectively.
• The segment file is named after the maximum number of messages offset from the previous global segment. Values are up to 64 bits long, 19 digits long, and no digits are padded with zeros.

Create a topicXXX that contains 1 partition Set the size of each segment to 500MB and start the producer to write a large amount of data to Kafka Broker. Figure 2 illustrates these two rules:

Taking the pair of segment file in Figure 2 as an example, the physical structure of index< — >data file in the segment is as follows:

In Figure 3 above, the index file stores a lot of metadata, and the data file stores a lot of messages. The metadata in the index file points to the physical offset address of Message in the corresponding data file. Taking metadata 3,497 in the index file as an example, the third message is represented in the data file (368772 message is represented in the global partiton) and the physical offset address of the message is 497.

Figure 3 shows that the Segment data file is composed of many messages. The physical structure of the message is as follows:

For example, to read the message whose offset=368776, perform the following two steps.

• Segment file

  Figure 2 above, for example, of which 00000000000000000000. The index said at the beginning of the file, start offset (offset) to 0. The second file 00000000000000368769. The index of the volume start offset of 368770 = 368769 + 1. Similarly, the third file 00000000000000737337. The index of initial offset of 737338 = 737337 + 1, so on other subsequent files, named after the start offset and sort these documents, as long as according to the offset binary search * * * * file list, You can quickly locate specific files.

  When the offset = 368776 to 00000000000000368769. The index | log

• Segment file: offset=368776 Positioning in order to 00000000000000368769. The index of metadata and physical location 00000000000000368769. Physical offset of the log, Then run 00000000000000368769. Log to offset=368776.

Kafka efficient file storage design features

• Kafka divides a parition large file into several small files in a topic. By using these small files, it is easy to periodically clean or delete consumed files and reduce disk usage.
• Indexing information allows you to quickly locate messages and determine the maximum size of a response.
• Index metadata can be mapped to memory to avoid I/O operations on segment files.
• Sparse storage of index files can greatly reduce the space occupied by index file metadata.

1. Kafka cluster Partition Replication automatically allocates and analyzes data by default

Create a topic with 4 partitions and 2 Replication in a Kafka cluster. The flow of data Producer is shown in the figure:

(1)

(2) When two nodes are added to the cluster and the number of partitions increases to six, the distribution is as follows:

The copy allocation logic rules are as follows:

• In a Kafka cluster, each Broker has equal opportunities to be the Leader of partitions.
• In Broker Partition in the figure above, the arrow points to a copy, for example, Partition-0: Parition -0 in Broker1 is the Leader, Partition-0 in Broker2 is the copy.
• In this pattern, each Broker(in BrokerId order) allocates the primary Partition in turn, and the next Broker is the copy, so the cycle allocates iteratively, with multiple copies following this rule.

• Sort all N brokers and I partitions to be allocated.
• Allocate the ith Partition to the (I mod n) Broker.
• Allocate the JTH copy of the ith Partition to the ((I + j) mod n) Broker.

4.2 Message Delivery and Lifecycle:
(This is the biggest difference from AMQ)



4.3 compressed









4.4 Message Reliability
























4.5 Backup Mechanism











4.6 Kafka efficiency related design



4.6.1 Message persistence
4.6.2 Constant time performance guarantee

1. Producer client applications generate messages:

   1. The client connection object wraps the message in a request and sends it to the server
   2. The server entry also has a connection object that receives the request and stores the message as a file
   3. The server returns the response to the producer client

2. Consumer client application consumes messages:

   1. The client connection object also wraps the consumption information in a request and sends it to the server
   2. The server fetches messages from the file storage system
   3. The server returns the response to the consumer client
   4. The client restores the response result to a message and begins processing the message

5.1  Producers






The Producer client controls which partitions the messages are pushed to.


















5.2  Consumers



In Kafka, the offset value for which message is currently read is maintained by the consumer
For example, a consumer can re-consume data that has been consumed by resetting the offset value. Kafka keeps data for a configurable period of time, whether it is consumed or not, and only deletes it when it expires. (This is different from AMQ, where AMQ messages are persisted in mysql and deleted when consumed.)






































5.3  

2. Use the page cache instead of creating a separate cache. Publishers publish sequentially, and subscribers are usually a little behind publishers. Using Linux’s Page cache directly results in better performance, reducing cache management and garbage collection overhead.

3. Maintain consumption relationship and consumption information of each partition.

6.2 Zookeeper Details:

1. After each broker is started, a temporary Broker Registry is registered on ZooKeeper, containing the BROKER’s IP address, port number, and stored topics and partitions information.

2. After each consumer starts, it registers a temporary Consumer Registry on ZooKeeper that contains the consumer group to which the consumer belongs and topics to which the consumer subscribes.

3. Each consumer group is associated with a temporary owner Registry and a persistent offset Registry. There is an Owner Registry for each partition that is subscribed to, containing the consumer ID that subscribed to the partition; It also contains an offset Registry containing the last subscribed offset.