I. File structure

instructions

  • Messages in Kafka are classified by topics. Producers send messages to Kafka brokers through topics, and consumers read data through topics.
  • On the physical level, topics can be divided into partitions. A topic can be divided into several partitions.
  • A partition can also be subdivided into segments. A partition physically consists of multiple segments.

Partition

2.1 Viewing the Partition Structure

You can view the Kafka data storage directory in the config/server.properties configuration file

[root@hadoopserver config]# pwd
/usr/local12-2.8.0 / config/kafka_2 [root @ hadoopserver config]# vim server.properties
log.dirs=/usr/local/ kafka_2. 12-2.8.0 / dataCopy the code

Access the partition directory to view partition files

[root@hadoopserver data]# PWD /usr/local/kafka_2.12-2.8.0/data [root@hadoopserver data]# ls-laCopy the code

instructions

The naming rule for partiton is topic name + ordered number. The number of partitions for the first partiton starts from 0, and the maximum number of partitions is reduced by 1.

2.2 Functions of Partitions

For performance reasons, if messages for each topic are not partitioned on only one broker, and all consumers consume messages from that broker, the single-node broker becomes a performance bottleneck. If there are partitions, messages sent by producers are stored on different partitions of each broker, so that consumers can read messages from different partitions of different brokers in parallel, achieving horizontal scaling.

2.3 Data Files in Partitions

View the demo-0 partition directory structure

[root@hadoopserver data]# tree demo-0/Demo - 0 / ├ ─ ─ 00000000000000000001. The index ├ ─ ─ 00000000000000000001. The log ├ ─ ─ 00000000000000000001. Timeindex ├ ─ ─ 00000000000000001018. The index ├ ─ ─ 00000000000000001018. The log ├ ─ ─ 00000000000000001018. Timeindex ├ ─ ─ 00000000000000002042. The index ├ ─ ─ 00000000000000002042. The log ├ ─ ─ 00000000000000002042. TimeindexCopy the code

Each partition contains a number of files, which are conceptually called segments, meaning that each partition consists of multiple segments.

Segment storage

3.1 Segment Composition and naming

The segment consists of the following sections:

  • Index: indicates the index file
  • Log: data file
  • Timeindex: timeindex file

The name of each subsequent segment is the offset value of the last message in the preceding segment. Values are up to 64 bits long, 20 digits long, and no digits are padded with zeros.

Advantages of naming rules: convenient binary search, fast location to the data file

3.2 the Segment effect

If no segment is introduced, a partition corresponds to only one file (log). This file grows as messages are sent. Kafka messages are not updated sequentially, so delete only the first part of the file. Delete the entire file to ensure that each segment is written in sequence.

3.3 Physical Structure of index and Log Files

instructions

The left part of the figure above is the index file, which stores key-values in pairs, and the right part is the data file corresponding to the index file.

  • Index file key: the number of the message in the data file (the corresponding log file), such as “1,3,6,8…” Represents the first message, the third message, the sixth message, and the eighth message in the log file respectively.
  • Index file value: indicates the physical offset address (location) of the message

advantages

The index file does not index every message in the data file. Instead, sparse storage is used to build an index every certain byte of data. The index file is saved from taking up too much space and can be kept in memory.

disadvantages

Messages that are not indexed cannot be located in the data file at once, requiring a sequential scan.

3.4 Message Physical Structure

As can be seen from the Segment data file in the right part of the figure above, the log data file does not store data directly, but consists of a number of messages. Messages contain the actual message data. The physical structure of messages will be explained in detail below.

The keyword explain
8 byte offset Each message within parition has an ordered ID number, called offset, which uniquely determines the position of each message within parition. That is, offset represents the number of messages in the partiion
4 byte message size Message size
4 byte CRC32 Verify message with CRC32
1 byte “magic” Indicates the Kafka protocol version of this release
1 byte “attributes” Represents a standalone version, or identifies the compression type, or the encoding type
4 byte key length Indicates the length of the key. If the key is -1, the K byte key field is left blank
K byte key optional
value bytes payload Represents actual message data

How do consumers find message through offset

If we want to read the message with offset=1066, we need to do the following two steps:

Step 1: Find segment file

00000000000000000000. The index said at the beginning of the file, start offset (offset) to 0. The second file 00000000000000001018. The index of the volume start offset of 1019 = 1018 + 1. Similarly, the third file 00000000000000002042. The index of initial offset of 2043 = 2042 + 1, so on other subsequent files, named after the start offset and sort these documents, as long as according to the offset binary search file list, can quickly locate the specific file. When the offset = 1066 to 00000000000000001018. The index | log.

Step 2: Find the message by segment file

By the first step in the localization to segment the file, when offset = 1066, locating in turn to 00000000000000001018. The index of metadata physical location and 00000000000000001018. Physical offset of the log, At this point, we can only get the physical offset address of 1065, and then search through 00000000000000368769.log until offset=1066. Each message has a fixed format and it’s easy to tell if it’s the next message.

Attached is a reference article link:

Segmentfault.com/a/119000002…

www.cnblogs.com/yitianyouyi…