CommitLog

A storage file for the original message content. Like Kafka, messages are variably long and written sequentially

Generation rules:

The default 1G for each file is 1024 * 1024 * 1024. The fileName of the commitlog commitlog is fileName. The name is 20 characters in length, and zeros are added on the left side. For example, 00000000000000000000 indicates the first file, the start offset is 0, and the file size is 1 073 741 824 bytes. When the file is full and the second file name is 00000000001073741824 and the start offset is 1073741824, messages are written to the file in sequence. When the file is full, messages are written to the next file

ConsumeQueue

The content of the message is not stored in the ConsumeQueue. Instead, the message offset is stored in the CommitLog. That is, the ConsumeQueue is actually an index file for CommitLog.

A ConsumeQueue file corresponds to a queue under topic

A ConsumeQueue is a fixed-length structure of 20 bytes per record. Obviously, when a Consumer consumes a message, it reads the message twice: first it reads the ConsumeQueue for offset, and then it finds the message content of the CommitLog via offset

The role of ConsumeQueue

  1. Messages are retrieved from the ConsumeQueue via the offset saved by the broker (the subscript of the ConsumerQueue saved in the offsetTable. Offset JSON file), and are quickly located to the commitLog message location
  2. Tag filtering is also done by traversing the ConsumeQueue (comparing hash tags to consumer tags).
  3. ConsumeQueue can also be stored in the operating system’s PageCache for caching to improve retrieval performance

Here is the ConsumeQueue THAT I parsed

public static void main(String[] args) throws IOException { decodeCQ(new File("D:\\00000000000000000000")); } static void decodeCQ(File consumeQueue) throws IOException { FileInputStream fis = new FileInputStream(consumeQueue); DataInputStream dis = new DataInputStream(fis); long preTag = 0; long count = 1; while (true) { long offset = dis.readLong(); int size = dis.readInt(); long tag = dis.readLong(); if (size == 0) { break; } preTag = tag; System.out.printf(" %d %d %d\n", tag, size, offset); } fis.close(); }Copy the code
hash(tag)|size|offset(commitLog) 3552231 191 180081
 3552231 191 180654
 3552231 191 180845
 3552231 191 182182
 3552231 192 182565
 121074 201 182757
 3552231 245 190411
 3552231 245 190656
 3552231 245 190901
 3552231 245 191146
 3552231 245 191391
 3552231 245 191636
 3552231 245 191881
 99 197 219910
 99 197 220107
 99 197 220304
Copy the code

OffsetTable. Offset (saved in JSON)

This is not the same thing as the commitLog offset, which is the subscript/line number consumed by the File ConsumeQueue, You can locate the message body directly by going to the ConsumeQueue and finding the commitlogOffset. The offset is the core of the message consumption schedule

Offset persistent

Type (parent is OffsetStore) :

  • Local file type

DefaultMQPushConsumer BROADCASTING mode, the consumers do not interfere with each other, using LoclaFileOffsetStore, the Offset stored in the Consumer local

  • Broker Type of generation storage

The CLUSTERING mode for DefaultMQPushConsumer, where the Broker side stores and controls the value of Offset, uses RemoteBrokerOffsetStore

{" offsetTable ": {" zxp_test_topic @ zxp_test_group2" : {0:16, 1:17, now, with}, "TopicTest @ please_rename_unique_group_name_4" : {0:25 0, 1, 250, 2:250, 3:250}, "%RETRY%zxp_test_group2@zxp_test_group2":{0:3 }, "%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0 }, "Zxp_test_group3 RETRY % % @ zxp_test_group3" : 0-0 at {}, "order_topic @ zxp_test_group3" : {0-0, 1:3, 2:3, 3-3}}}Copy the code

indexFile

If we need to look up messages based on message IDS, consumeQueue does not store message ids, and if nothing else, we have to go through commitlog files again. IndexFile is the file for this

Offset summary

Since the concept of multiple offsets has emerged, LET me summarize:

  1. Offset (message body offset) in CommitLog

Reflected in commitlog file name, corresponding to the commitlog file all the messages in the topic of the queue start offset (convenient via ConsumeQueue.com mitlogOffset found to consume news which exist in the current commitlog file)

  1. CommitlogOffset (message body offset) in ConsumeQueue

Locates the offset of the current message in the Commitlog

  1. Offsettable. Offset (subscript)

Locates the subscript of the currently consumed ConsumeQueue