RocketMQ Advanced Features (1)

Message store:

1. Message sending process:

Message Send -> Message store ->ACK confirm sent -> Consumer consumption ->ACK confirm -> Message delete

2,commitLog

Real message storage file -> file system, SSD drop disk fast,1G increasing size

3,consumerQueue 

The consumerQueue maintains a queue with a single data structure divided into offset(8byte),size(4byte), and tagCode(8byte). Consumequeue can store 30W entries in a single file

4,indexFile 

An indexFile is also an index, but the fields used to create the indexFile are key and time. Therefore, like restoring the index by time based on this index, an indexFile is a single file that can store 2000W index entries. An indexFile is named by timestamp Since indexFile is a hash underneath, it can be said that RocketMq’s underlying implementation is a hash

CommitLog can clean up to three days or 75% of the disk

5, questions:

RocketMQ, after all, stores disks. How can it be efficient

CommitLog writes, and random reads from the ConsumerQueue

Take advantage of the operating system’s PageCache+ zero copy technology

1, disk –DMA Copy– kernel buffer — CPU Copy– user buffer — CPU Copy– socket buffer –DMA Copy– NIC buffer kernel buffer can read cache, has been optimized

Zero copy is available in two ways:

If the map file is truncated by another process, the system call to the writer() method will be terminated by sigBUS signals because of invalid addresses

Note: SIGBUS means that the address corresponding to the pointer is a valid address, but the bus cannot use the pointer properly. This is usually the result of unaligned data access. The default behavior of SIGBUS is to terminate the current process and produce core dump.

2,mmap+ sendFile to avoid problems caused by write NIO mode

6. Flush mechanism: Asynchronously flush the information to the memory and then return