Kafka

Kafka basis

Through the ISR multiple copy mechanism
- Data consistency LEO and HW (High water level)
- LEO identifies the next position of the last message in each partition, and each copy of the partition has its own LEO. The smallest LEO in ISR is HW, commonly known as high water level, and consumers can only pull the message before HW

Reliability ACK analysis ensures high availability
- Relying on the number of copies to support reliability is not enough. Most people also think of generating the producer client parameters request.required. Acks.
- For a configuration with acks = 1, the production producer sends the message to the Leader replica, which notifies the production producer that it has committed successfully after being successfully written to the local daily log, as shown in the figure below. If the leader breaks down before the follower copy of the ISR collection can pull the newly written message from the leader, the sent message will be lost.
- For the ack = -1 configuration, the producer sends the message to the leader copy. After the leader copy is successfully written into the local daily log, the leader copy needs to wait for all the follower copies in the ISR to complete synchronization before it can tell the producer that it has successfully committed. Even if the leader replica goes down, messages are not lost
- The ack = -1 configuration means that the producer does not need to wait for confirmation from the self-broker before continuing to send the next batch of messages. In this case, the data transmission efficiency is the highest, but the data reliability is the lowest.
Disk-based storage

Kafka uses the sequential disk read/write mode to greatly improve read/write performance
High-performance page caching
Zero copy
Asynchronous send
Batch pull data reduces network I/OS

To use transactions use rocketMQ