The basic concept of Kafka

Basic Concepts:

Broker: a kafka instance that can host multiple partitions topic: a topic can have multiple partitions. A partition can be spread across multiple brokers. For each partition, you can specify A leader and multiple followers as replicas. For A broker, it can be the leader of partitions A and B as well as the follower of partitions C and D

For each message, it will have an offset as a unique identifier in the partition. The subscript LEO (Log End Offset) identifies the Offset of the next message. The concept is that each broker in each partition has a LEO HW (High Water Mark) message whose Offset satisfies HW-1. Is visible to the consumer. Each partition has only one HW, and HW takes the minimum value of LEO in this partition

The scene that

Assume that A partition of the order has 1 master and 2 slave, and both HW and LEO are 3. That is, there are three messages with offset 0,1, and 2 messages written by the leader. Due to the different synchronization efficiency of each follower, the leader has been completely synchronized with copy A, while only one message has been synchronized with copy B. At this time, the LEO of the leader and the two followers are 5, 5 and 4 respectively, and the HW at this time takes the minimum value of LEO in this partition, that is, 4. Messages 0,1,2,3 are consumable, and 4 is invisible

benefits

If all replicas are required to be replicated before messages are visible, performance is affected. If the messages are visible only after the leader completes the replication, messages may be lost. Kafka is an effective trade-off between data reliability and performance