Kafka guarantees that messages are not lost or repeated

Enable.auto.com MIT =false Disable the automatic submission offset. The offset is shifted after the automatic submission is complete. The production end resends the data: the production end resends the data from the deduplication table before the consumption. Here is the most troublesome solution: 1. Asynchronous buffers are full, block there, wait for the buffer to be available, can’t empty buffer 2. If the message is sent successfully, the next message will be sent. If the message fails, the message will be recorded in the log and scanned by the timing script. (The timing script may not really send the message, but does not receive the feedback.)

Data loss: 1) In synchronous mode, there are three states to ensure that messages are produced safely. If the state is set to 1 (only the leader partition is written successfully), data will be lost if the Leader partition hangs. 2) Messages can also be lost when using asynchronous mode. When the buffer is full, the data is immediately discarded if the buffer pool is set to 0 (empties the buffer pool as soon as it is full without acknowledgement).

As long as you can avoid both of these situations, you can guarantee that messages will not be lost. 1) This means that in synchronous mode, the acknowledgement mechanism is set to -1, that is, messages are written to the leader and all replicas. 2) Also, in asynchronous mode, if the buffer pool is full when the message is sent but no acknowledgement is received, the configuration file will set the blocking timeout time to unlimited, that is, to keep the production side blocked, so that data will not be lost.

Ack: The ACK confirmation mechanism is set to 0, indicating that the producer does not wait for the response, does not wait for borker’s confirmation information, and has the minimum delay. The producer cannot know whether the message is successful or not, and the message may be lost, but has the maximum throughput. The ACK mechanism is set to -1, that is, messages are written to the leader and all replicas. All replicas in the ISR list return confirmation messages. The ACK confirmation mechanism is set to 1. The leader has received the data confirmation information, and the Replica pulls the data asynchronously. The ACK mechanism is set to 2, which means that the broker returns success if the producer writes successfully to the partition leader and one other follower, regardless of whether the other follower writes successfully. The ACK mechanism is set to “all”, that is, the send method returns only when all copies are synchronized to the data. In this way, the data can be sent successfully. Theoretically, the data will not be lost. Min.insync.replicas =1 means that at least one replica returns successfully. Otherwise, the product is abnormal

Conclusion: The integrity of the message and the throughput of the system are mutually exclusive. In order to ensure that the message is not lost, the throughput of the system must be lost. Ack set -1 2. Set the minimum number of copies successfully synchronized to the number of copies -1 3. Increase retry times 4. Use the kafkaproducer. send(Record, callback) method instead of send(Record) method. 7. All replicas are collectively called Assigned Replicas (AR 8). When the client buffer is full, messages may be lost. 9, block.on.buffer. Full = true Consumer: 1, enable.auto.mit =false Disables the automatic commit shift

Unclean. Leader. Election. Enable set to false (the default parameter to true), mean, when have you the latest record of replication downtime, Kafka himself will elect a master node, If the default allows replication’s node that has not yet synchronized your latest data to be voted master, your data will be lost, so you should set the parameter to false as needed;

Retries is set to a larger value. Setting a value greater than 0 will cause the client to resend any data if it fails to send. Note that these retries are no different from the retries when the client receives a send error. Allow retries to potentially change the order of data. If both records are sent to the same partition, the first message fails and the second message succeeds, and the second message appears before the first.

The replication. The factor > min. Insync. Replicas. If the two are equal, the partition will not work properly when one copy dies. Replica.factor = min.insync.replicas + 1.

Messages in the same partition are out of order: Assume that a and B have two messages. A sends the message first and tries again due to a failure to send the message. In this case, the order will be after B’s message. You can set the Max. In. Flight. Requests.. Per connection = 1 to avoid Max. In the flight. Requests. Per. Connection: limit the client on a single connection to send did not respond to the request number. Setting this value to 1 means that kafka Brokers can no longer send requests to the same broker before responding to a request, but throughput decreases

Version after 0.11.0: Idempotent sending: Producer ID (PID) and Sequence Number are introduced to realize idempotent semantics of Producer.

• Producer ID: Each new Producer is assigned a unique PID when it is initialized
• Sequence Number: For each PID, each
  
  of the data sent by the Producer corresponds to a monotonically increasing Sequence Number starting from 0.
  

The Broker also maintains an ordinal number for each <PID, Topic, Partition> and increments the ordinal number each time a message is committed. For each received message, the Broker accepts it if its number is larger than the number the Broker maintained (that is, the number of the message that was last committed), otherwise it is discarded:

• If the message number is larger than the number maintained by the Broker, data has not been written to the message. In this case, the Broker rejects the message and Producer throws InvalidSequenceNumber
• If the message number is less than or equal to the number maintained by the Broker, the message is a duplicate message. The Broker directly discards the message and the Producer throws DuplicateSequenceNumber

This mechanism solves the problem of data duplication and data disorder. Transaction mechanism: Multiple operations either all succeed or all fail. The essence of a Kafka Transaction is that the message corresponding to a set of write operations (if any) and the update of Offset corresponding to a set of read operations (if any) are marked the same (i.e., Transaction Marker) so that all read and write operations involved in the Transaction are visible or invisible simultaneously.

Add ISR: HW is commonly known as HighWatermark, HighWatermark abbreviation, take the smallest LEO in the ISR corresponding to a partition as HW, consumer can only consume the location of HW at most. In addition, each replica has HW, and the leader and follower are responsible for updating their own HW status. For a new message written by the leader, the consumer cannot consume it immediately. The leader will wait for the message to be synchronized with replicas in all ISR replicas to update the HW, and then the message can be consumed by the consumer. This ensures that if the leader broker fails, the message can still be retrieved from the newly elected Leader. There are no HW restrictions on read requests from the internal broKer. The following figure details the flow of ISR, HW and LEO as producer produces messages to brokers:

Thus, Kafka’s replication mechanism is neither fully synchronous nor purely asynchronous. In fact, synchronous replication requires all working followers to copy before the message is committed. This replication greatly affects throughput. In asynchronous replication mode, the followers asynchronously copy data from the leader, and the data is considered to be committed as long as the log is written by the leader. In this case, if the followers have not finished copying and fall behind the leader, the leader breaks down suddenly. Data will be lost. Kafka uses ISR in a balanced way to ensure data loss and throughput.


Kafka ISR management is ultimately fed back to the Zookeeper node. Location: /brokers/topics/[topic]/partitions/[partition]/state. There are currently two places where the Zookeeper node is maintained:


1.Controller to maintain: One of the brokers in a Kafka cluster is elected as a Controller, responsible for Partition management and replica state management, as well as performing administrative tasks such as redistributing partitions. Under certain conditions, the LeaderSelector under Controller elects a new leader, the ISR and the new Leader_EPOCH and Controller_EPOCH are written to the related nodes of Zookeeper. Also initiate LeaderAndIsrRequest to notify all Replicas.


2. Maintenance by the Leader: The leader has a separate thread to periodically check whether the followers in the ISR break away from the ISR. If the ISR changes, the new ISR information is returned to the relevant Zookeeper node.

References: www.infoq.cn/article/dep…