background
The kafka value here is the broker. The boundaries of the broker message loss need to be aligned:
1 Messages that have been submitted
2 limited persistence
If the message fails, it is not broke that lost the message;
Limited persistence (Broker availability)
Producer loss message
producer.send(Object msg) ; This way of sending messages is asynchronous; Fire and forget, send regardless of the result;
The cause of the failure may be many, for example, network jitter, sending messages exceeding the size limit;
How? Always send messages with return values, i.e. Producer.send (MSG,callback)
Callback can accurately tell you whether the message is sent successfully, and you can also have a solution if the message fails to be sent.
Network jitter: retransmission
Send message out of size: Adjust the size of the message to send
It’s not that the broker loses the message, it’s that the message from the producer fails to commit.
Consumer lost message
Kafka consumption message model:
That is, consume the message and set the offset.
What about when consumers lose data? If the consumer fails, the message is lost and the broker does not know it.
How to break? Always consume the message before updating the displacement; This may cause the problem of repeated message consumption, but does not cause message loss;
Multithreaded consumption lost messages
That is, when automatic displacement submission is enabled, if a thread has a problem but still submits displacement, message loss will occur.
How to break? To turn off auto commit shift, the consumer configuration parameter: enable.auto.mit =false
Tune broker parameters to prevent message loss
The configuration is adjusted to ensure that Kafka messages are not lost.
The second is that the producer sending the message must use the callback function method and handle the failure of sending.
At the same time, when writing the consumer program, consume first and then submit.
summary
This section defines kafka messages that are not lost to the broker. The basic requirements are: The message has been committed to the broker and at least one broker is available.
Then it introduces how to prevent message loss in Kafka from three aspects: producer, consumer and configuration parameters.
Original is not easy, attention is precious, forwarding price is higher! Reprint please indicate the source, let us exchange, common progress, welcome communication. I will continue to share Java software programming knowledge and programmer development career path, welcome to pay attention to, I sorted out these years of programming learning all kinds of resources, pay attention to the public account ‘Li Fuchun continuous output’, send ‘learning materials’ to share with you!
