1. Ensure the order of partition messages. The same producer writing messages to the same partition must be ordered

2. All synchronous copies are considered committed only when messages have been written


3. Messages will not be lost as long as one replica is active


4. Consumers can only extract messages that have already been submitted








Broker’s handling of message reliability

1. Replication coefficient. That is, how many copies of a message should have (typically 3), how these copies are distributed across the rack to ensure that one broker fails or a rack route becomes unavailable.


2) Incomplete election of chief. Allow out-of-sync replicas as leaders. The downside is that for the same offset, the out-of-sync copy as the leader gets the new data, while the original copy stores the old data.

The scene could be

1. Suppose there are 3 replicas and 2 replicas are suspended. The chief replicas work normally, but the chief replicas are also suspended.

2. Among the three replicas, the leader replicas are normal, but there is a certain delay due to the network delay following the replication of the leader replicas. If the leader replicas die, the other replicas are not synchronized

3. Synchronize at least one copy. When the number of partition synchronization copies is less than the minimum synchronization copies, it stops accepting messages from the producer and throws an exception. In order to avoid incomplete election caused by the data write and read expectations inconsistent situation








The producer’s handling of message reliability

Producer-to-message reliability can be introduced in two ways.

  • First, assume that acks=1, but there are three copies. If the chief copy crashes and the other copies are considered synchronous, a message is lost to the producer.
  • Secondly, it is assumed that acks=all, that is, all three replicas are synchronized. If the chieftains copy crashes, the producer will only receive the response that the chieftains are unavailable during the election, and the producer needs to process the message by himself.

Therefore, two aspects need to be considered:

1. It is set to acks, but needs to deal with throughput and message loss

The more ACKS, the less probability of loss, but the less throughput, you have to wait for all received

The producer retry mechanism is used. For retries, kafka uses the internal retry mechanism. Non-retried errors are saved elsewhere and entered later.

The risk of retry is message duplication





Consumer handling of message reliability

The biggest problem for consumers is that if a message offset is submitted but not processed, the message will never be processed. So the key is how to deal with message offsets.

  • Automatic offset commit: Ensures that only offsets that have already been processed are committed
  • A strategy for manually offsetting commits: ensure that they are always committed later in processing, ensure that commits are not too frequent or too few, do appropriate retries, and ensure that scenarios requiring one-time semantics are met


What does zero copy kafka mean?

Zero copy depends on the operating system. Kafka has a lot of data persistence track disks and disk files sent over the network. The traditional way is to go through four copies. First, the system will call file data to read into the memory Buffer, and then the application program will read the kernel state into the user state Buffer. Then the user sends data through the socket to copy the user state to the kernel state Buffer, and finally copies the data to NIC through DMA copy. On linux2.4+, the sendfile system call is copied from DMA to the NIC Buffer via zero copy without CPU copying

Zero copy source, only two context switches

How long is the data retained?

You can set the retention duration or size for each topic. Each partition will have several fragments, and the fragment that is currently writing data (active fragment) will never be deleted. If data is configured to be retained for five days, it will be retained for five days

The default is 1 GB or 1 week, whichever is smaller. When a fragment is full, close the current file and open a new one, which is convenient for searching and deleting



The file format of the data store?

The storage format is the same as that sent by the producer to the consumer. The message contains not only build sum values, but also sizes, checksums, versions, compression algorithms, and timestamps

How do I delete a key directly?

The application sends a message with the same key but a value of NULL (called a tombstone message), and only the NULL message is kept for regular cleanup. After a certain period of time, the cleanup thread cleans up the tombstone message when the consumer finds the null record while consuming it and knows it should be deleted from the database

Consumers can’t work if they’re offline


Kafka’s Compact strategy?

Application scenario: The message contains the same key, but only the value of the latest key is required. When compact is executed, a map is created in the memory. Key is the hash of the message key, and the value is the offset of the message key. After reading each fragment of a certain amount of dirty messages, if the current message key exists, the offset is small, the value is expired, or the message is null, the map is discarded

How to stuff (read) data into Kafka?








How does the Connect Api handle interaction with other systems?