A lot of students ask me in the private letter Kafka performance optimization to do what measures, for the answer to the relevant question actually I have written a long time, is not systematic sort out a article, recently thought to take some time to sort out, next time students ask me related questions I can be natural and unrestrained throw a link. This is a common Kafka interview question, and it’s not hard for the interviewer to ask. There are plenty of articles on the web, such as “Why Kafka Is So Fast? I have read these articles and they are well written. The problem is that they only list part of the main points but not all of them are detailed. There’s more to this article than any of you can find online, and if you run into a question during an interview after reading this article, you’re sure to impress your interviewer.
PS: All the main points of this article are covered in detail in Understanding Kafka. If you’re not familiar with this topic, check it out again.
Batch processing
Message sending and consumption in traditional messaging middleware is monolithic. For the producer, it sends a message, and then the broker returns an ACK indicating receipt, generating two RPCS. For the consumer, it requests to receive the message, then the broker returns the message, and finally sends an ACK to indicate that it has been consumed, resulting in three RPCS (some message-oriented middleware optimizes that the broker returns multiple messages). Kafka uses batch processing: the producer aggregates a batch of messages and then does two MORE RPCS to store the message to the broker, an operation that would otherwise take many RPCS to complete. If you want to send 1000 messages, each 1KB in size, then traditional messaging middleware would need 2000 RPCS. Kafka might wrap these 1000 messages into a single 1MB message, which takes two RPCS to complete the task. This improvement was once considered a form of “cheating,” but now that the microbatch concept is in vogue, other messaging middleware is following suit.
Client optimization
Continuing with the concept of batch processing, the new version of the producer client abandons the single thread in favor of two threads: the main thread and the Sender thread. The main thread is responsible for putting messages into the client cache, and the Sender thread is responsible for sending messages from the cache, which aggregates multiple messages into a batch. Some message-oriented middleware will throw messages directly to the broker.
Log format
Kafka has undergone three log format changes since version 0.8: V0, V1, and V2. Kafka log format Kafka log format Kafka log format is more and more convenient for batch message processing, interested students can read this article to understand.
Log coding
If you’re familiar with Kafka’s log format (see above), you should know that the log (Record, or message) itself has a few additional fields in addition to the basic key and value. These additional fields take up a fixed amount of space (see above, left). The latest versions of Kafka use Varints and ZigZag encoding to reduce the size of these additional fields. As the log (message) becomes smaller, network transmission becomes more efficient, log storage becomes more efficient, and thus the performance of collation is improved.
Message compression
Kafka supports multiple message compression methods (GZIP, SNappy, and LZ4). Message compression can greatly reduce network traffic, reduce network I/O, and improve overall performance. Message compression is a time-for-space optimization method. It is not recommended to compress messages if there is a requirement for delay.
Create an index for quick location and query
Each log fragment file corresponds to two index files, which are mainly used to improve the efficiency of finding messages, which is also a way to improve performance. (The specific content is explained in detail in chapter 5 of the book. It seems that I forgot to publish it in the official account, so I searched around and failed to find it.)
partition
Many people overlook this factor, but partitioning is also a very effective way to improve performance, which is more obvious than logging encoding, message compression, and so on. Partitioning is also heavily involved in other distributed components, but the basics of why partitioning improves performance are not covered here. Note, however, that increasing the number of partitions in a Kafka theme does not always lead to improved performance. .
consistency
Most sources don’t mention consistency when they talk about Kafka’s performance optimization initiatives. We know of common consistency protocols such as Paxos, Raft, Gossip, etc. Kafka takes a different approach to PacificA’s approach. Instead of “slapping your thigh”, Kafka uses this model to make things more efficient. The details will follow in an article like The Feasibility analysis and pros and cons of replacing PacificA with Raft in Kafka.
Sequential write disc
Operating systems can make deep optimizations for linear reads and writes, such as read-ahead (reading a large disk block into memory ahead of time) and write-behind (combining many small logical writes into a large physical write operation). Kafka adopted in the design of file additional ways to write the message, which can only be appended at the end of the log file of the new interest rates, and are not allowed to modify written messages, belongs to the typical order to write this way disk operation, so even if Kafka use disk as a storage medium, it can carry throughput is to be reckoned with.
Page caching
Why does Kafka perform so well? When confronted with this problem many people will think of the above order to write disk this point. There is also a layer of PageCache optimization ahead of the sequential swash plate.
Page caching is a major disk cache implemented by operating systems to reduce disk I/O operations. To be specific, data on the disk is cached to the memory, and the access to the disk is changed to the access to the memory. To compensate for this disparity in performance, modern operating systems increasingly “aggressively” use memory as disk cache, even happily using all available memory as disk cache so that when memory is reclaimed there is little performance penalty and all reads and writes to disk go through a unified cache.
When a process is preparing to read the contents of a file on the disk, the operating system checks whether the page where the data is to be read is in the pagecache. If it is, the data is directly returned, thus avoiding I/O operations on the physical disk. If there is no hit, the operating system makes a read request to disk and stores the read pages into the page cache, which then returns the data to the process. Similarly, if a process needs to write data to disk, the operating system checks whether the corresponding page is in the page cache. If not, the corresponding page is added to the page cache first, and the data is written to the corresponding page. The modified page becomes a dirty page, and the operating system writes the data in the dirty page to disk at an appropriate time to maintain data consistency.
For a process, it may cache the data it needs to process internally, but this data may also be cached in the operating system’s page cache, so the same data may be cached twice. And page caching is hard to disallow unless Direct I/O is used. In addition, anyone who has used Java generally knows two things: objects have a very high memory overhead, often several times or more the size of real data, and space utilization is low; Java garbage collection becomes slower and slower as more data is stored in the heap. For these reasons, using a file system and relying on a page cache is clearly better than maintaining an in-process cache or any other structure. At the very least, we can save a portion of the in-process cache consumption and save more space by using compact bytecodes instead of objects. This way, we can use 28GB to 30GB of memory on 32GB machines without worrying about GC performance issues. Furthermore, even if the Kafka service is restarted, the page cache remains active, whereas the in-process cache needs to be rebuilt. This also greatly simplifies the code logic, as maintaining consistency between page caches and files is left to the operating system, which is more secure and efficient than in-process maintenance.
The heavy use of page caching in Kafka is an important factor in Kafka’s high throughput. Although messages are written to the page cache first, the operating system is responsible for the specific task of flushing.
Zero copy
I posted an article called “What is Zero Copy” a long time ago. If you don’t know Zero Copy, you can read it. Kafka uses Zero Copy technology to make consumption more efficient. Kafka writes the message to the page cache first. If the consumer reads the message from the page cache, it can read it directly from the page cache. This saves another copy overhead from disk to page cache. In addition, for the concept of reading and writing, you can learn more about what is write magnification and read magnification.
The attached
The following figure shows a disk I/O process:
Linux IO disk
Write in the last
This article lists some of Kafka’s performance optimizations. Everything in this article is explained in Understanding Kafka, but it’s just scattered here and there, arranged in a predetermined order, from the easy to the hard. If the length of the book is used to list similar topics, there will be knowledge to explain the redundancy, so there is no repetition in the book, but these contents will be published in the public number, the front has been arranged in accordance with other dimensions of several articles. If you need new dimension content, you can leave a message in the public number, if the appeal is very big, I will sort out this article, this article is so.
Welcome to support the author’s book: “Illustrated Kafka Guide” and “Illustrated Kafka’s Core Principles”
Welcome to support my new books: In-depth Understanding of Kafka: Core Design and Practice Principles and RabbitMQ Field Guide. Also welcome to follow my wechat official account: Zhu Xiaosi’s blog.
