preface
Java introduced NiO-related classes such as ByteBuffer in JDK 1.4, which allowed Java programmers to write and write files using block-based methods instead of stream-based ones. The JDK also introduces the Kings of IO performance optimization — zero-copy sendFile and Mmap. But how good are they? How fast is it compared to RandomAccessFile? Under what circumstances is fast? Is FileChannel faster or MappedByteBuffer faster……
(Zero Copy I: User-mode Perspective)
God, there are so many questions ！！！！！！
Let’s take our time.
Take a look at MQS who are good at taking advantage of IO zero copy
As we know, there are a lot of MQ in the Java world: ActiveMQ, Kafka, RocketMQ, Qunar MQ, and they are the big players in the Java world using NIO zero copy.
However, their performance is pretty much the same. Regardless of other factors, such as network transfer, data structure design, and file storage, let’s just discuss how the Broker side reads and writes files to see how they differ.
The following figure shows the file read and write methods used by each MQ.
- Kafka: Record reads and writes are based on FileChannel. Index read/write based on MMAP (page big prompt).
- RocketMQ: The read drive is based on MMAP, and the write drive uses MMAP by default. You can modify the configuration to configure FileChannel.
- QMQ: where MQ, read disk using MMAP, write disk using FileChannel.
- ActiveMQ 5.15: Reading and writing is all based on RandomAccessFile, which is why we abandoned ActiveMQ.
So, is MMAP better or FileChannel better?
MMAP It is well known that os-based MMAP memory mapping technology, through MMU mapping files, makes random read and write files and read and write memory similar speed.
The FileChannel? Is it zero copy? Unfortunately, no. FileChannel is fast, just because it’s block-based.
Next up, Benchmark Everything — Xu Ma.
Benchmark?
How to Benchmark? Benchmark?
Since it is reading and writing files, it is natural to look at the read and write performance, which is the most basic. Note, however, that MQ typically uses a timed flush to prevent data loss, and both MMAP and FileChannel have force methods for flushing pageCache data to hard disk. Does force affect performance? The answer is yes. To what extent? I don’t know. Does the size of each write affect performance, no doubt, but what are the rules? Is FileOutputStream really useless? The answer is not necessarily.
File tuning has always been an art, because there are so many factors that affect performance. First, the advent of SSDS has raised questions about traditional B+ tree-based tree structures. Second, each file system performs differently. The performance difference between Linux Ext3 and Ext4 is huge (the difference in file deletion performance is about 20 times). Fortunately, Apple finally released macOS High Sierra and iOS 10.3 in 2017, both of which dropped HFS+, Switched to APFS with higher performance. Each file system can have a different scheduling algorithm, and there are performance burrs caused by virtual memory page miss interrupts…….
Conscientious RocketMQ provides scripts for Linux IO tuning, which does a good job:)
Beside the point.
Java MappedByteBuffer & FileChannel & RandomAccessFile & FileXXXputStream You can also run it on your own machine.
The test environment
CPU: Intel i7 4-core 8-thread 4.2GHz memory: 40GB DDR4 disk: SSD read/write 2GB/s JDK1.8 OS: Mac OS 10.13.6 virtual memory: not disabled, 9GB
Test points to note:
- To prevent the PageCache cache, a new file is generated each time for reading.
- To test the impact of different packets on performance, you need to run multiple tests with packets of different sizes.
- Force has a significant impact on performance and should be tested separately.
- Test with 1GB files (small files have no reference and large files cannot be mapped by MMAP)
Pure reading test
1 gb file:
Test MappedByteBuffer & FileChannel & RandomAccessFile & FileInputStream.
From this graph, we can see that MMAP’s performance is overwhelming, especially for small data volumes. The other streams only start to de-kill Mmap at 4KB. Therefore, use Mmap to read data below 4KB.
Zoom in to see how mmap compares to FileChannel:
As you can see from the figure above, FileChannel and other non-zero copies beat mmap by a large margin when writing data packets larger than 4kb, except for the BT test that read 1 gigabyte at a time.
Therefore, if your packet is larger than 4KB, use FileChannel.
Pure write test
1 gb file:
Test MappedByteBuffer & FileChannel & RandomAccessFile & FileInputStream.
From the figure above, we can see that MMAP performance is equally stable. FileChannel isn’t bad either, but at 32 bytes of data, it’s not.
Look at the thumbnail again:
As you can see, 64 bytes is the dividing line between FileChannel and Mmap performance, and from 64 bytes, FileChannel works its way up until bitTorrent 1GB files lose a bit.
Therefore, we recommend: If your packet size is larger than 64 bytes, use FileChannel to write.
Asynchronous Force Testing
We know that RocketMQ uses asynchronous flush, but how does asynchronous force affect performance? Benchmark everything. Let’s use an asynchronous thread and brush every 16KB to see how well it performs.
Mmap continues to lag and perform poorly, remaining around 4000 except for a little jitter at 2048 bytes, and around 1500 without force. FileChannel is not affected by Force at all. In my tests, a force for a 1GB file took around 800 milliseconds. The larger the buffer, the more time, vice versa.
As an aside, Kafka has always discouraged the use of force for this reason. Of course, Kafka also has its own multi-copy policy for data security.
Here we conclude that if you need to perform force frequently, even asynchronously, do not use Mmap and use FileChannel instead.
Summary.
Based on the above tests, we came up with a chart:
Assuming that our system’s packets are around 1024-2048, what strategy should we use?
A: Read using Mmap and write only using FileChannel.
Looking back at the MQ implementers, QMQ seems to be the only one doing this. Of course, RocketMQ also provides FileChannel write options. The default mMAP write and asynchronous flush are the main culprits of the broker busy.
Kafka, because it is not force by default, also uses FileChannel for writing. Why use FileChannel for reading? Probably because the message size is over 4KB.
On this assumption, the design of these MQ seems to make perfect sense.
Finally, don’t use force when you can. If you want to use force, use FileChannel.