Out-of-heap memory, memory mapping, and Zero Copy in Java NIO

background

When I was in my former company, I participated in a coding competition and only got a middle score. However, DURING the competition, I learned many excellent ways of thinking from others and received guidance from my predecessors. Especially, my knowledge expansion during the competition was of great help to me. Some of the knowledge that is seldom exposed to will be helpful in the future work.

The title is simple, and it goes something like this:

• In a 4 gb memory machine to achieve a large file content by line sorting
• Each file is a string of lowercase letters that are not repeated. The maximum length is 128 bytes
• The file sizes are 1G/2G/5G/10G/20G. Obviously, some files cannot be loaded into memory

Without going into details of the process and results, here are some of the NIO techniques used, both in various frameworks such as Netty and in various middleware such as RocketMQ.

The basic concept

Out of memory

As we all know, the JVM allocates a chunk of memory for processing. Classes, objects, methods, and other data are stored in the JVM stack. The JVM is responsible for managing and recycling this chunk of memory.

In contrast, out-of-heap memory is memory allocated by direct calls to system malloc, which is not directly managed by the JVM and is not limited by the JVM’s maximum memory.

User mode and kernel mode

Applications cannot access resources such as memory and hard disk directly, but are invoked through interfaces provided by the operating system. In order to ensure security, the operating system classifies the system into the kernel mode with high permission and the user mode with low permission. Many operations of the user mode need to borrow the kernel mode for system scheduling, that is, state transition.

Unix System Architecture

The memory mapping

The OPERATING system (OS) provides a method to map the contents of a disk file to the memory. The modification of the memory data is automatically flushed to the disk file by the OS. You can skip a lot of THE IO in the process of memory mapping, and the flush process can be done even if the application crashes. This is memory mapping.

Memory-mapped files can be used to process files stored on disks without I/O operations. Therefore, memory-mapped files play an important role in processing files with a large amount of data. — Sogou Encyclopedia

ByteBuffer FileChannel channel of the buffer

A ByteBuffer is a buffer. All data in NIO is processed by the buffer. The underlying layer is usually a Byte array. In brief, ByteBuffer is a logically continuous piece of memory used for NIO read/write transfer. Properly designed, it can achieve zero-copy of data and reduce unnecessary data replication process.

Zero-copy: The process of reading and writing files at a time is as follows:

1. Copy from disk to kernel-mode cache (read data)
2. Read from the kernel state to the user-state cache where the application resides
3. Copy from user-mode cache to kernel-mode cache (write data)
4. Copy from kernel-mode cache to real write target, such as hard disk/network socket cache

As you can see, the data is read/write copied twice in the flow. The main problem is the replication between kernel and user caches. However, if you can make proper use of the capability provided by the kernel to directly Copy from the kernel-mode cache to the target cache location without going through the Copy between the user and kernel mode, the unnecessary Copy process, also known as zero-copy, will be significantly reduced.

The key way to

Some core methods that are hard to understand

In order to reuse Buffer to achieve zero copy, Buffer has many built-in cursors. The use of these cursors is the core of Buffer and the most difficult to understand:

• Mark is used to mark a particular location
• Position Current position
• Limit Indicates that the Buffer can be read from 0 to limit
• Capacity capacity

These operations do not really distinguish between read/write use, once the understanding of deviation will be difficult to achieve the correct processing logic, may take an afternoon to tune through, blood lessons

DirectByteBuffer Direct buffer

DirectByteBuffer, a special ByteBuffer, also requires a contiguous chunk of underlying memory that operates in the same manner as ordinary ByteBuffer, but is allocated off-heap memory by calling the native method unsafe.

The unsafe native method also does the memory freeing of the direct buffer; the DirectByteBuffer refers to memory that is held via PhantomReference and reclaimed by the JVM itself. However, if DirectByteBuffer is old after several GCS, it is likely to survive long periods due to the long interval between Full GCS, resulting in the pointed out-of-heap memory not being reclaimed. When manual collection is required, the Cleaner private method inside DirectByteBuffer is called through reflection.

Why use off-heap memory

Java applications typically operate on JVM managed heap memory, and a piece of data sent from the application to the network needs to be copied multiple times:

1. Copy from inside the heap to outside the heap
2. Copy from the heap to the socket cache
3. The socket buffer flush

Given the Java memory model, there may also be replication between working memory/main memory;

With GC in mind, there may also be replication between memory in the heap;

With out-of-heap memory, there is one less step of in-heap to out-of-heap replication.

Advantages of using direct buffers:

• This buffer memory is not directly reclaimed by the JVM
• The size is not limited by the maximum memory allocated by the JVM
• Some IO operations can avoid copying between off-heap and in-heap memory, such as network transfers
• Some large objects with long lifetimes can be kept in off-heap memory, reducing the impact on GC

Disadvantages:

• It is not managed directly by the JVM and is prone to out-of-heap memory leaks
• Since off-heap memory does not hold complex objects but only wrapper classes of basic types (both bottom byte arrays), serialization is required to store objects

Why must be copied to out-of-heap memory first

The reference points out that in BIO, Native allocates a block of memory outside the heap to copy the data in the heap before reading or writing a file:

1. When the underlying system calls are made through write, read, pwrite, and pread functions, the start address of buffer and buffer count are passed in as parameters. Using the Java Heap GC, we know that buffers in the JVM tend to be in the form of byte[], which is a special object where objects move in the heap due to the Java Heap GC. The address argument we passed to the system function is no longer the actual buffer address, so there will be an error in reading and writing. The C Heap is only affected by the Full GC and is relatively address stable.
2. There is no requirement in the JVM specification that Java byte[] must be contiguous memory space, which is often constrained by the type of the host language; The virtual address space allocated by the C Heap can be contiguous, whereas the above system call requires that we use contiguous address space as a buffer.

MappedByteBuffer Memory mapping buffer

Like other bytebuffers, MappedByteBuffer has a contiguous memory base. The difference is that this memory uses memory-mapped memory, which means that changes to this buffer are synchronized to the corresponding file.

FileChannel

NIO’s Channel type is a Channel that does not access data itself but interacts with Buffer.

The Channel class is mainly used to manipulate data, transfer data, and implement memory mapping.

Several kinds of Channel:

• FileChannel
• SocketChannel (Client TCP)
• ServerSocketChannel (Server TCP)
• DatagramChannel (UDP)

The key way to

Why use Channel

• The underlying implementation of transferFrom and transferTo depends on the OPERATING system API, and the operating system kernel is responsible for data replication. Since the back and forth replication from the kernel buffer to the user buffer and context switch are omitted, The transferFrom and transferTo methods of Channel are quite efficient
• Read and write uses ByteBuffer to reduce the number of copies
• MappedByteBuffer maps a chunk of memory that doesn’t block waiting for the flush to complete and doesn’t worry about losing data if an application crashes

FileChannel advantages:

• Memory-mapped content can prevent data loss caused by program crash (kill-9). This feature is very useful in many middleware systems.
• No blocking waiting, high efficiency
• Reduce replication times

Disadvantages:

• Since the size of mapped file needs to be specified for memory mapping, when the size of mapped file is larger than that of the written content, file gap will occur, that is, there is a part of empty padding after the FILE EOF, and garbled characters at the end of the file, which needs to be paid attention to in practical applications
• The mapped memory pages need to be replaced, resulting in complex memory management of the system

Some channels can operate in read/write mode

Efficiency is

public class UnitTest1 {
    private static final String prefix = "~/path/to/";

    public static void main(String[] args) throws Exception {
        streamCopy("input"."output1");
        bufferCopy("input"."output2");
        directBufferCopy("input"."output3");
        mappedByteBufferCopy("input"."output4");
        mappedByteBufferCopyByPart("input"."output5");
        channelCopy("input"."output6");
    }

    /** * use stream */
    private static void streamCopy(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        File inputFile = new File(prefix + from);
        File outputFile = new File(prefix + to);

        FileInputStream fis = new FileInputStream(inputFile);
        FileOutputStream fos = new FileOutputStream(outputFile);

        byte[] bytes = new byte[1024];
        int len;
        while((len = fis.read(bytes)) ! = -1) {
            fos.write(bytes, 0, len);
        }
        fos.flush();

        fis.close();
        fos.close();

        long endTime = System.currentTimeMillis();
        System.out.println("streamCopy cost:" + (endTime - startTime));
    }

    /** * use buffer */
    private static void bufferCopy(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        RandomAccessFile inputFile = new RandomAccessFile(prefix + from, "r");
        RandomAccessFile outputFile = new RandomAccessFile(prefix + to, "rw");

        FileChannel inputChannel = inputFile.getChannel();
        FileChannel outputChannel = outputFile.getChannel();

        ByteBuffer byteBuffer = ByteBuffer.allocate(1024);
        while(inputChannel.read(byteBuffer) ! = -1) {
            byteBuffer.flip();
            outputChannel.write(byteBuffer);
            byteBuffer.clear();
        }

        inputChannel.close();
        outputChannel.close();

        long endTime = System.currentTimeMillis();
        System.out.println("bufferCopy cost:" + (endTime - startTime));
    }

    /** * use out-of-heap memory */
    private static void directBufferCopy(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        RandomAccessFile inputFile = new RandomAccessFile(prefix + from, "r");
        RandomAccessFile outputFile = new RandomAccessFile(prefix + to, "rw");

        FileChannel inputChannel = inputFile.getChannel();
        FileChannel outputChannel = outputFile.getChannel();

        ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024);
        while(inputChannel.read(byteBuffer) ! = -1) {
            byteBuffer.flip();
            outputChannel.write(byteBuffer);
            byteBuffer.clear();
        }

        inputChannel.close();
        outputChannel.close();

        long endTime = System.currentTimeMillis();
        System.out.println("directBufferCopy cost:" + (endTime - startTime));
    }

    /** * Full memory mapping */
    private static void mappedByteBufferCopy(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        RandomAccessFile inputFile = new RandomAccessFile(prefix + from, "r");
        RandomAccessFile outputFile = new RandomAccessFile(prefix + to, "rw");

        FileChannel inputChannel = inputFile.getChannel();
        FileChannel outputChannel = outputFile.getChannel();

        MappedByteBuffer iBuffer = inputChannel.map(MapMode.READ_ONLY, 0, inputFile.length());
        MappedByteBuffer oBuffer = outputChannel.map(MapMode.READ_WRITE, 0, inputFile.length());

        // Direct buffer operations, no other IO operations
        oBuffer.put(iBuffer);

        inputChannel.close();
        outputChannel.close();

        long endTime = System.currentTimeMillis();
        System.out.println("mappedByteBufferCopy cost:" + (endTime - startTime));
    }

    /** * Memory mapping part */
    private static void mappedByteBufferCopyByPart(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        RandomAccessFile inputFile = new RandomAccessFile(prefix + from, "r");
        RandomAccessFile outputFile = new RandomAccessFile(prefix + to, "rw");

        FileChannel inputChannel = inputFile.getChannel();
        FileChannel outputChannel = outputFile.getChannel();

        for (long i = 0; i < inputFile.length(); i += 1024) {
            long size = 1024;
            // Avoid file gaps
            if (i + size > inputFile.length()) {
                size = inputFile.length() - i;
            }
            MappedByteBuffer iBuffer = inputChannel.map(MapMode.READ_ONLY, i, size);
            MappedByteBuffer oBuffer = outputChannel.map(MapMode.READ_WRITE, i, size);
            oBuffer.put(iBuffer);
        }

        inputChannel.close();
        outputChannel.close();

        long endTime = System.currentTimeMillis();
        System.out.println("mappedByteBufferCopyByPart cost:" + (endTime - startTime));
    }

    /** * zero copy */
    private static void channelCopy(String from, String to) throws IOException {
        long startTime = System.currentTimeMillis();

        RandomAccessFile inputFile = new RandomAccessFile(prefix + from, "r");
        RandomAccessFile outputFile = new RandomAccessFile(prefix + to, "rw");

        FileChannel inputChannel = inputFile.getChannel();
        FileChannel outputChannel = outputFile.getChannel();

        inputChannel.transferTo(0, inputFile.length(), outputChannel);

        inputChannel.close();
        outputChannel.close();

        long endTime = System.currentTimeMillis();
        System.out.println("channelCopy cost:"+ (endTime - startTime)); }}Copy the code

The size of the input file is 360MB, which is actually a small file.

This code outputs on my development machine as follows:

streamCopy cost:2718
bufferCopy cost:2604
directBufferCopy cost:2420
mappedByteBufferCopy cost:541
mappedByteBufferCopyByPart cost:11232
channelCopy cost:330
Copy the code

• Use STREAM as the benchmark
• ByteBuffer is a bit more efficient than the benchmark
• Out-of-heap memory is more efficient than in-heap memory in file replication
• Memory mapping large chunks of files to operate on is very fast, because it saves a lot of unnecessary IO
• If memory mapping is too small, efficiency is low due to frequent memory replacement
• ZeroCopy, come on, nothing to say

The resources

Zero copy in Java – Zhihu

Efficient data transfer with Zero copy – IBM Developer

What are the advantages of Java NIO Direct Buffer? – zhihu

This article moves my blog, welcome to visit!