preface

Recently, I am working on a new retail cashier app, which has relatively high requirements for the stability of the app. However, some problems that are difficult to reproduce will inevitably occur. To solve these problems, analyzing logs is sometimes a necessary means to solve them. Now we mainly analyze the implementation of log writing scheme. Please refer to AwesomeLog for detailed code. If I can help you, I hope to give you a STAR. Thank you.

Defects in conventional solutions

  • Performance issues: Log writing starts with standard I/O writing directly to the file. When a log is written, open the file first, then write to the log, and finally close the file. However, file writing is an I/O operation. As the number of logs increases, more I/O operations will inevitably cause performance bottlenecks. Why do you say that? Because the process of writing data from the program to disk actually involves two copies of data: one copy of user-space memory to the kernel space cache, and one copy of the kernel space cache to the disk when writing back. It also involves frequent switching between kernel space and user space when writing back occurs.
  • Log loss: To solve the performance problem, we can immediately think of reducing I/O operations, we can first cache the log in memory, when a certain number or when appropriate to write the log in memory to disk. This seems to reduce I/O operations, but if the app is forced to kill or Crash while writing the in-memory logs to disk, this can cause a more serious problem, loss of logs.

Looking at this, is there really no high-performance solution to log integrity? The answer is MMAP. What the hell is Mmap? Let’s move on.

What is the mmap

Mmap is a method to map files in memory. That is, a file or other object is mapped to the address space of a process to achieve a one-to-one mapping between the file disk address and a segment of virtual address in the virtual address space of a process. After implementing such mapping relation, process can adopt the way of pointer reads and writes a block of memory, the system will automatically back to write dirty pages to the corresponding files on disk, that is done to the operation of the file without having to call the read and write system calls such as function, on the contrary, the kernel space of this area change has a direct response to the user space, In this way, files can be shared between different processes.

Many articles on the web say that Mmap bypasses page caching entirely, which is not true. The physical memory we eventually mapped is still in the page cache, which provides the following benefits:

  • Reduce system calls. We only need one mmap() system call, and all subsequent calls operate as if they are in memory, without a lot of read/write system calls.
  • Reduce data copy. A normal read() call requires two copies of the data; Mmap only needs to be copied from disk once and does not need to be copied back to user space due to memory mapping.
  • High reliability. After mMAP writes data to the page cache, the kernel thread can be relied on to periodically write back to disk, as is the case with cache I/O latency.

From the diagram above, we only need one copy of data with MMAP.

Mmap usage scenarios

Mmap is suitable for frequent reads and writes to the same area, and threading is recommended.

  • User logs and data reporting meet this scenario. The Xlog module in the Mars framework of wechat open source is also implemented based on MMAP features.
  • Mmap is also a good choice for cross-process synchronization. Android has its own Binder mechanism for cross-process communication, which also uses MMAP internally.

The specific implementation

In Android, files can be mapped to memory using the Java-provided MappedByteBuffer and then read and write. (wechat xlog module mmap is based on C++ code)

MappedByteBuffer is located in the Java NIO package and is used to map file contents to buffers using the MMAP technology. Buffers are created using the FileChannel map method.

RandomAccessFile raf = new RandomAccessFile(file, "rw");
//position Start position of the mapping file. Size Size of the mapping file
MappedByteBuffer buffer = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, position, size);
// Write bytes to the buffer
buffer.put(log);
Copy the code

The unmap method is available in Java, but it is not available in Unmap. The unmap method is not available in Java, but it is private. We can use reflection to call unmap. Check out this blog post to get around the restrictions.)

    /** * unmap memory from file ** /
    private void unmap(MappedByteBuffer mbbi) {
        if (mbbi == null) {
            return;
        }
        try{ Class<? > clazz = Class.forName("sun.nio.ch.FileChannelImpl");
            Method m = clazz.getDeclaredMethod("unmap", MappedByteBuffer.class);
            m.setAccessible(true);
            m.invoke(null, mbbi);
        } catch(Throwable e) { e.printStackTrace(); }}Copy the code

A point here is that at first, when I was writing to a file, I used Mmap to write the log directly to the file, which required code to dynamically expand, which was quite troublesome. Later, IT turned out that Xlog was writing the log to the cache, which is the memory region mapped by MMAP. The mapped disk file is a newly created cache file. When the cache content reaches a certain threshold, the background thread is informed to write the contents of the cache to the file. Flush the cache to the target file when it is full, or flush the cache to the target file manually.

As for why wechat did this, it must be for performance reasons. For details, please refer to the framework analysis and migration ideas of wechat’s cross-platform component Mars-Xlog

conclusion

This article focuses on some of the drawbacks of the logging writing routine and the reasons for them, and then introduces the definition, advantages, and usage scenarios of MMAP. Finally, it mainly describes the concrete implementation of MMAP and how to apply it to log writing. Please refer to AwesomeLog for detailed code. If I can help you, I hope to give you a STAR. Thank you

Reference documents: Android Development master class, wechat Xlog