preface
As a super e-commerce application, kaola.com generates massive log information every day and has higher requirements on log writing performance and integrity.
Conventional scheme
The usual way to log on Android is to manipulate files through the Java Api. When a log is written, first open the file, then write to the log, and finally close the file. Using this scheme does not seem to have much impact on the program at present, but as the log volume increases, frequent IO operations in Java will easily lead to GC, frequent file opening, and easy to cause CPU spikes.
Let’s examine the process of writing directly to a file:
- The user initiates a write operation
- The operating system searches the page cache a. If no match is found, a page missing exception is generated, and the page cache is created to write the content passed in by the user to the page cache B. If it does, the user’s incoming content is written directly to the page cache
- The call of user write is complete
- After a page is modified, it becomes a dirty page. The operating system uses two mechanisms to write the dirty page back to disk a. The user manually calls fsync() b. The pdflush process periodically writes dirty pages back to disk
It can be seen that the process of writing data from the program to disk actually involves two copies of data: one is the copy of user space memory to the cache of kernel space, and one is the copy of the cache of kernel space to the disk when writing back. It also involves frequent switching between kernel space and user space when writing back occurs.
SSD storage also has a “write up” problem compared to mechanical hard drives. The problem has to do with the physical structure of SSD storage. After an SSD has been written, the data to be written cannot be directly updated but can be overwritten. Data must be erased before being overwritten. However, the smallest unit of write is Page, and the smallest unit of erase is Block, and Block is much larger than Page. Therefore, when writing new data, you need to first read the data from the Block and the data to be written together, then erase the Block, and finally write the read data to the storage. As a result, the data actually written may be much larger than the data originally written.
I didn’t expect so many operations to be involved in simple file writing, but only transparent to the application layer.
Since there are so many operations per write, can we cache the logs and write them to disk once they reach a certain number?
While this can significantly reduce the number of I/OS, it can cause another, more serious problem — lost logs
Logs are cached in memory so that the integrity of logs cannot be guaranteed when a program crashes or a process is killed.
A complete log solution needs to be met
- High efficiency, can not affect the system performance, not because of the introduction of the log module caused by the application lag
- Ensure log integrity. If log integrity is not guaranteed, log collection is meaningless
- If necessary, compress and encrypt logs
High performance solution
Since we can’t reduce the number of writes, can we optimize while writing files?
The answer is yes, use MMAP
Mmap is a method of memory-mapping files. It maps a file or other object to the address space of a process to achieve the mapping between the file disk address and a segment of virtual address in the process virtual address space. The function prototype is as follows
#include <sys/mman.h>
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
Copy the codeParameters that
Mmap mapping model
The sample code
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
main(){
int fd;
void *start;
struct stat sb;
fd = open("/etc/passwd", O_RDONLY); /* open /etc/passwd */ fstat(fd, &sb); /* get file size */ start = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0);if(start == MAP_FAILED) /* Check whether the mapping succeeds */return;
printf("%s", start); munma(start, sb.st_size); /* Closed (fd); }Copy the codeMmap operations provide a mechanism for user programs to access device memory directly, which is more efficient than copying data to each other in user and kernel space. It is commonly used in applications that require high performance.
In addition, MMAP ensures log integrity. Mmap write back time:
- Out of memory
- Process exits
- Call msync or munmap
- 30s-60s without MAP_NOSYNC (FreeBSD only)
Unmap function prototype
#include <sys/mman.h>
int munmap(void *addr, size_t length);
Copy the codeWhen a file is mapped, the program will apply for a space of the same size in native memory. Therefore, it is recommended to map a small section of content each time, such as 64K, and then remap the content behind the file when it is full.
It is important to note that for multi-process files, FileLock synchronization is possible using the Java Api, whereas Mmap does not work with multiple processes working on the same file. For multi-process applications, multiple files need to be mapped on demand.
Continue to optimize
According to the above scheme, JNI interface was designed, SO was packaged and introduced into the project.
Considering the size of the installation package, can we not use so?
In fact, Java already provides an implementation of memory mapping – MappedByteBuffer
MappedByteBuffer is located in the Java NIO package and is used to map file contents to buffers using the MMAP technology. Buffers are created using the FileChannel map method
MappedByteBuffer raf = new RandomAccessFile(file, "rw");
MappedByteBuffer buffer = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, position, size);
Copy the codeThere is a bit of a pit, although Java provides map method, but does not provide unmap method, through Google learned that unmap method is available, but is private
// FileChannelImpl.class
private static void unmap(MappedByteBuffer var0) {
Cleaner var1 = ((DirectBuffer)var0).cleaner();
if(var1 ! = null) { var1.clean(); } } // Cleaner.class public voidclean() {
if (remove(this)) {
try {
this.thunk.run();
} catch (final Throwable var2) {
AccessController.doPrivileged(new PrivilegedAction<Void>() {
public Void run() {
if(System.err ! = null) { (new Error("Cleaner terminated abnormally", var2)).printStackTrace();
}
System.exit(1);
returnnull; }}); }}}Copy the codeThat’s when reflection calls come to mind
public static void unmap(MappedByteBuffer buffer) {
if (buffer == null) {
return; } try { Class<? > clazz = Class.forName("sun.nio.ch.FileChannelImpl");
Method m = clazz.getDeclaredMethod("unmap", MappedByteBuffer.class);
m.setAccessible(true); m.invoke(null, buffer); } catch (Throwable e) { e.printStackTrace(); }}Copy the codeSince Android P already restricts access to private apis, this still needs to be optimized for Android P.
To test the efficiency of MappedByteBuffer, we wrote 64byte data to memory, MappedByteBuffer and disk files 500,000 times, and counted the time consuming
| methods | Time consuming |
|---|---|
| memory | 384ms |
| MappedByteBuffer | 700ms |
| Disk file | 16805ms |
It can be seen that although MappedByteBuffer is not as good as writing to memory, it has a qualitative improvement compared with writing to disk files.
Looking forward to
Currently, the log module only guarantees log writing performance and integrity, but does not implement log compression and encryption. Currently, cached logs are desensitized data. If services require secure storage in the future, the encryption function will be added.
conclusion
This paper mainly analyzes the problems existing in the way of directly writing files to record logs, and extends the high performance file writing scheme MMAP, which gives consideration to the writing performance and integrity. Finally, the implementation of memory mapping in Java layer is introduced to avoid the introduction of SO.
For more technical articles, visit the Koala Mobile Team technical blog.