A high-performance, real-time key-value persistence component of iOS

In the first half of this year, I saw such an article by wechat development team: MMKV– MMAP-based iOS high-performance universal key-value component. In the article, I mentioned the realization of a high-performance KV component with Mmap. Although I did not show too much specific code, the basic idea is very clear. At the end of the article, I mentioned the open source plan. After waiting for nearly six months to see the source code for this component, I decided to try to write one myself.

The wheel

According to the convention first on the wheel, you can give a star collection oh ~

FastKV github

About NSUserDefaults

Before you start writing this component, you should do some research on NSUserDefaults performance (PS: this is a mistake, I actually did it after WRITING this component).

As FAR as I know, NSUserDefaults has a layer of memory caching, so it provides a method called Synchronize for synchronizing disks and caches, but this method is now described in apple’s documentation for any other reason: You do not need to remove the synchronize call again.

The test results are as follows (write 1W times, value type is NSInteger, environment: iPhone 8 64G, iOS 11.4)

Non-synchronize time: 137ms

Synchronize time: 3758ms

Obviously, synchronize imposes a high performance loss on performance. In this paper, a high-performance and real-time key-value persistence component is required, which means that data can be persisted in extreme cases without affecting performance. In the so-called extreme case, for example, the data can be stored in the disk when the App crashes, and the data will not be lost due to the late synchronization between the cache and the disk.

From the data, we can see that the performance of non-synchronize is quite good, which seems to be much better than the test result in the above wechat article. The performance advantage of Mmap and NSUserDefaults is not obvious.

So let’s look at the aspect of high real time. Since Apple tells us to remove the Synchronize in its documentation, has Apple solved the problem of high real-time and high performance in NSUserDefaults? Holding the mentality of trying the author did a test, the answer is negative. Data loss still occurs in extreme cases without the use of Synchronize. So our MMAP still has its use, at least it guarantees high real-time while also taking into account the performance problem.

Please read this article before reading the rest for better understanding. MMKV– MMAP-based iOS high-performance universal key-value component

Data serialization

For specific implementation, the author referred to MMKV of wechat team above. That article has been described in detail, so the analysis of that article will not be carried out here.

One point to mention here is about data serialization. MMKV uses Google’s open source Protobuf in serialization. The author decided to customize a memory data format in consideration of various reasons during implementation, so as to avoid the dependence on Protobuf.

User-defined protocols are divided into three parts: Header Segment, Data Segment, and Check Code.

The Header Segment:

The length of this part is fixed, 160 bits or 288 bits.

VALUE_TYPE: indicates the data type. Currently, there are eight types: bool, nil, int32, int64, float, double, string, and data.

VERSION: indicates the VERSION when the data is recorded.

OBJC_TYPE Length: specifies the length of the OC class name string.

KEY length: indicates the KEY length.

DATA length: indicates the length of the value.

The Data Segment:

OBJC_TYPE: a character string of the OC class name.

KEY: the KEY.

DATA: the value.

Check Code:

CRC code: Indicates the crC-16 cyclic redundancy detection code for the last 16 bits of the data. It is used for later data verification.

Space growth

Allocation policy

The use of MMAP involves a memory allocation problem, and we provide two memory allocation strategies here.

One strategy mentioned in the MMKV article is to serialize the load when running out of memory in append. If it is not enough after serialization, double the size of the file until it is.

size_t allocationSize = 1;
while (allocationSize <= neededSize) {
    allocationSize *= 2;
}
return allocationSize;
Copy the code

Another strategy references the Memory allocation implementation of Python lists.

size_t allocationSize = (neededSize >> 3) + (neededSize < 9 ? 3:6); return allocationSize + neededSize;Copy the code

Memory jitter

These two memory allocation strategies are fine if you only consider adding new keys, but sequential reassignment operations may occur when keys are updated multiple times, as illustrated in the following example.

If the currently allocated Mmap size is only a fraction of the size currently in use, so that any subsequent append operation will trigger the reassignment, but since the key is being updated each time, if the current Mmap data is already the minimum set (no duplicate key data), Therefore, the MMAP size is just enough after the rearrangement, and there is no need to reallocate the MMAP size. The Mmap size is only a tiny bit more than the size currently in use, and any append will do the same again.

To solve this problem, I added a logic to the append operation: Normally, allocationSize is calculated according to the actual neededSize. If the key is being updated, allocationSize will be iterated twice. That is, the allocationSize calculated in the first calculation is the neededSize calculated in the second calculation.

size_t totalSize = dataLength + FastKVHeaderSize; size_t neededSize = updated ? [self _fkvAllocationSizeWithNeededSize:totalSize + size] : totalSize + size; If (neededSize > _mmsize | | (updated && [self _fkvAllocationSizeWithNeededSize: neededSize] > _mmsize)) {/ / redistribution mmap}Copy the code

Other optimization

The storage of some OC objects can be optimized, such as NSDate and NSURL. In actual storage, they can be serialized as double and NSString, which not only improves performance but also reduces space consumption.

Performance comparison

The test results are as follows (1W times, value type NSInteger, environment: iPhone 8 64G, iOS 11.4)

Add Time: 70ms (NSUserDefults Sync: 3469ms)

Update time: 80ms (NSUserDefults Sync: 3521ms)

Get time: 10ms (NSUserDefults: 48ms)

The performance of MMAP is indeed much better than that of NSUserDefults Sync, which is basically consistent with the performance test results of MMKV in the wechat article. In general, it is recommended to use the official NSUserDefults if the real-time requirements are not high.

Other Open Source works

TinyPart — Modular framework github nuggets

Coolog — Github mining for extensible log framework

WhiteElephantKiller – Useless code scanning tool github

reference

MMKV– MMAP-based iOS high-performance universal key-value component

alexlee002/mmkv