This article is out of date

Please clickMMKV 1.1.1

MMKV (Official introduction)

MMKV is a key-value component based on MMAP memory mapping. Protobuf is used to realize the underlying serialization/deserialization, which has high performance and strong stability. It has been used on wechat since the middle of 2015, and its performance and stability have been verified over time. Recently, it has been ported to Android/macOS/Windows platform and also open source.github

Detailed usage instructions and performance comparison are provided in the official documentation. We know that NSUserDedefaults is just a simple XML file, and even apple’s NSKeyArchive serialization of objc objects is just a copy of an XML file. So not only is MMKV thread safe but it’s also super performance NSUserdefaults;

Read the instructions

This article focuses on iOS source code parsing, but before we start, we need to understand three concepts: Mmap, Protobuf, and CRC.

mmap wiki

In computing, mmap(2) is a POSIX-compliant Unix system call that maps files or devices into memory. It is a method of memory-mapped file I/O. It implements demand paging, because file contents are not read from disk initially and do not use physical RAM at all. The actual reads from disk Are performed in a “lazy” manner, after a specific location is email exchange.

An article explains it in more detail:

Mmap is a method of memory-mapping files. A file or other object is mapped to the address space of a process to achieve the mapping between the file disk address and a segment of virtual address in the process virtual address space. After such mapping is achieved, the process can use Pointers to read and write the memory, and the system will automatically write back dirty pages to the corresponding file disk, that is, the operation on the file is completed without calling system call functions such as read and write. Conversely, changes made by the kernel space to this area directly reflect user space, allowing file sharing between different processes.

Simply put, a read/write file operation requires a page cache as a link between the kernel and the application layer, so a file operation requires two copies of data (kernel to page cache and page cache to the application layer), whereas MMAP allows direct interaction between user space and kernel space without the need for page cache. Mmap also formally maps memory directly, and its usage scenarios are limited. As apple docs put it:

File mapping is effective when:

You have a large file whose contents you want to access randomly one or more times.

You have a small file whose contents you want to read into memory all at once and access frequently. This technique is best for files that are no more than a few virtual memory pages in size.

You want to cache specific portions of a file in memory. File mapping eliminates the need to cache the data at all, which leaves more room in the system disk caches for other data.

Therefore, MMAP is most efficient when we need to access a small portion of a larger file at a high frequency.

In fact, not only MMKV, but also Wechat’s XLog and Meituan’s Logan log tools, as well as SQLight use Mmap to improve file access efficiency in high-frequency update scenarios.

Protocol Buffer wiki

Protocol Buffers (Protobuf) is a method of serializing structured data. It is useful in developing programs to communicate with each other over a wire or for storing data. The method involves an interface description language that describes the structure of some data and a program that generates source code from that description for generating or parsing a stream of bytes that represents the structured data.

Protobuf is a method of serializing structured data. It was originally developed to solve the compatibility problem between old and new protocols (high and low versions) on the server side. Therefore, it is called “protocol buffer”, but later gradually developed for data transmission and storage.

MMKV formally considered the good performance of Protobuf in performance and space, using a simplified version of Protobuf as a serialization scheme, but also extended the incremental update capability of Protobuf, after the incremental KV object serialized, directly append to the end of memory for serialization.

So how does Protobuf code efficiently?

The implementation of tag-value (tag-length-value) encoding reduces the use of delimiters and makes data storage more compact.
Using base 128 varINT (variable length encoding) principle to compress data, binary data is very compact, pb volume is smaller. But pb is not compressed to the limit, float, double float are not compressed.
There are fewer {,}, : symbols, and less volume than JSON and XML. Plus varint compression, gzip compression is even smaller!

CRC check

Cyclic Redundancy Check (CRC) is a hash function that generates a short fixed-digit check code based on network data packets or computer files to detect or verify errors that may occur after data is transferred or saved. The resulting number is calculated and appended to the data before transmission or storage, and then checked by the receiver to see if the data has changed.

Considering that file system and operating system have certain instability, MMKV adds CRC check to screen invalid data. In the iOS wechat live network environment, there are an average of about 700,000 times of data verification failed.

Implement

Before we begin, take a look at the file composition:

As the official introduction says, it’s really lightweight. In addition to the introduction of OpenSSL for AES encryption, Protobuf is also available in mini. The remaining classes are the main components of the MMKV implementation.

MMKV

+[MMKV initialize]

+ (void)initialize {
	if (self == MMKV.class) {
		g_instanceDic = [NSMutableDictionary dictionary];
		g_instanceLock = [[NSRecursiveLock alloc] init];

		DEFAULT_MMAP_SIZE = getpagesize();
		MMKVInfo(@"pagesize:%d", DEFAULT_MMAP_SIZE);

#ifdef __IPHONE_OS_VERSION_MIN_REQUIRED
		auto appState = [UIApplication sharedApplication].applicationState;
		g_isInBackground = (appState == UIApplicationStateBackground);
		MMKVInfo(@"g_isInBackground:%d, appState:%ld", g_isInBackground, appState);

		[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(didEnterBackground) name:UIApplicationDidEnterBackgroundNotification object:nil];
		[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(didBecomeActive) name:UIApplicationDidBecomeActiveNotification object:nil];
#endif
	}
}
Copy the code

Global g_instanceDic is initialized in Initialize and is given a recursive lock to keep the thread safe. ⚠️ note that g_instanceDic is not initialized with dispatch_once to ensure uniqueness. Commit history was originally used, but was later optimized as redundant code. CXX is a static variable that is safe to access globally.

The page size of the current system is then assigned to DEFAULT_MMAP_SIZE as a constraint on whether memory decluttering and file writing back are required, since mMAP’s Mach memory size is constrained by pageSize.

Finally, if you’re on an iPhone you’ll receive notifications to update the state of g_isInBackground to allow reads and writes in different threads.

+[MMKV mmkvWithID: cryptKey: relativePath:]

+ (instancetype)mmkvWithID:(NSString *)mmapID cryptKey:(NSData *)cryptKey relativePath:(nullable NSString *)relativePath  { if (mmapID.length <= 0) { return nil; } / / / id + relativePath generates kvKey nsstrings * kvKey = [MMKV mmapKeyWithMMapID: mmapID relativePath: relativePath]; CScopedLock lock(g_instanceLock); MMKV *kv = [g_instanceDic objectForKey:kvKey]; if (kv == nil) { NSString *kvPath = [MMKV mappedKVPathWithID:mmapID relativePath:relativePath]; if (! isFileExist(kvPath)) { if (! createFile(kvPath)) { MMKVError(@"fail to create file at %@", kvPath); return nil; } } kv = [[MMKV alloc] initWithMMapID:kvKey cryptKey:cryptKey path:kvPath]; [g_instanceDic setObject:kv forKey:kvKey]; } return kv; }Copy the code

This method obtains MMKV instance by corresponding ID, All instances are stored in g_instanceDic with (ID + relativePath) md5 keys, and each instance is stored in its corresponding file after protobuf encode after serialization. The file name is ID. To ensure data reliability, each file has a CORRESPONDING CRC check file.

In terms of data security, users can encrypt AES through the cryptKey passed in. MMKV is embedded with OpenSSL as the basis of AES encryption.

All MMKV files are stored in the same root directory, which can be set by -[initializeMMKV:] or setMMKVBasePath before all MMKV instances are initialized. The relativePath is configured relative to mmkvBasePath.

Similar to NSUserDefaults standerUserDefault MMKV provides defaultMMKV, whose DEFAULT_MMAP_ID is @” mmKv.default “. MMKV also provides -[migrateFromUserDefaults:] to facilitate data migration;

mmkv instance

NSRecursiveLock *m_lock; // recursive lock to keep m_DIC thread safe
NSMutableDictionary *m_dic; /// kv container, hold the real key-value pair
NSString *m_path; // MMKV file path
NSString *m_crcPath; // The CRC file path of MMKV
NSString *m_mmapID; // The unique ID is generated after (mmkvID + relativePath) MD5.
int m_fd; // The file operator
char *m_ptr; // Current kv file operation pointer
size_t m_size; // The size of the file mapped by mmap
size_t m_actualSize; // The current memory occupied by KV
MiniCodedOutputData *m_output; // Map the remaining space of memory
AESCrypt *m_cryptor; /// encryptor, the file content will be updated after the recalculation of the encrypted value
MMKVMetaInfo m_metaInfo; // Save the CRC file digest.Copy the code

Here is a list of MMKV instance variables, MMKV init will first initialize the following parameters and call loadFromFile to serialize the file to m_dict.

m_lock = [[NSRecursiveLock alloc] init];

m_mmapID = kvKey;

m_path = path;

m_crcPath = [MMKV crcPathWithMappedKVPath:m_path];

m_cryptor = AESCrypt((const unsigned char *) cryptKey.bytes, cryptKey.length);

+[MMKV loadFromFile]

LoadFromFile is one of the core methods of MMKV. The file is long, but the implementation is clear;

- (void)loadFromFile {/// 1. Obtain the summary of CRC file and store m_metaInfo [self prepareMetaFile]; if (m_metaFilePtr ! = nullptr && m_metaFilePtr ! = MAP_FAILED) { m_metaInfo.read(m_metaFilePtr); } if (m_cryptor) { if (m_metaInfo.m_version >= 2) { m_cryptor->reset(m_metaInfo.m_vector, sizeof(m_metaInfo.m_vector)); }} / / / 2. Obtain fild descriptor m_fd = open (m_path UTF8String, O_RDWR | O_CREAT, S_IRWXU); if (m_fd < 0) { MMKVError(@"fail to open:%@, %s", m_path, strerror(errno)); } else {/// 3. According to the file fd size and system pagesize (DEFAULT_MMAP_SIZE) to calculate the file m_size; m_size = 0; struct stat st = {}; if (fstat(m_fd, &st) ! = -1) { m_size = (size_t) st.st_size; } // round up to (n * pagesize) if (m_size < DEFAULT_MMAP_SIZE || (m_size % DEFAULT_MMAP_SIZE ! = 0)) { m_size = ((m_size / DEFAULT_MMAP_SIZE) + 1) * DEFAULT_MMAP_SIZE; if (ftruncate(m_fd, m_size) ! = 0) { MMKVError(@"fail to truncate [%@] to size %zu, %s", m_mmapID, m_size, strerror(errno)); m_size = (size_t) st.st_size; return; }} // 4. To carry on the memory mapping, mapping obtained through mmap starting position of the corresponding address m_ptr = (char *) mmap (nullptr m_size, PROT_READ | PROT_WRITE, MAP_SHARED, m_fd, 0). if (m_ptr == MAP_FAILED) { MMKVError(@"fail to mmap [%@], %s", m_mmapID, strerror(errno)); } else {/// 5. Use m_ptr to read a fixed 4-byte lenbuffer; // Use MiniCodedInputData to get the real data size of the file from lenBuffer, i.e. m_actualSize; const int offset = pbFixed32Size(0); NSData *lenBuffer = [NSData dataWithBytesNoCopy:m_ptr length:offset freeWhenDone:NO]; @try { m_actualSize = MiniCodedInputData(lenBuffer).readFixed32(); } @catch (NSException *exception) { MMKVError(@"%@", exception); } MMKVInfo(@"loading [%@] with %zu size in total, file size is %zu", m_mmapID, m_actualSize, m_size); if (m_actualSize > 0) { /// 6. If the check fails and the error code is MMKVOnErrorRecover, the data rollback is attempted. bool loadFromFile, needFullWriteback = false; if (m_actualSize < m_size && m_actualSize + offset <= m_size) { if ([self checkFileCRCValid] == YES) { loadFromFile = true; } else {/// 7. Check the CRC file to ensure that the file is not damaged, if failed, execute 'onMMKVCRCCheckFail' callback loadFromFile = false; if (g_callbackHandler && [g_callbackHandler respondsToSelector:@selector(onMMKVCRCCheckFail:)]) { auto strategic = [g_callbackHandler onMMKVCRCCheckFail:m_mmapID]; if (strategic == MMKVOnErrorRecover) { loadFromFile = true; needFullWriteback = true; } } } } else { MMKVError(@"load [%@] error: %zu size in total, file size is %zu", m_mmapID, m_actualSize, m_size); loadFromFile = false; if (g_callbackHandler && [g_callbackHandler respondsToSelector:@selector(onMMKVFileLengthError:)]) { auto strategic = [g_callbackHandler onMMKVFileLengthError:m_mmapID]; if (strategic == MMKVOnErrorRecover) { loadFromFile = true; needFullWriteback = true; [self writeActualSize:m_size - offset]; }} // start reading the contents of the file with the length of m_actualSize. AES decryption; // 2. Decode protobuf, assign value to m_dic, and save the remaining bytes of file in m_output for data preparation; if (loadFromFile) { MMKVInfo(@"loading [%@] with crc %u sequence %u", m_mmapID, m_metaInfo.m_crcDigest, m_metaInfo.m_sequence); NSData *inputBuffer = [NSData dataWithBytesNoCopy:m_ptr + offset length:m_actualSize freeWhenDone:NO]; if (m_cryptor) { inputBuffer = decryptBuffer(*m_cryptor, inputBuffer); } m_dic = [MiniPBCoder decodeContainerOfClass:NSMutableDictionary.class withValueClass:NSData.class fromData:inputBuffer]; m_output = new MiniCodedOutputData(m_ptr + offset + m_actualSize, m_size - offset - m_actualSize); if (needFullWriteback) { [self fullWriteBack]; } } else { [self writeActualSize:0]; m_output = new MiniCodedOutputData(m_ptr + offset, m_size - offset); [self recaculateCRCDigest]; } } else { m_output = new MiniCodedOutputData(m_ptr + offset, m_size - offset); [self recaculateCRCDigest]; } MMKVInfo(@"loaded [%@] with %zu values", m_mmapID, (unsigned long) m_dic.count); } } if (m_dic == nil) { m_dic = [NSMutableDictionary dictionary]; } if (! [self isFileValid]) { MMKVWarning(@"[%@] file not valid", m_mmapID); } tryResetFileProtection(m_path); tryResetFileProtection(m_crcPath); m_needLoadFromFile = NO; }Copy the code

A quick refresher:

Get the SUMMARY of CRC file, save m_metaInfo, save m_metaFd CRC file Descriptor.
Get MMKV Fild Descriptor;
According to the file fd size and system pagesize (DEFAULT_MMAP_SIZE) to calculate the file m_size;
To perform memory mapping, use Mmap to obtain the start address m_ptr of the file corresponding to the mapped memory.
Read the fixed 4-byte data lenBuffer in the header of the file through M_PTR, and obtain the real data size of the file from LenBuffer with MiniCodedInputData, that is, m_actualSize.
Check the file m_actualSize for correctness, or execute if it failsonMMKVFileLengthErrorCallback: If the check fails and the error code is MMKVOnErrorRecover, data rollback is attempted.
Check file CRC to ensure that the file is not damagedonMMKVCRCCheckFailThe callback.
Check that the file m_actualSize is correct and the file is not damaged. Start reading the contents of the file with the length of m_actualSize.
AES decryption;
Decode protobuf, assign value to M_DIC, and save the remaining bytes of file in M_Output for data preparation;
TryResetFileProtection Guarantees read and write permissions for files.

Setter

Let’s take a look at the assignment method of MMKV:

- (BOOL)setObject:(nullable NSObject<NSCoding> *)object forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setBool:(BOOL)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setInt32:(int32_t)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setUInt32:(uint32_t)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setInt64:(int64_t)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setUInt64:(uint64_t)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setFloat:(float)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setDouble:(double)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setString:(NSString *)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setDate:(NSDate *)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));

- (BOOL)setData:(NSData *)value forKey:(NSString *)key NS_SWIFT_NAME(set(_:forKey:));
Copy the code

Its interfaces are declared in much the same way as NSUserDefaults and support Swift apis. The difference is that after MMKV updates the corresponding value, it no longer needs to manually call sync/async, and it will carry out -[MMKV checkLoadData] to check the data and synchronously write the data back to output. Let’s look at the implementation:

Ojbc object type - (BOOL)setObject:(nullable NSObject<NSCoding> *)object forKey:(NSString *)key {if (key.length <= 0) {/// ojbc object type - (BOOL)setObject:(nullable NSObject<NSCoding> *)object forKey:(NSString *)key {if (key.length <= 0) { return NO; } if (object == nil) { [self removeValueForKey:key]; return YES; } NSData *data; if ([MiniPBCoder isMiniPBCoderCompatibleObject:object]) { data = [MiniPBCoder encodeDataWithObject:object]; } else { /*if ([object conformsToProtocol:@protocol(NSCoding)])*/ { data = [NSKeyedArchiver archivedDataWithRootObject:object]; } } return [self setRawData:data forKey:key]; } /// basic data type - (BOOL)setBool:(BOOL)value forKey:(NSString *)key {if (key.length <= 0) {return NO; } size_t size = pbBoolSize(value); NSMutableData *data = [NSMutableData dataWithLength:size]; MiniCodedOutputData output(data); output.writeBool(value); return [self setRawData:data forKey:key]; }Copy the code

-[MMKV setRawData: forKey:] -[MMKV setRawData: forKey:] But the data here has been processed by MiniCodedOutputData, data alignment.

So let’s start with BOOL, NSMutableData *data is initialized with the size required by the protobuffer encoding bool. Then new a MiniCodedOutputData output(data) with data as the parameter and put the corresponding Bool value Writes output. WriteBool is written to data in the byte order of MiniCodedOutputData to align data. Finally, call -[setRawData: forKey:].

MiniPBCoder encodeDataWithObject objC objects require protobuf type checks before writing, serializing the supported data types directly [MiniPBCoder encodeDataWithObject:], Does not support the call system [NSKeyedArchiver archivedDataWithRootObject:]. MiniPBCoder Supports serialization of objC types:

NSString

NSData

NSDate

Here don’t understand is that the internal implementation is to support the encoding Dictionary container, but only in – [isMiniPBCoderCompatibleObject:] these three returns to YES.

-[MiniPBCoder getEncodeData]

+[MiniPBCoder encodeDataWithObject:] initializes the MiniPBCoder object with the objC passed in as an argument and calls getEncodeData to return the serialized data. GetEncodeData is implemented as follows:

- (NSData *)getEncodeData { if (m_outputBuffer ! = nil) { return m_outputBuffer; } m_encodeItems = new std::vector<MiniPBEncodeItem>(); size_t index = [self prepareObjectForEncode:m_obj]; MiniPBEncodeItem *oItem = (index < m_encodeItems->size()) ? &(*m_encodeItems)[index] : nullptr; if (oItem && oItem->compiledSize > 0) { // non-protobuf object(NSString/NSArray, etc) need to to write SIZE as well as DATA, // so compiledSize is used, m_outputBuffer = [NSMutableData dataWithLength:oItem->compiledSize]; m_outputData = new MiniCodedOutputData(m_outputBuffer); [self writeRootObject]; return m_outputBuffer; } return nil; }Copy the code

The core of this is to convert an encode object into a MiniPBEncodeItem by -[MiniPBCoder prepareObjectForEncode:] and store it in STD :: Vector

*m_encodeItems uses a vector of CXX here because an encode object may be of type NSDictionary, When it is an NSDictionary object, it recursively calls prepareObjectForEncode to convert its key and value into MiniPBEncodeItem and store it in m_encodeItems.

Initialize m_outputBuffer to fetch m_encodeItems according to their compiledSize, just like the underlying data type, M_encodeItems is also finally converted to MiniCodedOutputData and called -[MiniPBCoder writeRootObject] for byte alignment. The internal implementation of writeRootObject is relatively simple, which is to align the Varint length of the protobuf according to the type of encodeItem and write the data to m_outputBuffer.

MiniPBEncodeItemType Supports the following types:

enum MiniPBEncodeItemType {
  PBEncodeItemType_None,
  PBEncodeItemType_NSString,
  PBEncodeItemType_NSData,
  PBEncodeItemType_NSDate,
  PBEncodeItemType_NSContainer,
};
Copy the code

EncodeItem compiledSize is the size required to encode the valueSize of an encodeItem into the Varint of a protobuf.

-[MMKV appendData: forKey:]

All methods of the setter API end up at -[MMKV setRawData: forKey:], the inner core of which is to call appendData to write data.

- (BOOL)appendData:(NSData *)data forKey:(NSString *)key { /// 1. Respectively to obtain the key length and data. The length calculation of writing data size size_t keyLength = [key lengthOfBytesUsingEncoding: NSUTF8StringEncoding]; auto size = keyLength + pbRawVarint32Size((int32_t) keyLength); // size needed to encode the key size += data.length + pbRawVarint32Size((int32_t) data.length); // size needed to encode the value /// 2. BOOL hasEnoughSize = [self ensureMemorySize:size]; if (hasEnoughSize == NO || [self isFileValid] == NO) { return NO; } // 3. Write m_actualSize BOOL ret = [self writeActualSize:m_actualSize + size]; If (ret) {/ / / 4. Write m_output ret = [self protectFromBackgroundWriting: size writeBlock: ^ (MiniCodedOutputData * output) { output->writeString(key); output->writeData(data); // note: write size of data }]; if (ret) { static const int offset = pbFixed32Size(0); auto ptr = (uint8_t *) m_ptr + offset + m_actualSize - size; if (m_cryptor) { m_cryptor->encrypt(ptr, ptr, size); } [self updateCRCDigest:ptr withSize:size increaseSequence:KeepSequence]; } } return ret; }Copy the code

M_lock is locked before each data is written, Data. length + key lenght protobuf Varint -[MMKV writeActualSize:] write m_actualSize, After the success of the write call again – [MMKV protectFromBackgroundWriting: writeBlock:] to complete data written. Finally, data is updated by appending to the end of M_OUTPUT, and m_DIC will be updated only after the appending is successful.

According to the official instructions, append new data directly in append mode for write optimization.

Standard Protobuf does not provide incremental update capability; each write must be written in full. Considering that the main usage scenario is frequent write update, we need the ability of incremental update: append the incremental KV object directly to the end of memory after serialization; In this way, there will be several new and old copies of the same key, with the latest data at the end. Therefore, when the program starts and turns on MMKV for the first time, the data can be guaranteed to be up-to-date and effective by constantly replacing the previous value with the value read later.

The problem with appending data directly to the end of m_output is that the space grows rapidly, resulting in uncontrollable file sizes. Therefore, -[MMKV ensureMemorySize:] is called for file refactoring before data is written. Official note:

Using Append for incremental updates brings up a new problem, which is that the file size can grow out of control if you append constantly. For example, if the same key is constantly updated, it may consume hundreds of M or even G space. In fact, the entire KV file is only one key, which can be saved in less than 1K space. This is clearly undesirable. We need to make a compromise between performance and space: apply space in unit of memory pagesize, append mode until space runs out; When append to the end of the file, file reorganization, key rearrangement, try serialization save the rearrangement result; If you don’t have enough space after reloading, double the size of the file until you have enough space.

Let’s see how this works:

// since we use append mode, when -[setData: forKey:] many times, space may not be enough // try a full rewrite to make space - (BOOL)ensureMemorySize:(size_t)newSize { [self checkLoadData]; M_fd, m_size, m_output, m_ptr have all been successfully initialized if (! [self isFileValid]) { MMKVWarning(@"[%@] file not valid", m_mmapID); return NO; } // make some room for placeholder constexpr uint32_t /*ItemSizeHolder = 0x00ffffff,*/ ItemSizeHolderSize = 4; if (m_dic.count == 0) { newSize += ItemSizeHolderSize; } /// 2. If there is not enough space to store new_size or m_DIC is empty, try to reorganize the file. /// Serialize the data stored in m_DIC and write it to m_output as the reorganized data. if (newSize >= m_output->spaceLeft() || m_dic.count == 0) { // try a full rewrite to make space static const int offset = pbFixed32Size(0); NSData *data = [MiniPBCoder encodeDataWithObject:m_dic]; size_t lenNeeded = data.length + offset + newSize; size_t avgItemSize = lenNeeded / std::max<size_t>(1, m_dic.count); size_t futureUsage = avgItemSize * std::max<size_t>(8, m_dic.count / 2); /// 3. In the case of insufficient memory, execute the do-while loop, continuously multiply m_size by 2 until there is enough space for full data write back or reserved enough space to avoid frequent expansion. // 1. no space for a full rewrite, double it // 2. or space is not large enough for future usage, double it to avoid frequently full rewrite if (lenNeeded >= m_size || (lenNeeded + futureUsage) >= m_size) { size_t oldSize = m_size; do { m_size *= 2; } while (lenNeeded + futureUsage >= m_size); MMKVInfo(@"extending [%@] file size from %zu to %zu, incoming size:%zu, future usage:%zu", m_mmapID, oldSize, m_size, newSize, futureUsage); // If we can't extend size, rollback to old state if (fTRUNCate (m_fd, m_size)! = 0) { MMKVError(@"fail to truncate [%@] to size %zu, %s", m_mmapID, m_size, strerror(errno)); m_size = oldSize; return NO; If (munmap(m_ptr, oldSize)! = 0) { MMKVError(@"fail to munmap [%@], %s", m_mmapID, strerror(errno)); } // 6. Re-map the memory according to the new m_size, Update m_prt pointer m_ptr = (char *) mmap (m_ptr m_size, PROT_READ | PROT_WRITE, MAP_SHARED, m_fd, 0). if (m_ptr == MAP_FAILED) { MMKVError(@"fail to mmap [%@], %s", m_mmapID, strerror(errno)); } // check if we fail to make more space if (! [self isFileValid]) { MMKVWarning(@"[%@] file not valid", m_mmapID); return NO; } /// 7. Re-generate m_OUTPUT and reset the data size offset. // keep m_output consistent with m_ptr -- writeAcutalSize: may fail delete m_output; m_output = new MiniCodedOutputData(m_ptr + offset, m_size - offset); m_output->seek(m_actualSize); } / / / 8. Data encryption after the reforming of the if (m_cryptor) {[self updateIVAndIncreaseSequence: KeepSequence]; m_cryptor->reset(m_metaInfo.m_vector, sizeof(m_metaInfo.m_vector)); auto ptr = (unsigned char *) data.bytes; m_cryptor->encrypt(ptr, ptr, data.length); } / / / 9. The real data write m_prt m_actualSize head corresponding to the size of the memory area if ([self writeActualSize: data, length] = = NO) {return NO; } // 10. Delete m_output; m_output = new MiniCodedOutputData(m_ptr + offset, m_size - offset); BOOL ret = [self protectFromBackgroundWriting:m_actualSize writeBlock:^(MiniCodedOutputData *output) { output->writeRawData(data); }]; if (ret) { [self recaculateCRCDigest]; } return ret; } return YES; }Copy the code

Check the validity of the file, m_fd, m_SIZE, m_output, m_ptr have been successfully initialized;
If the remaining space is insufficient to store new_size or m_DIC is empty, try file reorganization.

Serialize the data stored in M_DIC and write m_OUTPUT as the reformed data;

In the case of low memory, execute a do-while loop, multiplying m_size by 2 until there is enough space for full data write back or reserved enough space to avoid frequent expansion.
1. no space for a full rewrite, double it
2. or space is not large enough for future usage, double it to avoid frequently full rewrite
Clear files in preparation for data writing;
Remove old memory maps.
Remap the memory according to the new M_SIZE, update the M_PRT pointer;
Rebuild m_Output, resetting the data size to offset.
Re-encrypt the reformed data
Write the real data size, m_actualSize, to the memory area corresponding to the m_PRT header
Write the reformed data again

Getter

- (id)getObjectOfClass:(Class)cls forKey:(NSString *)key {
	if (key.length <= 0) {
		return nil;
	}
	NSData *data = [self getRawDataForKey:key];
	if (data.length > 0) {

		if ([MiniPBCoder isMiniPBCoderCompatibleType:cls]) {
			return [MiniPBCoder decodeObjectOfClass:cls fromData:data];
		} else {
			if ([cls conformsToProtocol:@protocol(NSCoding)]) {
				return [NSKeyedUnarchiver unarchiveObjectWithData:data];
			}
		}
	}
	return nil;
}

- (BOOL)getBoolForKey:(NSString *)key {
	return [self getBoolForKey:key defaultValue:FALSE];
}
- (BOOL)getBoolForKey:(NSString *)key defaultValue:(BOOL)defaultValue {
	if (key.length <= 0) {
		return defaultValue;
	}
	NSData *data = [self getRawDataForKey:key];
	if (data.length > 0) {
		@try {
			MiniCodedInputData input(data);
			return input.readBool();
		} @catch (NSException *exception) {
			MMKVError(@"%@", exception);
		}
	}
	return defaultValue;
}
Copy the code

Like setters, both base data types and objC types first call -[MMKV getRawDataForKey:] to get data. GetRawData simply returns the corresponding data directly via m_dict and checks the file status -[MMKV checkLoadData].

Basic data types are decoded by MiniCodedInputData IntPUT (data) and corresponding values are returned. Objc will call the decoder for types that support protobuf encodings. Instead, call the system’s +[NSKeyedUnarchiver unarchiveObjectWithData:].

Protobuf decoding implementation is relatively simple, the core implementation is:

- (id)decodeOneObject:(id)obj ofClass:(Class)cls {
	if(! cls && ! obj) {return nil;
	}
	if(! cls) { cls = [(NSObject *) obj class];
	}

	if (cls == [NSString class]) {
		return m_inputData->readString();
	} else if (cls == [NSMutableString class]) {
		return [NSMutableString stringWithString:m_inputData->readString()];
	} else if (cls == [NSData class]) {
		return m_inputData->readData();
	} else if (cls == [NSMutableData class]) {
		return [NSMutableData dataWithData:m_inputData->readData()];
	} else if (cls == [NSDate class]) {
		return [NSDate dateWithTimeIntervalSince1970:m_inputData->readDouble()];
	} else {
		MMKVError(@"%@ does not respond -[getValueTypeTable] and no basic type, can't handle".NSStringFromClass(cls));
	}

	return nil;
}
Copy the code

When we call +[MiniPBCoder decodeObjectOfClass: CLS fromData:], we create a MiniPBCoder object inside. And data into MiniCodedInputData *m_inputData in decodeOneObject according to different types of objects, read the stored data and initialization return.

Delete

The MMKV removeValueForKey is deleted by -[MMKV removeValueForKey:]

- (void)removeValueForKey:(NSString *)key {
	if (key.length <= 0) {
		return;
	}
	CScopedLock lock(m_lock);
	[self checkLoadData];

	if ([m_dic objectForKey:key] == nil) {
		return;
	}
	[m_dic removeObjectForKey:key];
	m_hasFullWriteBack = NO;

	static NSData *data = [NSData data];
	[self appendData:data forKey:key];
}
Copy the code

Much like setter, except that when the value corresponding to m_dict key is removed, appendData writes an empty data to m_Output. Finally, updates are written to the file during memory refactoring.

conclusion

MMKV is a MAP-based K-V repository similar to NSUerDefaults, but nearly a hundredfold more efficient.

It obtains the MMKV object corresponding to mmapID through mmkvWithID method, obtains m_PRT and m_output of the file through mmap, and writes the serialized data to m_dict.

MiniCodedOutData is used as an intermediate buffer to store data in bytes when writing data. Because of the nature of Mmap, data is written simultaneously to the file, and because the Protobuf protocol cannot do incremental updates, this is done by constantly appending new values to the file. When the write space is insufficient, the memory will be rearranged. After the file is expanded by double, k-v in m_dict will be serialized again.

When the data is queried, the Buffer is taken from the map, converted to the corresponding real type and returned.

When data is deleted, the key is found and deleted from the map, and the value of the key in the file is set to 0.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

MMKV 1.0 iOS (Deprecated)

This article is out of date

Please clickMMKV 1.1.1

MMKV (Official introduction)

Read the instructions

mmap wiki

Protocol Buffer wiki

CRC check

Implement

MMKV

+[MMKV initialize]

+[MMKV mmkvWithID: cryptKey: relativePath:]

mmkv instance

+[MMKV loadFromFile]

Setter

-[MiniPBCoder getEncodeData]

-[MMKV appendData: forKey:]

Getter

Delete

conclusion

MMKV 1.0 iOS (Deprecated)

This article is out of date

Please clickMMKV 1.1.1

MMKV (Official introduction)

Read the instructions

mmap wiki

Protocol Buffer wiki

CRC check

Implement

MMKV

+[MMKV initialize]

+[MMKV mmkvWithID: cryptKey: relativePath:]

mmkv instance

+[MMKV loadFromFile]

Setter

-[MiniPBCoder getEncodeData]

-[MMKV appendData: forKey:]

Getter

Delete

conclusion

Related Posts

Multithreading (pure dry dry dry goods version)

IOS functional programming (chain programming) FuncControl

Swift Academy of Algorithms – Longest Common subsequence algorithm