introduce

In the process of memory mapping, there is no actual data copy, files are not loaded into memory, but are logically placed into memory. Specifically, the code is to establish and initialize the relevant data structure (struct address_space). This process is implemented by system call mmap(), so the establishment of memory mapping is very efficient.

Since creating a memory map does not actually make a copy of the data, how can a process end up accessing files on hard disk directly through memory operations? That depends on a few related processes after memory mapping.

Mmap () returns a pointer PTR that points to an address in the process’s logical address space, so that the process can manipulate the file instead of calling read or write. However, the PTR points to a logical address, and to manipulate the data within it, the MMU must translate the logical address into a physical address, independent of memory mapping.

In this case, MMU cannot find the physical address corresponding to PTR in the address mapping table, that is, MMU failure will result in a page-missing interrupt. The interrupt response function of the page-missing interrupt will look for the corresponding page in swap. If it is not found (that is, the file is never read into memory), the file is read from the hard disk into physical memory through the mapping established by mmap(), as shown in Procedure 3 in Figure 1. This process has nothing to do with memory mapping.

If the physical memory is insufficient during data copy, unused physical pages are swapped to hard disks through the virtual memory mechanism (SWAP). This process has nothing to do with memory mapping.

Mmap memory mapping process

  1. The process starts the mapping process and creates a virtual mapping area for the mapping in the virtual address space
  2. The kernel space system call function Mmap (different from the user space function) is called to realize the one-to-one mapping between the physical address of the file and the virtual address of the process
  3. A process initiates an access to this mapped space, raises a page – missing exception, and copies the file contents into physical memory (main memory)

Suitable scene

  • You have a large file whose contents you want to access randomly one or more times
  • You have a small file whose contents you want to read into memory immediately and access frequently. This technique works best for files that are no larger than a few virtual memory pages. (A page is the smallest unit of address space, and virtual and physical pages are the same size, usually 4KB.)
  • You need to cache specific parts of the file in memory. File mapping eliminates the need to cache data, which makes more space for other data in the system disk cache

When accessing a very large file at random, it is usually best to map only a small portion of the file. The problem with mapping large files is that the files consume active memory. If the file is large enough, the system may be forced to paginate other parts of memory to load the file. Mapping multiple files into memory complicates this problem.

Inappropriate scenes

  • You want to read a file from start to finish
  • This file is hundreds of megabytes or larger. Mapping large files into memory fills memory quickly and can lead to paging, which cancels out the benefits of mapping files first. For large sequential read operations, disk caching is disabled and the file is read into a small memory buffer
  • This file is larger than the contiguous virtual memory address space available. For 64-bit applications, this is not a problem, but for 32-bit applications, it is a problem, right
  • The file is on a removable drive
  • The file is on a network drive

The sample code

// // viewController. m // TestCode // // Created by Zhangdasen on 2020/5/24. // Copyright © 2020 Zhangdasen. All rights reserved. // #import "ViewController.h" #import <sys/mman.h> #import <sys/stat.h> @interface ViewController () @end @implementation ViewController - (void)viewDidLoad { [super viewDidLoad]; NSString *path = [NSHomeDirectory() stringByAppendingPathComponent:@"test.data"]; NSLog(@"path: %@", path); NSString *str = @"test str2"; [str writeToFile:path atomically:YES encoding:NSUTF8StringEncoding error:nil]; ProcessFile(path.UTF8String); NSString *result = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil]; NSLog(@"result:%@", result); } int MapFile(const char * inPathName, void ** outDataPtr, size_t * outDataLength, size_t appendSize) { int outError; int fileDescriptor; struct stat statInfo; // Return safe values on error. outError = 0; *outDataPtr = NULL; *outDataLength = 0; // Open the file. fileDescriptor = open( inPathName, O_RDWR, 0 ); if( fileDescriptor < 0 ) { outError = errno; } else { // We now know the file exists. Retrieve the file size. if( fstat( fileDescriptor, &statInfo ) ! = 0 ) { outError = errno; } else { ftruncate(fileDescriptor, statInfo.st_size + appendSize); fsync(fileDescriptor); *outDataPtr = mmap(NULL, statInfo.st_size + appendSize, PROT_READ|PROT_WRITE, MAP_FILE|MAP_SHARED, fileDescriptor, 0); if( *outDataPtr == MAP_FAILED ) { outError = errno; } else { // On success, return the size of the mapped file. *outDataLength = statInfo.st_size; } // Now close the file. The kernel doesn't use our file descriptor. Close (fileDescriptor); } return outError; } void ProcessFile(const char * inPathName) { size_t dataLength; void * dataPtr; char *appendStr = " append_key2"; int appendSize = (int)strlen(appendStr); if( MapFile(inPathName, &dataPtr, &dataLength, appendSize) == 0) { dataPtr = dataPtr + dataLength; memcpy(dataPtr, appendStr, appendSize); // Unmap files munmap(dataPtr, appendSize + dataLength); } } @endCopy the code