Android anonymously shared memory
The familiar IPC methods in Android include sockets, files, ContentProviders, Binders, and shared memory. Shared memory is the most efficient and can achieve zero copy, which is useful in cross-process big data transmission and log collection scenarios. Shared memory is a built-in IPC mechanism in Linux. Android uses this model directly, but makes its own improvements to form anonymous shared memory in Android.
This article will analyze how to use anonymous shared memory through the source code of MemoryFile provided by Android, and implement a simple version of MemoryFile using native layer code.
MemoryFile simple to use
/ / MainActivity. Kt process 1
class MainActivity : AppCompatActivity() {
var mBinder: Binder? = null
val memoryFile:MemoryFile? = null
private var mConnection: ServiceConnection = object : ServiceConnection {
override fun onServiceConnected(className: ComponentName, service: IBinder) {
mBinder = service as Binder
}
override fun onServiceDisconnected(className: ComponentName) {
mBinder = null}}override fun onCreate(savedInstanceState: Bundle?). {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val intent = Intent(this, TestShareMemoryService::class.java)
startService(intent)
bindService(intent, mConnection, Context.BIND_AUTO_CREATE)
}
//1. Create shared memory and pass file descriptors through binder
fun createMemoryFile(view: View) {
// Parameter 1 file name can be null, parameter 2 file size
memoryFile = MemoryFile("test".1024) memoryFile? .apply { mBinder? .apply {val data = Parcel.obtain()
val reply = Parcel.obtain()
val getFileDescriptorMethod: Method =
memoryFile.getClass().getDeclaredMethod("getFileDescriptor")
val fileDescriptor = getFileDescriptorMethod.invoke(memoryFile)
// Serialize to transmit
val pfd = ParcelFileDescriptor.dup(fileDescriptor)
data.writeFileDescriptor(fileDescriptor)
transact(TestShareMemoryService.TRANS_CODE_SET_FD, data, reply, 0)}}}//2. Write data
fun write(data:ByteArray) {
memoryFile.write(data.0.0.data.size); }}Copy the code/ / MainActivity2. Kt process 2
class MainActivity2 : AppCompatActivity() {
var mBinder: IBinder? = null
private var mConnection: ServiceConnection = object : ServiceConnection {
override fun onServiceConnected(className: ComponentName, service: IBinder) {
mBinder = service
}
override fun onServiceDisconnected(className: ComponentName) {
mBinder = null}}fun read(view: View) {
val data = Parcel.obtain()
valreply = Parcel.obtain() mBinder? .apply {// Get the file descriptor passed by MainActivity from the server
transact(TestShareMemoryService.TRANS_CODE_GET_FD, data, reply, 0)
var fi: FileInputStream? = null
var fileDescriptor: FileDescriptor? = null
try {
val pfd = reply.readFileDescriptor()
if (pfd == null) {
return
}
fileDescriptor = pfd.fileDescriptor
fi = FileInputStream(fileDescriptor)
// Read data
fi.read(buffer)
}
} catch (e: RemoteException) {
e.printStackTrace()
} catch (e: IOException) {
e.printStackTrace()
} finally {
if(fileDescriptor ! =null) {
try {
fi.close()
} catch (e: IOException) {
e.printStackTrace()
}
}
}
}
}
override fun onCreate(savedInstanceState: Bundle?). {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main2)
val intent = Intent(this, TestShareMemoryService::class.java)
startService(intent)
bindService(intent, mConnection, Context.BIND_AUTO_CREATE)
}
}
Copy the code//TestShareMemoryService.kt
class TestShareMemoryService : Service() {
lateinit var fd: ParcelFileDescriptor
companion object {
const val TRANS_CODE_GET_FD = 0x0000
const val TRANS_CODE_SET_FD = 0x0001
}
override fun onBind(intent: Intent?).: IBinder {
return TestBinder()
}
inner class TestBinder : Binder() {
override fun onTransact(code: Int.data: Parcel, reply: Parcel? , flags:Int): Boolean {
when (code) {
TRANS_CODE_SET_FD -> {
// Save the file descriptor passed by the process that created the shared memory
fd = data.readFileDescriptor()
}
TRANS_CODE_GET_FD -> {
// Pass the file descriptor to the requesting processreply? .writeFileDescriptor(fd.fileDescriptor) } }return true}}}Copy the codeComb through the process
- Process 1 creates a MemoryFile and writes data to it
- Pass the MemoryFile descriptor to process 2 with Binder
- 3. Process 2 reads and writes data using the obtained file descriptor
There is a question in step 2 of the process, passing the file descriptor from process 1 to process 2. Is the file descriptor the same for both processes?
The answer is that the two file descriptors are not the same, but they both refer to the same file in the kernel.
File descriptor
What are file descriptors in Linux? Before we answer that question, what is a process in Linux?
In Linux, a process is actually a structure, and threads and processes use the same structure, part of the source code is as follows:
struct task_struct {
// Process status
long state;
// Virtual memory structure
struct mm_struct *mm;
/ / process
pid_t pid;
// A pointer to the parent process
struct task_struct __rcu *parent;
// List of child processes
struct list_head children;
// A pointer to file system information
struct fs_struct *fs;
// An array containing Pointers to open files for the process
struct files_struct *files;
};
Copy the codeYou can see that there is a files field in the structure, which records the pointer to the open file of the process, and the file descriptor is actually the index of the array of files, as shown below:
For the sake of illustration, both fd1 and fd2 are written as 1. In fact, when each process is created, the first three digits of files are filled in by default, pointing to the standard input stream, standard output stream, and standard error stream respectively. So by default, the file descriptor for a process is 0 for input, 1 for output, and 2 for error.
How can process 2 generate a file that points to the same file as fd1 using process 1’s fd1?
Recall how we converted FD1 to FD2 using Binder#transact, so Binder source code
//Binder.c
static void binder_transaction(struct binder_proc *proc,
struct binder_thread *thread,
struct binder_transaction_data *tr, int reply) {
switch(fp->type) {
case BINDER_TYPE_FD: {
int target_fd;
struct file *file;
// Get the real file from process 1's fp->handle, which is the only fd pointing to it in the kernel
file = fget(fp->handle);
// Get the file descriptor fd that is not used in the target process
target_fd = task_get_unused_fd_flags(target_proc, O_CLOEXEC);
// Match the file descriptor fd of the target process to the file, so that the target process can find the file through target_fd
task_fd_install(target_proc, target_fd, file);
} break; }}Copy the codeLooking at the source code, we found that the principle is very simple. In fact, Binder in the kernel helps us to convert, because the kernel has all user process information, so it can easily do this.
It should also be noted that file1,file2, and file3 are not necessarily physical files on disk, but can also be abstract files (virtual files). The anonymous shared memory referred to in this article is actually mapped to a virtual file. For this section, take a look at the Linux TMPFS filesystem.
MemoryFile source code parsing
After a brief introduction to the basics of shared memory, let’s take a look at what Android does. MemoryFile is a Java layer anonymous shared memory tool provided by Android that tracks the entire process through its source code.
List of relevant documents:
frameworks/base/core/java/android/os/
- MemoryFile.java
- SharedMemory.java
frameworks/base/core/jni/android_os_SharedMemory.cpp
system/core/libcutils/ashmem-dev.cpp
Copy the code//MemoryFile.java
public MemoryFile(String name, int length) throws IOException {
// create anonymous SharedMemory using SharedMemory
mSharedMemory = SharedMemory.create(name, length);
/ / map
mMapping = mSharedMemory.mapReadWrite();
}
Copy the code//SharedMemory
public static @NonNull SharedMemory create(@Nullable String name, int size)
throws ErrnoException {
// The native layer is actually called to create anonymous shared memory and return the file descriptor
return new SharedMemory(nCreate(name, size));
}
private static native FileDescriptor nCreate(String name, int size) throws ErrnoException;
Copy the code//android_os_SharedMemory.cpp
jobject SharedMemory_nCreate(JNIEnv* env, jobject, jstring jname, jint size) {
const char* name = jname ? env->GetStringUTFChars(jname, nullptr) : nullptr;
// Call ashmem_create_region to create anonymous shared memory
int fd = ashmem_create_region(name, size);
/ /...
jobject jifd = jniCreateFileDescriptor(env, fd);
if (jifd == nullptr) {
close(fd);
}
return jifd;
}
Copy the code// ashmem-dev.cpp
int ashmem_create_region(const char *name, size_t size)
{
int ret, save_errno;
// Open the virtual file corresponding to anonymous shared memory, and finally call __ashmem_open_locked()
int fd = __ashmem_open();
if (fd < 0) {
return fd;
}
if (name) {
char buf[ASHMEM_NAME_LEN] = {0};
strlcpy(buf, name, sizeof(buf));
// The TEMP_FAILURE_RETRY macro definition is used to set the name by ioctl so that false is returned
Ioctl is a system call. The user process interacts with the memory. The internal call copy_from_user gets the data passed by the user process
ret = TEMP_FAILURE_RETRY(ioctl(fd, ASHMEM_SET_NAME, buf));
if (ret < 0) {
gotoerror; }}// Set the size of anonymous shared files
ret = TEMP_FAILURE_RETRY(ioctl(fd, ASHMEM_SET_SIZE, size));
if (ret < 0) {
goto error;
}
return fd;
error:
save_errno = errno;
close(fd);
errno = save_errno;
return ret;
}
static std::string get_ashmem_device_path(a) {
static const std::string boot_id_path = "/proc/sys/kernel/random/boot_id";
std::string boot_id;
if(! android::base::ReadFileToString(boot_id_path, &boot_id)) {
ALOGE("Failed to read %s: %s.\n", boot_id_path.c_str(), strerror(errno));
return "";
};
boot_id = android::base::Trim(boot_id);
return "/dev/ashmem" + boot_id;
}
static int __ashmem_open_locked()
{
// Get the anonymous shared memory path. Android Q uses this method later
static const std::string ashmem_device_path = get_ashmem_device_path(a);if (ashmem_device_path.empty()) {
return - 1;
}
// Open the virtual file used by anonymous shared memory
int fd = TEMP_FAILURE_RETRY(open(ashmem_device_path.c_str(), O_RDWR | O_CLOEXEC));
// On devices prior to Android Q where fd < 0, use the original path "/dev/ashmem"
if (fd < 0) {
int saved_errno = errno;
// Open the virtual file used by anonymous shared memory
fd = TEMP_FAILURE_RETRY(open("/dev/ashmem", O_RDWR | O_CLOEXEC));
if (fd < 0) {
returnfd; }}/ /...
return fd;
}
Copy the codeThis is the process for obtaining anonymous shared memory file descriptors. To summarize the core parts, just give an example before Android Q:
- 1, open (“/dev/ashmem O_RDWR | O_CLOEXEC), open the virtual file
- Ioctl (fd, ASHMEM_SET_NAME, buf), set the name
- 3, ioctl(fd, ASHMEM_SET_SIZE, size), set the size
How do you map to shared memory through file descriptors
In the MemoryFile constructor, SharedMemory#create(name, size) is called to create an anonymous file, and sharedmemory.mapreadonly () is called to map the anonymous file to SharedMemory. Finally, the following method is called:
//SharedMemory.java
public @NonNull ByteBuffer map(int prot, int offset, int length) throws ErrnoException {
// Memory is mapped using the mFileDescriptor and returns the memory address
long address = Os.mmap(0, length, prot, OsConstants.MAP_SHARED, mFileDescriptor, offset);
boolean readOnly = (prot & OsConstants.PROT_WRITE) == 0;
// Cancel the memory mapping Runnable, the run method calls os.munmap (mAddress, mSize);
Runnable unmapper = new Unmapper(address, length, mMemoryRegistration.acquire());
// Use DirectByteBuffer to read and write directly to memory
return new DirectByteBuffer(length, address, mFileDescriptor, unmapper, readOnly);
}
Copy the codeFor those familiar with Linux, the os.mmap () and os.munmap () methods should tell you that memory maps are actually called Linux system functions mmap and munmap, as described in the man manual
mmap, munmap – map or unmap files or devices into memory
- Mmap, which maps files or devices to memory
- Munmap, cancel file or device to memory mapping
In practice, when processes share memory, they do not always read or write a small amount of data and then unmap, but re-establish the shared memory area when there is new communication. Instead, the shared area is kept until communication is complete, so the data content is kept in shared memory and not written back to a file. Content in shared memory is often written back to the file when the mapping is unmapped. Therefore, the use of shared memory communication is very efficient.
Take a look at the official comment, place a connection, open it and see sure enough that mMAP is called
//Os.java
/** * See mmap(2). */
public static long mmap(long address, long byteCount, int prot, int flags, FileDescriptor fd, long offset) throws ErrnoException { return Libcore.os.mmap(address, byteCount, prot, flags, fd, offset); }
Copy the codeAt this point, the core of the MemoryFile source code can be said to be analyzed.
Finally, a quick note on how Linux memory mapping works:
A physical page has a page number. If two processes a and B share 8K memory, for example, the code area is the same, then the page table (linear address to physical address conversion table) of both processes is used. Linux uses the same logical address as linear address) to map the linear address to the same physical page. In fact, there is only one copy of data in memory.
Native implements a simplified version of MemoryFile
Now customize a MemoryFile using the core method:
open(ASHMEM_NAME_DEF, O_RDWR);
mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
ioctl(fd, ASHMEM_SET_NAME, name);
ioctl(fd, ASHMEM_SET_SIZE, size);
munmap((void *) addr, size);
Copy the codeThe first step is to define the API interface, the code is as follows
class MyShareMemory(fd: Int) {
private val mFd: Int = fd
private val mSize: Int
init {
mSize = nGetSize(mFd)
require(mSize > 0) { "FileDescriptor is not a valid ashmem fd"}}// Get an object that can be used with Binder to transfer file descriptors across processes
fun getFileDescriptor(a): FileDescriptor {
return ParcelFileDescriptor.fromFd(mFd).fileDescriptor;
}
companion object {
init {
System.loadLibrary("mysharememory-lib")}fun create(name: String, size: Int): MyShareMemory {
require(size > 0) { "Size must be greater than zero" }
return MyShareMemory(nCreate(name, size))
}
// Create anonymous files that need to be mapped
@JvmStatic
private external fun nCreate(name: String, size: Int): Int
// Get the size
@JvmStatic
private external fun nGetSize(fd: Int): Int
// Close the file and unmap it
@JvmStatic
private external fun nClose(fd: Int)
// Write data, offset is set to destOffset, srcOffset is not set, can be modified, same with nRead
@JvmStatic
private external fun nWrite(fd: Int, size: Int, offset: Int.data: ByteArray): Int
/ / read the data
@JvmStatic
private external fun nRead(fd: Int, size: Int, offset: Int.data: ByteArray): Int}}Copy the codeLet’s implement these five JNI methods
extern "C"
JNIEXPORT jint JNICALL
Java_com_baidu_ryujin_ktc_MyShareMemory_nCreate(JNIEnv *env, jclass clazz, jstring name, jint size) {
char *addr;
int64_t ufd = 0;
const char *_name = env->GetStringUTFChars(name, 0);
// Open the anonymous file and map it. Addr is the address of the mapped memory and ufd is the file descriptor
int ret = create_shared_memory(_name, size, addr, ufd);
env->ReleaseStringUTFChars(name, _name);
return ufd;
}extern "C"
JNIEXPORT jint JNICALL
Java_com_baidu_ryujin_ktc_MyShareMemory_nGetSize(JNIEnv *env, jclass clazz, jint fd) {
return get_shared_memory_size(fd);
}extern "C"
JNIEXPORT void JNICALL
Java_com_baidu_ryujin_ktc_MyShareMemory_nClose(JNIEnv *env, jclass clazz, jint fd) {
char *addr;
// Open is called to map memory to get addr, because unmapping is needed, it is done for convenience, it can be saved in actual use
open_shared_memory(addr, fd);
close_shared_memory(fd, addr);
}extern "C"
JNIEXPORT jint JNICALL
Java_com_baidu_ryujin_ktc_MyShareMemory_nWrite(JNIEnv *env, jclass clazz, jint fd, jint size, jint offset, jbyteArray data_) {
char *addr;
int space = get_shared_memory_size(fd) - offset;
if (size - space > 0) {
return - 1;
}
// As with close, this is to get addr
open_shared_memory(addr, fd);
jbyte *data = env->GetByteArrayElements(data_, 0);
// Write data directly into the shared memory address
memcpy(addr + offset, data, size);
env->ReleaseByteArrayElements(data_, data, 0);
return 0;
}extern "C"
JNIEXPORT jint JNICALL
Java_com_baidu_ryujin_ktc_MyShareMemory_nRead(JNIEnv *env, jclass clazz, jint fd, jint size, jint offset, jbyteArray data_) {
/ /...
return 0;
}
Copy the codeCore implementation code
int create_shared_memory(const char *name, int64_t size, char *&addr, int64_t &fd) {
fd = open(ASHMEM_NAME_DEF, O_RDWR);//#define ASHMEM_NAME_DEF "dev/ashmem"
if (fd < 0) {
return - 1;
}
int len = get_shared_memory_size(fd);
if (len > 0) {// Get the address directly
addr = (char *) mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
return 1;
} else {/ / map
int ret = ioctl(fd, ASHMEM_SET_NAME, name);// Set the name
if (ret < 0) {
close(fd);
return - 1;
}
ret = ioctl(fd, ASHMEM_SET_SIZE, size);// Set the size
if (ret < 0) {
close(fd);
return - 1;
}
// Memory mapping
addr = (char *) mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}
return 0;
}
int open_shared_memory(char *&addr, int64_t fd) {
int size = get_shared_memory_size(fd);
if (size > 0) {
addr = (char *) mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
} else {
return - 1;
}
return 0;
}
int close_shared_memory(int64_t fd, char *&addr) {
int size = get_shared_memory_size(fd);
if (size < 0) {
return - 1;
}
// Cancel the mapping
int ret = munmap((void *) addr, size);
if (ret == - 1) {
return - 1;
}
ret = close(fd);
if (ret == - 1) {
return - 1;
}
return 0;
}
int get_shared_memory_size(int64_t fd) {
return ioctl(fd, ASHMEM_GET_SIZE, NULL);
}
Copy the codeIt is now possible to use custom MemoryFiles for cross-process data transfer just like MemoryFiles, as demonstrated by the Demo CustomAnroidShareMemory on Github.
Finally, I would like to discuss two questions. The following are only my personal reflections. Welcome to add and correct them:
Why Android design an anonymous shared memory, shared memory can not meet the requirements?
First of all, let’s consider the biggest difference between shared memory and Android anonymous shared memory. That is, shared memory usually maps a real file on a hard disk, while Android anonymous shared memory maps a virtual file. This indicates that Android wants to use shared memory for cross-process communication, but does not want to leave files, and also does not want other processes to accidentally open their own process files, so the benefits of using anonymous shared memory are:
- There is no need to worry about the data exception caused by the file being opened by another process.
- No files will be generated on the hard disk. Anonymous shared memory is used mainly for communication, and communication is very frequent. You don’t want to generate a lot of files for communication, or leave files behind.
Why anonymous shared memory? Have you set your name through iOTC?
In my opinion, the problem lies in my misunderstanding of the word “anonymous” before. In fact, anonymity does not mean that there is no name, but that it is impossible to find the actual object through the information on the surface, just like a vest. Anonymous shared memory is exactly the same, although we set the name, another process creates anonymous shared memory with the same name that does not point to the same memory (the code verifies), but the person behind the name has changed. This also answers the question of why anonymous shared memory doesn’t have to worry about being mapped by other processes to read or write data (unless it has its own consent, which is to pass file descriptors through binder to another process).