Reflection | the official can not save the day? Design and implementation of Android SharedPreferences

The Reflection blog series is my attempt at a new way of learning. The origins and contents of the series can be found here.

The origin of

Just a few days ago, I had the honor to read HiDhl’s article. After Tencent open source MMKV with similar functions, Google officially maintains Jetpack DataStore component. Does this mean that whether Tencent three parties or Google official perspective, Are SharedPreferences gone for good?

SharedPreferences don’t seem to matter to either MMKV supporters or DataStore fans; It is worth pondering that when I communicated with some of my peers through interviews or other means, I encountered the following situations:

When talking about SharedPreferences and MMKV, most people can talk about the pitfalls of the former and the performance advantages of the latter by orders of magnitude. However, in the detailed discussion of the shortcomings of the former, more profound results are often not obtained. A few problems are listed as follows:

• SharedPreferencesHow is it thread-safe, and what locks does its internal implementation use?
• Does an insecure process cause data loss?
• When data is lost, how is the ultimate barrier – file backup mechanism implemented?
• How is process security implementedSharedPreferences?

Beyond that, there are some architecture-related questions that are also worth thinking about from a designer’s perspective:

• whySharedPreferencesThere will be these defects, how to try to improve these defects?
• Why do you want to reinvent the wheelDataStoreComponent to replace the former?
• makeGoogleEngineers wonder, what is the fundamental reason why these defects still exist today?

To solve some of the deepest puzzles, start with the design and implementation of SharedPreferences itself.

The outline of this paper is as follows:

SharedPreferences

As we all know, Jetpack Security was officially introduced at Google I/O 2019, which is designed to secure files and SharedPreferences. EncryptedSharedPreferences.

Not only that, but the internal implementation of SharedPreferences is slightly different in the source code before and after Android 8.0. Android officials have been trying to “save” SharedPreferences.

So it’s worth taking a fresh look at SharedPreferences before we throw it away for a new solution.

1. Design and implementation: Establish the basic structure

SharedPreferences is a lightweight storage class that is used to store App configuration information. It is an XML file that is stored in the /data/data/shared_prefs directory as a key-value pair.

In the early 2000s, when Android was born, it was a good idea to use XML files to store your application’s lightweight data. Json was just born at that time, and although it was becoming a mainstream lightweight data exchange format, its greater advantage was readability, which I suspect is one of the reasons for using XML instead of JSON.

Now we have built the basic model for this lightweight storage class, using key and value pairs in XML to save the corresponding data to a local file. In this way, each time the data is read, the XML file is parsed to obtain the value corresponding to the specified key. Each time the data is updated, the corresponding value is also updated by the key in the file.

2. Optimization of read operations

In this way, although we have built the simplest file storage system, the performance is not very good. Every time a key value is read, the file must be read again. There is an obvious need to avoid cumbersome I/O operations as much as possible.

So the designer has carried on the simple optimization for read operation, when a SharedPreferences object for the first time in the Context. GetSharedPreferences () to initialize, the XML file on a read, All the contents of the file (that is, all the key-value pairs) are cached in a Map in memory, so that all subsequent read operations only need to be fetched from this Map:

final class SharedPreferencesImpl implements SharedPreferences {
  private final File mFile;             // The corresponding XML file
  private Map<String, Object> mMap;     // The Map cache all the key-value pairs in the XML file
}
Copy the code

The reader can’t help but wonder if this memory caching mechanism, while saving ON I/O operations, poses a risk of high memory footprint when there is a large amount of data in XML.

This is one of the reasons many developers complain about SharedPreferences. So, on both sides of the coin, is the high memory footprint really the designer’s problem?

Not really, because SharedPreferences are designed to be a lightweight storage of data, and even a large number of simple application-like configuration items (such as a Boolean or int) do not have a high amount of built-in usage. For complex data (such as serialized strings of complex objects), developers should use a solution like Room rather than store it all in SharedPreferences.

Therefore, rather than saying that SharedPreferences cause excessive memory usage, I would like to summarize this more objectively:

Although the memory cache mechanism On the surface looks as if he is a kind of trading space for time, frequent actually avoid the short time I/O operations on the impact of performance, and through good code specification, also can avoid the mechanism may lead to the side effects of memory footprint is too high, so this design is worth it for sure.

3. Optimize write operations

For write operations, a series of interfaces are also designed to optimize performance.

We know that updating the key-value pair is done via msharedPreferences.edit ().putString().commit() — what edit() is, what commit() is, Why not just at the beginning of the design of the mSharedPreferences. PutString () interface?

Designers expect that in complex businesses, where one operation sometimes results in multiple updates of key and value pairs, instead of updating the file multiple times, we would prefer to combine these updates into a single write operation to optimize performance.

Therefore, for a SharedPreferences write, the designer abstracts an Editor class so that no matter how many times an operation updates a couple of key-value pairs in the XML by calling putXXX() several times, it only writes to the file when it calls commit() :

// Simple business, one key-value pair at a time update
sharedPreferences.edit().putString().commit();

// Complex business, update multiple key/value pairs at a time, still only one IO operation (file write)
Editor editor = sharedPreferences.edit();
editor.putString();
editor.putBoolean().putInt();
editor.commit();   // Commit () to update the file
Copy the code

With this in mind, the reader should understand that it is often undesirable, and even design-backward, to overlook the design philosophy and usage scenarios of Editor.com MIT () by simply encapsulating the purview of code savings like sputils.putxxx ().

Another point of view to consider is that file I/O is by nature a very heavy operation, and putting it directly in the main thread will cause ANR in some scenarios (such as excessive data), so it makes more sense to put it in a child thread.

So the designer also provides an apply() method for the Editor to synchronize file data asynchronously, and recommends that developers use apply() instead of commit().

Editor+apply() seems to be a big optimization for write operations, but there are more problems, such as child threads updating files, which inevitably raise thread-safety issues; In addition, does the apply() method really avoid ANR as expected? The answer is no, and we’ll come back to that later.

4. Data update & file quantity tradeoff

With the increase of business complexity, we need to face a new problem is that the amount of data in XML files is increasing, and the cost of writing a file is also increasing.

How is the data updated in XML? The reader can simply interpret this as a full update — from above, we know that the data in the XML file is cached in an in-memory mMap, and every time you call editor.putxxx (), you actually store new data in mMap, and when you call commit() or apply(), All mMap data will eventually be fully updated to the XML file.

It can be seen that the size of data in XML does have a certain impact on the cost of write operation. Therefore, designers suggest that data of different business modules be stored in different XML files, that is, data should be stored in different XML files according to business.

Therefore, different XML files should have different SharedPreferences objects. If you want to work on an XML file, you need to pass a different file identifier to get the corresponding SharedPreferences:

@Override
public SharedPreferences getSharedPreferences(String name, int mode) {
  // The name parameter is the name of the file to get the specified SharedPreferences object
}
Copy the code

Therefore, when the XML file is too large, consider breaking it down into several smaller files, depending on the business; But too many small files can also lead to too many SharedPreferences objects that are difficult to manage and confusing. In actual development, developers should balance according to the needs of the business.

Second, thread safety issues

Are SharedPreferences thread safe?

There is no doubt that SharedPreferences are thread safe, but this is only for the finished product. There is clearly a gap in our current implementation.

In order to ensure thread safety, we should not add a lock.

Add a lock? That’s the start! Three locks. Don’t say that’s too many. You have to study the psychology of writing code, and you’re willing to put a lot of extra development into the code, and you don’t care if you add two more.

1. Ensure the readability of complex process code

To make SharedPreferences thread safe, Google’s designers use a total of three locks:

final class SharedPreferencesImpl implements SharedPreferences {
  // 1. Use comments to mark the order of locks
  // Lock ordering rules:
  // - acquire SharedPreferencesImpl.mLock before EditorImpl.mLock
  // - acquire mWritingToDiskLock before EditorImpl.mLock

  // 2. Specify which lock to hold
  @GuardedBy("mLock")
  private Map<String, Object> mMap;

  @GuardedBy("mWritingToDiskLock")
  private long mDiskStateGeneration;

  public final class EditorImpl implements Editor {
    @GuardedBy("mEditorLock")
    private final Map<String, Object> mModified = newHashMap<>(); }}Copy the code

How can you improve the readability of your code for such a complex class? SharedPreferencesImpl makes a good example of this by specifying the order of locking through comments and using the @Guardedby annotation for the locked member.

For simple read operations, we know that the principle is to read the mMap value in memory and return it. To ensure thread safety, we only need to add a lock to ensure that mMap is thread safe:

public String getString(String key, @Nullable String defValue) {
    synchronized (mLock) {
        String v = (String)mMap.get(key);
        returnv ! =null? v : defValue; }}Copy the code

So, can we also achieve thread-safety with a lock for writes?

2. Ensure thread-safe write operations

For write operations, putXXX() does not immediately update the data in mMap, this is of course, if the developer did not call apply(), then the data should be discarded, but if the update directly in mMap, then the data is difficult to recover.

Therefore, the Editor itself should also hold a mEditorMap object to store updates to the data; Only when apply() is called will an attempt be made to merge mEditorMap with mMap for data update purposes.

Therefore, we need another lock to keep mEditorMap thread safe. The reason for not sharing the same lock with mMap is that getXXX and putXXX should not conflict until apply() is called.

Code implementation reference is as follows:

public final class EditorImpl implements Editor {
  @Override
  public Editor putString(String key, String value) {
      synchronized (mEditorLock) {
          mEditorMap.put(key, value);
          return this; }}}Copy the code

When you really need to apply() for write operations, mEditorMap and mMap are merged, then two locks must be used to ensure the thread safety of mEditorMap and mMap, to ensure that mMap can be updated successfully, and finally update to the corresponding XML file.

File updates also require a lock of course:

// SharedPreferencesImpl.EditorImpl.enqueueDiskWrite()
synchronized (mWritingToDiskLock) {
    writeToFile(mcr, isFromSyncCommit);
}
Copy the code

In the end, we made the entire write operation thread safe by using a total of three locks.

Detailed extended space limitations, this article does not carry on the source code, interested readers may refer to SharedPreferencesImpl. EditorImpl class the apply () the source code.

3. An ANR you can’t get rid of

The apply() method is designed to avoid ANR problems caused by main thread I/O operations. Is the ANR problem effectively solved?

No, in this article by bytedance’s technical team, it is clearly stated that a considerable part of ANR statistics in online environments come from SharedPreference. Therefore, it can be seen that apply() does not completely avoid this problem, so what causes ANR?

After our optimization, SharedPreferences are thread-safe, and the internal implementation of Apply () does give I/O to child threads, which is fine in itself, but the reason for that is a different mechanism in Android.

In the apply () method, first create a waiting for locks, depending on the source code version, will eventually to the task of update file QueuedWork. SingleThreadExecutor () a single thread or HandlerThread to execute, when the file after the update releases the lock.

But when the Activity.onStop() and Service handle the onStop related methods, queuedWork.WaittoFinish () is executed to wait for all wait locks to be released, so if SharedPreferences never finish updating, It can get stuck in the main thread and eventually time out and cause ANR.

When does SharedPreferences never complete a task? For example, too frequently and incontinently apply(), resulting in too many tasks, which also illustrates the drawbacks of the rude design of sputils.putxxx ().

Why did Google do this? Here’s a guess from the Bytedance tech team:

Both Commit and Apply generate ANR, but Google hasn’t fixed this bug from the beginning of Android to the current Android8.0, so what’s the problem if we fix it? The only reason Google blocks the main thread to process SP until the Activity and Service call onStop is to keep the data persistent as much as possible. If a crash occurs during the run, the SP is not persisted, and the persistence itself is an IO operation, which will also fail.

It seems that the reason for this defect does have its own design considerations. Fortunately, at the end of this article, there is also a compromise solution, which interested readers can understand, and this article will not go into details.

Third, process security issues

1. How to ensure process security

Is SharedPreferences process secure? Let’s open the SharedPreferences source code and take a look at the comments for the top class:

/ * * *... * This class does not support use across multiple processes. * ... * /
public interface SharedPreferences {
  // ...
}
Copy the code

As a result, SharedPreferences are process insecure because cross-process locks are not used, and there is a possibility of data loss in frequent reads and writes across processes, which is clearly not what we want.

So how do you keep the SharedPreferences process safe?

The implementation of many ideas, such as the use of file lock, ensure that only one process at a time to access the file; Or for Android development, ContentProvider is an officially promoted cross-process component. Other processes can access SharedPreferences through customized ContentProvider, which can also ensure the security of SharedPreferences process. And so on.

Space reason, interested in the realization of readers, can refer to Baidu or at the end of the article reference.

2. File corruption & backup mechanism

SharedPreferences is once again taking on a new challenge.

Because of unpredictable reasons (such as a kernel crash or a sudden power failure), writes to XML files stop abnormally, and the Android file system itself has a lot of protection, but there are still cases of data loss or file corruption.

As a designer, how do you get around this? The answer is to back up the file. Before writing to SharedPreferences, you will first back up the file and rename the initial file to a backup file with a.bak suffix added:

// Try to write to the file
private void writeToFile(...). {
  if(! backupFileExists) { ! mFile.renameTo(mBackupFile); }}Copy the code

After this, an attempt is made to write to the file, and when the write is successful, the backup file is deleted:

// If the data is written successfully, delete the existing backup file immediately
// Writing was successful, delete the backup file if there is one.
mBackupFile.delete();
Copy the code

On the other hand, if the write fails due to an exception (for example, the process is killed), if the backup file exists after the process restarts, the backup file is renamed as the source file, and the incomplete file is directly discarded:

// When the load is initialized from disk
private void loadFromDisk(a) {
    synchronized (mLock) {
        if(mBackupFile.exists()) { mFile.delete(); mBackupFile.renameTo(mFile); }}}Copy the code

Now, with a file backup mechanism, we can guarantee that only the last update will be lost, while previously successfully saved data will remain valid.

Four, summary

Taken together, the issues that have been looked at in SharedPreferences have their own considerations from a design perspective.

As you can see, although SharedPreferences are better overall, why does it still have a significant performance gap compared to MMKV and Jetpack DataStore?

This reason is more comprehensive and complex, even the author is still in the shallow level of understanding, such as the latter two in their data serialization of the more advanced Protocol protobuf, MMKV’s own data incremental update mechanism and so on, if there is an opportunity to start a new article to share.

On the other hand, rather than just defining good and bad between components, I believe that by looking at and learning them dialectically, even SharedPreferences can still be rewarding.

Reference & Thanks

Careful readers should be able to notice that the reference & Thanks section is more and more written by the author, for no other reason, the author never thought that a single article can explain a body of knowledge in all aspects, and this is the same.

Therefore, readers should have the option of looking at other quality content, and even adding a brief introduction to it (since the titles are mostly similar), rather than leaving a bunch of links that start with HTTPS at the end of the article.

As a tribute to the content creators, if you enjoyed this post, I hope you enjoy the following.

1. Please do not abuse SharedPreference @weishu

How do we define good writing? In my opinion, depth and attraction are both indispensable. Depth ensures the durability of the article, and attraction represents the smooth writing skills and structure of the article. This article thoroughly deconstructs the ANR principle caused by Apply () in an easy to understand way. I think it is the best article for advanced learning of SharedPreferences.

2. Android source code analysis SharedPreferences @xiaoweiz

As for how to systematically master the theory of a technology, the author’s understanding is to learn to understand its design idea, improve the whole system structure step by step from scratch, and finally verify each other through the source code.

For SharedPreferences, learn about design ideas, see this article; Source code parsing, see this article.

Android’s SharedPreferences (below) @godliness

The title is similar to 1, but the content is more in-depth, this paper aimed at the file security problem and file backup mechanism under the multi-process source code level analysis, worthy of collection.

4, analysis of SharedPreference Apply caused by ANR problems @bytedance technical team

The problem with ANR caused by the Apply () method is well worth reading.

5. Goodbye SharedPreferences hug Jetpack datastore@hidhl

Some of the bloggers I’ve been following recently have written in-depth articles about SharedPreferences in a systematic way, and it was this article that inspired me to write this article about SharedPreferences.

6. Use ContentProvider to implement SharedPreferences process to share data @king Dragon 123

SharedPreferences @ Trace,

For the SharedPreferences multi-process security implementation scheme, interested readers can read as an extension.

---

About me

Hello, I am the only one who is interested in this article. If you think this article is of value to you, please feel free to follow me at ❤️, or on my blog or GitHub.

If you think the writing is a little bit worse, please pay attention and push me to write a better writing — just in case I do someday.

• My Android learning system
• About article correction
• About Paying for Knowledge
• About the Reflections series