Android Jetpack DataStore

1. DataStore Introduction

Jetpack DataStore is a new and improved data storage solution that allows the use of protocol buffers to store key-value pairs or typed objects. DataStore stores data in an asynchronous, consistent transactional manner, overcoming some of the drawbacks of SharedPreferences (collectively referred to as SP). DataStore is implemented based on Kotlin coroutines and Flow, and can migrate SP data, aiming to replace SP.

DataStore provides two different implementations: Preferences DataStore and Proto DataStore. The Preferences DataStore is used to store key-value pairs. Proto DataStore is used to store typed objects. The following examples are provided.

SharedPreferences Disadvantages

Before DataStore came along, the most common storage method we used was undoubtedly SP, which was widely praised for its simplicity, ease of use. However, SP is defined by Google as lightweight storage. If less data is stored, there is no problem in using SP. When a large amount of data needs to be stored, SP may cause the following problems:

When SP loads data for the first time, it needs to load the data in full. When the data is large, it may block the UI thread and cause a lag
SP reads and writes files that are not type safe, have no mechanism for signaling errors, and lack transactional apis
Commit ()/apply() operations can cause ANR problems:

Commit () is a synchronous commit, which directly performs IO operations on the UI main thread. If the write operation takes a long time, the UI thread will be blocked, resulting in ANR. Although apply() commits asynchronously, if the onStop() method of Activity/Service is executed, it will also wait for SP to finish writing asynchronously. ANR problems will also occur if the wait time is too long. For apply(), let’s expand:

SharedPreferencesImpl#EditorImpl. Java finally executes apply() :

public final CountDownLatch writtenToDiskLatch = new CountDownLatch(1); public void apply() { final MemoryCommitResult mcr = commitToMemory(); final Runnable awaitCommit = new Runnable() { public void run() { try { mcr.writtenToDiskLatch.await(); } catch (InterruptedException ignored) { } } }; / / 8.0 before QueuedWork. Add (awaitCommit); / / 8.0 after QueuedWork. AddFinisher (awaitCommit); / / asynchronous execution of disk writes SharedPreferencesImpl. Enclosing enqueueDiskWrite (MCR, postWriteRunnable); / /... Other...... }Copy the code

Construct a Runnable task called awaitCommit and add it to QueuedWork, which internally calls the countdownlatch.await () method to perform wait operations directly on the UI thread. It depends on when the task is executed in QueuedWork.

The QueuedWork class is implemented differently on Android8.0 and above:

8.0 queuedwork.java:

public class QueuedWork { private static final ConcurrentLinkedQueue<Runnable> sPendingWorkFinishers = new ConcurrentLinkedQueue<Runnable>(); public static void add(Runnable finisher) { sPendingWorkFinishers.add(finisher); } public static void waitToFinish() { Runnable toFinish; // Fetch the task from the queue: if the task is empty, it breaks out of the loop and the UI thread can continue executing; / / vice task is not empty, take out the tasks and execution, the actual execution CountDownLatch. Await (), namely the UI thread will be blocked waiting for a while ((toFinish = sPendingWorkFinishers. Poll ())! = null) { toFinish.run(); }} / /... Other...... }Copy the code

8.0 queuedwork.java:

public class QueuedWork { private static final LinkedList<Runnable> sFinishers = new LinkedList<>(); public static void waitToFinish() { Handler handler = getHandler(); StrictMode.ThreadPolicy oldPolicy = StrictMode.allowThreadDiskWrites(); ProcessPendingWork (); } finally { StrictMode.setThreadPolicy(oldPolicy); } try { while (true) { Runnable finisher; Synchronized (sLock) {// Remove tasks from the queue finisher = sFinishers. Poll (); } if (finisher == null) {break; if (finisher == null) {break; } // The task is not empty, execute countdownlatch.await (), that is, the UI thread will block waiting for finisher. Run (); } } finally { sCanDelay = true; }}}Copy the code

As you can see, whether before or after 8.0, waitToFinish() attempts to fetch tasks from the Runnable task queue, if any, and executes them directly, looking directly at where waitToFinish() was called:

ActivityThread.java

private void handleStopActivity(IBinder token, boolean show, int configChanges, int seq) { //...... Other...... QueuedWork.waitToFinish(); } private void handleStopService(IBinder token) { //...... Other...... QueuedWork.waitToFinish(); }Copy the code

You can see that the waitToFinish() method is called in both the handleStopActivity and handleStopService methods in the ActivityThread. Both Activity onStop() and Service onStop() wait for the write task to complete before continuing.

So even though apply() writes to disk asynchronously, if onStop() of the Activity/Service is executed at this time, it may still block the UI thread and cause ANR.

Voice-over: ANR problems caused by SP use can be optimized through some Hook means, such as byte published toutiao ANR optimization practice series – farewell SharedPreference waiting. SP used in our company’s project is also optimized according to this, and the optimized effect is relatively significant. Therefore, SP has not been migrated in the project at present (such as to MMKV or DataStore), but it does not affect our learning of new storage posture.

3. Use DataStore

DataStore advantage:

DataStoreData updates are handled in a transactional manner.
DataStoreBased on theKotlin FlowAccess data, default inDispatchers.IOTo operate asynchronously to avoid blockingUIThread, and can occur when reading dataExceptionProcess.
Does not provideThe apply (), commit ()Methods for persisting data.
supportSPAutomatic one-time migration toDataStoreIn the.

3.1 the Preferences DataStore

Adding dependencies

Implementation 'androidx. Datastore: datastore - preferences: 1.0.0'Copy the code

Build the Preferences DataStore

Val context.bookdatastorepf: DataStore<Preferences> by preferencesDataStoreCopy the code

With the above code, we have successfully created the Preferences DataStore, where preferencesDataStore() is a top-level function with the following parameters:

Name: createPreferences DataStoreFile name.
CorruptionHandler: ifDataStoreWhen attempting to read data, the data cannot be deserialized and thrownandroidx.datastore.core.CorruptionExceptionIs executedcorruptionHandler.
ProduceMigrations:SPGenerate transfer toPreferences DataStore.ApplicationContextPassed as parameters to these callbacks, the migration runs before any access to the data.
Scope: coroutine scope, defaultIOOperating inDispatchers.IOExecute in thread.

/data/data/ project package name /files/ creates a file named pf_datastore as follows:

You can see that the suffix is not XML, but.preferences_pb. One thing to note here: Do not write the above initialization code into the Activity, otherwise when you repeat the Activity and use Preferences DataStore, you will try to create a.preferences_pb file with the same name, because it has already been created once. Throws exceptions directly:

java.lang.IllegalStateException: There are multiple DataStores active for the same file: xxx. You should either maintain your DataStore as a singleton or confirm that there is no two DataStore's active on the same file (by confirming that the scope is cancelled).Copy the code

An error class in androidx. Datastore: datastore – core: 1.0.0 androidx/datastore/core/SingleProcessDataStore below:

internal val activeFiles = mutableSetOf<String>() file.absolutePath.let { synchronized(activeFilesLock) { check(! activeFiles.contains(it)) { "There are multiple DataStores active for the same file: $file. You should " + "either maintain your DataStore as a singleton or confirm that there is " + "no two DataStore's active on the same file (by confirming that the scope" + " is cancelled)." } activeFiles.add(it) } }Copy the code

Which is the file via file (applicationContext filesDir, “datastore / $fileName”) generated file, namely the Preferences datastore final address to operation in the disk file, ActiveFiles stores the path of the generated file in the memory. If the activeFiles already exist, an exception is thrown directly, that is, repeated creation is not allowed.

Save the data

First declare an entity class, BookModel:

data class BookModel(
    var name: String = "",
    var price: Float = 0f,
    var type: Type = Type.ENGLISH
)

enum class Type {
    MATH,
    CHINESE,
    ENGLISH
}
Copy the code

Store operations performed in bookrepo.kt:

const val KEY_BOOK_NAME = "key_book_name" const val KEY_BOOK_PRICE = "key_book_price" const val KEY_BOOK_TYPE = Key<T> type object PreferenceKeys {val P_KEY_BOOK_NAME = stringenceskey (KEY_BOOK_NAME)  val P_KEY_BOOK_PRICE = floatPreferencesKey(KEY_BOOK_PRICE) val P_KEY_BOOK_TYPE = stringPreferencesKey(KEY_BOOK_TYPE) } /** * Preferences DataStore to save data */ suspend fun saveBookPf BookModel) { context.bookDataStorePf.edit { preferences -> preferences[PreferenceKeys.P_KEY_BOOK_NAME] = book.name preferences[PreferenceKeys.P_KEY_BOOK_PRICE] = book.price preferences[PreferenceKeys.P_KEY_BOOK_TYPE] = book.type.name } }Copy the code

In the Activity:

lifecycleScope.launch { val book = BookModel( name = "Hello Preferences DataStore", price = (1.. 10).random().tofloat (), // Where the price changes with each click, to show that the UI layer can listen for data changes at any time type = type.math) mbookrebo.savepfData (book)}Copy the code

Through bookDataStorePf. Edit (transform: Suspend (MutablePreferences) -> Unit) suspends and stores a function that accepts a transform block that updates state in the DataStore transactionally.

Take the data

/** * Preferences */ val bookPfFlow: Flow<BookModel> = context.bookDataStorePf.data.catch { exception -> // dataStore.data throws an IOException when an error is encountered when reading data if (exception is IOException) { emit(emptyPreferences()) } else { throw exception <T> val bookName = preferences[preferencekeys.p_key_book_name]? : "" val bookPrice = preferences[PreferenceKeys.P_KEY_BOOK_PRICE] ? : 0f val bookType = Type.valueOf(preferences[PreferenceKeys.P_KEY_BOOK_TYPE] ? : Type.MATH.name) return@map BookModel(bookName, bookPrice, bookType) }Copy the code

In the Activity:

lifecycleScope.launch {
    mBookViewModel.bookPfFlow.collect {
        mTvContentPf.text = it.toString()
    }
}
Copy the code

The return from bookdatastorepf. data is Flow

, which is then used to do a series of processing on the data. IOExceptions are thrown if an error occurs when reading data from a file. You can use the catch() operator before map() and issue emptyPreferences() when the exception thrown is IOException. If another type of exception occurs, rethrow it.

Note: Key< T> and T can only store Int, Long, Float, Double, Boolean, String, Set< String>. This limitation in androidx/datastore/preferences/core/PreferencesSerializer class participation serialized getValueProto () method:

private fun getValueProto(value: Any): Value { return when (value) { is Boolean -> Value.newBuilder().setBoolean(value).build() is Float -> Value.newBuilder().setFloat(value).build() is Double -> Value.newBuilder().setDouble(value).build() is Int -> Value.newBuilder().setInteger(value).build() is Long -> Value.newBuilder().setLong(value).build() is String -> Value.newBuilder().setString(value).build() is Set<*> -> @Suppress("UNCHECKED_CAST") Value.newBuilder().setStringSet( NewBuilder ().addallstrings (value as Set<String>)).build() Else -> throw IllegalStateException("PreferencesSerializer does not support type: ${value.javaclass. Name}")}}Copy the code

As you can see in the last else logic, if it’s not of the above type, it throws an exception directly. Because the Key is the Preferences. The Key type < T >, the default system help us package a layer, is located in androidx. Datastore. Preferences. Core. PreferencesKeys. Kt:

public fun intPreferencesKey(name: String): Preferences.Key<Int> = Preferences.Key(name)

public fun doublePreferencesKey(name: String): Preferences.Key<Double> = Preferences.Key(name)

public fun stringPreferencesKey(name: String): Preferences.Key<String> = Preferences.Key(name)

public fun booleanPreferencesKey(name: String): Preferences.Key<Boolean> = Preferences.Key(name)

public fun floatPreferencesKey(name: String): Preferences.Key<Float> = Preferences.Key(name)

public fun longPreferencesKey(name: String): Preferences.Key<Long> = Preferences.Key(name)

public fun stringSetPreferencesKey(name: String): Preferences.Key<Set<String>> =
    Preferences.Key(name)
Copy the code

Because the above declarations are in the top-level function, they can be used directly. For example, if we want to declare a preference. Key< T>, we can declare it as follows:

val P_KEY_NAME: Preferences.Key<String> = stringPreferencesKey("key")
Copy the code

Migrate the SP to the Preferences DataStore

If you want to migrate SP, add produceMigrations parameter in the Preferences DataStore build link as follows:

//SharedPreference file const val BOOK_PREFERENCES_NAME = "book_preferences" val context.bookdatastorepf: DataStore<Preferences> by preferencesDataStore( name = "pf_datastore", // Migrate SP to Preference DataStore, produceMigrations = {context -> listOf(SharedPreferencesMigration(context, BOOK_PREFERENCES_NAME)) } )Copy the code

The SP file will be deleted from the Preferences DataStore as follows:

Before migration (SP file already exists) :

After creating the Preferences DataStore and performing the migration (the SP file has been deleted) :

3.2 Proto DataStore

One disadvantage of the SP and Preferences DataStore is the inability to define the schema to ensure that the correct data types are used when accessing keys. Proto DataStore can use the Protocol Buffers buffer definition schema to solve this problem.

The Protobuf buffer is a mechanism for serializing structured data. By using protocols, Proto DataStore can know the type of storage and provide the type without using a key.

Adding dependencies

1. Add the protocol buffer plug-in and Proto DataStore dependencies

To use Proto DataStore and have the protocol buffer generate code for our architecture, we need to introduce the Protobuf plugin in build.gradle:

plugins { ... "Version" id "com. Google. Protobuf 0.8.17"} android {/ /... Other Configurations.................. SourceSets {main {java.srcdirs = [' SRC /main/ Java '] proto {// Specify proto source file address srcDir 'SRC /main/protobuf' include '**/*.protobuf'}}} // Proto buffer protocol buffer-related configurations for DataStore protobuf {protoc {// Protoc version See also: https://repo1.maven.org/maven2/com/google/protobuf/protoc/ an artifact = "com. Google. Protobuf: protoc: 3.18.0"} / / Generates  the java Protobuf-lite code for the Protobufs in this project. See // https://github.com/google/protobuf-gradle-plugin#customizing-protobuf-compilation // for more information. GenerateProtoTasks {all().each {task -> task.builtins {Java {option 'lite'}}}} // Modify the location where Java classes are generated by default $buildDir/generated/source/proto generatedFilesBaseDir = "$projectDir/src/main/generated" } } dependencies { api 'androidx. Datastore: datastore: 1.0.0' API "com. Google. Protobuf: protobuf - javalite: 3.18.0"... }Copy the code

The number of libraries that need to be configured or imported seems quite large, so consider putting these configurations into a separate Module.

Define and use protobuf objects

Once you define how the data is structured, the compiler generates source code that makes it easy to write and read structured data. Protobuf: SRC /main/protobuf: proto: SRC /main/protobuf: proto: SRC /main/protobuf

Book.proto file contents:

Syntax = "proto3" must be specified on the first line. // javA_package: specifies the package name of the Java class generated by proto file option java_Package = "org.ninetriPods.mq.study "; // javA_outer_className: Specifies the name of the Java class generated by the proto file. Option javA_outer_className = "BookProto"; enum Type { MATH = 0; CHINESE = 1; ENGLISH = 2; } message Book { string name = 1; Float price = 2; Type Type = 3; / / type}Copy the code

After the above code is written, execute Build -> ReBuild Project, and the corresponding Java code will be generated in the path of the generatedFilesBaseDir configuration, as follows:

3. Create the serializer

The serializer defines how to access the data types we define in the PROto file. If there is no data on disk, the serializer also defines a default return value. Here we create a serializer called BookSerializer:

object BookSerializer : Serializer<BookProto.Book> {
    override val defaultValue: BookProto.Book = BookProto.Book.getDefaultInstance()

    override suspend fun readFrom(input: InputStream): BookProto.Book {
        try {
            return BookProto.Book.parseFrom(input)
        } catch (exception: InvalidProtocolBufferException) {
            throw CorruptionException("Cannot read proto.", exception)
        }
    }

    override suspend fun writeTo(t: BookProto.Book, output: OutputStream) {
        t.writeTo(output)
    }
}
Copy the code

Where, bookproto.book is code generated through the protocol buffer. If the Bookproto.book object or related method is not found, the project can be cleaned and rebuilt to ensure that the protocol buffer generates the object.

Build Proto DataStore

// Construct Proto DataStore val context.bookdatastorept: DataStore<BookProto.Book> by dataStore( fileName = "BookProto.pb", serializer = BookSerializer)Copy the code

DataStore is a top-level function. The following parameters can be passed:

FileName: createProto DataStoreFile name.
serializer: Serializer SerializerDefines how to access formatted data.
CorruptionHandler: ifDataStoreData cannot be deserialized while trying to read, thrownandroidx.datastore.core.CorruptionExceptionCorruptionHandler is invoked.
ProduceMigrations:SPMigration toProto DataStoreWhen the execution.ApplicationContextPassed as parameters to these callbacks, the migration runs before any access to the data
Scope: coroutine scope, defaultIOOperating inDispatchers.IOExecute in thread.

/data/data/ project package name /files/ create a file named bookproto.pb as follows:

Save the data

LifecycleScope. Launch {/ / build BookProto. Book val bookInfo = BookProto. Book. The getDefaultInstance () toBuilder () .setName("Hello Proto DataStore") .setPrice(20f) .setType(BookProto.Type.ENGLISH) .build() bookDataStorePt.updateData { bookInfo } }Copy the code

Proto DataStore provides a suspended function, datastore.updateData (), to store data, and when storage is complete, the coroutine completes.

Take the data

/** * Proto DataStore */ val bookProtoFlow: Flow<BookProto.Book> = context.bookDataStorePt.data .catch { exception -> if (exception is IOException) { Emit (BookProto. Book. GetDefaultInstance ())} else {throw exception}} / / Activity lifecycleScope launch { mBookViewModel.bookProtoFlow.collect { mTvContentPt.text = it.toString() } }Copy the code

The Proto DataStore retrives data in the same way as the Preferences DataStore.

The SP is migrated to the Proto DataStore

// Construct Proto DataStore val context.bookdatastorept: DataStore<BookProto.Book> by dataStore( fileName = "BookProto.pb", serializer = BookSerializer, // Migrate SP to Proto DataStore, produceMigrations = {context -> listOf( androidx.datastore.migrations.SharedPreferencesMigration( context, BOOK_PREFERENCES_NAME ) { sharedPrefs: SharedPreferencesView, currentData: bookProto.book -> String = sharedPrefs.getString(KEY_BOOK_NAME, "") ? : "" val bookPrice: Float = sharedPrefs.getFloat(KEY_BOOK_PRICE, 0f) val typeStr = sharedPrefs.getString(KEY_BOOK_TYPE, BookProto.Type.MATH.name) val bookType: BookProto.Type = BookProto.Type.valueOf(typeStr ?: Currentdata.tobuilder ().setName(bookName).setprice (bookPrice) .setType(bookType) .build() } ) } )Copy the code

Proto DataStore defines SharedPreferencesMigration class. Migrate specifies the following two parameters:

SharedPreferencesView: can be used fromSharedPreferencesRetrieve data from
Bookproto. Book: current data

Similarly, if produceMigrations are passed during creation, SP files will be migrated to the Proto DataStore and deleted after migration.

Here also need to note that the Preferences DataStore, Proto DataStore used to perform the migration during SharedPreferencesMigration classes, but these two places using the corresponding package name is different, Such as Proto DataStore package name path is androidx DataStore. Migrations. SharedPreferencesMigration, when writing them in a file and pay attention to one of them to use full path.

Four,

SP and DataStore:

See the full sample code in this article:Jetpack DataStore sample

Five, the reference

[1] official: Use Jetpack DataStore to store data

1. DataStore Introduction

SharedPreferences Disadvantages

3. Use DataStore

3.1 the Preferences DataStore

Adding dependencies

Build the Preferences DataStore

Save the data

Take the data

Migrate the SP to the Preferences DataStore

3.2 Proto DataStore

Adding dependencies

Build Proto DataStore

Save the data

Take the data

The SP is migrated to the Proto DataStore

Four,

Five, the reference

Related Posts

Here are a few of the Android Studio plug-ins I’m using

Complete the development of Flutter application step by step – Complete the home page of gold digging App

JAVA thread pool