This article is the second part of the Paging3 source code analysis, which will focus on the implementation principle of RemoteMediator. There are many articles on the web that describe the multilevel data source utility class, but it is somewhat problematic that RemoteMediator does not fully understand the entire request process. This article will look at the implementation of RemoteMediator from a source code perspective and share some tips on RemoteMediator.

Jetpack source code Analysis (5) – Paging3 source code analysis (1)

The main contents of this paper are as follows:

  1. Request process for multilevel data sources.
  2. Analysis respectivelyRemoteMediatorandPagingSourceImplementation details of.
  3. The differences between Refresh, Prepend, and Append operations in multi-level and single data sources.
  4. aboutRemoteMediatorSome tips for use.

References for this article:

  1. Page from network and database
  2. Paging 3 is used for Paging loading
  3. Android Jetpack Database Room

Note that the source code for Paging in this article is from version 3.0.0-alpha08.

1. Request process of RemoteMediator

Compared to a single data source, RemoteMediator has one more process — pulling data from the network and putting it into a database. So in a multilevel data source, how do you get the data from the database to the UI layer to display? This is PagingSource.

Before that, I will explain PaingSource and RemoteMediator to make it easier for everyone to understand, because they work differently.

  1. PaingSource: is used to retrieve data needed by the UI layer, and can only be retrieved from one place, this is calledSingle data source. The data required by the UI layer is obtained through this class, including the multilevel data source,PaingSourceResponsible for retrieving data from the database.
  2. RemoteMediator: The main function is also to get data, but it is from the network (or other places) to get data, and then put into the local database, forPaingSourceGet data from a local database. In particular,RemoteMediatorThe data is not directly displayed in the UI, but stored in the database.

In the meantime, I’ve drawn a diagram to help you understand how the two classes work together. (It’s easy to think that the data from RemoteMediator will also be displayed at the Ui level.)

The entire request process is as follows:

First of all, RemoteMediator will perform a refresh operation on the first request, which will request the first batch of data and put it into the local database. At this time, the corresponding PagingSource will load data from the database. RemoteMediator and PagingSource are used together. The PagingSource request process is similar to the normal request process, which we have introduced in the previous article, interested students can look at: Jetpack source code analysis (5) – Paging3 source code analysis (1). The data retrieved by PagingSource is eventually passed to the Ui layer by sending a PageEvent. But in this process, we have two problems:

  1. How do I notify PagingSource when it finds that there is not enough dataRemoteMediatorKeep asking.
  2. In addition to loading the first page data,RemoteMediatorHow do I load data for other pages, and how do I notify PagingSource when it’s loaded?

These two questions will not be analyzed here. We will focus on these and many other problems. For example, when we introduce PagPresenter, we say that it stores all the data internally, which is not true in RemoteMediator. Also, PagingSource performs Refresh operations even if it is not refreshed manually.

So how does a multilevel data source get data from a database? When we define the Flow inside the ViewModel, we get a PagingSource object directly from the Dao. We didn’t define it ourselves. This PagingSource is actually a LegacyPagingSource, which is an implementation within the Paging3 framework. The entire process of fetching data from the database is set to the LimitOffsetDataSource via the LegacyPagingSource load method. LimitOffsetDataSource is a subclass of PositionalDataSource that internally handles the operation of getting data from a database. In other words, the LegacyPagingSource is really just a Wrapper to smooth out the differences between Paging2 and Paging3.

2. Trigger the RemoteMediator

We know that the RemoteMediator request is made through the load method, so where is this method called? Inside PageFetcher and PageFetcherSnapshot there is no direct call to the RemoteMediator method. Instead, it is assisted by RemoteMediatorAccessor. RemoteMediatorAccessor encapsulates a lot of the call logic of RemoteMediator, including the first load and load more, mainly through the internal launchRefresh and launchBoundary to complete the call to RemoteMediator. Another thing to note about RemoteMediatorAccessor is that the object is only created once on the first refresh, which is very different from PageFetcherSnapshot. Because RemoteMediatorAccessor has a lot of global state that cannot be lost to Refresh.

RemoteMediator triggers two types of requests: Refresh and Append(Prepend). Let’s take a look at their details separately.

(1). Refresh

PageEventFlow = pageEventFlow; pageEventFlow = pageEventFlow; pageEventFlow = pageEventFlow;

if (triggerRemoteRefresh) { remoteMediatorConnection? .let { val pagingState = stateLock.withLock { state.currentPagingState(null) } it.requestLoad(REFRESH, pagingState) } }Copy the code

This code is very simple, but contains a lot of information, three main points:

  1. First, determine whether or nottriggerRemoteRefreshIf the value is true, Refresh is performed. Why do we have to judge this variable? Because if this code calls, it means that Refresh is being performed, but it does notRemoteMediatorData must be refreshed (RemoteMediatorWhen refreshing data, you need to clean up the old data in the database. . In a single data source, there may never be a second Refrsh operation as long as there is no manual Refresh.PagingConfigThe inside of thejumpThresholdField breaks this rule), but inRemoteMediatorPagingSource may trigger Refresh multiple times (triggered by a normal slide) if there is not enough data in the local database, so the above code may be called multiple times. So you have to go throughtriggerRemoteRefreshTo filter the conditions. On the other hand, other places can call PagingSource manuallyinvalidateAnd AdapterrefreshMethod to trigger a refresh, so what’s the difference between the two methods:

(1). Invalidate only means that the current PagingSource is invalid and a new PagingSource will be created. This process does not affect the original existing data. This method generally does not allow external manual calls. (2). Refresh Indicates that all data needs to be cleared and a new request is made. For example, if we do a drop-down refresh, this method will be called.

Refresh triggerRemoteRefresh = true refresh Remoterefresh = true refresh Remoterefresh = true The invalidate method passes false, meaning triggerRemoteRefresh is false. For the sake of clarity and simplicity, THE refresh triggered by refresh is called a complete refresh, and the refresh triggered by invalidate is called an incomplete refresh. 2. Set the PagingState inside PageFetcherSnapshotState to null. This step is mainly to assist the full refresh. During Paging Refresh, the system obtains a Refresh key to determine which data to load. If this key is empty, it is completely refreshed. If it is not empty, it is not completely refreshed. This part of the code is in the getRefreshKey method of LegacyPagingSource. 3. Invoke the requestLoad method of RemoteMediatorConnection to request refreshed data. The requestLoad method is important because this is how RemoteMediator triggers network requests.

Next, let’s take a look at the requestLoad method and get straight to the code:

override fun requestLoad(loadType: LoadType, pagingState: PagingState<Key, Value>) { // 1. Adds a task to the task queue. Val newRequest = accessorstate. use {it. Add (loadType, pagingState)} // Make a network request. if (newRequest) { when (loadType) { LoadType.REFRESH -> launchRefresh() else -> launchBoundary() } } }Copy the code

The requestLoad method does two main things internally:

  1. Use the add method to add a task to the task queue. inRemoteMediatorAccessImplInside, maintained onependingRequestsQueue, which stores tasks of three loadTypes. There are two main things to check when adding: First, check whether the current task queue already has the corresponding LoadType task; Second, whether the current task is in an unlocked state. Only when the two conditions are met, the vm can be added successfully and the next step can be performed.
  2. calllaunchRefreshMethod to make a network request.

Let’s take a look at launchRefresh:

private fun launchRefresh() { scope.launch { var launchAppendPrepend = false isolationRunner.runInIsolation( priority = PRIORITY_REFRESH ) { val pendingPagingState = accessorState.use { it.getPendingRefresh() } pendingPagingState?.let { // Call RemoteMediator's load method to make the network request. val loadResult = remoteMediator.load(LoadType.REFRESH, PendingPagingState) launchAppendPrepend = when (loadResult) {is mediatorResult. Success -> {// Updates the status and removes the related task from the queue. accessorState.use { it.clearPendingRequests() it.setBlockState(LoadType.APPEND, UNBLOCKED) it.setBlockState(LoadType.PREPEND, UNBLOCKED) it.setError(LoadType.APPEND, Null) it. SetError (loadType.prepend, NULL)} false} is MediatorResult.Error -> {// If the request fails, then check whether the queue is Append or PREPEND, If so, then // execute. accessorState.use { it.clearPendingRequest(LoadType.REFRESH) it.setError(LoadType.REFRESH, LoadState.Error(loadResult.throwable)) it.getPendingBoundary() ! = null } } } } } if (launchAppendPrepend) { launchBoundary() } } }Copy the code

Two main things are done in the launchRefresh method:

  1. callRemoteMediatorLoad method. Because the load method is self-defined, we all know what it does, so WE won’t expand it here.
  2. Second, update the corresponding state. If the request is successful, theAPPENDandPREPENDRelease to ensure the normal operation behind; If the request fails, in addition to updating the status, callgetPendingBoundaryCheck whether the current task queue existsAppendandPrependThe task, if any, is called to make the request, that is, to calllaunchBoundary.

(2). Append

For the sake of simplicity, we will only look at the scenario for Append. Prepend is similar to Append, so we won’t repeat it here.

Now that we understand Refresh firing, let’s look at Append firing. How does the Append trigger? After the PagingSource has finished loading data, the Append operation of RemoteMediator needs to be triggered based on the requested data (including Refresh of PagingSource and Append). For example:

if (remoteMediatorConnection ! = null) { if (result.prevKey == null || result.nextKey == null) { val pagingState = stateLock.withLock { state.currentPagingState(lastHint) } if (result.prevKey == null) { remoteMediatorConnection.requestLoad(PREPEND, pagingState) } if (result.nextKey == null) { remoteMediatorConnection.requestLoad(APPEND, pagingState) } } }Copy the code

The Refresh criteria for PagingSource and Append for RemoteMediator are very similar: the nextKey(prevKey) of the requested data is null. If the value is empty, the Append operation of RemoteMediator needs to be performed, that is, new data needs to be pulled from the network. So what does nextKey mean when it’s null?

In general, empty nextKey indicates that the current data has been loaded at the edge of the existing data in the database. At this point, the next page of data must be loaded from the network, otherwise the user will immediately be unable to slide. So why does nextKey indicate that the data boundary has been reached? In Paging2, the data request has a concept called totalCount. If the position of the last data item is equal to totalCount, then the boundary has been reached and nextKey is empty. NextKey here involves the loading process of PagingSource, as well as the loading of LegacyPagingSource and LimitOffsetDataSource. We will not analyze this for now, and the following content will focus on this.

The launchBoundary method of RemoteMediatorAccessor is used to create the launchBoundary function. The launchBoundary method of RemoteMediatorAccessor is used to create the launchBoundary function.

private fun launchBoundary() { scope.launch { isolationRunner.runInIsolation( priority = PRIORITY_APPEND_PREPEND ) { while (true) { val (loadType, pendingPagingState) = accessorState.use { it.getPendingBoundary() } ?: break when (val loadResult = remoteMediator.load(loadType, pendingPagingState)) { is MediatorResult.Success -> { accessorState.use { it.clearPendingRequest(loadType) if (loadResult.endOfPaginationReached) { it.setBlockState(loadType, COMPLETED) } } } is MediatorResult.Error -> { accessorState.use { it.clearPendingRequest(loadType) it.setError(loadType,  LoadState.Error(loadResult.throwable)) } } } } } } }Copy the code

What launchBoundary does is very simple. It calls the load method of RemoteMediator and requests the next batch of data into the database. The operation is basically similar to Refresh, but I won’t analyze it here.

That’s the end of the RemoteMediator trigger process, but let’s do a little summary to keep it in mind.

The internal refresh of Paging3 can be divided into two types: complete refresh (calling the Refresh method of PageFetcher) and incomplete refresh (calling the invalidate method of PageFetcher). The difference between the two refreshes is that RemoteMediator clears existing data and requests data again when the refresh is complete. Not completely refreshing does not. This is mainly controlled through the triggerRemoteRefresh of PageFetcherSnapshot. RemoteMediator refresh is triggered by the requestLoad method of RemoteMediatorConnection, which internally invokes the launchRefresh method, and thus invokes the Load method of RemoteMediator. It is important to note that the requestLoad method is the only way that an external (PageFetcherSnapshot) trigger RemoteMediator to load network data. RemoteMediator loads more of the trigger after the PagingSource request completes, and when it reaches the data boundary, loads the data for the next page using the requestLoad method. The launchBoundary method triggers the load method of RemoteMediator internally. The data boundary is mainly determined by the fact that nextKey is empty, which involves the loading process of PagingSource, which we will analyze in a moment.

3. PagingSource and DataSource loading

As mentioned earlier, in a multilevel DataSource, PagingSource is LegacyPagingSource and DataSource is LimitOffsetDataSource. LegacyPagingSource only acts as a bridge to ensure that the DataSource of Paging2 can be used in Paging3.

Let’s look directly at the LegacyPagingSource load method:

override suspend fun load(params: LoadParams<Key>): LoadResult<Key, Value> { val type = when (params) { is LoadParams.Refresh -> REFRESH is LoadParams.Append -> APPEND is LoadParams.Prepend -> PREPEND } val dataSourceParams = Params( type, params.key, params.loadSize, params.placeholdersEnabled, @Suppress("DEPRECATION") params.pageSize ) return withContext(fetchDispatcher) { dataSource.load(dataSourceParams).run {  LoadResult.Page( data, @Suppress("UNCHECKED_CAST") if (data.isEmpty() && params is LoadParams.Prepend) null else prevKey as Key?, @Suppress("UNCHECKED_CAST") if (data.isEmpty() && params is LoadParams.Append) null else nextKey as Key?, itemsBefore, itemsAfter ) } } }Copy the code

The load method implementation is simple and ends up calling the LimitOffsetDataSource load method. But there is one caveat:

When the data returned by the request is null, the key is null. Empty data indicates that the local database has no more data to load, meaning that it has already loaded to the boundary, so you need to tell RemoteMediator to request more data from the network. In this case, there is a problem: when executing PagingSource finds that there is no more data and it needs to fetch data from the network, how will PagingSource be notified to reload the data when the data request comes back? When Room is initialized, it creates a trigger for our table that listens for updates, inserts, and deletes. When new data is updated to the database, the trigger sends an invalidate notification. This notification calls the Invalidate method of the PagingSource, causing the PagingSource to be created and reloaded. The process of sliding up and down also triggers the Refresh operation of the PagingSource. Those interested in the logic of triggers can look at one of Room’s classes: InvalidationTracker. After the PagingSource is rebuilt, Refresh should be completely new. This is due to the presence of the initialKey, because the pre-creation key is retrieved when the scan method is used, as follows:

    val flow: Flow<PagingData<Value>> = channelFlow {
        // ......
        refreshChannel.asFlow()
            .onStart {
                // ......
            }
            .scan(null) {
                // ......
                @OptIn(ExperimentalPagingApi::class)
                val initialKey: Key? = previousGeneration?.refreshKeyInfo()
                    ?.let { pagingSource.getRefreshKey(it) }
                    ?: initialKey

                // ......
             }
             // ......
    }
Copy the code

The initialKey is obtained primarily through the getRefreshKey method of PagingSource. Internally, the key is computed mainly by the anchorPosition of PagingState, which is either 0 or empty for a full refresh, so it does not connect to previous data.

LoadInitial = load initial = load initial = load initial = load initial = load initial

       internal suspend fun loadInitial(params: LoadInitialParams) =
        suspendCancellableCoroutine<BaseResult<T>> { cont ->
            loadInitial(
                params,
                object : LoadInitialCallback<T>() {
                    override fun onResult(data: List<T>, position: Int, totalCount: Int) {
                        if (isInvalid) {
                           //......
                        } else {
                            val nextKey = position + data.size
                            resume(
                                params,
                                BaseResult(
                                    data = data,
                                    // skip passing prevKey if nothing else to load
                                    prevKey = if (position == 0) null else position,
                                    // skip passing nextKey if nothing else to load
                                    nextKey = if (nextKey == totalCount) null else nextKey,
                                    itemsBefore = position,
                                    itemsAfter = totalCount - data.size - position
                                )
                            )
                        }
                    }
                    // ......
            )
        }
Copy the code

The loadInitial method does two things:

  1. Call anotherloadInitialMethod to get data. thisloadInitialThe way to do that is to get the data from the database, so if you’re interested, you can look at it, but I won’t expand it here.
  2. Return a BaseResult based on the result of the request. What we’re paying special attention to here is whennextKey == totalCount“, returns nextKey, which verifies our previous statement.

At this point, the Refresh loading of PagingSource is complete. I omit the analysis of the Append procedure because the Append procedure is very similar to the Refresh procedure, except that they are calling different methods. Refresh calls the loadInitial method, Append calls the loadRange method, and so on.

Refresh and Append PagingSource Refresh and Append PagingSource Refresh are related to each other. Next, I will continue to answer your questions.

4. Associate Refresh of PagingSource with Append

In a single data source, we all know that a complete load of PagingSource consists of a Refresh + multiple appends + multiple prepends. This is not the case in multilevel data sources, where the full PagingSource loading process is: [Refresh + multiple Append + multiple Prepend] + [Refresh + multiple Append + multiple Prepend]…… .

Note that in a multistage data source, the complete loading of RemoteMediator is Refresh + Append + Prepend. This is not the full process of PagingSource, so be clear.

The complete loading process of PagingSource has been briefly described above, and we will explain it in detail here. Mainly from two aspects:

  1. Refresh + AppendAfter Refresh, local fetch (not only one page of data, but also one page of data) will be performed. We will Append the fetched data to the UI layer one page at a time by sliding down. When the Refresh quantity is consumed this time, that is, the slide reaches the boundary (or the prefetched data has been fully loaded into the UI layer), the nextKey is empty and fires againRemoteMediatorNetwork request (at this timeRemoteMediatorLoadType of is Append.RemoteMediatorAfter the request completes and the update is made to the database, the trigger notifies PagingSource to recreate and Refresh(partially) to start another round of Append. The same is true for Prepend.
  2. Prepend: First, herePrependWe’re not talking about getting new data from the database, we’re talking about getting old data. For example, at the moment, we swipe down to position 200, and when we swipe up, we Prepend (that is, get the previous data from the database). Why is that? Because PagePresenter does not save all data in a multilevel data source (breaking the rule of saving all data, as mentioned earlier), it does at mostinitialLoadSizeSize of data. All other data will be replaced with placeholders, i.ePagePresenterThe inside of theplaceholdersBeforeandplaceholdersAfterWhen needed again, it is reloaded and displayed from the database.

For example, the PositionalDataSource loadInitial method:

                    override fun onResult(data: List<T>, position: Int, totalCount: Int) {
                        if (isInvalid) {
                            // NOTE: this isInvalid check works around
                            // https://issuetracker.google.com/issues/124511903
                            cont.resume(BaseResult.empty())
                        } else {
                            val nextKey = position + data.size
                            resume(
                                params,
                                BaseResult(
                                    data = data,
                                    // skip passing prevKey if nothing else to load
                                    prevKey = if (position == 0) null else position,
                                    // skip passing nextKey if nothing else to load
                                    nextKey = if (nextKey == totalCount) null else nextKey,
                                    itemsBefore = position,
                                    itemsAfter = totalCount - data.size - position
                                )
                            )
                        }
                    }
Copy the code

As we scroll down position becomes larger and larger, so itemsBefore becomes larger and larger. However, our total amount of valid data does not get larger, it is always initialLoadSize. Therefore, when using RemoteMediator, don’t try to fetch data from anywhere, because it might be empty.

5. Tips for using RemoteMediator

Now that we’re done with the source code analysis, I have a few tips for using RemoteMediator.

(1). Do not arbitrarily call the getItem method of Adapter to obtain data at any location

Because PagePresenter only retains valid data of initialLoadSize, and all other locations are filled with NULL, the data obtained by getItem is likely to be empty, causing unnecessary errors.

(2). Set the initialLoadSize of PageConfig as large as possible

Since the Append and Prepend operations of PagingSource consume the initialLoadSize obtained by RemoteMediator during the sliding process, set initialLoadSize to a larger value. You can reduce the number of RemoteMediator requests. Secondly, it is best to set initialLoadSize to an integer multiple of pageSize to avoid page breaking during Append and Prepend.

(3). It is better to set the number of RemoteMediator requests to the same and initialLoadSize

For RemoteMediator, the loadType of each request is different, but the essence is similar. PagingSource finds that the data is insufficient and needs to be added. So each request for the same data, to ensure simple and unified logic. The reference code is as follows:

    override suspend fun load(
        loadType: LoadType,
        state: PagingState<Int, Message>
    ): MediatorResult {
        val startIndex = when (loadType) {
            LoadType.REFRESH -> 0
            LoadType.PREPEND -> return MediatorResult.Success(true)
            LoadType.APPEND -> {
                val stringBuilder = StringBuilder()
                state.pages.forEach {
                    stringBuilder.append("size = ${it.data.size}, count = ${it.data.count()}\n")
                }
                Log.i("pby123", stringBuilder.toString())
                count += state.config.initialLoadSize
                count
            }
        }
        Log.i("pby123", "CustomRemoteMediator,loadType = $loadType")
        return try {
            val messages = Service.create().getMessage(state.config.initialLoadSize, startIndex)
            DataBaseHelper.dataBase.withTransaction {
                if (loadType == LoadType.REFRESH) {
                    mMessageDao.clearMessage()
                }
                mMessageDao.insertMessage(messages)
            }
            MediatorResult.Success(messages.isEmpty())
        } catch (e: Exception) {
            MediatorResult.Error(e)
        }

    }
Copy the code

(4). The PagingState of the load method of RemoteMediator stores data that is not all data

PagingState stores data internally not all of the data, but the data from the last Refresh. Do not try to count the total of all data through this variable. However, it can be calculated using the following code, but it is not 100% guaranteed, as itemsBefore and itemsAfter may be invalid values.

        val pages = state.pages
        var totalCount = 0
        if(pages.isNotEmpty()){
            pages.forEach {
                totalCount += it.data.size + it.itemsBefore + it.itemsAfter
            }
        }
Copy the code

6. Summary

Here, Paging3 source analysis content is over, I made a simple summary:

  1. Under normal circumstances,RemoteMediatorRefresh only once, unless manually Refresh; The PagingSource may Refresh several times, except for the first Refresh initialization, whenRemoteMediatorFetched from the network, PagingSource also Refresh when placed in the database.
  2. In Paging3, there are two types of refresh, incomplete refresh, calledPageFetchertheinvalidateMethod, in which case the data in the local database is not emptied, only added; Flush completely, that is, callPageFetchertherefreshMethod, which clears the local database of data.
  3. The complete loading process of PagingSource in multi-level data source is as follows: [Refresh + multiple Append + multiple Prepend] + [Refresh + multiple Append + multiple Prepend]…… This is different from a single data source.