LOFTER App Offline package solution and related performance analysis

1. The background

H5 pages in an APP are usually hosted in a WebView, and page presentation is highly dependent on the network environment. The off-line package scheme aims to reduce the impact of network on page performance with low compatibility cost, while retaining the advantages of H5 itself.

The offline package packages the page resources, including HTML, JS and CSS, into compressed packages, the APP is pre-downloaded to the local, and the WebView directly loads static resources from the local, so as to reduce the impact of the network environment on the page loading. Offline packages focus on the white screen process of establishing links -> accepting pages/styles/scripts.

Front-end engineers can use the offline package plug-in to compress page resources and upload them to the offline package configuration platform. Through the configuration platform, associated applications can be managed, deployed, published, and viewed.

LOFTER APP has a lot of H5 applications, such as fair, certification, customer service message, etc., we hope that by offline package faster page will be visible to the user, improving the user experience, such applications have relatively stable as a whole, need more appropriate way to access the offline package, we plan to do so for the purpose to carry on the design and implementation.

2. Overall offline package solution

The overall scheme design is as follows:

The front-end application (Web) maintains the original development and deployment process, modifying the packaging process only through the WebPack Plugin, establishing a connection with the configuration platform, and generating a compressed file to upload to NOS. This makes it easy for existing H5 applications to have offline package capability, and for subsequent offline applications to also have static release capability.

The Offline package Configuration Platform is used to manage offline package applications, including adding, deleting, modifying and checking applications, version management, Configuration viewing, setting and checking updated URLS, and other functions. The Configuration generated by the WebPack Plugin is stored in the library through the interface. Each version update is stored and takes effect through the online operation.

The APP back end (APP Server) provides an interface for querying offline package configurations on the APP side. It also includes whether to enable or disable ios and Android offline packages. Offline package applications that take effect on the configuration platform are provided to apps in a list.

The client (APP) obtains the offline package list through the interface. If an application version needs to be updated, delete the local package resource before updating it. Through the overall consideration, we do not do incremental update and differential package maintenance at present. In order to reduce the update problems caused by excessive application volume, we provide subcontracting configuration, which can realize offline step by step. In the WebView with the offline package function enabled, reverse proxy is performed for all resource requests. If the resource is cached locally, use the local cache. If not, request the online resource. The client can check for configuration updates as follows: 1. Pull the full offline package to apply the configuration during startup. 2. Matches the updated URL configured on the platform.

2.1 Offline Package Application Configuration List

The following is an example:

[{" appId ":" appId offline package application ", "name" : "offline application package name", "version" : "Offline package application version ", "updateTime":" offline package application configuration item updateTime, Can be used to determine whether need to override configuration items ", "refreshUrls" : (" www.lofter.com/mp/lofter878789/html/page1.html "), "packages" : [{"packageName":" quietly ", "quietly ":true, "dependencies":["common trunk "], "urls": [ "//www.lofter.com/market/page1.html", "//www.lofter/market/page2.html" ], "zipUrl": "https://xxx.nosdn.127.net/offline/xxxx.zip", "md5":"xxxxxxxxxxxxxxxx" },{ "packageName":" package_2", "Dependencies ":["common Trunk package "], "urls": [ "//www.lofter.com/market/page3.html", "//www.lofter/market/page4.html" ], "zipUrl": "Https://xxx.nosdn.127.net/offline/xxxx.zip", "md5" : "XXXXXXXXXXXXXXXX"}, {" packageName ":" common trunk package ", "loadingQuietly":true, "zipUrl": "https://xxx.nosdn.127.net/offline/xxxx.zip", "md5":"xxxxxxxxxxxxxxxx" } ] } ]Copy the code

Main configuration items:

• RefreshUrls define the urls that need to be checked for version updates when matched.
• Urls define links that require offline package downloads to trigger offline package downloads when not loaded silently.
• LoadingQuietly defines whether the current package needs to be silently loaded. The APP loads the package that needs to be silently loaded first after obtaining the configuration.
• Dependencies Defines the dependencies between packages. The client needs to download the dependent package first.
• ZipUrl is a compressed package of resource files and mapping files, which also generates MD5 for client verification.

2.2 Offline Package Plug-in

Offline package plug-in workflow is as follows:

Before webPack generates the output file, generate the mapping file manifest.json according to the resource information, upload the resource bundle to NOS for APP download, and submit the offline package application related information program.json to the offline package configuration platform. Subcontracting is also provided in the form of configuration, where unconfigured resources are automatically divided into a package. The configuration of additional static resources is provided so that if a page needs to import a third-party resource at high priority, a local copy can be saved and the access link associated with the resource can be read by the plug-in, thus avoiding the impact of such resource request time on page loading.

Use mode:

const OfflinePackage = require('lofter-mp-webpack-plugin'); const offlineEntry = getEntry(); module.exports = { ... plugins: [ new HtmlWebpackPlugin(), new OfflinePackage({ packageNameValue: 'test', appId: 'lofter1', uploadConfigUrl: 'http://www.lofter.com/mp/api/appVersion/add', baseUrl: `//www.lofter.com/mp/lofter1/`, cdnUrl: `//xx.cdn.com/mp/lofter1/`, nosConfig: nosConfig, ignoreFileTypes: ['txt', 'js.map'], excludeFileName: ['page2'], entry: offlineEntry, extraManifest: { "static/jquery.min.js": "/ / cdn.bootcss.com/jquery/2.2.4/jquery.min.js"}, dividePackageConfig: [{loadingQuietly: true, entryFile: ['index'], },{ loadingQuietly: false, entryFile: ['page1'], }] }) ] }Copy the code

Need to be configured after the HtmlWebpackPlugin, easy to directly fetch after the processing of resources.

2.3 Client Implementation

Android forms offline package SDK, WebView registered SDK to have the ability to intercept resources, through the shouldInterceptRequest method to intercept WebView resource address, and the existing offline package needs to intercept the URL match, if not match, do not intercept. Directly access remote resources. Otherwise, use WebView to obtain resources. After downloading a compressed package containing static resources from a remote CDN, decompression is not necessary. You can directly read the files in the package.

IOS currently uses NSURLProtocol to intercept WKWebView network request, but it will lead to the problem of body loss of POST request. Hook JS code is injected to do some processing of fetch and XMLHttpRequest. In the actual debugging, We also completed Headers and implemented a simple same-origin policy. We are still looking for a better implementation, otherwise as business complexity increases, we will inevitably implement a complex custom browser on the client side.

When using the App to verify the offline package function, it is impossible to effectively judge the load of resources only by capturing packets. Here, Android students provide a log filtering tool, and you can view the logs of offline package intercepting on the computer after connecting the mobile phone. The debugging floating layer of iOS also adds the logs of offline package resource matching. In addition, we agreed that the JSBridge that collects web logs should be added to the log file of the app through active invocation by the Web for troubleshooting.

3. Statistics on the offline package effect

Here, we analyze the effect of the application of Lehu Market. Lehu Market is a multi-page H5 application with a large number of users in the site. It has been connected to the WAPM platform of cloud Music for performance monitoring before the offline package is launched, and the changes of the page performance indicators can be observed with the offline package function. The offline package feature of the client WebView container came online in version 6.11.1. We selected all versions 6.10.0 to 6.10.3 as a comparison. Statistics are collected based on average DNS time, average TCP time, Average DomReady time, and average load time.

This is the calculation method of indicators and is collected and reported by the SDK of WAPM.

The statistical results are summarized as follows:

After graphical processing, the statistical figure of percentage accumulation is obtained:

As you can see, the DNS time and TCP time are significantly reduced at both ends. Dom Ready and load times have also been reduced, with Android as a whole seeing an average drop of 30%. It should be noted that the home page HTML is part of SSR, currently go is online resources, so even if the data decreased, compared with other pages in front of the two indicators did not tend to 0.

For the majority of pages that get primary data through asynchronous interfaces, actually using WebVitals metrics can better show the real experience on the user side (WebVitals introduces portal). Through data collection and analysis, offline package can effectively reduce the time of FCP (first content rendering) and LCP (maximum content rendering), and has little impact on CLS (cumulative layout displacement) and FID (first input delay). Since only browsers based on Chromium kernel can collect WebVitals indicators at present, only the data of Android terminal will be displayed here.

After graphical processing, the bar statistical chart is obtained:

As you can see, the offline package is significantly optimized for FCP and LCP. It can be seen from the table that the absolute value optimization of offline packages in P95 statistics is mostly higher than that of P75, indicating that the fluctuation of overall data can be reduced and the experience optimization of some users with slow loading can be more effective.

4. Analyze offline package loading

According to the statistics, loading static resources from the local server can improve performance indicators. Switching the main static resources to the local server can significantly reduce the time of DNS and TCP, which is expected. The difference between DomReady and page load completion time is quite surprising. Here we look at how the page performs when the offline package application is loaded in debug mode, and how the non-offline package mode saves time.

Because the data of Android terminal is relatively stable, this time we will focus on the analysis of offline applications of Android WebView, and select the home page and business page which are more representative as examples.

The device is a Huawei Glory 9, Android 9, running memory 6GB, processor Hisilicon Kirin 960, network environment: Unicom 4G, connected to the phone using performance module for loading analysis.

4.1 Analysis of the market home page

The data on the first screen of the market home page is pre-filled into HTML, and is not stored as offline resources. Currently, the main JS and CSS resources are stored offline.

Here is the overall loading timeline for the page:

As you can see, for resources loaded locally, there is only waiting time and file read parsing time.

After homepage.js is loaded, the browser performs the long task of building a page, and DomReady is done. In the offline package scenario, this long task takes about 371.7ms. In the case that the load-time of homepage.js is reduced by the offline package, the DCL time is moved forward.

Assuming that the user-side network environment is general, Fast 3G network environment is used to simulate:

The external JS with higher priority blocks the rendering process. Even though homepage. JS has been loaded, the rendering task does not start until later.

Compare how the page loads without offline packages:

Fast 3G simulation environment:

Compared to many cases where the long task time of building the DOM is similar, the time spent on several high priority resource requests directly determines the time point of the DCL.

As for the homepage, the optimization of the loading time of offline package is mainly the reduced loading time of home.js itself or the loading time difference between it and the external resources with high priority (index.js or index.css), that is, the part circled in the red box in the figure, which is about several hundred milliseconds, which is consistent with the statistics.

4.2 Detailed page analysis of marketers

The business page of the best seller list was selected as the test page.

Because of the skeleton screen, you can see that both FP and FCP are at a very early point in time.

Simulation of Fast 3G network environment:

The loading time of offline resources is similar in different network environments. Similar to the loading effect of the home page, the DCL time point is significantly delayed due to the high priority resources blocking the page rendering.

As can be seen from the figure, the time point of DCL and Onload is often formed before the main interface returns, and the page information is not completely displayed at this time. The LCP indicator can better reflect the real rendering situation of the page, and the statistics also show the optimization of the LCP indicator by the offline package.

5. To summarize

In the offline package scenario, all external resources with high priority should be loaded as offline resources to avoid blocking the main rendering process. The offline packet scheme can effectively reduce the DNS and TCP time of the first page access. In the case of extremely complex user-side network environment, improve the stability of the main resource loading, so that containers can perform rendering tasks earlier.

This article is published by the front end team of Netease Yuanqi Business Department. Any unauthorized reprint of the article is prohibited. Welcome to exchange with us front-end related technical problems and experience, at the same time, the team and departments are recruiting front-end, server and client each position of the developer, the above can contact [email protected] for communication.