1. Introduction
This section we will revolve around the alipay App build optimizing parsing the qixin series, segmentation and dismantling the client in the “code management”, “certificate management”, “version management”, the construction of the “package” discusses the dimensions of the concrete implementation plan, lead us to learn more about pay treasure in the App under the building blocks of continuous optimization.
This section mainly records the process of rearranging alipay Android Apk files to improve I/O performance.
Background 2.
On the Android platform, due to the rapid launch of a large number of businesses, Android long tail models and other reasons, The performance and experience of Alipay App in the startup stage and on some core links are not ideal, thus affecting the user’s use experience. From a purely business perspective, performance can be improved by optimizing UI layout, code structure, bundle loading, etc. As an engineering team, the conventional wisdom seems to be that you can’t contribute much to performance optimization. After investigating some solutions, we tried to improve the App performance by optimizing the compilation products and intervening in the construction process.
Principle 3.
Before and after the layout, the actual files in Apk do not change substantially, only the location. So why does this tweak have a performance impact? This goes back to Linux’s filesystem mechanism.
As shown in the following figure, there is a Pagecache layer between the VFS App processes in the Linux underlying file system. The Pagecache layer consists of physical pages in the memory, and the contents of the pagecache layer correspond to blocks on the disk. The Pagecache size is dynamic and can be enlarged or shrunk when memory is running out. A Cache’s storage device is called a backing store. A page usually contains multiple blocks, which are not necessarily contiguous.
When the kernel makes a read request (for example, a process makes a read() request), it first checks to see if the requested data is cached in Pagecache. If so, it is read directly from memory, without the need to access the disk. This is called a cache hit. If the requested data is not in the cache, a cache miss, the data must be read from the disk.
The kernel then caches the read data to the cache so that subsequent read requests can hit the cache. A Page can cache only part of a file, rather than the entire file. Caching disk data to improve performance is based on two factors:
-
First, disk access is several orders of magnitude slower than memory (milliseconds to nanoseconds).
-
The second is the data that has been accessed, with a high probability of being accessed again.
In the Android system, every time the upper-layer App reads the disk, the file system will press 16 * 4K block by default to read the data from the disk and put the data into pagecache. If the next file read is already in Pagecache, then real disk IO does not occur. Instead, the file is read directly from Pagecache, greatly improving the read speed. There is a cache there is recycling, pagecache another important work is to free page, thereby freeing up memory space. The task of Cache reclaim is to select the appropriate page to release, and if the page is dirty, write the page back to disk before releasing it.
The ideal would be to release the page that is the longest away from the next visit, but obviously this is not realistic. The LRU improved two-list is a strategy used by Linux. This recycling strategy is very similar to the image loading caching strategy that is common in business development areas. The LRU algorithm selects the most recently visited page, i.e. kills the page that has not been visited recently. The problem with the original LRU algorithm is that some files are accessed only once, but according to LRU’s algorithm, even if they are never accessed again, they are not selected if they have just been accessed.
The two-list policy maintains Two lists, active List and Inactive List. Pages on the active list are considered hot and cannot be released. Only pages on the Inactive List can be released. The page of the data cached for the first time is added to the Inactive List, and the page that is already on the Inactive List is moved to the Active List if accessed again. Both lists are maintained using a pseudo-LRU algorithm, where new pages are added at the end and removed at the head, just like queues.
If the number of pages in the active List is much larger than the inactive List, then the pages in the active List header are moved to the Inactive List to maintain balance between the two tables. Simply put, the purpose of file relayout is to arrange the files needed in the startup phase together in the APK file, and make use of the Pagecache mechanism as much as possible to read as many files as needed in the startup phase with the least disk IO times to reduce IO overhead, so as to achieve the purpose of improving startup performance.
4. Landing plan
After understanding the principle, we need to consider how to use engineering solutions on the Alipay App, mainly from the following three processes to design and implementation.
- Measure:
The premise of relayout must be accurate measurement, targeting the files that can be adjusted and need to be adjusted. This process needs to be accurate enough, otherwise it will result in poor results after the relayout. The ultimate purpose of measurement is to, statistics to alipay startup phase, which files loaded, and is the occurrence of real disk IO, or hit the Pagecache cache. We provide a measurement tool that dumps the I/O behavior of the file system by modifying the kernel source code, and patches the specific Android ROM to measure the boot time file behavior. Some data are as follows:
In the data, the first column indicates the file in which the IO behavior occurred, and the second column indicates the part of the file in which the IO behavior occurred for this offset.
The third column indicates where I/O occurred. If 0, real disk I/O occurred. If it is 1, the content was read from the Pagecache cache.
You can see from the data that some of the files in Apk are actually disk I/OS, and you can try to arrange the files that are used in Apk during the startup phase in the hope that with a small amount of I/OS, you’ll be able to read all of them. The next step is to parse the ZIP package structure to map the file offsets in the above results to the detailed file names. First of all, you need to get the file layout in the installation package, which can be obtained through a tool like 010 Editor. For engineering purposes, you can also refer to the ZIP format definition and analyze the ZIP file through script.
Then, by matching the parse results with the previous statistical results, you can find out which files in the ZIP are read during startup and provide data support for relayout.
- Layout:
After getting a list of files from the startup phase, the second step is to arrange the files together in Apk during the build packaging phase based on this list of files. Here you need to modify the source 7Z compression tool. In alipay construction process, in order to improve the compression efficiency and reduce the package size, 7Z tool was used to compress the Apk at last. The reason for this rearrangement is briefly explained here. Regardless of the type of compression tool, the file order in a ZIP is the default file system order, which is numerically and alphabetically. If you want to specify that files are grouped together, you have to break this rule. To extend a command line parameter, we use the up arrow ‘^’. We can pass in list.txt, and then when 7z executes the output file stream, change the final output order according to the order of the files in the list, so as to achieve the purpose of rearrangement. For example, the following command compresses all files in the source directory and places the specified files in the list at the beginning of the zip package.
7z a -tzip archive.zip source* ^list.txt
In this way, the simple process of file rearrangement is realized. Of course, the construction process of Alipay is more complicated, and a series of processes such as repackaging and re-signature are involved in the process. We’ll talk about that later. Here’s a catch: when we first started reordering files, we measured that it didn’t work very well. It turns out that the original list of files that we adjusted, just during the measurement phase, was all the files that were experiencing disk IO, and we grouped them together in the mistaken belief that if they were adjusted, the overall IO situation would improve. However, if only these files are adjusted, new disk I/O may occur if the files that are previously arranged behind these files are used in the cache through the prefetch mechanism during the startup. The correct tuning method should count all the files in the startup phase in precise time order, and arrange them together so that a small amount of IO occurs, and all of them can be read into the cache. A brief look at the effect of the file adjustment before and after in the main Apk of one experiment is as follows, with several configuration related files moved to the head of the file.
Before the adjustment
After the adjustment
- Regression testing:
After all the documents are adjusted according to all the plans, it is time to verify the effect. There are mainly the following verification methods and ideas:
-
Record the screen offline, and then disassemble the video frame, test the intuitive start time.
-
Use the tool offline to measure THE I/O status, check whether the NUMBER of DISK I/OS decreases during the startup, and quantify a concept of cache miss rate.
-
Under the line, the scheme of buried point and script were used to simulate cold start for several times, and the average value was measured to eliminate possible errors and observe the trend.
-
In other optimizations and similar code, just by adjusting the IO, compare the boot time changes between the two versions. In the experimental stage of relayout scheme, there are two schemes, one and two, and three or four schemes should be used in the subsequent engineering landing and normal optimization.
Evolution of 5.
Through the above landing scheme, after the completion of preliminary experiments in offline and some online grayscale versions, we consider the engineering and normal implementation of this matter. Before engineering, the measurement process is expanded and a simple measurement method is explored.
- Metric optimization:
The original measurement scheme had a deep technical content. In this scheme, it was necessary to be familiar with and understand the underlying file system of Linux, and it was necessary to have the ability to modify the source code. This scheme was implemented under the guidance of other senior experts. In order to keep the overall solution under control, we thought of logging the resource loading process of the Android source code directly, and then analyzing the log directly, so that the file loading is clear at startup. Of course, the defect is obvious, there is no way to determine whether the file is read through disk IO or the Pagecache cache. To intervene the resource loading record, either hook way, or directly change the framework, and swipe a ROM. Considering the factors of engineering automation test, the way of modifying the framework is adopted, so as to facilitate the subsequent test platform, and directly use specific mobile phone to run script execution. The following uses Android 7.0 as an example to describe how to modify drawable and XML processes. If other versions do test measurement model, the modification is similar.
- XML loading process is modified. In parsing XML file process, log directly.
/**
* Loads an XML parser for the specified file.
*
* @param file the path for the XML file to parse
* @param id the resource identifier for the file
* @param assetCookie the asset cookie for the file
* @param type the type of resource (used for logging)
* @return a parser for the specified XML file
* @throws NotFoundException if the file could not be loaded
*/
@NonNull
XmlResourceParser loadXmlResourceParser(@NonNull String file, @AnyRes int id, int assetCookie,
@NonNull String type)
throws NotFoundException {
if(id ! =0) {
try {
synchronized (mCachedXmlBlocks) {
if(! getResourcePackageName(id).equalsIgnoreCase("android")) {
Log.i("AlipayRes"."ResourceId: " + Integer.toHexString(id) + " ResourcePackage name: " + getResourcePackageName(id) + " Loading xml: " + file);
}
final int[] cachedXmlBlockCookies = mCachedXmlBlockCookies;
final String[] cachedXmlBlockFiles = mCachedXmlBlockFiles;
final XmlBlock[] cachedXmlBlocks = mCachedXmlBlocks;
// First see if this block is in our cache.
final int num = cachedXmlBlockFiles.length;
for (int i = 0; i < num; i++) {
if(cachedXmlBlockCookies[i] == assetCookie && cachedXmlBlockFiles[i] ! =null
&& cachedXmlBlockFiles[i].equals(file)) {
return cachedXmlBlocks[i].newParser();
}
}
……
……
}
Copy the code- Drawable modify
/** * Loads a drawable from XML or resources stream. */
private Drawable loadDrawableForCookie(Resources wrapper, TypedValue value, int id,
Resources.Theme theme) {
if (value.string == null) {
throw new NotFoundException("Resource \"" + getResourceName(id) + "\" ("
+ Integer.toHexString(id) + ") is not a Drawable (color or path): " + value);
}
final String file = value.string.toString();
if (TRACE_FOR_MISS_PRELOAD) {
// Log only framework resources
if ((id >>> 24) = =0x1) {
final String name = getResourceName(id);
if(name ! =null) {
Log.d(TAG, "Loading framework drawable #" + Integer.toHexString(id)
+ ":" + name + " at "+ file); }}}if (DEBUG_LOAD) {
Log.v(TAG, "Loading drawable for cookie " + value.assetCookie + ":" + file);
}
if(! getResourcePackageName(id).equalsIgnoreCase("android")) {
Log.i("AlipayRes"."ResourceId: " + Integer.toHexString(id) + " ResourcePackage name: " + getResourcePackageName(id) + " Loading drawable: "+ file); }... ... }Copy the codeBrush into the ROM, replace the modified framework, cold start Alipay, clear cache, through log filtering can get a complete startup file loading list.
adb shell am force-stop com.eg.android.AlipayGphone
adb shell
echo 1 > /proc/sys/vm/drop_caches
Copy the code
- Engineering:
So the single point capability is basically available and once you have the single point capability, you need to find a solution that can automate it as much as possible. The specific flow chart is as follows. The ReApk (Optimized Apk) process can then be extended to other build build product optimizations.
6. Results and Prospects
At present, the overall scheme has been launched in The Android App of Alipay Wallet. This single item, startup performance, has an optimization effect of about 5% under the overall number of users, and the effect is more obvious on low-end computers. According to different models, the startup performance can be optimized by about 10%.
Facebook’s tool chain optimization solution, Redex, has developed a whole set of open source solutions for dex optimization, from measurement to regression testing. As for the relayout of ZIP, we hope to make this whole set of solutions as “out of the box” as possible in the future, enabling more apps inside and outside the company.
7. Summary
In this section, we have a preliminary understanding of how Alipay optimizes I/O performance by rearranging installation packages on The Android client. Due to space constraints, we cannot cover many technical points. And the corresponding technical kernel, we also applied in mPaaS and output, welcome to start the experience:
Tech.antfin.com/docs/2/4954…
We are also looking forward to your feedback on the design ideas and specific practices of Android startup performance optimization. Welcome to discuss and communicate with you.
Past reading
The opening | modular and decoupling type development in ant gold mPaaS theorypractice probing depth,
“Practice of Alipay Mobile Terminal Dynamic Solution”
Alipay Client Architecture Analysis: A Preliminary Study of iOS Container Framework
“Alipay Client Architecture Analysis: A Preliminary Study of Android Container Framework”
Alipay Client Architecture Analysis: Android Client Startup Speed Optimization of “Garbage collection”
