This paper mainly introduces the research and practice of Dewu App in improving the compilation speed of Android project.Copy the code

background

With the rapid development of Dewu App business, the number of code and components of Android projects increases rapidly, and the compilation time of projects also increases significantly. At the beginning of this year, the average incremental compilation time was close to 3.9 minutes, which seriously affected the development efficiency and prompted us to explore various measures to shorten the compilation time and improve the development efficiency.

In early April, the time was reduced to 2.3 minutes through a series of routine optimizations, such as incremental annotation processor modification, incremental Transform, componentization, engineering, and optimized project configuration. In June, the first version of the Wade Plugin went live, further shortening it to 1.3 minutes. In August, Wade’s second major release was released, eventually reducing the add-on time to 0.8 minutes. This paper mainly introduces the technical principle and implementation idea of Wade Plugin.

Introduction to the

Wade Plugin is an Android Gradle Plugin designed to improve compilation speed. After the exhaustion of conventional optimization methods, the add-ons of the project still took 2.3 minutes, in which DexArchiveBuilder, MergeProjectDex, MergeLibDex and MergeExtDex accounted for 1.7 minutes. It is not difficult to see that optimization space of DexBuild and DexMerge should be explored if time consumption is to be further reduced.

Wade Plugin replaces the native DexBuildTask with WadeDexBuild and the native DexMergeTask with WadeDexMerge by Hook Android native compilation process. Native DexBuild takes an average of 60 seconds, while WadeDexBuild takes 12 seconds. Native DexMerge takes 42 seconds on average, WadeDexMerge takes 2 seconds.

WadeDexBuild

The principle of

The process of converting Class to Dex by calling Dx or D8 is called DexConvert, which takes up most of the time of DexBuild. Native DexBuild performs DexConvert in the granularity of Jar and Class. On average, a Jar contains 200+. Classes in the acquisition project, which means that every change of a class will trigger 200 classes to perform DexConvert.

Ideally, only modified classes participate in DexConvert.

Optimization scheme

Before DexConvert is executed, unzip the Jar and execute DexConvert with.class granularity. And only the changed. Class is involved, and the unchanged. Class execution results are reused from the previous compilation cache. There are four types of change corresponding to the cache reuse policy: for the new. Class, you need to participate in DexConvert; Modified. Class participates in DexConvert; Removed. Class does not participate in DexConvert; What has not changed does not participate.

In this case, the removal of.class is treated specially. For example, after demo. class is removed, in addition to the corresponding product demo. dex, we also need to find its inner class product Demo$1.dex, Demo$2.dex, etc.

The main implementation

Input (Task Inputs)

Input the.class file path for WadeDexBuild according to the Consumer Transform Inputs Transform. When a Consumer Transform is connected, its output path is compiled. If there is no Consumer Transform, the output paths of Java Compile and Kotlin Compile are compiled. If there are multiple Consumer Transforms, take the output path of the last Transform as the DexBuild input.

Incremental compilation trigger conditions

Triggering conditions determine whether this compilation is incremental and whether the cache from the previous compilation is available. The incremental conditions of WadeDexBuild include 28 in 5 categories (AGP3 and AGP4 are slightly different):

  • Gradle configuration
  1. AndroidJarClasspath
  2. DesugaringClasspathClasses
  3. ErrorFormatMode
  4. MinSdkVersion
  5. Dexer
  6. UseGradleWorkers
  7. InBufferSize
  8. Debuggable
  9. Java8LangSupportType
  10. ProjectVariant
  11. NumberOfBuckets
  12. DxNoOptimizeFlagPresent
  • Wade configuration
  1. WadeExtension.scope
  2. WadeExtension.duplicateClass
  3. WadeExtension.dexBucketSize
  4. WadeExtension.jarBucketSize
  • Wade cache
  1. ProjectWorkspaceDir
  2. SubProjectWorkspaceDir
  3. ExternalLibWorkspaceDir
  4. MixedScopeWorkspaceDir
  • The input file
  1. ProjectClasses
  2. SubProjectClasses
  3. ExternalLibClasses
  4. MixedScopeClasses
  • Product files
  1. ProjectOutputDex
  2. SubProjectOutputDex
  3. ExternalLibOutputDex
  4. MixedScopeOutputDex

Gradle configuration conditions are similar to native trigger conditions.

For example, if the Dexer in Gradle configuration is changed from D8 Dexer to DX Dexer, the last compilation cache cannot be reused and a complete compilation is required.

Incremental compilation is triggered when, for example, a Kotlin class is modified so that the MixedScopeClasses in the input file changes and the compile cache should be reusable.

Dex Convert

The key step in WadeDexBuild is to Convert the native Dex Convert from Jar to Class granularity. First, unzip the Jar, and write the decompressed. Class files into the cache directory. Then compare the classes that participated in the last compilation with the classes that participated in this compilation one by one.

Performance optimization

After the granularity of Dex Convert is converted from Jar to Class, the time is significantly reduced. However, there are 423 jars in the project and 83000+ classes after decompression, leading to the two steps of decompression and file comparison before Dex Convert being very time-consuming. There are three main aspects to the optimization of these two steps.

  • ForkJoinPool

ForkJoinPool is used instead of traditional ExecutorService concurrency because its Work Steeling algorithm is ideal for small-file, high-task scenarios that maximize CPU idle time.

  • mmap

File comparison is an I/O intensive task, and the read/write speed of common file flows is slow. Wade Plugin All I/O operations are implemented with MMAP, including read, write, copy, etc. Replacing file streams with Mmap has a noticeable effect on overall speed.

  • Replace the MD5 CRC – 32

A common way to compare two files is to compare the length of the files and then check whether the MD5 of the files is the same. Due to the large number of classes, the calculation of MD5 takes considerable time. Using the CRC-32 algorithm to compute the file Hash as Checksum instead of MD5 can reduce the time for file comparison.

The reliability of the Checksum calculated by CRC-32 is not as good as that calculated by MD5. In theory, there will be Hash collisions, which will cause the modified Class to be misjudged as unmodified, and then the cache will be used instead of the latest file to compile, which means that the modified Class is invalid. However, the actual probability of occurrence is very low, so it is worth sacrificing theoretical correctness to ensure the efficiency of each compilation.

  • The optimization effect

The average time of decompression and write cache after optimization is 5700ms. The file comparison time is only 10ms thanks to the CRC-32 algorithm. The overall time of DexBuild is reduced from the original 60 seconds to 12 seconds.

WadeDexMerge

Optimization scheme

DexMerge Combines. Dex files to reduce the number and volume of dex files in the APK and improve the installation speed and initial operation speed. The disadvantage of native DexMerge is that it does not support incremental compilation, and the time required is proportional to the number of Dex files. DexMerge takes between 30 and 60 seconds to generate an item.

DexMerge may not be performed for projects with a small amount of code and a small number of classes. AGP also automatically skips the DexMergingTask. If MinSDKVersion is greater than 23, DexMerge is not executed when the number of Dex is less than 500. If MinSDKVersion is less than 23 and the number of Dex is less than 50, DexMerge is automatically skipped.

The Hook DexMergingTask can forcibly skip DexMerge by ignoring the Dex quantity threshold of AGP. However, for projects with a large number of Dex, forcibly skipping DexMerge has obvious side effects. Forcibly skipping DexMerge on the object Acquisition App will increase the package volume by about 40M, increase the APK installation time by 15 seconds, and increase the initial startup time by about 10 seconds.

WadeDexMerge allows you to forcibly skip DexMerge and incremental Merge. By default, incremental Merge is used. It is easy to skip the DexMerge implementation, just note that the subsequent PackageTask only recognizes the. Dex file, not the. Jar file, first process the. Jar file in the DexBuild product, and then copy it with the. Dex product into the inputDir of the PackageTask. The inputDir can through reflection PackageAndroidArtifact. GetDexFolders (). This section mainly introduces the implementation of WadeDexMerge incremental compilation.

The main implementation

DexMerge input files include. Jar and. Dex, and output. Dex files. The core of incremental implementation is to Merge input files into buckets, only Merge changed buckets, and reuse the cache of other buckets.

If only one file in Bucket0 changes and none of the other buckets change, Merge Bucket0. After bucking, you need to find out which files have changed since the last compilation and what types of changes they have made. This scenario is similar to the classic algorithm question “How do I find the different elements in two arrays?” , so you can use the fast or slow pointer to calculate file changes.

As shown in the figure, the slow pointer points to the array of files compiled last time, while the fast pointer points to the array of files compiled this time. Compare the files with the two Pointers. If the two Pointers are the same, the fast pointer points to the next file until the difference is found. The pseudocode is as follows:

long fast = 0 long slow = 0 while (slow < prev.size()) { long temp = fast while (temp < curr.size()) { if (prev[slow] ==  curr[temp]) { break } temp++ } if (temp ! = curr.size()) { fast = temp boolean isModified = isModified(prev[slow], curr[fast], reuseScope) if (isModified) { //found difference fileChanges.add(new DefaultFileChange(prev[slow], ChangeType.MODIFIED)) } } else {//not found fileChanges.add(new DefaultFileChange(prev[slow], ChangeType.REMOVED)) } slow++ }Copy the code

The total number of buckets and Bucket Size directly affect the incremental effect. Theoretically, the more buckets, the better. If you have 100 buckets, the full Merge takes 1/100 of the incremental time. However, more buckets means more dex in APK, which will affect package volume, installation time, and initial startup time. After several tests, the combined effect is best when the total number of buckets is 50-100, and the time of Merge is significantly reduced with few side effects. At present, there are a total of 66 Bucket projects, including 23 Jar types and 43 Dex types.

High availability

In the aspect of high availability construction, data statistics, compilation monitoring, compilation index weekly report were used to timely obtain the market situation and find problems; Compatibility with different AGP and Gradle versions to improve plugin compatibility; Continuously monitor compilation exceptions and iterate to fix problems to improve stability.

The seven indicators

Seven metrics reflect the team’s overall build performance:

  • Incremental compilation time
  • Average compilation time
  • Full compilation time
  • Incremental compilation takes 50 qubits
  • Incremental proportion
  • Compilation success rate
  • Total compilation time per capita

The calculation of indicators depends on the buried point data, and the values of some fields in the buried point are difficult to obtain. For example, whether the JavaCompileTask compiled this time is incremental needs to be realized by AGP and Gradle piling. There are three Hook points to cut into.

Wade’s early version used scheme 1, and found poor compatibility of Hook Gradle classes in actual use. Currently use 2, Hook AGP com. Android. Build. Gradle. Tasks. JavaCompileCreationAction class, Injection WadeJavaCompile class instead of the original org.gradle.api.tasks.com running. JavaCompile class. Javaccompile WadeJavacCompile is a JavaCompile wrapper class, rewrite compile() to the Javac identify. isIncremental The pseudocode is as follows:

public class WadeJavaCompile extends JavaCompile { ... private static File mFile; @Override protected void compile(IncrementalTaskInputs inputs) { ... boolean isIncremental = inputs.isIncremental(); try { FileUtils.writeStringToFile(mFile, "isIncremental:" + isIncremental + "\n", true); } catch (IOException e) { ... } super.compile(inputs); }... }Copy the code

The Hook process for AGP native classes can be roughly divided into three steps: get Gradle’s VisitableURLClassLoader, edit the bytecode of the target class with ASM or Javassist, and load the edited bytecode with a reflex call to classLoader.defineclass ().

Gradle processes and Gradle daemons usually live in the background. When Android Studio is opened, the first compilation will trigger the loading of AGP class bytecode. Any subsequent compilation will not trigger the loading of class bytecode. You must ensure that the Hook bytecode is “pre-loaded” to the VisitableURLClassLoader before AGP. Therefore, Wade plug-in access requires the apply Wade plugin in Root Project to ensure that Hook code can be executed before the Apply Android plugin in App Project.

compatibility

Compatible with AGP3 and AGP4, Gradle5 and Gradle6 versions.

Key steps in the plug-in, such as incremental compilation trigger conditions, reflection fetch Consumer Transform, WadeDexMergeTask, etc., are adapted for different versions.

The stability of

In the actual use of the process encountered a variety of difficult diseases, here is a list of the top 10 common abnormalities.

  1. Java.io.IOException: The input doesn't contain any classes. Did you specify the proper '-injars' options?
  2. java.io.FileNotFoundException: /Users/panes/app/build/intermediates/compile_and_runtime_not_namespaced_r_class_jar/debug/R.jar (No such file or directory)
  3. Under Caused by: com. Android. View the r8. Utils. B: Error: YeezyCompleteListener. Class, Type com. XXX is defined multiple times
  4. Caused by: org.gradle.api.UncheckedIOException:java.util.zip.ZipException: error in opening zip file
  5. Caused by: com.android.tools.r8.utils.b: Error: Class content provided for type descriptor xxx.r actually defines class com.xxx.R
  6. A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
  7. com.android.builder.dexing.DexArchiveMergerException: Error while merging dex archives: Type com.xxx.R is defined multiple times
  8. base.apk code is missing
  9. Archive is not readable : /Users/panes/android/app/build/intermediates/mixed_scope_dex_archive/developerDebug/out/c6795cc73f81ff9c1c0b5d0adb06b1b4 161c540cbf761ba11415aae4856b11b4_4.jar
  10. Could not determine dependencies of app:wadeInputChangesInspect

After nearly 30 iterations, these issues have been resolved. The latest version V2.6.4 has been compiled for 6800 times, with 4 exceptions.

The benchmark

Benchmark scores an average of 14.4 seconds for 10 incremental compilations (where only one line of code is changed) and 6.2 seconds for 10 infinite compilations (where the code stays the same). Running time cleans up background tasks and shuts down other resource-hogging processes, but the actual compilation environment is much more complex than ideal, and benchmarks are only used to verify that the theory works.

conclusion

The Development of Wade Plugin was fraught with difficulties, and it was not easy to rewrite the Android native compilation process to greatly improve speed and ensure stability and reliability. There are more details that have not been introduced, such as identification of hot codes during add-ups, reuse of files to change calculation results, Hook PackageTask to do the bottom pocket of files in Apk to prevent packet anomalies. Expect more improvements in future releases as well.

Author: Acquisition Technology Team – Panes