Compile speed optimized mind map outline

Because the content is more, so split two parts to explain. Welcome to give the author some motivation thanks thanks. If you have any ideas or ideas, please contact me directly for discussion. The overall content is mainly divided into six parts to introduce:

1, why do we need to optimize compile speed (time is money) 2, Android compile packaging process (know yourself and know your opponent can win all battles) 3, compile time detection (bystanders review, the local puzzle) 4, compile optimization routine scheme (to grow a beard before adults) 5, compile speed depth optimization (perseverance, 6, future optimization control (wife my wife and other people’s wife)

Github：github.com/blindmonk

I. Introduction to compilation speed optimization

1, three years of development, five years of compilation

Gradle is a build tool that compiles too slowly on medium to large projects, averaging 7 minutes for full compilation and 3 minutes for incremental compilation of a single line of code. ** With the expansion of business, future project code will gradually increase, modular split more detailed, compilation speed will only be slower. Developers waste most of their time compiling projects, and while three years of development and five years of compilation are an overstatement, compilation speed takes up most of a developer’s time. Seriously affect the development experience and development efficiency.

2. Increase development efficiency

Once a developer iterates on a new requirement, it is likely to be evaluated/developed/tested. The development and compilation process is as follows:

Requirements –> full build –> incremental build –> Install –> Verify

Full compilation/incremental compilation/installation is the majority of the process. Assuming full compilation is in 7 minutes, we run it 6 times a day, so we’re wasting 42 minutes of code compilation per day, or about 13 hours of code compilation per month based on our 20 days of work. From this point of view, compilation speed optimization is imperative.

So we focus on optimizing for full/incremental/install processes. By the way, explain the difference between full compilation and incremental compilation.

Full compilation: The project was not run, no specific cache files were generated, and the resource code was compiled and merged for the first time. Let’s say clean. Most are used at post-compile time for the initial compilation of a project, packaging/clearing the cache.

Incremental compilation: Builds on the basis of full compilation. Changes/additions/deletions/additions will be changed without affecting the entire compiled file. Most are used for initial compilation/post-compilation of the cache files generated for the project.

3. Be happy and reduce bugs

I believe you have such a similar scene:

For example, UI acceptance only needs to change to a line of text, the size of a button display

Data values on the back end need to be adjusted by a field, and business logic needs to be adjusted by a line or several lines of code

In this case, you only need to change one or a few lines of code, and you need to quickly verify correctness. The 2-3 minute incrementally compiled wait is one of the most frustrating, and it’s one of the most frustrating situations for developers to get upset. ** It’s easy to lose focus on the next change, resulting in minor bugs.

Combined with the above situation began to optimize the compilation speed, before optimization, it is necessary to understand the compilation process is how? Which piece of compilation is dragging down our entire compilation process.

Ii. Compilation process

As you can see, the Android compilation and packaging process is quite complex to generate APK from various source files and code, but it can be broken down into seven steps:

  1. Aapt: Package resource files
  2. Aidl: Processes AIDL files
  3. Javac: generates a. Class file
  4. Dex: converts. Class to a dex file
  5. Apkbuilder: Generates APK packages
  6. Jarsigner: Sign the APK
  7. Zipalign: signature alignment

Because each project has a different amount of resources and code, the entire packaging process is fixed. As a result, the speed of compilation and packaging is proportional to the amount of resources and code on the project. For each node of the packaging process, you can find the object tool in the Android-SDK directory. The entire android compilation process is handled according to Gradle. How to convert our newly added activity. class and drawable into APK step by step for ART virtual machine to recognize and decode operation? Here’s a look at the flow under each node and how Gradle handles it.

Aapt: Package resource files

Use AAPT to package RES resource files to generate R.java, resources.arsc and RES files. Res files are divided into binary and non-binary files. Typical non-binary files such as RES /raw and images are left as is and not compiled.

The RES directory has nine types of directories, as follows:

– animator: This resource is stored as an XML file in the res/animator directory and is used to describe attribute animations. – anim: These resources are stored as XML files in the res/anim directory and are used to describe tween animations. -color: This type of resource is stored in the RES /color directory as an XML file and selected by describing the object color state. – drawable: This resource is stored in the RES /drawable directory as an XML or Bitmap file and is used to describe drawable objects. For example, we can place images (.png,.9.png,.jpg,.gif) inside as background images for the application interface view. Note that Bitmap files saved in this directory may be optimized during packaging. For example, a true-color PNG file that doesn’t need more than 256 colors might be converted to a PNG panel with only an 8-bit palette so that images can be compressed losslessly to reduce memory resources taken up by images. -layout: These resources are stored as XML files in the RES /layout directory and are used to describe the application interface layout. – menu: This resource is saved as an XML file in the RES /menu directory and is used to describe the application menu. – RAW: These resources are stored in the RES/RAW directory in any format. They are packaged in apK files just like assets. However, they are given resource ids, so that we can access them by ID in the program. For example, if you have a file named filename in the res/ RAW directory, and it is being compiled and given a resource ID of r.raw.filename, you can access it using the following code: Resources res = getResources(); InputStream is = res .openRawResource(R.raw.filename); – values: These resources are stored as XML files in the RES /values directory and are used to describe simple values such as arrays, colors, sizes, strings, and style values. In general, These six different values are stored in files named arrays.xml, colors.xml, Dimens. XML, strings. XML, and styles.xml, respectively. – XML: These resources are stored as XML files in the RES/XML directory and are generally used to describe the configuration information of applications.

2, AIDL: processing AIDL files

Aidl: Is one of the IPC methods in Android, mainly used for cross-process communication, common projects rarely have such files.

The tool used in this process is aidl. Exe, located in the Android-SDK /tools directory. Android interface definition language, Android provides a unique implementation of Inter Process Communication (IPC). This stage processes the.aidl file and generates the corresponding.java file. If the AIDL file is not used in the project.

Javac: generates. Class files

Use Java Compiler to compile all Java code in the project, including.java files generated by R. Java,.aidl files, Java source files, and.class files. The relevant code can be found under the corresponding build.

4. Dex: Convert. Class to a dex file

The dx tool, located in the Android-sdk /tools directory, generates classes.dex files for android VMS to execute. You can find the corresponding code under Build and use the dex command to convert directly. At this stage any third-party libraries and. Class files will be converted to. Dex files. The main work of dx tool is to convert Java bytecode to Dalvik bytecode, compress constant pool, eliminate redundant information, etc.

Dx --dex --output=D:\test.dex D:\testCopy the code

5, APkBuilder: generate APK package

The packaged tool apkBuilder is located in the Android-SDK /tools directory. Apkbuilder for a script file, the actual call is android SDK/tools/lib/sdklib jar files in com. Android. Sdklib. Build. ApkbuilderMain class. All uncompiled Resources(e.g. Res /raw, images, etc.), Other Resources(assets files), compiled Resources,.dex files, resources.arsc and Androidmanifest.xml Will be packaged by the ApkBuilder tool into the final.APk file.

6. Jarsigner: Sign APK

Once an APK file is generated, it must be signed before it can be installed on the device. In the development process, the main use is two types of signed keystore. One is debug.keystore for debugging, which is mainly used for debugging. The other is the keystore used to publish the official version.

Zipalign: signature alignment

If you publish an official APK, you must align the APK using zipalign, which is located in the Android-SDK /tools directory.

Zipalign is a tool for organizing APK files on android platform. It makes 4-byte alignment of uncompressed data in APK. The main process of alignment is to offset all resource files in APK package to a 4-byte integer multiple from the start of the file. Ordinary files can be manipulated as if they were read from memory. Without 4-byte alignment, it must be read explicitly, which is slow and consumes extra memory.

GradleTask

Gradle works when we click on Run ‘app’. In the Build window, we can see the detailed Task log. The main function is to process each step of the 7 packaging processes mentioned above. In the window log we can see the familiar keywords such as compileDevDebugAidl in the second line from the name processing Aidl. And generate BuildConfig files generateDevDeubgBuildConfig is. Our common buildConfig.isdebug is handled in this Task. Of course, each Gradle Task is a refinement of the above 7 packaging steps. I put the whole system used in the Task and implementation class list out interested partners can study the source code. Let’s simple analysis under GenerateBuildConfig Task source, source code based on com. Android. View the build: gradle: 4.1.0 Kotlin version.

Abstract class GenerateBuildConfig: NonIncrementalTask(){// versionName @get:Input @get:Optional abstract val versionName: Property<String? > // version @get:Input @get:Optionalabstract abstract val versionCode: Property<Int? > // The only abstract method of the parent NonIncrementalTask class, Override Fun doTaskAction() {// The properties in the getClass include some custom properties val buildConfigData = BuildConfigData.Builder() .setBuildConfigPackageName(buildConfigPackageName.get()) .apply { If (hasversioninfo.get ()) {versioncode.ornull? If (hasversioninfo.get ()) {versionCode. .let { addIntField("VERSION_CODE", it) addStringField("VERSION_NAME", "${versionName.getOrElse("")}") } } // val generator: GeneratedCodeFileCreator = the if (bytecodeOutputFile isPresent) {/ / create a JVM bytecode BuildConfig. Kotlin version is reformed BuildConfigByteCodeGenerator (byteCodeBuildConfigData)} else { / / create a Java file BuildConfig, Java version of the GenerateBuildConfig has always been the plan BuildConfigGenerator (sourceCodeBuildConfigData)}} JavaWriter generator.generate()}}Copy the code

You can see that GenerateBuildConfig has been changed to Kotlin, and all other system tasks have been changed to Kotlin versions. It seems that Google has also made a big bet. Kotlin will also write articles about coroutines, suspend, non-blocking suspend functions, extension functions, and generics. As normal, GenerateBuildConfig inherits NonIncrementalTask, which is the parent of Kotlin’s version and almost all other system tasks inherit from this class. The main function is an incremental compiler handling class. There is an abstract method inside, doTaskAction, which is the main logical implementation method in GenerateBuildConfig. There is also a cleanUpTaskOutputs method called before doTaskAction to ensure that the task output is deleted before the task runs.

The main logical flow for generating Java classes:

doTaskAction-->buildConfigData -->BuildConfigGenerator-->JavaWriter
Copy the code

The main logical flow for generating bytecode classes:

doTaskAction-->buildConfigData -->BuildConfigByteCodeGenerator-->ClassWriter
Copy the code

Major process split

  1. Generate the buildConfigData class, which is a Design pattern for Builder

  2. Add default properties like BUILD_TYPE, FLAVOR, DEBUG, etc

  3. Generate BuildConfigGenerator isPresent generate BuildConfigByteCodeGenerator otherwise

  4. Add custom properties with BuildConfigGenerator via items.get()

  5. The call to generate generates the concrete implementation class internally using JavaWriter or ClassWriter

Other tasks, corresponding implementation classes and functions of the system

3. Compilation time detection

Gradlew command

For larger projects or implementing a large number of custom Transfrom-API projects, you may need to delve into the build process to find bottlenecks. To do this, you can dissect the time it takes Gradle to execute each stage of the build life cycle and each build task.

To generate and view build performance profiling reports, follow these steps:

1. Open the cli terminal in the root directory of the project. 2. Run the following command to execute Claen. Because if the input to a task (such as the source code) hasn’t changed, Gradle skips it. So a second build with unchanged input will always run faster because the task will not be repeated. Running the Clean task before build ensures that you can profile the complete build process.

//mac
gradlew clean
//window
gradle clean
Copy the code

3. After the Clean command is executed, run the following command based on the required build environment

//mac
gradlew assembleDebug --profile
//window
gradle assembleDebug --profile
Copy the code

4. After the build is complete, the corresponding HTML report can be found in the /build/reports/profile/ directory in the root directory of the project

You can view each TAB in the report to see your build. For example, the Task Execution TAB shows how long It took Gradle to execute each build Task. It is important to note here that the Summary’s Task Execution is a summation of each module, which is actually running in parallel.

Summary of the time spent building the Configuration DependencyResolution of the TaskExecution time

2. Customize Gradle lifecycle implementation methods

It can be seen that each gradleTask will print the time consuming after each running build and compilation. Therefore, we can optimize the tasks with serious time consuming tasks and monitor the tasks that spend more than a certain amount of time. If a threshold is triggered, an alarm is generated to ensure that subsequent tasks remain low in time because there is more to this monitoring scheme, which will be covered in more detail in chapter 2.

Other lifecycle methods are omitted as follows:

import java.util.concurrent.TimeUnitclass 
TimingsListener implements TaskExecutionListener, BuildListener { 
  
    private long startTime    
    private timings = []

    @Override    
    void beforeExecute(Task task) {        
        startTime = System.nanoTime()    
    } 
    
    @Override    
    void afterExecute(Task task, TaskState taskState) {        
        def ms = TimeUnit.MILLISECONDS.convert(
        System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
        timings.add([ms, task.path])
        task.project.logger.warn "${task.path} took ${ms}ms"
    }

    @Override
    void buildFinished(BuildResult result) {
        println "Task timings:"
        for (timing in timings) {
            if (timing[0] >= 50) {
                printf "%7sms  %s\n", timing
            }
        }
    }
}
gradle.addListener new TimingsListener()
Copy the code

Fourth, compile and optimize the conventional scheme

As the saying goes, “good things in advance, must first sharpen its tools”, “do not mistakenly cut wood work”, “first to move” and so on. Roughly meaning that is the first need to use the tools of the upgrade in order to play more no injury or in the fight before the plan when to start, when to use the killing skills, etc.. According to the above conclusions, there are several optimization schemes for compilation speed:

1. Use the latest version of tools

Google has also been well aware of the development pain, revamping the system’s Gradle Task and creating Studio tools for faster builds such as Instant Run and Apply Changes. Instant Run is based on Transfrom-API technology. Many hot repair frameworks in the Transfrom-API industry are based on this idea, but due to too many complaints, Instant Run was abandoned in Android Studio 3.5. Apply Changes, which relies on the Android 8.0 Start VIRTUAL Machine Support special directive (JVMTI) to make class replacements. These two tools will not be covered in detail in the section on deep compilation speed optimization.

With almost every update, the Android tools have some build optimization, so we can upgrade the following tools to the latest version:

  • Android Studio and SDK tools
  • Android Plugin for Gradle

2. The Debug environment compiles only the resources needed

Avoid compiling unnecessary resources

Avoid compiling and packaging resources that are not tested (for example, other language localization and screen density resources). To do this, you can specify a language resource and screen density only for versions of “dev” or “debug,” as shown in the following example:

android { ... productFlavors { debug { ... ResConfigs "zh", "xxhdpi"} resConfigs "zh", "xxhdpi"}... }}Copy the code

Disable Crashlytics for debug builds

If you do not need to run the Crashlytics report, disable the plug-in as follows to speed up debug builds:

android {  
    ...  
    buildTypes {    
        debug {      
            ext.enableCrashlytics = false
    }
}
Copy the code

Do not automatically generate the build ID

If you want to use Crashlytics for debug builds, you can speed up incremental builds by preventing Crashlytics from updating application resources with unique build ids during each build. Because this build ID is stored in the resource file referenced by the manifest, disabling automatic build ID generation can also be used to debug builds with Apply Changes and Crashlytics. If you want to prevent Crashlytics from automatically updating its build ID, you can configure it as follows:

android {  
    ...  
    buildTypes {    
        debug {      
            ext.alwaysUpdateBuildId = false    
    }
}
Copy the code

3. Version converts images to WebP

WebP is an image file format that provides both lossy compression (like JPEG) and transparency (like PNG), although it provides better compression than JPEG or PNG. Reducing the image file size can speed up builds (without having to compress them at build time), especially if the application uses a lot of image resources. However, when decompressing WebP images, you will notice a small increase in CPU usage on your device. Using Android Studio, you can easily convert images to WebP format. The steps are as follows:

  1. Right-click on an image file or folder containing some image files, and click Convert to WebP.

  2. The Converting Images to WebP dialog opens. The default Settings depend on the minSdkVersion setting of the current module.

  3. Click OK to begin the conversion. If you want to convert multiple images, the conversion can be done in a single step, and the conversion can be undone to restore all the converted images at once.

  4. If lossless conversion is selected above, the system converts immediately. The image will be transformed in its original position. If lossy conversion is selected, proceed to the next step.

  5. If you select the lossy conversion and choose to see a preview of each converted image before saving, Android Studio displays each image during the conversion to check the conversion results.

  6. Click Finish. The image will be transformed in its original position.

The original JPG image is on the left and the lossy encoded WebP image is on the right. The original and converted image file sizes are displayed in the dialog box. You can drag the slider left or right to change the quality Settings and instantly see the effect of the encoded picture and the file size.

4. PNG format is disabled

If you can’t (or don’t want to) convert PNG images to WebP format, you can still speed up your build by disabling automatic image compression every time you build your application. If you are using Android plugin 3.0.0 or later, PNG processing is disabled by default only for the “debug” build type. To disable this optimization for other build types, add the following code to the build.gradle file:

android { ... BuildTypes {debug{// disable PNG compression. crunchPngs false } }Copy the code

5. Enable Gradle cache

The build cache can store specific output generated by the Android Plugin for Gradle when the project is built (for example, unpackaged AArs and remote dependencies preprocessed by dex). Clean builds are significantly faster when caching is used because the build system can directly reuse these cached files for subsequent builds without having to recreate them.

Org.gradle.caching =true Android. enableBuildCache=trueCopy the code

6. Enable incremental and parallel compilation for Kotlin

Incremental =true kotlin.incremental. Java =true kotlin.incremental.js=true kotlin.caching.enabled=true kotlin.parallel.tasks.in.project=trueCopy the code

7. Use static dependency versions

In build. Gradle dependency declaration in the file, you should avoid using the version number with a plus sign at the end, for example ‘com. Android. View the build: gradle: 2. +’. Using dynamic version numbers can lead to unexpected version updates and difficult resolution of version differences, and can slow down builds as Gradle checks for updates. Static/hard-coded version numbers should be used.

8. Adjust the heap size properly

Org.gradle.jvmargs = -xmx4000mCopy the code

9, KAPT optimization

APT: Java provides a compile-time plug-in that scans the source code at compile time, finds annotations in the code, generates new Java files according to the parsing rules defined by the developer, and executes the generated code that will be compiled by Javac along with the code you wrote manually.

KAPT: Three solutions have been iterated over to Kapt3 and the third solution has been chosen:

  1. Redesigned, but violates the coexisting with Java principle.

  2. Generate a “stub class” in which all the methods have an empty body, i.e. only the class structure, and compile the “stub classes” into the JavAC CLASspath. Method return types are expressions that need to be parsed, which can greatly reduce compilation speed

  3. The Kotlin code compiles to a binary that is recognized by the Java compiler

    # optimization kapt kapt. Use. Worker. API = true / / run in parallel kapt. Incremental. Apt = true / / incremental compilation kapt.include.com running. The classpath = false / / open the cache kapt { useBuildCache = true }

Use an incremental annotation handler

The Android Gradle plugin 3.3.0 and later improves support for incremental annotation processing. Therefore, if you want to speed up incremental builds, you can update the Android Gradle plug-in and use only incremental annotation processors if possible.

In addition, if Kotlin is used in your application, you need to use KAPT 1.3.30 or later to support incremental annotation processors in Kotlin code. If you must use one or more annotation handlers that do not support incremental builds, annotation processing will not be incremental. However, if the project uses Kapt, Java compilation is still incremental.

Third-party incremental annotation processors support:

summary

This article introduces the first four parts, There are a lot of open source resources on the web that are pretty much the same, Gradle Task, we see BuildConfig source code analysis, and of course there are some other tasks that are really interesting, especially after the Google developers converted to Kotlin, It’s still quite different from previous JAVA versions. Kotlin is also a future direction for Google. There are also some general optimization schemes can refer to the above code configuration into their own projects, I believe you can see a significant speed increase. Of course, the key is that the subsequent in-depth optimization includes:

  • Module AAR scheme

  • The transform controls

  • DexBuilder optimization for incremental compilation

There are also some mature solutions in the industry, such as Facebook Buck, Ali Freeline, Youzan Savitar, as well as system solutions: Instant Run, Apply Changes and the principle of their implementation. Because the content is too long, they are divided into two parts and will be introduced in the subsequent article. I will continue to post other types of articles, and not just on Android. At the same time to form their own knowledge system including architecture design, performance optimization, interview related, programming language, multimedia, data structure algorithm, Framework, plug-in. Welcome to follow, leave a message, like. Exchange thanks with the author.