This paper introduces QQ Music team’s exploration and practice in incremental compilation component research and development.

1. Preface

Engineering compilation is an important part of Android application development. As the amount of engineering code expands, the compilation time becomes longer and longer, which slows down the development efficiency.

This problem is not uncommon in mid-to-large teams. Take QQ Music as an example, Android engineering code volume reached more than 1.2 million lines, each line of code modification, have to wait more than 4 minutes to see the change effect on the phone.

To address this problem, we developed an incremental compilation component of our own. After one year of continuous optimization, the component has been able to support daily development work within the team, effectively improving compilation efficiency in local development scenarios.

This paper will introduce the exploration and practice of THE QQ Music team in the development of incremental compilation components.

2. Problem analysis

During native development, we would iterate through the process of modifying the code – compiling the project – installing APK- and running validation.

Therefore, we can analyze the causes of slow compilation from the two dimensions of compilation and installation.

The first is the compile phase.

The main process is to collect all the resource files in the project for compilation and get the resource package and the resource index class. The resource index class is then compiled into bytecode files along with all the code files of the project, and the bytecode files need to be further compiled into Dex files so that they can be recognized by the Android virtual machine.

After resource packages and Dex files are ready, they will be packed and compressed together, followed by signature, alignment and other processes, and finally compiled to obtain an APK installation package.

In this process, both resource and code compilation takes time proportional to the number of files to compile. We typically only change a few code files during development and then trigger compilation. Ideally, the compiler should compile only the files that are being modified. However, this is difficult to achieve with native tools because of the dependencies of the code.

Since version 3.0, Android Gradle Plugin has abandoned the compile keyword and introduced the implementation keyword to declare dependencies, hoping to accelerate the compilation speed of large projects from the granularity of module. However, for some single engineering projects that do not split multiple modules, the effect is not ideal.

Now look at the installation phase.

The installation package needs to be transferred to the mobile phone through ADB tool first, and then the system carries out signature verification. After the verification is successful, decompress and copy files. For example, copy Dex files and so files.

In addition, if the system version is 5.0 or 6.0, due to the AOT mechanism, the installation process will be precompiled and the bytecode in Dex will be changed into machine code to improve the efficiency of application running, which will further prolong the installation time.

As you can see, the size of the installation package and the version of the mobile phone system will affect the installation time.

3. Optimize your thinking

Based on the above analysis, there are three main solutions.

To do a good job, we must sharpen our tools first. First, we can try to optimize the construction tool chain of the project.

The most common way is to update the Android Gradle Plugin, Gradle and other tools, and adjust the build parameters. However, after practice, the optimization effect they bring is not ideal.

Of course, in addition to Gradle build tools, consider using Facebook’s Buck as a build tool. According to the official introduction, it makes use of the idea of multi-module, multi-task parallel compilation, which can greatly shorten the compilation time.

However, for large projects, migrating build tools can be costly. Many plugins and development tool chains are based on Gradle system. If you migrate, you will lose the support of these functions. In addition, if the project involves the collaboration of other teams and projects, the construction solution cannot be changed at will.

Another approach is to optimize the project code to minimize the amount of code involved in compilation.

Here you can do a lot of things, such as sorting out the business to delete redundant code, multi-project separation, implementation of componentalization (modular) transformation, etc. However, due to the existence of objective factors such as deep code coupling and tight development pace, the difficulty of code optimization is usually relatively large, and the implementation cycle of each scheme will be relatively long. So it can’t solve the problem of slow compilation quickly in the short term.

So, can you provide a compilation tool that only compiles a small amount of the changed code at a time during local development, preferably skipping the APK installation process and only pushing and loading the new changed code? In this way, the compile time can be significantly reduced from both compile and install dimensions.

This is the core idea of incremental compilation tools. For tool access parties, it is possible to quickly improve local development efficiency at a lower cost without requiring drastic tool chain upgrades or engineering changes.

Up to now, the industry has two main schemes to refer to.

Instant Run is Google’s first generation of incremental compilation solutions. However, in large projects, the speed increase is not significant, and in some scenarios it can even make the build time longer.

First, prior to Gradle 4.6, if an annotation processor was used in a project, every code change was fully compiled. In addition, if the modified class contains public static constants, this will also cause the change to be fully compiled.

There were some compatibility issues with Instant Run, but since it was integrated into Android Studio, it was a black box for us, unable to solve problems by itself. We had to passively feedback problems and wait for the release of new versions. So overall, this scheme is not suitable to introduce.

In the latest Android Studio, Instant Run has been scrapped in favor of the Apply Changes scheme, which is based on JVMTI technology. However, it only supports Android 8.0 or higher versions of mobile phones, and the actual speed increase effect in the project is not obvious.

Another is The Freeline solution, which makes full use of cached files to quickly compile and deploy code changes to the device in a matter of seconds. However, it also has some problems that can not be ignored. The first is the lack of support for Kotlin, which is fatal now that Kotlin has been officially announced by Google as the preferred language for Android development. In addition, resources with ids cannot be deleted. Otherwise, errors may occur during resource compilation.

Another potential problem is that Freeline sacrifices some of its correctness to ensure faster compilation. For example, when a public static constant is changed, only the corresponding class file is compiled. Other classes that reference the constant are not compiled. Constant inline optimization can cause these classes to run with the same old values, and the changes won’t take effect.

In summary, the existing solutions in the industry cannot meet our needs. So in early 2019, we started working on our own incremental compiled components.

The birth of incremental compilation

In June 2019, the incremental compilation component completed the first version of development and started to be officially connected to the QQ Music project.

After access, the speed effect of local development is more obvious. According to the actual team statistics, a full compilation takes about 418 seconds, while an incremental compilation takes only 13 seconds. On a daily basis, the total time spent on engineering compilation per person decreased from 3.95 hours to 1.02 hours, an improvement of 74%.

The incrementally compiled component is fully based on Gradle standards and implemented as a Gradle plug-in with good multi-platform compatibility and minimal intrusion into the target project. Users simply need to access our Gradle plugin to perform specific Gradle tasks and enter incremental compilation mode.

In terms of function support, the component supports fast compilation of Java, Kotlin and other code files as well as all types of resource files. Incremental support from DataBinding was added earlier this year. To further reduce usage costs, we have also included a companion Android Studio plugin in the latest release that allows developers to visually access component functionality.

The following diagram illustrates the overall principle of the component, which divides the development cycle into compile time and run time.

The first compilation (also known as full compilation) requires a complete compilation project to obtain the original installation package, which takes the same time as the original packaging task. When compilation is triggered, it enters the incremental compilation mode, which takes a very short time. The component collects the changed code, compiles it, gets the incremental product, and pushes it to the phone.

The runtime is responsible for dynamic loading and running of incremental products on the phone.

The implementation of several key modules will be introduced later in this article.

5. Core principles

The code to compile

(1) Obtain the modified file and compile it

The first question to consider is, how do you identify which files the user has changed?

Our approach is to collect the last modification time of all project files after each successful compilation and save it as a file snapshot. At the start of the next compilation, the component generates the latest snapshot of the file and compares it with the previous snapshot to collect the files that the user has changed.

To be able to compile these files separately, you also need to resolve the problem of class references.

On the first full compilation of the project, the component collects all the generated class files and places them in the cache directory. When compiling the modified file, the native Javac or Kotlinc program is called and the cache directory is passed in as the classpath to solve the compile-time code reference problem.

(2) Conduct code dependency analysis

As mentioned above, providing the CLASspath allows the compile phase to execute successfully, but does not guarantee that the code logic at run time is correct. For example, if a class modifies the argument list of a method, other classes that depend on the class need to be recompiled in addition to that class. Otherwise, NoSuchMethodException occurs during runtime.

Therefore, it is not enough to compile the code that has been changed by the user because of the interdependencies between the codes. You may also need to find its subdependency sets for compilation.

Along this line, two more questions need to be considered:

• How do I get the change type of a modified class? Changes of a type, such as the internal implementation of a modification method, do not affect the child dependency sets. In order to minimize the amount of code involved while ensuring that the compilation is correct, we need to get the change type of the modified class so that we can decide whether to recompile its subdependencies.
• How do I get the subdependencies of a modified class? This makes sense because a component can’t know what to compile until it calculates the subset of dependencies of a class.

To obtain these two items of information, it is necessary to analyze the internal structure of the class and extract data such as class name, class modifiers, member variables and methods. We use the ASM tool to parse the class file and save the parsed information into our custom ResolvedClass data structure.

The next solution looks like this:

1. During full compilation, the component synchronously starts an independent process that traverses all the class files to obtain the corresponding ResolvedClass information, which is stored in the local file. If a class is found to reference another class, the class name of the current class is added to the list of subdependencies of the referenced class (resolvedBy field).

2. When incremental compilation is triggered, the component first compiles the changed class, resulting in a new class file. Then start the code dependency analysis process to parse the new ResolvedClass and compare it with the old ResolvedClass that was resolved at full compile time to get the changed type of the class.

When the current class’s change type is found in the table below, the component retrieves the subdependency set, starts a second compilation, and obtains the corresponding class file of the subdependency set.

In this way, we minimize the amount of code that needs to be compiled while ensuring that it is compiled correctly.

All class files generated during incremental compilation are then further compiled by the DX tool into Dex files and pushed to the phone via ADB for dynamic loading.

Compile resource

(1) Resource increment

The basic idea for this piece is similar to code increments. That is, the modified resources are collected and then compiled.

The native resource compilation process mainly uses AAPT, or AAPT2.

At the beginning, our project still used AAPT, based on which it was relatively difficult to increase resources. The aAPT tool does not support compilation of a single resource. Freeline through the modification of aAPT source code, the realization of a single resource increment function. However, this part of their solution is not open source, and the change still does not support the deletion of resources with ID, so they did not consider introducing it in the component.

Take a look at AAPT2. The biggest difference with AAPT is that it naturally supports compilation from a single resource. Internally, the resource packaging is divided into two steps: compile and link. In the compilation stage, it is responsible for compiling single or multiple resources into binary files. The link stage is responsible for merging all binaries and then packaging.

So we first upgraded the tool chain of the project and introduced AAPT2, and then the component redesigned the resource increment scheme based on this.

After the project is compiled for the first time, the component collects all compiled resource binaries into a cache directory. When the resources are subsequently changed, the compilation function of AAPT2 is first called to compile the changed resources into binary files. Then copy the new binary file to the resource cache directory, overwriting the file with the same name.

It then uses aapT2’s linking function to generate the final incremental resource bundle for this directory and push it to the phone for dynamic loading.

After such transformation, the time of incremental compilation of resources in QQ Music project was reduced from 32 seconds to 12 seconds, and the efficiency was further improved.

(2) The resource ID is fixed

There is one file that needs special attention during resource compilation: the R.java file.

To enable developers to reference resources in their code, the resource compiler assigns an index ID to each resource during compilation and stores it in an R.java file as a public static constant. All you need to do is refer to the resource in your code in the form of r.color.text.

When the compiler compiles source code and finds a reference to a constant (modified with both static and final keywords) that is a literal original data type or string, the compiler replaces that reference with a constant value.

That is, references in code such as r.col.text are replaced with corresponding numbers in the class file.

During resource compilation, resources are sorted by name and indexed incrementally. If a resource is added or deleted, the indexes of subsequent resources may be misplaced.

In this scenario, if a class references a resource whose index has changed, it needs to re-participate in compilation. Otherwise, you run into the problem of deranged resource references.

But this would result in a large number of classes being compiled in increments, which is the opposite of what we intended.

Therefore, the ID in R.java needs to be fixed. In simple terms, this means that the ID assigned to the same resource remains the same between compilations. In fact, the hot repair scenario also has the same appeal. For patch packs, there are strict size requirements. If we want to hotfix the resource, it is impossible to recompile all the code using the resource into the patch package for distribution, so we need to fix the resource ID.

The corresponding solution is also more common in the industry. If you try to output the aapT2 command line tool’s help document, you can see that there are two parameters:

• — stables: File containing a list of names to ID mapping
• –emit-ids: emit a file at the given path with a list of name to ID mappings, suitable for use with — stables -ids

Therefore, when compiling resources, we can inject aapT2 with the EMIT – IDS parameter and output the mapping between resource names and resource ids in the specified file. In addition, when AAPT2 is started next time, the mapping relationship is passed in through stables – IDS to achieve the effect of fixed resource IDS.

Dynamic loading

(1) Code injection

After compiling, several incremental Dex packages can be obtained and pushed to the specific directory of the mobile phone.

So at runtime, what we need to do is interfere with the native class loading process, so that the changed code is loaded first, so that the changes take effect.

Let’s take a look at the native Android class loading process.

After the application starts, a class loader named PathClassLoader is used to load the Dex file in the installation package. When a class needs to be loaded, the system traverses the Dex array from front to back until it finds the corresponding class.

Based on this, the incremental component inserts the incremental Dex file into the front of the Dex array by reflection when the application starts. When a class needs to be loaded later, the system mechanism traverses from front to back, so the modified class is preferentially searched from the incremental Dex and hit. It should be noted that all incremental Dex will be inserted into the Dex array in reverse order according to the generation time, such as INC_3. Dex, inc_2. Dex and INC_1. Dex. In this way, it can ensure that the latest implementation of a class is always loaded after multiple incremental modifications.

Handle the problem that class changes do not take effect

After the first release, we received feedback from colleagues that the modern code changes would not take effect on Android 7.0 or later. After analysis, you can ensure that the incremental code compiled successfully, but the problem occurred during the runtime class loading phase.

This is due to a change in the virtual machine code compilation strategy starting with Android 7.0.

The instructions in Dex need to be translated into machine code before they can be executed. With the change of system version, the compilation strategy of Dex bytecode also has different performance.

On systems up to 5.0, the Dalvik virtual machine is used. Whenever a new class is encountered at application runtime, the JIT compiler compiles the class on the fly, and the compiled code is optimized into fairly lean primitives, making it faster to execute the same logic the next time. However, since the compilation work is carried out during the application running and there is no cache, the application starts slowly, the operation efficiency is affected, and the power consumption is high.

Therefore, starting with Android 5.0, Google replaced the Dalvik VIRTUAL machine with the ART virtual machine. The biggest difference from Dalvik is that ART virtual machine adopts AOT ahead of schedule compilation mechanism. During application installation, the system will use the dex2OAT tool to precompile all Dex files in the installation package and generate an OAT file that can be run on the local machine. In this way, every time the subsequent application runs, there is no need to compile, and the efficiency of application startup and operation is greatly improved. However, AOT takes too long to execute each time, which makes installation extremely slow.

So, starting with Android 7.0, the Hybrid Mode ART VIRTUAL machine supports Interpreter, JIT, and AOT modes at the same time. They are used interchangeably to achieve the best balance between installation time, memory usage, battery consumption and performance.

At application runtime, the virtual machine first uses Interpreter to interpret and execute code. If a hot function is found, the JIT compiler is enabled and the compilation results are stored in a local profile; When the Android device is idle or charging, the system periodically performs AOT compilation of the profile in the background, producing a “hot code”.

On the next application restart, the compiled hot code is inserted once and for all into the class loader’s cache ClassTable. During the subsequent class loading process, the ClassTable is first searched for the cache, and if so, returned directly, skipping the subsequent class lookup process.

At this point, we can explain why mixed compilation causes accidental incremental code changes to fail.

To load the incrementally changed class A, there are two cases:

1. No class A in the hot code: Ideally, the system will look for class A in the incremental Dex because it is not hit in the ClassTable, and the incremental code will work in this case.
2. Hot code contains class A: During class loading, the system will preferentially hit class A in the ClassTable before the change, resulting in the problem that the increment does not take effect.

Tinker’s solution to this problem is to first copy the Dex array of the native classloader to create a completely new custom classloader. All class loaders referenced by the application process are then directed to the custom class loader, which is responsible for all subsequent class loading and patch code injection.

Since hot code is not inserted into the ClassTable cache of the custom class loader, subsequent patches can be loaded without hot code interference and can work normally.

However, the incrementally compiled component is a debug package for local development, so a simpler solution is possible: the component automatically specifies Android :vmSafeMode=”true” in androidmanifest.xml. This switch deactivates the AOT compiler. Hot code cannot be generated, and the above problems will not occur.

(2) Resource injection

Dynamic loading of resources is relatively simple. In Instant Run, the addAssets method of AssetsManager is called by reflection, and the incremental resource bundle is loaded into memory. The new Resources object is then replaced by all the Resources held by ActivityThread and so on. This is the basic idea in most hotfix frameworks.

6. Conclusion

Reviewing the practice of incremental compilation components, it is actually a comprehensive application of Android application compilation, hot repair, bytecode piling, Gradle and other technologies. For large projects, local development efficiency can be improved quickly and at low cost.

In the meantime, we have a few suggestions for optimizing compilation speed. First of all, it is suggested to update the latest compilation tool chain in time and use the latest optimization results of the official. The profile analysis tool provided by Gradle is used to analyze specific tasks and solve some unreasonable time-consuming scripts. At the same time, it is also recommended to synchronize modular transformation, code separation and so on. This step may take a long time, but the late benefits are not only improved compilation efficiency, but also improved code reuse at the business module level.

At present, the component has been connected to QQ Music, National K Song and other team applications, and has been open source within the company. Incremental compilation components also have some features that need further development. Such as four component incremental support, Module incremental support, etc. At the same time, we are constantly optimizing the components through the issues that come to light in the actual development work scenarios.

After further refinement, an external open source program for components will be implemented. We look forward to helping more teams in need with open source, enabling seamless integration and easy improvement of development efficiency without having to worry about implementation details.

QQ Music recruitment Android/iOS client development, click here to submit resume ~ can also send resume to email: [email protected]