Android isn’t cool, it’s just harder than before

A few years ago Android exploded into the new trillion-dollar market, and millions of gold diggers rushed in expecting to get their hands on it.

At that time, under the environment of mobile games, as long as you can have a running Lianliankan can find a job, onto the track to be pushed forward by the tide, this industry is not not bright.

Now, the tide is slowing down, and Android messages are flying all over the place, with “cool” being the word.

But rest assured, Android is not cool!

Android entry is relatively simple, the primary Android many, many, near saturation, you open a recruitment app can be very intuitive judgment, now Android access conditions are getting higher and higher.

background

Android projects generally use Gradle as a build packaging tool, and its execution speed has been criticized for being slow. For large projects like the first Android project today, a full compilation time can be as high as six or seven minutes. In some cases, quick verification is required. Incremental compilation of a single line of code can take two to three minutes to change. This slow process can have a significant impact on the development experience and efficiency, so it is important to optimize gradle build time.

In toutiao Android project, the optimization and deterioration of compilation and construction speed have been carried out alternately. In 2018, due to the influence of modularization and splitting, the clean build time reached the peak of around 7 minutes and 30 seconds. Relevant students accelerated through module AAR and Maven agent. As well as incremental Java compilation, the clean build time was optimized to 4 minutes and incremental compilation to 20-30s. However, with the large-scale use of Kotlin, the proliferation of custom Transform and APT libraries, the incremental compilation speed is slowed down to 2 minutes and 30 seconds, and the trend is further deteriorating. In order to optimize the existing unreasonable compile time consuming, to prevent further deterioration, recent 5 or 6 bi-monthly again aiming at compile time consuming to do a series of special optimization (kapt, transform, dexBuilder, build – cache, etc.) and add some deterioration prevention control scheme. From 4.27 to 6.29, the overall optimization effect is as follows:

Historical optimization scheme

Since students related to basic client technology have made many gradle-related optimizations for Toutiao Android project in the past 18 years or so, and these optimizations are the basis of recent optimizations, I first select a few representative schemes to introduce them as background synchronization for the following.

Maven agent optimizes sync time

background

Gradle projects will add a list of Maven repository locations in their repositories, which will be used as a repository for component dependencies. Earlier in the Toutiao project, we configured dozens of Maven repositories for dependencies, but dependencies are located in the order of maven repository configuration. If a component is in the last repository, the previous dozen or so repositories will have to initiate a network lookup in turn, and only look up the next one when the network request fails. If most of the components in the project are in the later repository, the total lookup time can be very long.

Optimization scheme

1. Use maven private server, set up proxy warehouse on the private server, configure proxy for other warehouses (such as Google, JCenter, mavenCentral, etc.), after the proxy warehouse is created, turn off its Cache in Negative Cache configuration item: If the maven repository fails to find the dependency, the maven repository will not search for the corresponding dependency for a period of time. Even if the maven repository already contains the dependency of that version, the search will still return the failure result.
2. Set up a warehouse group and put all the warehouses into a unified warehouse group. When you rely on the search, you only need to search the warehouse in this group, which can greatly reduce the time consuming of repeatedly initiating network requests to traverse the warehouse.

Module aar,

background

The project underwent several component-based and modular refactorings, resulting in more than 200 submodules. If all of these submodules were included in the project, the code of all submodules would need to be recompiled during clean Build, and for most developers, Basically, you only care about the few modules you are responsible for, and you don’t need to change the code of other modules at all, and the configuration and compilation time of these other projects becomes an unnecessary cost.

Optimization scheme

The solution to the excessive number of submodules mentioned above is to publish all modules as AAR, rely on these compiled components through Maven by default in the project, and when a module needs to be modified, change the dependency form of the module to source dependency through configuration items, so that only the modified modules are compiled at compile time. But doing so can lead to modules gradually reverting to source-dependent form altogether, unless the developer is required to manually publish the module as an AAR and change back to dependent form after each modification. This relies heavily on the consciousness of developers, and it will be extremely tedious when the number of modules is large and the dependency relationship is complex. Therefore, in order to facilitate the development stage, a set of more complete and detailed schemes are designed:

1. At the time of development, the code pulled from the main branch must be fully AAR dependent, and no sub-modules other than the APP module are source imported.
2. To modify the corresponding module, modify the INCLUDES parameter in local.properties to specify the module imported from the source code.
3. After the completion of development, push the code to the remote end and trigger the code merging process. In the CI precompilation process, compare with the target branch of code merging, detect the modified modules, release these modules into AAR according to their dependencies, and modify the dependencies into aar of the new version in the project. This step ensures that after the completion of each code merging, The dependencies on the main branch are all AAR dependent.

earnings

Through the above transformation, the clean build time is reduced from 7 or 8 minutes to 4 or 5 minutes after the source module is switched to AAR dependency, and the profit is close to 50%.

Incremental Java/Kotlin compilation

background

In non-clean build cases, changing Java/Kotlin code does incremental compilation, but to be absolutely correct, Gradle selects the code that needs to be recompiled based on a series of dependencies. This calculation is coarser-grained. It may cause a lot of code to re-execute APT, compile and other processes.

Because gradle is a universal framework, it is designed to be absolutely correct, so it is easy to fail incremental compilations. In practical development, we can make a compromise between compile correctness and compile speed.

1. The original Javac and kotlinCompile tasks are disabled, and the modified code is only compiled.
2. Dynamically disable KAPt-related tasks to reduce the time consumption of tasks such as KAPT and kaptGenerateStub.

Although the above scheme (hereinafter referred to as fastBuild) has some problems in constant modification and method signature change (constant inlining, etc.), incremental compilation can be changed from more than 2 minutes to 20~30s, greatly improving compilation efficiency, and the problematic scenarios are not common. Therefore, on the whole, the scheme has more advantages than disadvantages.

Compile time deterioration

Through several optimization schemes and other optimization methods introduced above, the overall compilation speed of Toutiao Android project in 2018 (clean Build 4 5min, incremental compilation of FAST 20 30s) is relatively fast among large projects of the same magnitude. However, as the business needs to develop later, the compilation script adds a lot of new logic:

1. Kotlin is widely used, and Kapt has added a lot of annotation processing logic.
2. Introduced support for the java8 syntax, whose desugar(desugar) operation increases compilation time.
3. A large number of bytecode staking requirements, with many transforms added, significantly increased incremental compilation time.

With the introduction of this logic, incremental compilation time deteriorates to 2 minutes and 30 seconds. Even with FastBuild, it takes as much as 1 second to compile a single line of code, making the development experience very poor. The following will focus on the recent optimization process of the above problems.

Recent optimization plan

App shell module kAPT optimization

background

After several modular and component-oriented reconstruction of Toutiao project, most codes of app module (NewsArticle) have been migrated to sub-module (as mentioned above, sub-module can be aar for compilation speed optimization, and app module is only a shell.

An exception case was found in the build profile data: Obviously there are only two classes of app module KAPT (annotationProcessor annotation processing) correlation takes nearly 1 minute.

On closer inspection, although the APP module is split with only two simple classes of code, it uses six kAPT libraries, and only one annotation, ServiceImpl, actually takes effect (internal ServiceManager framework, which instructs to produce Proxy classes, Decouple code calls between modules). Such an operation is like a tiger, but only two fixed Proxy classes are generated each compilation, and the input-output ratio is very low compared to the high time consuming of 53s.

Optimization scheme

Move the fixed generated Proxy class from generate directory to SRC directory, then disable kAPt-related tasks in app module, and add related control scheme (as shown in the picture below: throw exception immediately after detecting unreasonable situation) to prevent others from adding new Kapt library.

earnings

1. Average revenue in MAC Clean builds is 40s
2. Average gain of 20s in CI Clean build

Kapt isolation optimization

background

By introducing the kAPT case of the abnormality found in the APP module above, we found that a library.gradle was defined in the project for convenience. The function of this file is to define the common Android DSL configuration and common basic dependencies in the project. So all the submodules in the project apply this file, but this file is gradually added to the new KAPT annotation processing library by different businesses. When compiled from full source, all submodules have to execute all six KAPT defined in the library module. Even if the module does not have any annotation related processing.

The problem of the above situation is that, compared with the annotation processing of pure Java module, Kotlin code needs to convert KT file into Java through kaptGenerateStub first, so that APT processor can uniformly scan and process annotations for Java. However, as mentioned above, in fact, there are many modules that do not have any actual KAPT processing, but do a KT to Java operation for nothing. The more modules are introduced into the source code, this meaningless time adds up to be very considerable.

In order to find out which submodules really use Kapt and which are useless enough to disable Kapt-related tasks, we scanned all submodules in the project:

1. By obtaining all dependencies of kAPT Configuration, you can obtain jar packages of kAPT dependency library, and use ASM to obtain all annotations.
2. Traverse all.java,.kt source files under the sourceset of all subprojects and parse import information to see if there are annotations parsed in Step 1
3. After the package task was completed, all subprojects were traversed through all Java files generated under generate/apt and generate/kapt directory

Using the above scheme, through full source packaging, the final scan is probably 70+ modules will not do any actual output of KAPT, and these kAPT will not output, KaptGenerateStub’s task time adds up to a higher 217s (the actual total time may be lower because the task executes concurrently).

After obtaining modules that do not actually generate kapt content, start a fine-grained split of these modules. Change them from apply library.gradle to library-api-gradle without kapt. Consistent with library logic.

But this is to secretly make some changes behind the back, it is likely that the new students do not know this optimization method, may add annotations but there is no output and can not find the reason, and the optimization effect is best to bring less trouble to the business students. To avoid this, the isolation optimization for the library-API module dependent annotations is implemented by automatically excluding all of the module dependent annotation libraries. When attempting to use annotations, the dependency removal problem is first detected because no reference is obtained (as shown in the figure below).

On the other hand, in the event of a compilation error, the gradle plug-in will automatically resolve the missing symbol. If the symbol is found to be an isolated optimized annotation, it will prompt you to replace library-API with Library to minimize the negative impact of the optimization solution on the business.

earnings

1. The MAC full source scenario has an acceleration benefit of around 58s.
2. On CI machines, the concurrent performance of tasks is better due to the larger number of CPU cores, with only about 10 seconds of benefit.

Resource sharing

Click:

** Android Architecture Video +BAT Interview Topics PDF+ Study Notes ** available for free

There is a lot of information about Learning Android online, but if the knowledge learned is not systematic, and problems are only tasted, no further research, then it is difficult to achieve real technology improvement. I hope this systematic technical system has a direction reference for everyone.

2020 May be a bumpy year for Android, but don’t panic. Make your own plans and learn your own lessons. Competition is everywhere, in every industry. Believe in yourself, there is nothing impossible, only unexpected. I wish you all the best in 2021.