Bytedance Terminal Technology — Wang Longhai feng Guang LAN Junjian

The background,

This article is the first in a series of articles on kAPT optimization, which will be followed by build Cache, Kotlin, and dex optimization. This article is produced by Client Infra->Build Infra team, powered by Wang Longhai, Feng Guang, LAN Junjian

Kapt is no stranger to Android development, and there have been many articles about kapt in the compilation optimization process, mainly for incremental compilation scenarios.

Douyin Volcano students encountered a more serious problem when accessing HILt: triggering OOM on a 16GB PC. For example, in the process of executing KAPT in volcano project, no matter using AAR dependency or full source compilation, kAPT cannot be compiled. It can be considered that KAPT will have a relatively large impact on memory.

Before we look at this problem, let’s talk about how KAPT works.

Second, the principle of Kapt

  1. Source and use of KAPT

Kapt can be understood as a tool for annotation processing in the context of Kotlin development. As for apt, which can be fully equivalent to Java. Because Java apt cannot handle Kotlin source files, kAPT was developed to implement apt tasks for hybrid projects or pure Kotlin projects.

It’s very simple to use:

You just need to introduce the kapt plugin, replace the original annotationProcessor with kapt, and let kotlin help you complete the original apt work.

kapt "groupId:artifactId:version"

apply plugin: 'kotlin-kapt'
Copy the code

When you introduce ‘kotlin-kapt’ in a module, KaptGenerateStu ‘ ‘b’ ‘s’ ‘${variant}k’ ‘otlin’ and kapt ‘ ‘${variant}’ ‘Kotlin’ tasks are automatically generated during module construction. Therefore, the introduction principle should be minimized and introduced on demand to avoid the impact of large compilation time.

  1. The principle of analysis

The module that introduces KAPT adds two tasks that handle annotation generation classes. Let’s take a quick look at how these two tasks work.

Here you can see that the whole process of kAPT processing is divided into two steps: “generate Stub file” and “call APT processing annotations”. You can see very clearly that kapt has nothing new in fact, the bottom is still called Java APT to complete the whole task. And by the way,

Why did Kotlin’s team do this?

Apt for Java is implemented by implementing JSR269. JSR269 defines an API for the APT plug-in, which is implemented in Java APT.

As an upstart, it is easy to think of two ways to implement similar features:

  • Rewrite the JSR269 API. Implement APT for both Kotlin files and Java files
  • Find ways to reuse Java apt

Obviously, the second path is simpler and more mature, plus there are precedents in the industry before Kotlin even thinks about it, such as Groovy’s support for APT. This is not difficult to understand kotlin’s design ideas, just to find a way to kotlin source code into Java source.

It is easy to understand why kAPT processing is divided into two steps: “generate Stub file” and “call APT processing annotation”.

Here is the general flow of the two steps.

Generating Stub files

This process is undertaken by kaptGenerateStubs${variant} Kotlin. As shown in the figure above, A.kt and B.kt are treated to produce A. Java and B. Java. Let’s see if the product is different from what we thought it would be.

On the left is a.kt file and on the right is a.java file generated by kaptGenetateStub.

As you can see, instead of generating the Equivalent Java source for Kotlin’s source code, you just generate the Java source in abi form, as long as you ensure that the corresponding method and field descriptors can be found, without dealing with the implementation content of the method body.

Call APT to handle annotations

The general flow of this process:

  • KaptTask finds the Kapt plugin registered with Kapt and finds all processors.
  • KaptTask calls JDK methods that parse the source file and generate the corresponding AST (abstract syntax tree).
  • KaptTask calls the JDK for annotation processing, and the processors found in #1 are called back inside the JDK.
  • The logic for writing a new Java file will be done in the business processors, at which point the JDK will take the new Java file for the second and third rounds of process. New Java files may also reference processor registered annotations.

So much for the whole concept of KAPT. Let’s take a look at the problems that Kapt can cause. Part of this section will be devoted to the process of solving the problem mentioned in the background.

3. Memory problems caused by KAPT

  1. Problem description

Here is a brief description of the issues mentioned in the background of this article.

During the process of Hilt access, the volcano project failed to be packaged on 16G MAC and was frequently reported to OOM. The corresponding stack is as follows:

At first glance, the stack is reported by the compiler itself, and it looks like memory has burst, but there is no obvious breakthrough from the stack.

  1. Troubleshooting and analysis process

Since it is a memory problem, we first try to find a way to repeat, recommend using VisuaxlVM for analysis, do not understand the tool students can click the link to learn, is a better tool to troubleshoot JVM problems.

  • Memory analysis

We use VisualVM to analyze the memory of Gradle daemon. It turns out that during The KAPT process, memory really keeps going up.

In order to know what’s going on in the code at these sudden memory spikes, we need to find ways to debug the code.

  • The preparatory work

Kotlin debug is slightly more troublesome than Gradle, and kotlin Compiler runs in three modes.

  • In-process: The entry to the Kotlin Compiler that is invoked in the currently launched process, where Gradle and Kotlin are in the same process.
  • Out-process: A separate process is compiled using a command line tool. The main process waits for the separate process to complete compilation.
  • Daemons: Daemons are long-running daemons that run in the background. Like Gradle daemons, if gradle finds a living daemon, it reuses it. If not, a new daemon is created.

By default, kotlin Compiler code runs in Kotlin’s daemon, so we can specify in-process mode for convenience. In this way, you can debug gradle daemons in the same way as gradle daemons.

./gradlew app:assembleCnFullDebug --stacktrace -Dorg.gradle.debug=true -Dkotlin.compiler.execution.strategy=in-process
Copy the code

  • A detailed analysis

Once breakpoint debugging is enabled, debug Kotlin can easily tease out the entire execution flow of KAPT, as shown in the following figure:

The enterTrees() method was confirmed to have OOM, which can only be followed up in JDK code.

After a walk through the JDK code, we have a general idea of how apt is handled in the JDK.

We started heap dump and the result is as follows:

As you can see from the figure, more than 10 million Scope$Entry[] objects have been created, which is obviously not normal.

However, the volcano project is very large, with a heap dump of more than 10 GIGABytes. If you directly select a Scope$Entry[] object for GC Root analysis, it will not be finished in a day.

So a demo with HILt plugged in was used for testing.

Starting with the first round, select a Scope$Entry[] object whose GC Root is as follows:

At this point, its GC Root is a Java Frame, and it should be executing a method and using it, so it’s normal to have a GC Root.

In the second round, GC Root is as follows:

It hasn’t been released yet, which is actually a little out of line.

Notice the JavacProcessingEnvironment has a piece of code like this:

/** Create a new round. */ private Round(Round prev, Set<JavaFileObject> newSourceFiles, Map<String,JavaFileObject> newClassFiles) { this(prev.nextContext(), prev.number+1, prev.compiler.log.nerrors, prev.compiler.log.nwarnings, null); this.genClassFiles = prev.genClassFiles; List<JCCompilationUnit> parsedFiles = compiler.parseFiles(newSourceFiles); roots = cleanTrees(prev.roots).appendList(parsedFiles); // Check for errors after parsing if (unrecoverableError()) return; enterClassFiles(genClassFiles); List<ClassSymbol> newClasses = enterClassFiles(newClassFiles); genClassFiles.putAll(newClassFiles); enterTrees(roots); . }Copy the code

CleanTrees () does the following:

private static <T extends JCTree> List<T> cleanTrees(List<T> nodes) {

    for (T node : nodes)

        treeCleaner.scan(node);

    return nodes;

}
Copy the code

TreeCleaner is defined as follows:

private static final TreeScanner treeCleaner = new TreeScanner() {

    public void scan(JCTree node) {

        super.scan(node);

        if (node != null)

            node.type = null;

    }

    public void visitTopLevel(JCCompilationUnit node) {

         node.packge = null;

         super.visitTopLevel(node);

    }

    public void visitClassDef(JCClassDecl node) {

         node.sym = null;

         super.visitClassDef(node);

    }

    public void visitMethodDef(JCMethodDecl node) {

         node.sym = null;

         super.visitMethodDef(node);

    }

    public void visitVarDef(JCVariableDecl node) {

         node.sym = null;

         super.visitVarDef(node);

    }

    public void visitNewClass(JCNewClass node) {

         node.constructor = null;

         super.visitNewClass(node);

    }

    public void visitAssignop(JCAssignOp node) {

         node.operator = null;

         super.visitAssignop(node);

    }

    public void visitUnary(JCUnary node) {

         node.operator = null;

         super.visitUnary(node);

    }

...
Copy the code

Clearly, the JDK designers wanted to leave objects in the syntax tree, including the symbol table, empty by traversing JCTree, giving them a chance to be freed.

However, such an operation does not free up the symbol table reference, which is stored here in the diagFormatter object of the log.

However, if this were all it would not be a serious problem, as you can see from the GC Root diagram that log.DiagFormatter only saves the previous symbol table each time.

Log. DiagFormatter doesn’t save it either, but it turns out that it still has GC Root, as follows:

Apparently, some JNI Global Reference holds it and it cannot be released.

To be sure, due to the design of JDK8, the symbol table created by each round of processing annotations will remain in memory until all processing is complete, resulting in projects with large code or a large number of processors (such as hilt introducing 13 processors). It’s easy to get OOM because it takes up too much memory. This pot JDK has to carry.

In fact, KAPT also takes precautions against this situation, so it does a memory leak detection after the annotation processing is done:

To determine whether the KAPT process has memory leaks, you can configure the log switch to look.

If an OOM occurs during the Annotation Processing, it can only throw an exception and never even go to the memory leak detection step. It can be seen that this memory leak detection does not play a significant role in the troubleshooting work of this article.

The solution

Although the problem is located in JDK, it is impossible for the official to solve it in a short time, let alone this is an old JDK version. We’ll have to find another way.

As annotation processing is carried out in JDK, the input Java file will be syntax-analyzed first, symbol table will be constructed, and many objects like Scope$Entry[] will be created.

In debug, one source file corresponds to one JCCompilationUnit, and one JCCompilationUnit contains a syntax tree.

It can be inferred from this that the memory footprint of annotation processing is proportional to the input source file.

Is it possible to reduce memory footprint by filtering incoming source files?

We analyzed the input files and found that there are a large number of R.java involved in KAPT compilation in the App Module. For medium and large projects, there are at least thousands to thousands of R.Java involved in KAPT compilation. The role of R.Java in APP compilation will not be described here.

For the App Module, r.Java is just a compilation aid. Generally speaking, app Modules are lightweight and rarely contain a lot of code, but because R.Java is involved in auxiliary compilation, IT is replaced by javaSourceRoots by AGP. However, there are many modules, and each module stores the R value of its underlying module, so the R.Java of app Module is very large and very large. In AGP 3.6.0, Google replaced R.java with R.jar to help compile.

On volcano’s project, 95% of the input files were R.java, and each R.Java had thousands of lines of code. Because r.Java is full of fields without annotations.

It can be said that r.Java files are kapt-independent and do not need to participate in parsing, adding extra execution time and memory.

So, change the javaSourceRoots code in KaptTask to the following to filter out the generated R.Java.

@get:Internal protected val javaSourceRoots: Set<File> get() = unfilteredJavaSourceRoots.filterTo(HashSet(), ::isRootAllowed).filterTo(HashSet(), { ! (it.absolutePath.contains("generated/not_namespaced_r_class_sources/")) } )Copy the code

earnings

Currently, this feature is a magic version of Kotlin that has been added to the volcano, Toutiao and other projects.

For Volcano, app: Kapt Task changed from 18min to OOM to 15s, which not only reduced the compilation time, but also saved 13G+ memory space.

As for other KAPT tasks that did not generate OOM before, they are also profitable. The picture below shows the comparison before and after the test in Toutiao:

The execution time of kAPT Task was reduced from 30.810s to 1.431s, and the speed increased by 20 times.

One more thing: while debugging the JDK, I found that JDK 8 did not do a good job in module decoupling and memory management, but I can understand that, after all, this is mainly code completed in 2013. So, from a compilation optimization perspective, upgrading the JDK version you are using in your project as soon as possible is also a big benefit (in fact, it can be done using JDK9, although it is still slow).

It should be noted that the above optimization applies to AGP 3.6.0 before AGP 3.6.0. After AGP 3.6.0, since r.Jar is involved in compilation instead of R.Java, there is no such problem. This paper focuses on the principle of KAPT, the troubleshooting process of related problems and the idea of optimization. Finally, some suggestions are given for Kapt optimization.

4. Suggestions and optimization of Kapt

In order to use Kapt without introducing large compile-related negative benefits, we have the following suggestions:

  1. Converges the KAPT scope

Project team many times before, in order to facilitate will create a library. The gradle/base gradle file, the file defines a lot of general kapt dependence, as the project modular, modular transformation, project module number more and more, Some API modules that only contain Model classes and interfaces and do not need KAPT at all also use these KAPT dependencies uniformly, making a large number of modules in the project to carry out meaningless KAPT time-consuming. Therefore, we suggest:

  • Try not to add a uniform kapt dependency for all modules in a file like library.gradle.
  • Gradle, library- API. Gradle, select the appropriate template file according to the module type, such as API type modules do not need to apply kotlin-kapt plugin, do not need to rely on kapt library

  1. Access optimization Tool

This article only describes a related optimization of kapt for memory problems, in fact, Kapt and Kotlin compilation there are many problems worth optimizing. Currently, within Byte, our team has developed a series of optimization tools to solve these problems without awareness to speed up incremental compilation. Due to the limitation of space, there will be a separate article to explain the relevant content.

  1. Try to find alternatives to Kapt

Using kapt in a project requires nothing more than a common code generation logic to reduce repetitive code writing, and kapt is not the only solution that can achieve similar results:

  • You can use the transform API officially provided by Google to create bytecode directly after compiling Java code into bytecode. Besides, the company already has byteX, any-Register and other transform frameworks. It is very convenient to write section code peg logic based on these frameworks, while utilizing the IO reuse ability of these frameworks, but also can further improve the compilation speed.
  • You can use reflection for debug packaging and continue to use Kapt for release packaging for both development experience and operational efficiency.

  1. Look forward to KSP, timely embrace

Kapt needs to go through kaptGenerateStub to convert Kotlin code to Java code before handing it over to the JDK, which is obviously too cumbersome. So, can annotation processing be performed directly in Kotlin Compiler? The answer is yes, in fact, Kotlin officially has such a scheme in a higher version, called Kotlin Symbol Processing(KSP), but it is still in the alpha stage and needs to wait for the adaptation of major processors. After stabilization, we will introduce the best practices of KSP to help people better develop annotation processing.

Join us

The Build Infra team is committed to solving the problems of Android development experience, improving the Android compilation experience, and is responsible for ensuring and improving the efficiency of r&d construction across the company’s business lines. If you are passionate about technology and pursuit of perfection, welcome to join us, we look forward to your growth with us. Currently, we have recruitment requirements in Beijing, Shanghai and Hangzhou. Please send your resume to:

[email protected], with the subject line: name-Devops-Build Infra.

🔥 Volcano Engine APMPlus application Performance Monitoring is a performance monitoring product for volcano Engine application development suite MARS. Through advanced data collection and monitoring technologies, we provide enterprises with full-link application performance monitoring services, helping enterprises improve the efficiency of troubleshooting and solving abnormal problems. At present, we specially launch “APMPlus Application Performance Monitoring Enterprise Support Action” for small and medium-sized enterprises, providing free application performance monitoring resource pack for small and medium-sized enterprises. Apply now for a chance to receive a 60-day free performance monitoring service with up to 60 million events.

👉 Click here to apply now