Android build speed optimized black tech – RocketX

I. Background description

Second, effect display

3. Problem analysis and module building

Fourth, problem solving and implementation

Five, a small surprise each day

Vi. Future prospects

I. Background description

As the size of the project grows, so does the compilation speed, sometimes requiring several minutes of compilation time for a change. Based on this common situation, RocketX was introduced to speed up full compilation by dynamically replacing the Module with an AAR during the compilation process. Let you experience the aar speed of all modules, but keep all modules easy to modify, perfect!

Second, effect display

2.1. Test project introduction

• Total of target projects3W+Class and resource files, full compilation4minOr so
• throughRocketXEffect after full increment (average 3 times for each operation)

• The project dependencies are shown below.appRely onbmBusiness module,bmBusiness modules depend on the top layerbase/commThe module

• rx(RocketX)Compile – you can see thatrx(RocketX)In no matter which module compilation speed is basically in control of 30s or so, because only compilationappAnd modified modules, other modules areaarPackages do not participate in compilation.
• Native compilation – whenbase/commModule changes. All modules at the bottom must be compiled. becauseapp/bmxxxThe module may be in usebaseInterface or variable in the module, and do not know whether changes to. (Then it’s very slow.)
• Native compilation – whenbmDiscoverIt’s changed. It just needs toappThe module andbmDiscoverTwo modules participate in compilation (fast)

Iii. Problem analysis and module building

3.1 Analysis of ideas and problems

• The last mind map involved the following questions:

1. Need to pass throughgradle pluginThe form of dynamic modification has not been changedmoduleDependence is corresponding toaarDepend on, ifmoduleModify, degenerate intoprojectProject dependencies, so that only changes are made at a timemodule 和 appTwo modules compile.
2. Need to putimplement/api moduleB, modified toimplement/api aarBAnd need to know how to add in the plug-inaarDependence and eliminate original dependence
3. We need to buildlocal mavenStore unmodifiedmoduleThe correspondingaar(You can also passflatDirInstead of faster)
4. The compilation process starts and needs to find which onemoduleDo the modification
5. You have to go through each of themmoduleTo replace the dependence of,moduleHow are dependencies obtained? Can all module dependencies be retrieved at once, or can they be called individually by module? Does modifying one module dependency block subsequent module dependency callbacks?
6. eachmoduleChange intoaarAnd then, self dependentchildDependencies (network dependencies,aar), toparent moduleHow do I find them allparent module)? Or do you just give it to me?app module? Is there anyapp 到 moduleRisk of dependency break? There needs to be a technical solution.
7. Need to behookCompile the process and complete the displacementloacal mavenIs modified inaar
8. provideASThe status barbuttonTo implement the on/off function, speed up compilation and still allow developers to use the triangles they have become accustomed torunbutton

3.2. Module construction

• According to the above analysis, although there are many problems, the whole project can be roughly divided into the following parts:

Iv. Problem Solving and Implementation:

4.1. How do I add them manuallyaarDependence, analysisimplementSource code implementation entry inDynamicAddDependencyMethodsIn thetryInvokeMethodMethods. It’s a dynamic languagemethodMissingfunction

• tryInvokeMethodThe code analysis

public DynamicInvokeResult tryInvokeMethod(String name, Object... Arguments) {// omit some code... return DynamicInvokeResult.found(this.dependencyAdder.add(configuration, normalizedArgs.get(0), (Closure)null)); }Copy the code

• dependencyAdderImplementation is aDirectDependencyAdder

private class DirectDependencyAdder implements DependencyAdder<Dependency> { private DirectDependencyAdder() { } public Dependency add(Configuration configuration, Object dependencyNotation, @Nullable Closure configureAction) { return DefaultDependencyHandler.this.doAdd(configuration, dependencyNotation, configureAction); }}Copy the code

• Finally, in theDefaultDependencyHandler.this.doAddI’m going to add, andDefaultDependencyHandler 在 projectYou can get

public interface Project extends Comparable<Project>, ExtensionAware, PluginAware { ... DependencyHandler getDependencies(); . }Copy the code

• whiledoAddMethod three parameters passdebugSource code discovery,configurationis"implementation", "api", "compileOnly"The object generated by these three strings,dependencyNotationIs aLinkHashMapThere are two key-value pairsname:aarName.ext:aarAnd the last oneconfigureAction 传 nullThat’s it. Callproject.dependencies.addEventually it will bedoAddMethod, that is, simply call add.

public Dependency add(String configurationName, Object dependencyNotation) { return this.add(configurationName, dependencyNotation, (Closure)null); } public Dependency add(String configurationName, Object dependencyNotation, A Closure configureClosure) {/ / direct call to the doAdd return here enclosing doAdd (this) configurationContainer) getByName (configurationName),  dependencyNotation, configureClosure); }Copy the code

• So add it in the same wayaar/jarImplementation code:configName 是 childProjectIn theconfigName, that is,"implementation", "api", "compileOnly"These three strings, taken intact:

fun addAarDependencyToProject(aarName: String, configName: String, project: // implement/XXX (name: 'libaccount-2.0.0', ext) {// implement/XXX (name: 'libaccount-2.0.0', ext) Val map = linkedMapOf<String, String>() map.put("name", aarName) map.put("ext", "aar") project.dependencies.add(configName, map) }Copy the code

4.2,localMavePreferred to useflatDirThis is done by specifying a cache directorygetLocalMavenCacheDirThe generatedaar/jarThe package is thrown in, and the dependency is modified by adding the corresponding package through 4.1 aboveaarYou can:

  fun flatDirs() {
        val map = mutableMapOf<String, File>()
        map.put("dirs", File(getLocalMavenCacheDir()))
        appProject.rootProject.allprojects {
            it.repositories.flatDir(map)
        }
    }
Copy the code

4.3 The compilation process starts, which one needs to be foundmoduleDo the modification

• Use files that traverse the entire projectlastModifyTimeTo do the implementation
• Have eachmoduleFor one granularity, recursively traverses the currentmoduleFile, put each filelastModifyTimeThe integration computes out a unique identifiercountTime
• throughcountTimeCompare with last time, same indicates no change, different changes. And it needs to be synchronizedcountTimeInto the local cache
• As a whole3WTime per file1.2 sAcceptable, currently in classChangeModuleUtils.ktimplement

4.4 Obtaining module dependency relationship

• The purpose is to find the time to generate a dependency graph for the entire project, and to generate a dependency graph parser here. The time to runCompilation oftaskPreviously, make sure that the substitution takes effect after the dependency is acquired, and globallymoduleAfter the dependency graph has been generated (that is, after executionbuild.gradle), so there are currently two ways:DependencyResolutionListener 和 projectsEvaluatedBut so far there are problems.

1. By listening DependencyResolutionListenerAnd, inbeforeResolveThe callback method handles:

  public interface DependencyResolutionListener {
    void beforeResolve(ResolvableDependencies var1);

    void afterResolve(ResolvableDependencies var1);
}

   project.gradle.addListener(DependencyResolutionListener listener)
Copy the code

2. But here’s the problem beforeResolveCallbacks are made multiple times and each one is executed modulethebuild.gradleResolving the dependency will result in a callback. So if you do it at the business level, wait until the last onemoduleCallback complete, pass againproject.configurationsGet allmoduleDependency graph of? The answer is yes, but it’s too late. Wait until the last one moduleCallback after parsingbeforeResolveIf a dependency cannot be modified, the following exception will be reported:

Cannot change dependencies of dependency configuration ':app:implementation' after it has been included in dependency resolution.
Copy the code

3. Change the way through the project. Gradle. ProjectsEvaluated {} after the callback to all module dependencies and modification. Dependency diagrams are available, but changing dependencies will still raise exceptions, which is too late after all.

4. So let’s go through gradle’s life cycle

5. Discovery Completed build.gradleThe life cycle after that is only: Project afterEvaluate 和 Gradle.projectsEvaluated(It didn’t work in 3). So onlyafterEvaluate ，afterEvaluateOr eachmoduleThe dependency is called back once and listens for the last one moduleThe timing of the callback and modification still report the same error. So there’s no time in the life cycle
6. Directly to findgradleSource code to find the exception occurs in:DefaultConfiguration.preventIllegalMutationThis method

private void preventIllegalParentMutation(MutationType type) { if (type ! = MutationType.DEPENDENCY_ATTRIBUTES) { if (this.resolvedState == InternalState.ARTIFACTS_RESOLVED) { throw new InvalidUserDataException(String.format("Cannot change %s of parent of %s after it has been resolved", type, this.getDisplayName())); } else if (this.resolvedState == InternalState.GRAPH_RESOLVED && type == MutationType.DEPENDENCIES) { throw new InvalidUserDataException(String.format("Cannot change %s of parent of %s after task dependencies have been resolved", type, this.getDisplayName())); }}}Copy the code

7. The main reason for this anomaly is that type 为 MutationType.DEPENDENCY_ATTRIBUTES, so where is it assigned toDEPENDENCY_ATTRIBUTESAnd modify the dependency before it is resolved to give the correlationgradleSource code call process:

8. So basically passapply plugin: 'com.android.application'Start calling in and pass the Settings

Project. Gradle. ProjectsEvaluated {} to monitor the callback, the type set to DEPENDENCY_ATTRIBUTES, Gradle also uses projectsEvaluated to resolve dependencies in the lifecycle. This is when all module dependency diagrams are generated.

9. So the final solution is inprojectsEvaluatedModify dependencies, but beforeGradlePluginUtilsBefore a listener inside it, it filters out everything set in by reflectionprojectsEvaluated The listenerActionAnonymous internal object), executed firstrockectXPlugin“, followed by other listeners. Something like this:

//AppProjectDependencies.kt init { val projectsEvaluatedList = hookProjectsEvaluatedAction() Project. Gradle. ProjectsEvaluated {/ / perform first heavy reliance on resolveDenpendency () / / executed after removing the listener (mainly adjust execution order, heavy dependence to work and not an error, May have AGP version compatibility) val clazz = Class. Class.forname (" org. Gradle. API. Invocation. Gradle ") val method = clazz.getDeclaredMethod("projectsEvaluated", Action::class.java) val mMethodInvocation = MethodInvocation(method, ArrayOf (it)) projectsEvaluatedList. ForEach {it. Dispatch (mMethodInvocation)}}} / / all monitored the projectsEvaluated removed an anonymous inner class fun hookProjectsEvaluatedAction(): List<BroadcastDispatch<BuildListener>> { var removeDispatch = mutableListOf<BroadcastDispatch<BuildListener>>() try { var buildListenerBroadcast: ListenerBroadcast<BuildListener>? = null val fBuildListenerBroadcast = DefaultGradle::class.java.getDeclaredField("buildListenerBroadcast") fBuildListenerBroadcast.isAccessible = true buildListenerBroadcast = fBuildListenerBroadcast.get(project.gradle) as? ListenerBroadcast<BuildListener> val fBroadcast = ListenerBroadcast::class.java.getDeclaredField("broadcast") fBroadcast.isAccessible = true val broadcast: BroadcastDispatch<BuildListener>? = fBroadcast.get(buildListenerBroadcast) as? BroadcastDispatch<BuildListener> val fDispatchers = broadcast? .javaClass? .getDeclaredField("dispatchers") fDispatchers? .isAccessible = true val dispatchers: ArrayList<BroadcastDispatch<BuildListener>>? = fDispatchers? .get(broadcast) as? ArrayList<BroadcastDispatch<BuildListener>> val clazz = Class.forName("org.gradle.internal.event.BroadcastDispatch\$ActionInvocationHandler") val iterator = dispatchers?.iterator() iterator?.let { while (iterator.hasNext()) { try { val next = iterator.next() val fDispatch = next.javaClass.getDeclaredField("dispatch") fDispatch.isAccessible = true val dispatch: Any? = fDispatch.get(next) val fMethodName = clazz.getDeclaredField("methodName") fMethodName.isAccessible = true val methodName = fMethodName.get(dispatch) as? String if (methodName?.contains("projectsEvaluated") == true) { removeDispatch.add(next) iterator.remove() } } catch (ignore: Exception) { } } } } catch (ignore: Exception) { } return removeDispatch }Copy the code

10. This is a perfect solution to the change dependency timing problem. See Issue18 for more details

• How to get eachmoduleDependence, dependence is hidden inConfiguration.dependencies, then throughproject.configurations.maybeCreate(configName)Find allConfigurationObject, you can get eachmodulethedependencies

4.5,moduledependenciesprojectreplaceaarTechnical solution

• eachmoduleThe traversal order of dependency substitution is unordered, so the technical solution needs to support unordered substitution
• The scheme currently used is: if the current moduleANo change, need to putAthroughlocalMavenTo swapA.aarAnd put theA.aarAs well asA 的 childDependency, given to the first levelparent moduleCan. (You may wonder what to do if the parent Module is also an AAR. In fact, there is no problem with this part. I won’t elaborate on it here because it is too long.)
• Why should I give it to youparentYou can’t give it directlyappBelow is a simple example ifB.aarDon’t giveAModule,AuseBModule interface missing, causing compilation failure

• Give the technical solution demonstration of the overall project replacement:

• The overall realization is inDependenciesHelper.ktIn this class, because it is too long to talk about, interested in the open source library code

4.5,hookCompile the process and complete the displacementloacal mavenIs modified inaar

• Click on the trianglerun, the command executed isapp:assembleDebugThat need to be inassembleDebugI’ll fill it inuploadLocalMavenTaskThrough thefinalizedByourtaskRun to synchronize the modifiedaar ：

val localMavenTask = childProject.tasks.maybeCreate("uploadLocalMaven"+buildType.capitalize(),LocalMavenTask::class.java) localMavenTask.localMaven = this@AarFlatLocalMaven bundleTask? .finalizedBy(localMavenTask)Copy the code

4.6, provideASThe status barbuttonThe little rocket buttons one spitfire and one without spitfire representenable/disableA broomclean rockectxThe cache needs to be writtenintellij idea pluginThat is, there are currently two plugins, onegradleA pluginASPlug-in:

5. A Little surprise a day (bugMore)

5.1 Discovery clickrunButton to execute the commandapp:assembleDebug, eachmodule 在 outputIt’s not packed outaar

${Flavor}${BuildType}Aar is a task that runs gradle :assembleDebug.

        android.applicationVariants.forEach {
            getAppAssembleTask(ASSEMBLE + it.flavorName.capitalize() + it.buildType.name.capitalize())?.let { task ->
                    hookBundleAarTask(task, it.buildType.name)
                }
        }
Copy the code

5.2. Repeated problems of multiple JAR packages were found after operation

Implementation fileTree(dir: “libs”, include: [“*.jar”]) implementation fileTree(dir: “libs”, include: [“*.jar”]) It can be determined by the following code:

// There are two kinds of dependencies: No need to add to parent, Because the jar package directly into its aar of libs folder if (childDepency is DefaultSelfResolvingDependency && (childDepency. Files is DefaultConfigurableFileCollection | | childDepency. Files is DefaultConfigurableFileTree)) {/ / here is dependent on the following two: No need to add to parent, // implementation RootProject. files("libs/tingyun-ea-agent-android-2.15.4.jar") // implementation rootProject.files("libs/tingyun-ea-agent-android-2.15.4.jar" implementation fileTree(dir: "libs", include: ["*.jar"]) } else { parentProject.key.dependencies.add(childConfig.name, childDepency) }Copy the code

5.3, discovery,aar/jarThere are multiple ways to rely on it

 implementation (name: 'libXXX', ext: 'aar') 
 implementation files("libXXX.aar")
Copy the code

Solution: Using the first and the second to merge with the AAR leads to class duplication problems

5.4, discovery,aarNew postural dependence

configurations.maybeCreate("default")
artifacts.add("default", file('lib-xx.aar'))
Copy the code

Default Config is actually the holder of the final output AAR of the Module. Default Config can hold a list of AAR. So manually adding an AAR to default Config is equivalent to packaging the current Module.

Solution: by childProject. Configurations. MaybeCreate (” default “). Find all artifacts added aar, release localmaven separately

fun getAarByArtifacts(childProject: Project): MutableList<String> {// Find all current artifacts. Add ("default", File ('xxx.aar')) relies on the coming aar var listArtifact = mutableListOf< defaultartifact >() var aarList = mutableListOf<String>() childProject.configurations.maybeCreate("default").artifacts? .foreach {if (it is DefaultPublishArtifact && "aar".equals(it.type)) {listartifact.add (it)}} // copy a copy to localMaven listArtifact.forEach { it.file.copyTo(File(FileUtil.getLocalMavenCacheDir(), it.file.name), Add (removeExtension(it.file.name))} return aarList} true) // remove suffix (.aar) aarList.add(removeExtension(it.file.name))} return aarList}Copy the code

5.5, discovery,android moduleIt could be packed upjar

Solution: The code is implemented in jarflatLocalMaven.kt by finding a task named JAR and injecting uploadLocalMaven Task after the JAR task

5.6, discovery,arouter 有 bug.transformDon’t passoutputProvider.deleteAll()Clean up the old cache

The arouter problem has been resolved and the code has been merged. However, no new plugin version was released to mavenCentral, so we solved it for Arouter first. However, arouter didn’t start incremental compilation, causing DexArchiveBuilderTask to run very slowly. In the project, I changed the arouter plug-in source code to support the TransForm increment speed to be doubled. The details will be discussed in the next section and dex speed optimization together.

Vi. Future prospects

At present, the preliminary version has been able to run in the project, but there are still many small problems emerging and solving, there is a long way to go, I will continue to search.

Next plan:

• dexBuild taskTo optimize the
• Resolve compatibility issues

At present, the plugin tends to be stable, and friends who like to try it can access through github tutorial, and pay attention to the later progress together. The solutions of subsequent issues will be updated in this article. If you like this article, please give us star. Github open source address