In the last article, how to accelerate the construction speed of APK and how to reduce the compilation time from 130 seconds to 17 seconds, I talked about the idea and preliminary implementation of optimization. After a period of time, the performance and stability of optimization have been greatly improved. Here I would like to thank you for your suggestions and the issue on Github. This article introduces the main optimization points and new functions as well as the pit.
Project address: github.com/typ0520/fas… Corresponding to the tag: github.com/typ0520/fas… Demo code: github.com/typ0520/fas…
Note: It is recommended to leave the Fastdex code and the Demo code behind. Most of the examples in this article can be run directly in the demo project. /gradlew will run tasks on MAC, or gradlew will run tasks on Windows
# # #, transformClassesWithJarMergingForDebug interception mission
Patch before packaging, is the no change of class from app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar is removed, there are two problems with this approach
- 1. This file is combined transformClassesWithJarMergingForDebugThis task exists only when multidex is enabled. If multidex is not enabled, the task is executed transformClassesWithDexForDebugTask input and not as a combined. The jar, but project classes directory (app/build/intermediates/classes/debug) output and rely on library jar jar and third-party libraries;
- 2, if existtransformClassesWithJarMergingForDebugIt would be inefficient to spend a lot of time synthesing Combined. Jar first and then remove the unchanged class from Combined. It would be much more efficient to bypass the synthesing of Combined
Now you need to first get transformClassesWithJarMergingForDebug task before and after the life cycle, Implementation way is similar to intercept transformClassesWithDexForDebug in scheme, complete test code address github.com/typ0520/fas…
public class MyJarMergingTransform extends Transform {
Transform base
MyJarMergingTransform(Transform base) {
this.base = base
}
@Override
void transform(TransformInvocation invocation) throws TransformException, IOException, InterruptedException {
List<JarInput> jarInputs = Lists.newArrayList();
List<DirectoryInput> dirInputs = Lists.newArrayList();
for (TransformInput input : invocation.getInputs()) {
jarInputs.addAll(input.getJarInputs());
}
for (TransformInput input : invocation.getInputs()) {
dirInputs.addAll(input.getDirectoryInputs());
}
for (JarInput jarInput : jarInputs) {
println("==jarmerge jar : ${jarInput.file}")}for (DirectoryInput directoryInput : dirInputs) {
println("==jarmerge directory: ${directoryInput.file}")
}
File combinedJar = invocation.outputProvider.getContentLocation("combined", base.getOutputTypes(), base.getScopes(), Format.JAR);
println("==combinedJar exists ${combinedJar.exists()} ${combinedJar}")
base.transform(invocation)
println("==combinedJar exists ${combinedJar.exists()} ${combinedJar}") } } public class MyDexTransform extends Transform { Transform base MyDexTransform(Transform base) { this.base = base } @Override void transform(TransformInvocation transformInvocation) throws TransformException, IOException, InterruptedException { List<JarInput> jarInputs = Lists.newArrayList(); List<DirectoryInput> dirInputs = Lists.newArrayList();for (TransformInput input : transformInvocation.getInputs()) {
jarInputs.addAll(input.getJarInputs());
}
for (TransformInput input : transformInvocation.getInputs()) {
dirInputs.addAll(input.getDirectoryInputs());
}
for (JarInput jarInput : jarInputs) {
println("==dex jar : ${jarInput.file}")}for (DirectoryInput directoryInput : dirInputs) {
println("==dex directory: ${directoryInput.file}")
}
base.transform(transformInvocation)
}
}
project.afterEvaluate {
android.applicationVariants.all { variant ->
project.getGradle().getTaskGraph().addTaskExecutionGraphListener(new TaskExecutionGraphListener() {
@Override
public void graphPopulated(TaskExecutionGraph taskGraph) {
for (Task task : taskGraph.getAllTasks()) {
if(task.getProject().equals(project) && task instanceof TransformTask && task.name.toLowerCase().contains(variant.name.toLowerCase())) { Transform transform = ((TransformTask) Task).getTransform() // This task is available if multidex is enabledif((((transform instanceof JarMergingTransform)) && ! (transform instanceof MyJarMergingTransform))) { project.logger.error("==fastdex find jarmerging transform. transform class: " + task.transform.getClass() + " . task name: " + task.name)
MyJarMergingTransform jarMergingTransform = new MyJarMergingTransform(transform)
Field field = getFieldByName(task.getClass(),'transform')
field.setAccessible(true)
field.set(task,jarMergingTransform)
}
if((((transform instanceof DexTransform)) && ! (transform instanceof MyDexTransform))) { project.logger.error("==fastdex find dex transform. transform class: " + task.transform.getClass() + " . task name: "MyDexTransform fastdexTransform = new MyDexTransform(transform) Field Field = getFieldByName(task.getClass(),'transform')
field.setAccessible(true) field.set(task,fastdexTransform) } } } } }); }}Copy the codeRun the above code in app/build.gradle./gradlew assembleDebug
-
Log output when multidex(multiDexEnabled True) is enabled **
:app:mergeDebugAssets :app:transformClassesWithJarMergingForDebug ==jarmerge jar : / Users/tong/Projects/fastdex - test - project/jarmerging - test/app/libs/exist - in - the app - libs - 2.1.2. Jar = = jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.android.support/multi Dex / 1.0.1 / jars/classes. The jar = = jarmerge jar: / Users/tong/Applications/android SDK - macosx/extras/android/m2repository/com/android/support/support - annotations / 23.3.0 / s Upport - annotations - 23.3.0. Jar = = jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.jakewharton/butterkni Fe / 8.0.1 / jars/classes. The jar = = jarmerge jar: / Users/tong /. Gradle/caches/files/modules - 2-2.1 / com. Jakewharton/butterknife b89f45d02d8b09400b472fab annotations / 8.0.1 / in 345 7 b7e38f4ede1f/butterknife - annotations - 8.0.1. Jar = = jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar ==jarmerge jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/un specified/jars/classes.jar ==jarmerge directory: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/classes/debug ==combinedJar existsfalse/Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/ combined.jar ==combinedJar existstrue/Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/ combined.jar :app:transformClassesWithMultidexlistForDebug :app:transformClassesWithDexForDebug ===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/ combined.jar :app:mergeDebugJniLibFoldersCopy the code -
Disable log output when multidex(multiDexEnabled False) **
:app:mergeDebugAssets :app:transformClassesWithDexForDebug ===dex jar : / Users/tong/Projects/fastdex - test - project/jarmerging - test/app/libs/exist - in - the app - libs - 2.1.2. Jar = = = dex jar: / Users/tong/Applications/android SDK - macosx/extras/android/m2repository/com/android/support/support - annotations / 23.3.0 / s Upport - annotations - 23.3.0. Jar = = = dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/com.jakewharton/butterkni Fe / 8.0.1 / jars/classes. The jar = = = dex jar: / Users/tong /. Gradle/caches/files/modules - 2-2.1 / com. Jakewharton/butterknife b89f45d02d8b09400b472fab annotations / 8.0.1 / in 345 7 b7e38f4ede1f/butterknife - annotations - 8.0.1. Jar = = = dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar ===dex jar : /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/un specified/jars/classes.jar ===dex directory: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/classes/debug :app:mergeDebugJniLibFoldersCopy the codeAs you can see from the log output above, all you need to do is generate patch.jar as indicated by the red arrow below
If the entry corresponds to the project code, the injection will be performed. If the entry corresponds to the project code, the injection will be skipped by the third-party library. Now you intercept the Jarmerge task and inject all classes in the DirectoryInput directory. This is much more efficient than before
### support library projects directly dependent on
Take the following project as an example github.com/typ0520/fas…
This project consists of three sub-projects
- app (android application project)
- aarlib (android library project)
- javalib (java project)
The APP project relies on Aarlib and Javalib
dependencies {
compile fileTree(dir: 'libs', include: ['*.jar'])
compile 'com. Jakewharton: butterknife: 8.0.1'
apt 'com. Jakewharton: butterknife - compiler: 8.0.1'
compile project(':javalib')
compile project(':aarlib')
compile project(':libgroup:javalib2')}Copy the codeTo use the compile project (‘ : ‘XXX) project depend on this way, in the process of the construction of the apk is as jar processing, from the intercept transformClassesWithJarMergingForDebug tasks log output can be proved
===dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar ===dex jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/un specified/jars/classes.jarCopy the codeThe patch package that changed the Library project didn’t work because it only removed the changed class from DirectoryInput and didn’t remove the library project output JAR. At this point you need to know which JarInput items belong to the Library project and which belong to third-party libraries. The most direct way is by file system path, but this requires the exclusion of library project dependencies such as jars placed directly in the libs directory
= = jarmerge jar: / Users/tong/Projects/fastdex - test - project/jarmerging - test/app/libs/exist - in - the app - libs - 2.1.2. JarCopy the codeSecondly, if the library directory and APP project are not in the same directory, we should make a fault tolerance judgment
==jarmerge jar: /Users/tong/Projects/fastdex-test-project/jarmerging-test/libgroup/javalib2/build/libs/javalib2.jarCopy the codeFinally, I gave up the way to determine the path, and turned to the API of Android Gradle to get the output JAR path of each library project. After browsing the source code, I found that 2.0.0, 2.2.0 and 2.3.0 correspond to different APIS, which can be solved by judging the version. The code is as follows
public class LibDependency {
public final File jarFile;
public final Project dependencyProject;
public final boolean androidLibrary;
LibDependency(File jarFile, Project dependencyProject, boolean androidLibrary) {
this.jarFile = jarFile
this.dependencyProject = dependencyProject
this.androidLibrary = androidLibrary
}
boolean equals(o) {
if (this.is(o)) return true
if(getClass() ! = o.class)return false
LibDependency that = (LibDependency) o
if(jarFile ! = that.jarFile)return false
return true
}
int hashCode() {
return(jarFile ! = null ? jarFile.hashCode() : 0) } @Override public StringtoString() {
return "LibDependency{" +
"jarFile=" + jarFile +
", dependencyProject=" + dependencyProject +
", androidLibrary=" + androidLibrary +
'} ';
}
private static Project getProjectByPath(Collection<Project> allprojects, String path) {
returnFind {it.path.equals(path)}} /** * Scan dependencies (<= 2.3.0) * @param library * @param libraryDependencies */ private static final void scanDependency(com.android.builder.model.Library library,Set<com.android.builder.model.Library> libraryDependencies) {if (library == null) {
return
}
if (library.getProject() == null) {
return
}
if (libraryDependencies.contains(library)) {
return
}
libraryDependencies.add(library)
if (library instanceof com.android.builder.model.AndroidLibrary) {
List<com.android.builder.model.Library> libraryList = library.getJavaDependencies()
if(libraryList ! = null) {for (com.android.builder.model.Library item : libraryList) {
scanDependency(item,libraryDependencies)
}
}
libraryList = library.getLibraryDependencies()
if(libraryList ! = null) {for (com.android.builder.model.Library item : libraryList) {
scanDependency(item,libraryDependencies)
}
}
}
else if (library instanceof com.android.builder.model.JavaLibrary) {
List<com.android.builder.model.Library> libraryList = library.getDependencies()
if(libraryList ! = null) {for(com.android.builder.model.Library item : LibraryList) {scanDependency(item,libraryDependencies)}}}} /** * Scan dependencies (2.0.0 <= android-build-version <= 2.2.0) * @param library * @param libraryDependencies */ private static final void scanDependency_2_0_0(Object library,Set<com.android.builder.model.Library> libraryDependencies) {if (library == null) {
return
}
if (library.getProject() == null){
return
}
if (libraryDependencies.contains(library)) {
return
}
libraryDependencies.add(library)
if (library instanceof com.android.builder.model.AndroidLibrary) {
List<com.android.builder.model.Library> libraryList = library.getLibraryDependencies()
if(libraryList ! = null) {for(com.android.builder.model.Library item : LibraryList) {scanDependency_2_0_0(item,libraryDependencies)}}}} /** *'xxx')
* @param project
* @return
*/
public static final Set<LibDependency> resolveProjectDependency(Project project, ApplicationVariant apkVariant) {
Set<LibDependency> libraryDependencySet = new HashSet<>()
VariantDependencies variantDeps = apkVariant.getVariantData().getVariantDependency();
if (Version.ANDROID_GRADLE_PLUGIN_VERSION.compareTo("2.3.0") >= 0) {
def allDependencies = new HashSet<>()
allDependencies.addAll(variantDeps.getCompileDependencies().getAllJavaDependencies())
allDependencies.addAll(variantDeps.getCompileDependencies().getAllAndroidDependencies())
for (Object dependency : allDependencies) {
if(dependency.projectPath ! = null) { def dependencyProject = getProjectByPath(project.rootProject.allprojects,dependency.projectPath); boolean androidLibrary = dependency.getClass().getName().equals("com.android.builder.dependency.level2.AndroidDependency");
File jarFile = null
if (androidLibrary) {
jarFile = dependency.getJarFile()
}
else {
jarFile = dependency.getArtifactFile()
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,androidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
}
else if (Version.ANDROID_GRADLE_PLUGIN_VERSION.compareTo("2.2.0") >= 0) {
Set<Library> librarySet = new HashSet<>()
for (Object jarLibrary : variantDeps.getCompileDependencies().getJarDependencies()) {
scanDependency(jarLibrary,librarySet)
}
for (Object androidLibrary : variantDeps.getCompileDependencies().getAndroidDependencies()) {
scanDependency(androidLibrary,librarySet)
}
for (com.android.builder.model.Library library : librarySet) {
boolean isAndroidLibrary = (library instanceof AndroidLibrary);
File jarFile = null
def dependencyProject = getProjectByPath(project.rootProject.allprojects,library.getProject());
if (isAndroidLibrary) {
com.android.builder.dependency.LibraryDependency androidLibrary = library;
jarFile = androidLibrary.getJarFile()
}
else {
jarFile = library.getJarFile();
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,isAndroidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
else {
Set librarySet = new HashSet<>()
for (Object jarLibrary : variantDeps.getJarDependencies()) {
if(jarLibrary.getProjectPath() ! = null) { librarySet.add(jarLibrary) } //scanDependency_2_0_0(jarLibrary,librarySet) }for (Object androidLibrary : variantDeps.getAndroidDependencies()) {
scanDependency_2_0_0(androidLibrary,librarySet)
}
for (Object library : librarySet) {
boolean isAndroidLibrary = (library instanceof AndroidLibrary);
File jarFile = null
def projectPath = (library instanceof com.android.builder.dependency.JarDependency) ? library.getProjectPath() : library.getProject()
def dependencyProject = getProjectByPath(project.rootProject.allprojects,projectPath);
if (isAndroidLibrary) {
com.android.builder.dependency.LibraryDependency androidLibrary = library;
jarFile = androidLibrary.getJarFile()
}
else {
jarFile = library.getJarFile();
}
LibDependency libraryDependency = new LibDependency(jarFile,dependencyProject,isAndroidLibrary)
libraryDependencySet.add(libraryDependency)
}
}
return libraryDependencySet
}
}Copy the codePut the above code and the following code into build.gradle
project.afterEvaluate {
android.applicationVariants.all { variant ->
def variantName = variant.name.capitalize()
if ("Debug".equals(variantName)) {
LibDependency.resolveProjectDependency(project,variant).each {
println("==androidLibrary: " + it.androidLibrary + " ,jarFile: " + it.jarFile)
}
}
}
}
task resolveProjectDependency<< {
}Copy the codeExecuting./gradlew resolveProjectDependency yields the following output
==androidLibrary: true,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/app/build/intermediates/exploded-aar/jarmerging-test/aarlib/un specified/jars/classes.jar ==androidLibrary:false ,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/javalib/build/libs/javalib.jar
==androidLibrary: false ,jarFile: /Users/tong/Projects/fastdex-test-project/jarmerging-test/libgroup/javalib2/build/libs/javalib2.jarCopy the codeGiven these paths, we can make matches in traversing JarInput, as long as the path list belongs to the output jars of the Library project. There are two places to use this
-
The library output jar classinject.groovy is injected when fully packaged
public static void injectJarInputFiles(FastdexVariant fastdexVariant, HashSet<File> jarInputFiles) { def project = fastdexVariant.project long start = System.currentTimeMillis() Set<LibDependency> libraryDependencies = fastdexVariant.libraryDependencies List<File> projectJarFiles = new ArrayList<>() ArrayList<>()':xxx')) for (LibDependency dependency : libraryDependencies) { projectJarFiles.add(dependency.jarFile) } if (fastdexVariant.configuration.debug) { project.logger.error("==fastdex projectJarFiles : ${projectJarFiles}")}for (File file : jarInputFiles) { if(! projectJarFiles.contains(file)) {continue } project.logger.error("==fastdex ==inject jar: ${file}") ClassInject.injectJar(fastdexVariant,file,file) } long end = System.currentTimeMillis() project.logger.error("==fastdex inject complete jar-size: ${projectJarFiles.size()} , use: ${end - start}ms")}Copy the code - Remove the changing class Jaroperation.groovy from the Library project output JAR when the patch is packaged
public static void generatePatchJar(FastdexVariant fastdexVariant, TransformInvocation transformInvocation, File patchJar) throws IOException { Set<LibDependency> libraryDependencies = fastdexVariant.libraryDependencies Map<String,String> jarAndProjectPathMap = new HashMap<>() List<File> projectJarFiles = new ArrayList<>() // Get the output jars of all dependent projects.':xxx'))
for(LibDependency dependency : libraryDependencies) { projectJarFiles.add(dependency.jarFile) jarAndProjectPathMap.put(dependency.jarFile.absolutePath,dependency.dependencyProject.projectDir.absolutePath) } Set<File> directoryInputFiles = new HashSet<>(); Jar Set<File> jarInputFiles = new HashSet<>();for (TransformInput input : transformInvocation.getInputs()) {
Collection<DirectoryInput> directoryInputs = input.getDirectoryInputs()
if(directoryInputs ! = null) {for (DirectoryInput directoryInput : directoryInputs) {
directoryInputFiles.add(directoryInput.getFile())
}
}
if(! projectJarFiles.isEmpty()) { Collection<JarInput> jarInputs = input.getJarInputs()if(jarInputs ! = null) {for (JarInput jarInput : jarInputs) {
if (projectJarFiles.contains(jarInput.getFile())) {
jarInputFiles.add(jarInput.getFile())
}
}
}
}
}
def project = fastdexVariant.project
File tempDir = new File(fastdexVariant.buildDir,"temp")
FileUtils.deleteDir(tempDir)
FileUtils.ensumeDir(tempDir)
Set<File> moudleDirectoryInputFiles = new HashSet<>()
DiffResultSet diffResultSet = fastdexVariant.projectSnapshoot.diffResultSet
for (File file : jarInputFiles) {
String projectPath = jarAndProjectPathMap.get(file.absolutePath)
List<String> patterns = diffResultSet.addOrModifiedClassesMap.get(projectPath)
if(patterns ! = null && ! patterns.isEmpty()) { File classesDir = new File(tempDir,"${file.name}-${System.currentTimeMillis()}")
project.copy {
from project.zipTree(file)
for (String pattern : patterns) {
include pattern
}
into classesDir
}
moudleDirectoryInputFiles.add(classesDir)
directoryInputFiles.add(classesDir)
}
}
JarOperation.generatePatchJar(fastdexVariant,directoryInputFiles,moudleDirectoryInputFiles,patchJar);
}Copy the codeNew snapshot comparison module
Fastdex currently has three areas for comparison
- A snapshot is taken for the current dependent library during the full package, and a change is made during the patch package
- Check whether all androidmanifest.xml in the app project and all dependent Android Library projects has changed since the last package.
- Snapshots are taken of all Java and Kotlin files during full packaging, and changes are made to those source files during patch packaging
In the first scenario, for example, to illustrate the principle of comparison, a full package generates a text file with the current dependencies written into it to be separated by a newline
/ Users/tong/Projects/fastdex/sample/app/libs/FM - SDK - 2.1.2. Jar /Users/tong/Projects/fastdex/sample/javalib/build/libs/javalib.jarCopy the codeWhen the patch is packaged, the text file is first read into the ArrayList, and then the current dependency list page is put into the ArrayList. You can obtain new items and delete items by performing the following operations. If there are deleted items and new items, the dependency is considered to have changed
ArrayList<String> old = new ArrayList<>();
old.add("/ Users/tong/Projects/fastdex/sample/app/libs/FM - SDK - 2.1.2. Jar");
old.add("/Users/tong/Projects/fastdex/sample/javalib/build/libs/javalib.jar");
ArrayList<String> now = new ArrayList<>();
now.add("/ Users/tong/Projects/fastdex/sample/app/libs/FM - SDK - 2.1.2. Jar");
now.add("/Users/tong/Projects/fastdex/sample/javalib/build/libs/new.jar"); Set<String> deletedNodes = new HashSet<>(); deletedNodes.addAll(old); deletedNodes.removeAll(now); // Add an item Set<String> increasedNodes = new HashSet<>(); increasedNodes.addAll(now); // If you don't use ArrayList to set a layer, sometimes you can't remove it. increasedNodes.removeAll(old); Set<String> needDiffNodes = new HashSet<>(); needDiffNodes.addAll(now); needDiffNodes.addAll(old); needDiffNodes.removeAll(deletedNodes); needDiffNodes.removeAll(increasedNodes);Copy the codeNote: Text comparisons do not update, but file comparisons do
All snapshot comparisons are based on an abstraction of the code above, which can be found at github.com/typ0520/fas…
Four, dex merge
After full packaging, more and more source files will change according to the normal development pace, and more and more classes will participate in the generation of dex, resulting in slower and slower patch packaging. An easy way to solve this problem is to put each patch.dex generated into the dex cache at the time of the full package (it must precede the previous dex) and update the source code snapshot, which has two disadvantages
- 1. Every patch package must be injected into the class file to fix the pre-verify bug mentioned in the previous article
- 2. Patch. Dex needs to be cached every time a patch is packaged, resulting in more and more dex in the following directory
app/build/intermediates/transforms/dex/debug/folders/1000/1f/mainCopy the code
The solution to the second problem is to merge the class in patch.dex into the cached dex, so that there is no need to keep all the patch.dex. A tricky problem is that if the number of methods in the cached dex is already 65535, adding the new class will definitely burst. Finally, Fastdex chooses to merge the patch.dex directly into the cache (merged-patch.dex) when triggering dex merge for the first time. In the future, when dex merge is triggered, merge patch.dex and merged-patch.dex (this also has potential problems, if there are too many changes in the class may lead to a 65535 error in dex merge).
The first issue was resolved by adding a configurable option to merge when more than three source files change by default. This allows you to restore the status of multiple source files without having to inject and merge them every time
This dex merge tool was found in Freeline. If you are interested, you can download it and try calling github.com/typ0520/fas…
java -jar fastdex-dex-merge.jar output.dex patch.dex merged-patch.dexCopy the code
Support for annotation generators
Annotations are becoming more and more popular in Android development, with ButterKnife, EventBus, and more choosing to use annotations for configuration. Annotations are divided into two types based on processing time, runtime annotations and compile-time annotations, which have been criticized by some for performance issues. The core of compile-time annotations relies on APT(Annotation Processing Tools). The principle is to add annotations to some code elements (such as types, functions, fields, etc.). At compile time, the compiler checks subclasses of AbstractProcessor. The process function of that type is called, and all the annotated elements are passed to the process function so that the developer can process them at compile time, for example, generating new Java classes from annotations. This is the basic principle of open source libraries such as ButterKnife and EventBus. The Java API already provides a framework for scanning source code and parsing annotations. You can extend the AbstractProcessor class to provide your own parsing annotation logic
– reference blog.csdn.net/industrious…
Although it can improve the efficiency of the runtime, it also brings some trouble to the development
-
AbstractProcessor classes are only used at compile time, not at run time, but if you compile dependent packages, they will be packaged into the dex
Take this project as an example (I suggest pulling down the code, which will be used in several places later) github.com/typ0520/fas…
Butterknife7.0.1 is relied on in the app
dependencies { compile 'com. Jakewharton: butterknife: 7.0.1' }Copy the codeThe annotation generator in ButterKnife7.0.1 is called the ButterKnifeProcessor
Perform. / gradlew app: assembleDebug
As can be seen from the above ButterKnifeProcessor. Class is packaged into dex
- In order to avoid the above situation, it can be introduced by annotationProcessor. Butterknife8.8.1 makes ButterKnifeProcessor independent into butterknife-Compiler module. The ButterKnife module only preserves code that is needed at runtime
Butterknife8.8.1 is relied on in App2
apply plugin: 'com.jakewharton.butterknife'
dependencies {
compile 'com. Jakewharton: butterknife: 8.8.1'
annotationProcessor 'com. Jakewharton: butterknife - compiler: 8.8.1'
}Copy the codePerform. / gradlew app2: assembleDebug
It can be seen from the figure above that all codes under the Butterknife.com Piler package are not packed into dex. Relying on AbstractProcessor-related code via annotationProcessor has the above benefits, but makes incremental compilation unavailable, In brief is a normal project execution compileDebugJavaWithJavac task calls javac will only compile the content change of Java source files, If you use the annotationProcessor every compileDebugJavaWithJavac mission all projects are involved in compiling the Java file, imagine if the project has hundreds of thousands of Java file to compile the acid bright. We can do a test and still use the project github.com/typ0520/fas…
Annotation-generators contain three subprojects
-
App rely on 7.0.1
compile 'com. Jakewharton: butterknife: 7.0.1'Copy the code -
App2 rely on 8.8.1
dependencies { compile 'com. Jakewharton: butterknife: 8.8.1' annotationProcessor 'com. Jakewharton: butterknife - compiler: 8.8.1' }Copy the code - App3 does not contain any AbstractProcessor
The three sub project contains two Java file com/lot/typ0520 / annotation_generators/HAHA. Java com/github/typ0520/annotation_generators/MainActivity.java
The test idea is to check the update time of the mainactivity. class file, change the hahaha. Java to compile the file, and check whether the update time of the mainactivity. class file is the same as that before compilation
Run the test from the increment_compile_test.sh shell script (you can test V_V manually if you use Windows)
#! /bin/bash
sh gradlew assembleDebug
test_increment_compile() {
echo "= = = = = = = = testThe ${1}Whether deltas are supported,The ${2}"
str=$(stat -x The ${1}/build/intermediates/classes/debug/com/github/typ0520/annotation_generators/MainActivity.class | grep 'Modify')
echo $str
echo 'package com.github.typ0520.annotation_generators; ' > The ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo 'public class HAHA {' >> The ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo " public long millis = $(date +%s);" >> The ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
echo '} ' >> The ${1}/src/main/java/com/github/typ0520/annotation_generators/HAHA.java
sh gradlew The ${1}:assembleDebug > /dev/null
str2=$(stat -x The ${1}/build/intermediates/classes/debug/com/github/typ0520/annotation_generators/MainActivity.class | grep 'Modify')
echo $str2
echo ' '
if [ "$str"= ="$str2" ];then
echo "The ${1}Java, mainActivity. class unchanged"
else
echo "The ${1}Change only hahaha. Java, mainActivity. class changes"
fi
}
test_increment_compile app "The compile 'com. Jakewharton: butterknife: 7.0.1'"
test_increment_compile app2 "AnnotationProcessor com. Jakewharton: butterknife - compiler: 8.8.1 '"
test_increment_compile app3 "Not using any AbstractProcessor."Copy the codePerform sh increment_compile_test. Sh
The output of the log proves what is described above
Since native does not support this, we will do this in our custom Java compile task. We can compare the Java source files that have changed with the previous snapshot module, so we can concatenate the Javac command parameters and call the Java files that only compile the changes
Demo has written a compilation task to facilitate you to understand how these parameters are spliced, too much code here will not be posted github.com/typ0520/fas… Github.com/typ0520/fas…
You can call./gradlew mycompile1 or./gradlew mycompile2 to see what the final command will be
The code for the corresponding module in Fastdex can be found at github.com/typ0520/fas…
Six, filled pit
Solve the bug was not ready to say this, because the most valuable thing is not to solve the problem, but how to discover and recreate the problem, this is really not good description V_V, at the request of JianYou or picked some relatively nutritious said problems, mainly said the solution, as for the question is how to locate and return can only try our best to describe.
1, issues# 2
Github.com/typ0520/fas… @hexi
The cause of this problem is that the original YtxApplication class in the project has been replaced with FastdexApplication, and ClassCastException is reported when something like the following is performed in the activity
MyApplication app = (MyApplication) getApplication();Copy the codeThe solution is found in the source code of instant-Run, which replaces all references to Application in the Android API with instances
public static void monkeyPatchApplication( Context context,
Application bootstrap,
Application realApplication,
String externalResourceFile) {
try {
// Find the ActivityThread instance forthe current thread Class<? > activityThread = Class.forName("android.app.ActivityThread");
Object currentActivityThread = getActivityThread(context, activityThread);
// Find the mInitialApplication field of the ActivityThread to the real application
Field mInitialApplication = activityThread.getDeclaredField("mInitialApplication");
mInitialApplication.setAccessible(true);
Application initialApplication = (Application) mInitialApplication.get(currentActivityThread);
if(realApplication ! = null && initialApplication == bootstrap) { mInitialApplication.set(currentActivityThread, realApplication); } // Replace all instance of the stub applicationin ActivityThread#mAllApplications with the
// real one
if(realApplication ! = null) { Field mAllApplications = activityThread.getDeclaredField("mAllApplications");
mAllApplications.setAccessible(true);
List<Application> allApplications = (List<Application>) mAllApplications
.get(currentActivityThread);
for (int i = 0; i < allApplications.size(); i++) {
if (allApplications.get(i) == bootstrap) {
allApplications.set(i, realApplication);
}
}
}
// Figure out how loaded APKs are stored.
// API version 8 has PackageInfo, 10 has LoadedApk. 9, I don't know. Class
loadedApkClass; try { loadedApkClass = Class.forName("android.app.LoadedApk"); } catch (ClassNotFoundException e) { loadedApkClass = Class.forName("android.app.ActivityThread$PackageInfo"); } Field mApplication = loadedApkClass.getDeclaredField("mApplication"); mApplication.setAccessible(true); Field mResDir = loadedApkClass.getDeclaredField("mResDir"); mResDir.setAccessible(true); Field mLoadedApk = null; try { mLoadedApk = Application.class.getDeclaredField("mLoadedApk"); } catch (NoSuchFieldException e) { // According to testing, it's okay to ignore this.
}
for (String fieldName : new String[]{"mPackages"."mResourcePackages"}) {
Field field = activityThread.getDeclaredField(fieldName);
field.setAccessible(true);
Object value = field.get(currentActivityThread);
for(Map.Entry<String, WeakReference<? >> entry : ((Map<String, WeakReference<? >>) value).entrySet()) { Object loadedApk = entry.getValue().get();if (loadedApk == null) {
continue;
}
if (mApplication.get(loadedApk) == bootstrap) {
if(realApplication ! = null) { mApplication.set(loadedApk, realApplication); }if(externalResourceFile ! = null) { mResDir.set(loadedApk, externalResourceFile); }if(realApplication ! = null && mLoadedApk ! = null) { mLoadedApk.set(realApplication, loadedApk); } } } } } catch (Throwable e) { throw new IllegalStateException(e); }}Copy the codeFor details, please refer to the code of the test project github.com/typ0520/fas…
2, issues# 6
Github.com/typ0520/fas… @YuJunKui1995
This error indicates that if the project contains baidumapapi_v2_0_0.jar, normal packaging is fine, as long as you use Fastdex, the following error will be reported
Error:Error converting bytecode to dex: Cause: PARSE ERROR: class name (com/baidu/platform/comapi/map/a) does not match path (com/baidu/platform/comapi/map/A.class) ... while parsing com/baidu/platform/comapi/map/A.classCopy the codeAfter analysis using Fastdex packaging will have decompression JAR and then in the compression operation, use the following code to do the test github.com/typ0520/fas…
task gen_dex2<< {
File tempDir = project.file('temp')
tempDir.deleteDir()
project.copy {
from project.zipTree(project.file('baidumapapi_v2_0_0.jar'))
into tempDir
}
File baidumapJar = project.file('temp/baidu.jar')
project.ant.zip(baseDir: tempDir, destFile: baidumapJar)
ProcessBuilder processBuilder = new ProcessBuilder('dx'.'--dex'."--output=" + project.file('baidu.dex').absolutePath, baidumapJar.absolutePath)
def process = processBuilder.start()
InputStream is = process.getInputStream()
BufferedReader reader = new BufferedReader(new InputStreamReader(is))
String line = null
while((line = reader.readLine()) ! = null) { println(line) } reader.close() int status = process.waitFor() reader = new BufferedReader(new InputStreamReader(process.getErrorStream()));while((line = reader.readLine()) ! = null) { System.out.println(line); } reader.close(); try { process.destroy() } catch (Throwable e) { } }Copy the codePerform. / gradlew gen_dex2
If the jar contains a.lass, the jar will contain a.lass. If the jar contains a.lass, the jar will contain a.lass. If the jar contains a.lass, the jar will contain A.lass. Git can’t detect a change in the case of a file name. I didn’t think about this problem at that time, and it seems to be the same problem now. It’s easy to fix the problem if you know how it happened. Since there are problems with operating jars in the file system, you can do it in memory. The corresponding Java APIS are ZipOutputStream and ZipInputStream.
If the MAC file system is case-insensitive, run the following command on the terminal to experience the output
echo 'a' > a.txt;echo 'A'> A.txt; cat a.txt; cat A.txtCopy the code
3, issues# 8
Github.com/typ0520/fas… @dongzy
Error:Execution failed for task ':app:tinkerSupportProcess_360DebugManifest'. java.io.FileNotFoundException: E:\newkp\ kuaipiAndroid \ newKP \app\ SRC \main\ Java \com\dx168\fastdex\ Runtime \ fastdexapplication. Java (system could not find the specified path)Copy the codeThe reason for this error is that the project of @dongzy uses the one-click access of Tinkerpatch. Tinkerpatch gradle plug-in also has the function of Application replacement. You must ensure that the fastdexProcess{variantName}Manifest task is executed at the end
FastdexManifestTask manifestTask = project.tasks.create("fastdexProcess${variantName}Manifest". FastdexManifestTask) manifestTask.fastdexVariant = fastdexVariant manifestTask.mustRunAfter variantOutput.processManifest variantOutput.processResources.dependsOn manifestTask //fix issue# 8
def tinkerPatchManifestTask = null
try {
tinkerPatchManifestTask = project.tasks.getByName("tinkerpatchSupportProcess${variantName}Manifest")
} catch (Throwable e) {}
if(tinkerPatchManifestTask ! = null) { manifestTask.mustRunAfter tinkerPatchManifestTask }Copy the code4, issues# hanky-panky
This paragraph is not to solve the problem, I can not help but ridicule this buddy, think it is a waste of his time, come up is “pro test has no soft use, I suggest you do not use what”, make me very depressed, Decisive in response to an article on zhihu past zhuanlan.zhihu.com/p/25768464 after communication later found this elder brothers in a normal packaging 3 seconds on the project of testing, I was speechless
To be honest, I really hope you have more respect for open source projects and use them if you think they are helpful to you. If you feel bad, you can choose to make suggestions or leave silently. If you have time and ability, you can participate in optimization and solve your own work problems while serving everyone. In this fast-paced society, everyone’s time is precious. Do you think it is a waste of time to test and make fun of it? Have you ever thought that the authors of open source projects sacrifice a lot of personal time to solve one problem after another, to solve the technical points of new functions by testing and comparing solutions one by one?
Note: If the project’s DEX generation is less than 10 seconds, it is recommended not to use Fastdex, the effect will almost not be perceived.
Gradle compilation speed optimization recommendations
- Don’t use similar to com. Android. View the build: gradle: 2. + dynamic dependence, or request maven server when start to compile each time compared to the current is the new version
-
Use compile Project (‘: XXX ‘) to compile the library project. In addition, each Library project contains a large number of tasks, each requiring a comparison of inputs and outputs, and the cumulative time consumption of these small tasks can be considerable. It is recommended to put the Library project into an AAR package and put it on the maven server of the company. Don’t tell me that library is often changed during the development phase and it is convenient to rely on it directly. It is not so troublesome to package each change to the Maven server. The project of our team has only one clean application project, all library codes have been thrown into maven server, the number of dex methods is about 12W, several Java files have been modified by Fastdex, and the package, patch and app restart can be completed in about 8 seconds
-
Do not use Flavor in the Library project under any circumstances
For details, please refer to this article by @Still Fantexixi for 17 tips on how to optimize the speed of APP building on Android
5, issues# 17
Github.com/typ0520/fas… @junchenChow
[ant:javac] : warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false forrepeatable builds [ant:javac] /Users/zhoujunchen/as/xx/app/build/fastdex/DevelopDebug/custom-combind/com/xx/xx/xx/xx/CourseDetailActivity.java:229: Error: - source does not support lambda expressions in 1.7 [ant: javac] wrapperControlsView. PostDelayed (() - > wrapperControlsView. InitiativeRefresh (), 500L); [ant:javac] ^ [ant:javac] (use -source 8 or later to enable lambda expressions) [ant:javac] /Users/zhoujunchen/as/android-donguo/app/build/fastdex/DevelopDebug/custom-combind/com/xx/xx/xx/xx/CourseDetailActivity. Subscribe (conf -> sharehelper.share (this, conf), Throwable::printStackTrace); [ant:javac] ^ [ant:javac] (use -source 8 or higher to enable method references) [ant:javac] two errors App: fastdexCustomCompileDevelopDebugJavaWithJavac FAILED what options did not open Does not support the lambda?Copy the codeThis error is caused by the fact that the previous custom compilation task was written to death. Use 1.7 to compile gradle-retrolambda.
https://github.com/evant/gradle-retrolambda/blob/master/gradle-retrolambda/src/main/groovy/me/tatarka/RetrolambdaPluginA ndroid.groovy private static configureCompileJavaTask(Project project, BaseVariant variant, RetrolambdaTransform transform) { variant.javaCompile.doFirst { def retrolambda = project.extensions.getByType(RetrolambdaExtension) def rt ="$retrolambda.jdk/jre/lib/rt.jar"
variant.javaCompile.classpath = variant.javaCompile.classpath + project.files(rt)
ensureCompileOnJava8(retrolambda, variant.javaCompile)
}
transform.putVariant(variant)
}
private static ensureCompileOnJava8(RetrolambdaExtension retrolambda, JavaCompile javaCompile) {
javaCompile.sourceCompatibility = "1.8"
javaCompile.targetCompatibility = "1.8"
if(! retrolambda.onJava8) { // Set JDK 8for the compiler task
def javac = "${retrolambda.tryGetJdk()}/bin/javac"
if(! checkIfExecutableExists(javac)) { throw new ProjectConfigurationException("Cannot find executable: $javac", null)
}
javaCompile.options.fork = true
javaCompile.options.forkOptions.executable = javac
}
}Copy the codeFrom this code we can learn the following
- You need to use javac in jdk1.8 to compile
- SourceCompatibility and targetCompatibility must be set to 1.8
- Add 1.8 rT.jar to classpath
With this information you can process it in your custom build task
if (project.plugins.hasPlugin("me.tatarka.retrolambda")) {
def retrolambda = project.retrolambda
def rt = "${retrolambda.jdk}${File.separator}jre${File.separator}lib${File.separator}rt.jar"
classpath.add(rt)
executable = "${retrolambda.tryGetJdk()}${File.separator}bin${File.separator}javac"
if (Os.isFamily(Os.FAMILY_WINDOWS)) {
executable = "${executable}.exe"
}
}
List<String> cmdArgs = new ArrayList<>()
cmdArgs.add(executable)
cmdArgs.add("-encoding")
cmdArgs.add("UTF-8")
cmdArgs.add("-g")
cmdArgs.add("-target")
cmdArgs.add(javaCompile.targetCompatibility)
cmdArgs.add("-source")
cmdArgs.add(javaCompile.sourceCompatibility)
cmdArgs.add("-cp")
cmdArgs.add(joinClasspath(classpath))Copy the codeFor details, see github.com/typ0520/fas…
6, issues#24 #29 #35 #36
Github.com/typ0520/fas… @wsf5918 @ysnows @jianglei199212 @tianshaokai @Razhan
Caused by: java.lang.RuntimeException: ==fastdex jar input size is 117, expected is 1
at com.dx168.fastdex.build.transform.FastdexTransform.getCombinedJarFile(FastdexTransform.groovy:173)
at com.dx168.fastdex.build.transform.FastdexTransform$getCombinedJarFile.callCurrent(Unknown Source)
at com.dx168.fastdex.build.transform.FastdexTransform.transform(FastdexTransform.groovy:131)
at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:185)
at com.android.build.gradle.internal.pipeline.TransformTask$2.call(TransformTask.java:181)
at com.android.builder.profile.ThreadRecorder.record(ThreadRecorder.java:102)
at com.android.build.gradle.internal.pipeline.TransformTask.transform(TransformTask.java:176)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$IncrementalTaskAction.doExecute(DefaultTaskClassInfoStore.java:163)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:134)
at org.gradle.api.internal.project.taskfactory.DefaultTaskClassInfoStore$StandardTaskAction.execute(DefaultTaskClassInfoStore.java:123) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:95) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:76) . 78 moreCopy the codeNormally open multidex and minSdkVersion < 21 when transformClassesWithJarMergingForDebug tasks, Used to merge all JarInput and DirectoryInput and output to build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar, The error is that the jarMerging task is missing, so when you go to dexTransform you expect only a combined. Jar, but there is no merge so the number of JAR inputs is 117. Because at that time has been unable to reproduce the problem, so using the plus sign means to solve, concrete is when went to executedJarMerge FastdexJarMergingTransform and after the completion of the execution is set to true, If multidex is enabled and executedJarMerge==false, the jarMerge task is missing. At this time of the call com. Android. Build. Gradle. Internal. Transforms. JarMerger manually merge can solve, See GradleUtils’ executeMerge method github.com/typ0520/fas…
The pattern of missing jarMerging tasks found later in development is as follows
- Com. Android. Tools. Build: > = 2.3.0 gradle version
- Build-type is debug
- The command line call does not work when only clicking studio’s Run button packs
- Click the Run button to package the device selected is >=6.0
The command line and the studio click on Run are both running gradle processes. If the command line and studio click on Run are running gradle processes, put the following code into build.gradle
println "projectProperties: " + project.gradle.startParameter.projectPropertiesCopy the codeClick the Run button in Studio to select a 6.0 device
You get the following output
projectProperties: [android.injected.build.density:560dpi, android.injected.build.api:23, android.injected.invoked.from.ide:true, android.injected.build.abi:x86]Copy the codeUsing the above these parameters one by one to do the test, found is android. The injected. Build. The API = 23 the influence of the parameters, we can do test in this test project github.com/typ0520/fas…
Perform. / gradlew clean assembleDebug – Pandroid. Injected. Build. API = 23 note: gradle custom parameters are – P at the beginning
From the above log output, you can see that the missing jarMerge task reappears. Let’s summarize the conditions for reproducing this problem
- Com. Android. Tools. Build: > = 2.3.0 gradle version
- Build-type is debug
- Launch parameters include android. Injected. Build. API and > = 23
The reason why 2.3.0 is such a behavior is because of the introduction of build-cache mechanism, which does not merge in order to do jar level dex cache, so that the third party library will only participate in the generation of dex transform for the first time, and dex will not be merged in order to improve efficiency. If the project is relatively large, there may be dozens or even hundreds of Dex in APK
At present, the jar merge of Fastdex is equivalent to banning this feature. Later, we will consider not merging with dex cache, so that the speed of full packaging can be improved a lot. In addition, it can be introduced into build-type packaging except debug, and the device must be larger than 6.0. Theoretically, after 5.0, the system can load multiple dex. I wonder why the threshold is set to 6.0 instead of 5.0
========================== originally wanted to put these months to do the function and optimization of all together in this article, write to write a brief book prompt words quickly exceed the limit, but can only be divided to write, the next chapter is mainly about the implementation of free installation module and IDEA plug-in. Mid-Autumn Festival is coming soon. I wish you a happy Mid-Autumn Festival in advance. To be continued, see you soon…
If you like this article, send us the star at github.com/typ0520/fas…
Speed up the build time of APK from 130 seconds to 17 seconds
Reference projects and articles
Instant Run Tinker Freeline Dynamic hot patch repair for Android App This section describes how to compile and package Android application resources
Key words: Faster APK compilation speed Faster Android App compilation speed Faster Android Studio compilation speed Slow Android Studio compilation speed optimized Android Studio Gradle compiles slowly
This article from typ0520 Jane books blog www.jianshu.com/p/53923d8f2…