preface

To be a good Android developer, you need a completeThe knowledge systemHere, let’s grow up to be what we want to be.

In the knowledge system of Android performance optimization, package volume optimization has always been a low priority, which leads to many developers do not pay attention to the size of their applications. The development process of the project can be generally divided into the following three stages:

Initial => growth => maturityCopy the code

Generally speaking, package volume optimization of the system is only considered when the application is in the middle and later stages of the growth period, so the benefits of package volume optimization are significant only at this stage and beyond.

So what are the benefits of package volume optimization? How to systematically analyze and optimize the application package volume? In this article, we will conduct in-depth analysis and exploration together.

Mind mapping outline

I. Introduction of slimming optimization and Apk analysis scheme

1. Slimming advantage

Let’s first introduce why we need to do APK slimming optimization.

APK slimming optimization reasons

There are three main reasons:

1. Download conversion rate

APK slimming optimization has a low priority in actual projects, because its benefits are not so obvious after it is done, especially for those projects that have not yet reached the stable period. As we all know, the development process of App is from the initial stage => growth stage => stable period. For projects in the early stage and growth stage, Startup optimization and caton optimization may be performed, but slimming optimization is generally not performed. The most important benefit of slimming optimization is the impact on application download conversion rate, which is one of the important indicators of App business operation and is very important in the phase of refined operation of the project. Because if your App has a smaller Apk than other apps of the same type, your App download rate is likely to be higher. Also, the smaller the packet size, the shorter the wait time for the user to download, so the more successful the download conversion. Therefore, the relationship between install package size and download conversion rate is roughly inversely proportional, that is, the larger the install package, the smaller the download conversion rate. For an 80MB application, even if the user clicks to download it, the download may fail because of slow network speed and sudden regret. With a 20MB application, the user can click on the download button and it may be over before they decide to download it.

In addition, many large apps now usually have a Lite version of the App, this is also due to the download conversion rate considerations.

2. Application market

The Google Play App market mandates that apps over 100MB can only be uploaded using an APK extension file. When uploading using the APK extension file, Google Play hosts the extension file for our app and provides it to the device for free. The extension file will be saved to the shared storage location of the device (SD card or USB mountable partition; Also known as “external” stores) where applications can access them. On most devices, Google Play downloads the extension at the same time as the APK, so the app has everything you need when you first open it. However, in some cases, our app must download files from Google Play when the app is launched. If you want to avoid using extended files and want the download size of the application to be greater than 100 MB, you should upload the application using Android App Bundles, at which point the application can be provided with a compressed download size of up to 150 MB. Android App Bundles are Android App Bundles that allow apps to solve the problem of large APK size by adding dynamic functional modules. The following is the composition structure diagram of Android App Bundle APK composed of one basic module and two dynamic function modules:

3. Requirements of channel partners

In addition, another reason is that when our App gets bigger, we may need to cooperate with various mobile phone manufacturers to pre-install it. These channel partners will make detailed requirements for your App, and only after meeting the corresponding requirements can your App be pre-installed on the mobile phone. Also, the larger the App, the higher the unit price. Therefore, slimming is also a problem we will encounter after the project is bigger.

Impact of large volume on App performance

In addition, the package volume will not only affect the download conversion rate of an App, but also affect the performance of the App in three aspects, as shown below:

1) Installation time: such as file copy, Library decompression, and when compiling ODEX, especially for Android 5.0 and 6.0 system, it takes a long time, but after Android 7.0 has mixed compilation, so it is acceptable. Finally, as the App gets bigger, the signature verification time will also get longer.
2) Runtime memory: Resource, Library, and Dex class loading will occupy a part of the application’s memory.
3) ROM space: If the application installation package size is 50MB, it is likely to be more than 100MB after the startup decompression. And if there is insufficient space on the flash memory, “write amplification” is likely to happen, it is the flash memory and solid-state drives (SSD) is a kind of undesirable phenomenon, flash before to write data must first be erased, and erase the granularity of the operation was much lower compared to write operation, performing these operations will move many times (or adapt) user data and metadata. Therefore, to rewrite data, you need to read some of the used parts of flash memory, update them, and write them to a new location, along with erasing if the new location has been used before. Because of the way flash works, the part of flash that has to be erased is much larger than the new data actually needs. That is, you can end up with many times the amount of physical data being written.

2. APK composition

As we all know, Android projects eventually compile into a.apk file, which is essentially a zip package. As a result, there are many different types of files inside it. These files, in terms of size, fall into the following four categories:

1) Code related: classes.dex: the Java file we wrote in the project will generate a.class file after compilation, and all the.class files will eventually be compiled by the DX tool to generate a classes.dex.
2) Resource related: res, assets, compiled binary resource file resources.arsc, manifest file, etc. The difference between res and assets is that files in the res directory generate the corresponding resource ID in the.r file, while assets does not automatically generate the corresponding ID, but obtains it through the interface of the AssetManager class. In addition, whenever a file is placed in the res folder, aapt automatically generates the corresponding ID and saves it in the.r file. However, the.r file only ensures that the compiler does not report errors. In fact, when the application is running, the system will find the corresponding resource path based on the ID. The resources.arsc file is the file that records the mapping between these ids and the location of the resource file.
3), So related: files in the lib directory, this file is actually very large optimization space.

In addition, there is meta-INF, which stores the signature information of the application. There are three main files, as shown below:

1) Manifest.MF: Each of these resource files has a corresponding SHA-256-digest (SHA1) signature, MF file SHA256 (SHA1) encoded in base64 is the SHA256 (SHA1) -digest-manifest value in cert.sf.
2), cert.sf: except the SHA256 (SHA1) -digest-manifest value defined at the beginning, the following values are the base64 encoded values of each item in the manifest.mf file by SHA256 (SHA1).
3) cert. RSA: contains the public key, encryption algorithm and other information. First, the cert.sf generated in the previous step was digitally abstracted using SHA256 (SHA1) and encrypted using RSA, followed by signing using the developer private key. The public key is then used for decryption at installation time. Finally, this is compared to the unencrypted summary information (the manifest.mf file), and if it matches, the content has not been modified.

3. APK analysis

Next, let’s take a look at four common methods of APK analysis.

1. Analyze APK using ApkTool decompile tool

The first way is to use ApkTool, a decompiler tool. Its official website is as follows:

ApkTool official website

The decompile command is as follows:

apktool d xxx.apk
Copy the code

Now, let’s use ApkTool to decompile the application.

ApkTool decompile actual combat

1. Download and configure apktool

Apktool Download the official configuration document

I only describe the download configuration for Mac OS X. For other platforms, click the link above.

1) Download the script and save it as apktool file.
2) download the latest version of apkTool.jar (fanqiang)
3) Rename the downloaded JAR package to apktool.jar.
4),Configuring Environment Variables, there are two scenarios, as shown below:
- The first is to move apktool and apktool.jar directly to /usr/local/bin, but this requires root permission, sudo before the command, press enter and enter the password.
- The second is under the ~/.bash_profile file, first create a new apktool folder, put the two files in this file, open the terminal, use vim to add environment configuration, and the command is as follows:

// 1. Use the vim command to open the.bash_profile file on the command line and edit it on the command line //. /apktool:$PATH // 2. /apktool:$PATH // 3. /apktool:$PATH // 3 source ~/.bash_profileCopy the code

5) Finally, use the following command to set the two file permissions to executable:

sudo chmod a+x file

2. Analyze APK using ApkTool

To decompile APK, enter the following command on the command line, as follows:

Java -jar apktool_2.3.4.jar apktool d app-release.apkCopy the code

When the decompile is complete, it will generate the app-release directory under the current folder. The directory structure is as follows:

So we can see the actual composition of the current App. Now, let’s introduce the second approach to APK analysis.

Use the Analyze APK provided after AS 2.2

Analyze APK provides the following functions:

1) You can visually view the composition of APK, such as size, proportion and so on.
2) Check the composition of the dex file.
3) Comparative analysis of different APK.

Use Analyze APK to open apK automatically. Use apK to Analyze apK to open apK automatically. Then, we can see the absolute size of the APK file and the percentage of each component file, as shown in the figure below:

As you can see, the size of the classes.dex for the Awesome WanAndroid app is 3.3MB, accounting for 42.2% of the total. In addition, the lib and Res directories also have 1.9MB, accounting for about 25% in total. Therefore, the optimization direction for Awesome WanAndroid App should be dex mainly, while SO and RES are secondary. In addition, we can also check what classes are included in classes.dex, as shown in the figure below:

When we do competitive analysis, it is very convenient for us to see which third-party SDKS are used in our App’s competitive products. It is also easy to view the final version of the APK file from the manifest file, because Analyze APK can parse the manifest file directly.

In addition, there is a Compare with Previos APK button in the upper right corner of the application. After clicking this button, we can Compare the current APK with another version of APK, so that we can Compare the file size of the old and new VERSIONS of APK.

Next, let’s introduce the third method of APK analysis.

3. Use NimbleDroid to analyze APK performance

Nimbledroid website

Nibledroid is a system for analyzing the performance indicators of Android apps developed by a doctoral entrepreneurial team from Columbia University in the United States. The analysis methods are static and dynamic, as shown below:

1) Static analysis: it can analyze the ranking of large files in the APK installation package, the number of Dex methods and the number of well-known third-party SDK methods and their proportion in the overall code.
2) Dynamic analysis: It can give the cold start time, list the specific Methods of Block UI, memory usage, and Hot Methods. From these analysis reports, specific optimization points can be located.

It is very simple to use, you only need to directly upload APK. The nimbleDroid website then automatically analyzes the APK and produces a comprehensive APK analysis report.

Now, let’s look at the final APK analysis tool, the binary checking tool Android-ClassShark.

4. Use Android-Classshark to perform APK analysis

Android-classshark project address

Android-classshark is a standalone binary checker for Android developers. It can browse any Android executable and check for information, such as class interfaces, member variables, etc. It also supports a variety of formats. Such as APK, Jar, Class, So, and all the Android binaries such as manifest files, etc. Let’s use Android-Classshark to get started.

Android – classshark of actual combat

First, let’s download the corresponding classyshark.jar from its Github address, as shown below:

Classyshark.jar – Download address

Then we double-click to open classshark.jar and drag our APK into its workspace. Next, we can see the analysis interface of Apk. Here, we click classes.dex under classes. On the left side of the analysis interface, we can see the number of methods and file size of this dex. As shown in the figure below:

In addition, we can switch to the method count ring icon statistics interface by clicking Methods Count in the upper left corner. We can not only visually see the number and relative size of Methods under each package, but also see the number and relative size of Methods under each subpackage. As shown in the figure below:

Two, code slimming scheme exploration

Before explaining how to optimize Dex, many students may not have enough understanding of Dex. Here we will first learn about Dex in detail.

1. Dex

Dex is an executable file of the Android system, which contains all operation instructions and runtime data of the application. Because Dalvik is a Java virtual machine specially designed for embedded devices, Dex files are fundamentally different from standard Class files in structural design. When a Java program is compiled into a class file, you need to use the dx tool to combine all the class files into a dex file, so that the dex file combines the common information in each class file. The goal is to ensure that each of these classes can share data, which reduces information redundancy to some extent and also makes the file structure more compact. Compared to traditional JAR files, Dex files can be reduced in size by about 50%. The comparison between the Class file and the Dex file is as follows:

If you want to learn more about the Dex file format, you can see Google official tutorial – Dex format.

Dex generally occupies a large proportion in the volume of an application package. The more Dex, the longer the installation time of an App will be. So, optimizing them is a top priority. Now, let’s look at some ways to optimize the volume of this part of Dex.

2, ProGuard

Java is a cross-platform, interpreted language, and Java source code is compiled into intermediate “bytecode” stored in Class files.

So why do we use code obfuscation?

Due to cross-platform needs, Java bytecode contains a lot of source code information, such as variable names, method names, and access to variables and methods by these names. These symbols carry a lot of semantic information and can be easily decompiled into Java source code. To prevent this, we can use a Java obfuscate to obfuscate Java bytecode.

Code obfuscation, also known as flower instructions, converts the code of a computer program into a form that is functionally equivalent, but difficult to read and understand directly. Obfuscation is to reorganize and process the released program so that the processed code can perform the same function as the pre-processed code. However, it is difficult to decompile the obfuscated code. Even if the decompilation is successful, it is difficult to obtain the true semantics of the program. The function of a prover is not only to protect the code, but also to reduce the size of the compiled program. It can reduce the size of the application package by shortening the names of variables and functions, and by losing some useless information.

Obfuscated form of code

Currently, there are three main forms of code obfuscation, as follows:

1) Change the names of various elements in the code, such as classes, functions and variables, to meaningless names. For example, convert hasValue to a single letter a. This way, the decompiler can’t guess the use from the name.
2) Rewrite some of the logic in your code to make it functionally equivalent, but difficult to understand. For example, it changes the instructions and structures of the loop.
3) Shuffling the code format, such as adding more Spaces or removing Spaces, or writing a line of code into multiple lines or changing multiple lines of code into one line.

The role of the Proguard

Proguard, a free Java class file compression, optimization, obfuscating, and pre-validation tool, is integrated with the Android SDK. Its main functions can be summarized in two points, as follows:

1) Lean: It detects and removes unused classes, methods, fields, and instructions, redundant code, and deeply optimizes bytecode. Finally, it changes the names of fields, methods, and classes in the class to short, meaningless names.
2) Security: Increase the difficulty of code decompilation to ensure the security of the code to a certain extent.

So confusion is not only the first barrier to secure Android source code, but to some extent, it can be used to reduce the size of the optimized bytecode. The process of optimizing bytecode is shown in the figure below:

And its function can be divided into three points, as follows:

1, Shrinking (Shrinking)

This function is enabled by default to reduce the size of the application, remove unused classes and members, and will be performed after the optimization action has been performed, as it may expose unused classes and members again. We can turn off compression by using the following rule:

-dontshrink disables compressionCopy the code

2. Optimization

Enabled by default, performs optimizations at the bytecode level to make your application run faster. The following rules can be used to optimize related operations:

- Dontoptimize Disables optimization. - Optimizationpasses n Specifies the number of times ProGuard has optimized code iterations. On Android, the average is 5Copy the code

3. Obfuscation

This is enabled by default, making decompilation more difficult. Classes and class members are randomly named unless protected by rules such as bytecode optimization. Use the following rules to turn off obfuscation:

-dontobfuscate close confusionCopy the code

Optimization details of Proguard

There are more than 30 optimizations in Proguard, including inlining, modifiers, merging classes, and methods. In certain situations, it optimizes as much as possible. Some of the details are listed below:

1) Optimized the use of Gson library.
2) Mark all classes as final.
3) Simplify enumeration types to constants.
4) Vertically merge some classes into the current class structure.
5) Merge some classes horizontally into the current class structure.
6) Remove the write-only field.
7) mark the class as private.
8) Pass field values across methods.
9) Mark some methods as private, static, or final.
10) Remove the synchronized flag from the method.
11) Remove method parameters that are not used.

Proguard configuration

Confusion, the default will be in the project directory app/build/outputs/mapping/release generated under a mapping. TXT file, this is the rule of confusion, so we can according to the file read the obfuscated code back into the original code. To use obfuscation, we just need to configure the following code:

BuildTypes {release {// 1, enable minifyEnabled true // 2, enable zipAlign to align resources in the installation package by 4 bytes, ZipAlignEnabled True // 3. Remove useless resource files: // When ProGuard removes unwanted code, the resources referenced by the code are also marked as useless, and the system removes them through resource compression. // The resource compressor does not currently remove resources (such as strings, sizes, styles, and colors) defined in the values/ folder. // When enabled, the Android build tool uses ResourceUsageAnalyzer to check which resources are useless. When useless resources are detected, the resource is replaced // with a predefined version. PNG,.9.png, and.xml provide predefined versions of TINY_PNG, TINY_9PNG, and TINY_XML byte arrays. // The compression tool works in safe compression mode by default. You can use strict compression mode to achieve better weight loss. ShrinkResources true // 4, confuse the location of the file, where proguardandroid. TXT is the SDK's default confuse configuration, / / it is located the location of the android SDK/tools/proguard/proguard - android. TXT, / / in addition, proguard - android - optimize. TXT for the SDK also confused the default configuration, // But it turns on the optimization switch by default. Also, we can set android.util.Log to an invalid code in the config obfuscation file, // to remove the code that prints the Log in apK. Proguard-rules.pro is the obfuscating configuration under this module. proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } }Copy the code

First, in note 1, we can decide whether to obfuscate by configuring minifyEnabled.

Then, in note 2, configuring zipAlignEnabled to true makes the resources in the installation package 4-byte aligned, which reduces the memory consumption of the application at run time.

Next, set shrinkResources to true in comment 3 to remove unusable resource files: When ProGuard removes some unusable code, the resources referenced by the code are also marked as unusable, and the system removes them through resource compression. Note that the resource compressor does not currently remove resources defined in the VALUES/folder (such as strings, sizes, styles, and colors). When enabled, the Android build tool uses ResourceUsageAnalyzer to check which resources are useless and replaces them with predefined versions when they are found. PNG,.9.png, and.xml provide predefined versions of TINY_PNG, TINY_9PNG, and TINY_XML byte arrays. The resource compression tool runs in safe compression mode by default, and you can turn on strict compression mode to achieve better weight loss.

Finally, in April, we can configure the confused location of files, including proguard – android. TXT for the SDK confused the default configuration, its location is located in the android SDK/tools/proguard/proguard – android. TXT, moreover, Proguard-android-optimy.txt is also the SDK’s default obfuscations, but it defaults to optimizer. In addition, we can also set android.util.Log to invalid code in the configuration obblur file to remove the code that prints the Log in APK. Proguard-rules.pro is the obfuscating configuration under this module.

${project. BuildDir}/outputs/mapping/${flavorDir}/

The file name	describe
dump.txt	The internal structure of all class files in APK
mapping.txt	Provide the original and obfuscated class, method and field names conversion between, can by proguard. Obfuscate. MappingReader to parse
seeds.txt	List the classes and members that are not confused
usage.txt	Lists the code removed from APK

Now, let’s review the basic rules of confusion.

Ground rules for confusion

# * indicates that only the class names under the package are retained, And package under the name of the class will still be confused - keep class com. Json. Chao. Wanandroid. # * * * said the class under this package and contains package names remain - keep class com. Json. Chao. Wanandroid. * * # Keep the name of the class, and keep the content inside is not be confused - keep class com. Json. Chao. Wanandroid. * {*; } # You can also use basic Java rules to protect specific classes from being confused, such as extend, Implement public class * extends android.app.activity # Keep all public content in the MainPagerFragment inner JavaScriptInterface class from being confused - KeepClassMembers class com.json.chao.wanandroid.ui.fragment.MainPagerFragment$JavaScriptInterface { public *; } # Use <init> when you want to protect only certain parts of the class; // Match all constructors <fields>; // Match all fields <methods>; / / # can match all method in the above clause with the private, public and native to further specify don't be confused in the capacity to keep the class com. Json. Chao. Wanandroid. App. WanAndroidApp { public <fields>; } # can also add parameters, The following said in Java. Lang. String as the constructor parameter will not be confused - keep class com. Json. Chao. Wanandroid. App. WanAndroidApp {public <init>(java.lang.String); If you have a member, use -KeepClassesWithMembers to keep the class and its membersCopy the code

With these obfuscation rules in mind, we should be able to write the obfuscation rules for our current application. Note that the default classes in AndroidMainfest are not confused, so the default classes in all four components and Application subclasses and Framework layers are not confused, and the default custom View is not confused either. Therefore, we do not need to manually add the following code to proguard-rules.pro:

-keep public class * extends android.app.Activity -keep public class * extends android.app.Appliction -keep public class  * extends android.app.Service -keep public class * extends android.content.BroadcastReceiver -keep public class * extends android.content.ContentProvider -keep public class * extends android.app.backup.BackupAgentHelper -keep public class * extends android.preference.Preference -keep public class * extends android.view.View -keep public class com.android.vending.licensing.ILicensingServiceCopy the code

The Application and the four components must be registered at AndroidMainfest, so trying to reduce the size of the APK by confusing the four components with the Application and custom View is not going to work. Because there are no rules to configure how to confuse the four components with the Application. Therefore, for the optimization of obfuscation, we can only try to minimize the keep range, so as to maximize the degree of application obfuscation. Adding the following rules to the obfuscation configuration can also output the final obfuscation configuration after the obfuscation:

Print the final configuration of ProGuard - printConfiguration configuration. TXTCopy the code

Confusion of actual combat

Here, we confuse the Awesome WanAndroid app and see how the APK volume changes before and after the confusion.

The figure below is the structure diagram of APK before the confusion of Awesome-WanAndroid. It can be seen that the volume of APK is about 8.3MB, of which the dex part takes up 3.6MB.

What happens to the volume of APK after this confusion? Let’s take a look at the mixed APK composition diagram, as shown below:

As can be seen, the original two dex files have become one, and the size of dex has been reduced to 2.2MB, which is a full reduction of 1.4MB, and the compression effect of dex is nearly 40%. The overall compression effect of APK is 17%. So, confusion is the first choice means of APK thin body really. In addition, Android Studio 3.1 and later will use D8 as the Dex compiler by default, and R8 will be integrated into the Android Gradle Plugin by default as of October 2019. Below, let’s see how D8 and R8 optimize the DEX part of APK.

3. D8 and R8 optimization

The D8 optimization

The optimization effect

In general, the optimization effects of D8 can be summed up as follows:

1) The compilation time of Dex is shorter.
2) the. Dex file is smaller.
3).dex files compiled by D8 have better runtime performance.
4) Processing that includes Java 8 language support.

Open the D8

In Android Studio 3.0, you need to proactively add the following in gradle.properties file:

android.enableD8 = true
Copy the code

Android Studio 3.1 or later D8 will be used as the default Dex compiler.

The R8 optimization

R8 official document (currently open source)

R8 is a replacement for the compression and optimization part of Proguard, and it still uses the same keep rules as Proguard. If we only want to use R8 in Android Studio, R8 will be integrated into the Android Gradle Plugin by default when we turn on obfuscations in build.gradle.

If we are currently using Android Studio 3.4 or Android Gradle plugin 3.4.0 or higher, R8 will be the default compiler. Otherwise, we must configure gradle.properties to support R8 for App obfuscation, as shown below:

android.enableR8=true
android.enableR8.libraries=true
Copy the code

So what’s R8’s advantage over obfuscation?

Both ProGuard and R8 apply basic name confusion: both renames classes, fields, and methods with short, meaningless names. They can also remove debugging properties. However, R8 is more efficient in inline container classes and is more aggressive in removing unused classes, fields, and methods. For example, R8 itself is integrated into ProGuard V6.1.1, which reduces the size of the COMPRESSED APK by about 10% compared to ProGuard’s 8.5%. And, with Kotlin now the first language of Android, R8 offers some kotlin-specific optimizations that ProGuard hasn’t yet provided.

On the surface, ProGuard and the R8 look very similar. They all use the same configuration, so switching between them is easy. When you zoom in, there are some differences. R8 does a better job of inlining container classes, avoiding object allocation. However, ProGuard has its own advantages, including the following:

1) ProGuard is much more powerful in reducing enumeration types to raw integers. It also passes constant method parameters, which are often useful for general-purpose libraries called with specific Settings for the application. ProGuard’s multiple optimization walks can often result in a series of optimizations. For example, the first pass can pass a constant method parameter so that the next pass can remove the parameter and pass the value further. The effect of multiple passes is especially noticeable when deleting logging code. ProGuard is more effective at removing all traces, including string operations that make up log messages.
2) The pattern matching algorithm applied in ProGuard can identify and replace short instruction sequences, thus improving code efficiency and opening up opportunities for more optimization. You can benefit from optimizing the order of traversal, especially mathematical and string operations.
3. Finally, ProGuard has the unique ability to optimize code that serializes or deserializes objects to JSON using the GSON library. The library relies heavily on reflection, which is convenient but inefficient. ProGuard’s optimized functionality can replace it with more efficient, direct access.

R8 optimized combat

Next, let’s take a look at how the APK volume changes with Awesome WanAndroid using R8, as shown below:

As can be seen, the size is reduced by 0.1MB compared with the APK only after confusion. The optimization effect of Dex is about 5%, and the overall compression effect of APK is about 1.5%. Although 0.1MB is small in terms of APK size, the proportion is not small. If you are in charge of an App like wechat or Taobao, the size of the App is usually around 100MB. Using R8 can reduce the size of the App by 1.5MB.

In addition, if you want to use R8 separately for Dex or JAR package, you can quickly run it in Python environment according to the official document at the top. The detailed steps are as follows:

1. Ensure that Python 2.7 or later is installed locally.

2. Since the R8 project uses dePOT_Tools provided by chromium project to manage dependencies, install depot_Tools first. (The installation for the Mac version is covered below)

1) Obtain depot_tools

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
2) Obtain the current directory of depot_tools

pwd
3) Add environment variables
- Vim ~/.bash_profile: Open the last line and add it. If this file is not available, add this file
- Export PATH=”$PATH:/PWD/depot_tools” : PWD is the PATH obtained in the second step
4) Effective environment variables

source ~/.bash_profile

3. Downloading and building R8 project

git clone https://r8.googlesource.com/r8
cd r8
tools/gradle.py d8 r8
Copy the code

The tools/gradle.py script will generate two JAR files: build/libs/d8.jar and build/libs/r8.jar.

The following code uses R8 to generate the optimized dex file in the out directory:

java -jar build/libs/r8.jar --release --output out --pg-conf proguard.cfg input.jar
Copy the code

D8 and R8 are very powerful, and Jake Wharton has been working on them for over a year. For more details on the implementation of D8 and R8, check out his blog.

4. Delete debugging messages and line numbers

Before explaining what is deubg information and line number information, we need to know something about Dex first.

As we all know, the JVM runs with.class files, while Android invented the Dalvik and ART virtual machines to make package sizes more compact and runtime more efficient. Both virtual machines run.dex files. Of course, the ART virtual machine can also run oAT files.

Therefore, the information content in the Dex file is the same as that in the Class file. The difference is that the Dex file has deduplicated the information in the Class file. A Dex contains many Class files and has a relatively large difference in structure. Dex is a partition structure. Each block in Dex is indexed by offset.

In order to display the corresponding debugging information during debugging, report the crash, or proactively obtain the call stack by obtaining the corresponding line number through the debugItem, we will add the above and below rules in the confusion configuration:

-keepattributes SourceFile, LineNumberTable
Copy the code

In this way, debug and line number information in Dex will be retained. At this time, the structure diagram of Dex is as follows:

As can be seen from the figure, the structure of Dex file is mainly divided into four sections: header area, index area, data area and map area. Our debug and line number information is stored in the debugItems area of the data area. Debug_items contains two types of information, as shown below:

1) Debugging information: contains function parameters and all local variables.
2) Troubleshooting information: contains the correspondence between the line numbers of all instruction sets and the line numbers of source files.

According to the official data of Google, debugItem generally accounts for about 5% of Dex. If we can remove debug and line number information, we can further reduce the size of Dex, but the function of debugging information will be lost. So, is there any way to remove debugItem? At the same time, can crash get the correct line number when reporting?

We can try to modify the Dex file directly and keep a small piece of debugItem so that the instruction set line number is the same as the source file line number when the system looks for the line number, so that any line number reported by the monitor is directly changed to the instruction set line number.

Each method will have a debugInfoItem, and each debugInfoItem has a mapping between the instruction set line number and the source file line number. Therefore, we directly delete all the redundant DebugInfoItems, only keeping one debugInfoItem. All methods will point to the same debugInfoItem, and the line number of the instruction set in the debugInfoItem will be the same as the line number of the source file, so that no matter how you look for the line number, you will get the instruction set line number. It should be noted that this scheme needs to be compatible with all virtual machine lookup methods, so it is not enough to keep only one debugInfoItem, and the debugInfoItem table should not be too large.

The StripDebugInfoPass of the ReDex is used to remove debugging information from the Dex. The configuration is as follows:

{
    "redex" : {
        "passes" : [
            "StripDebugInfoPass",
            "RegAllocPass"
        ]
    },
    "StripDebugInfoPass" : {
        "drop_all_dbg_info" : false,
        "drop_local_variables" : true,
        "drop_line_numbers" : false,
        "drop_src_files" : false,
        "use_whitelist" : false,
        "cls_whitelist" : [],
        "method_whitelist" : [],
        "drop_prologue_end" : true,
        "drop_epilogue_begin" : true,
        "drop_all_dbg_info_if_empty" : true
    },
    "RegAllocPass" : {
        "live_range_splitting": false
    }
}
Copy the code

On the debugInfo practical we will start in a minute, before that, let’s talk about another optimization point in Dex subcontracting.

5. Dex subcontracting optimization

Dex Subcontracting optimization principle

When our APK is too large, the number of methods of Dex will exceed 65536. Therefore, MUTILdex must be used for subcontracting. However, at this time, each Dex may call methods in other Dex.

1) Redundant method ID: Cross-dex invocation will cause the current Dex to retain the method ID in the invoked Dex. This redundancy will lead to fewer classes that can be stored in each Dex, and eventually lead to an increase in the number of compiled Dex, and the increase of Dex data will further aggravate this problem.
2) Information redundancy caused by other cross-dex calls: In addition to recording the id of the method called, it is also necessary to record the definition information of its class and current method, which will result in the redundancy of information of string_IDS, type_IDS and proto_IDS.

In order to reduce cross-dex calls, we must try to assign the classes and methods that have calling relationships to the same Dex. However, the calling relationship between classes is very complex, so it is difficult to achieve the optimal case. Fortunately, ReDex’s CrossDexDefMinimizer class analyzes the calling relationships between classes and uses a greedy algorithm to calculate a local optimal solution (some balance between the compile effect and the dex optimization effect). Use the “InterDexPass” configuration to put the classes that reference each other into the same Dex as much as possible, and increase the pre-verify of the classes to improve the cold startup speed of the application.

The configuration code of using Dex subcontracting in ReDex to optimize information redundancy caused by cross-dex invocation is as follows:

{
    "redex" : {
        "passes" : [
            "InterDexPass",
            "RegAllocPass"
        ]
    },
    "InterDexPass" : {
        "minimize_cross_dex_refs": true,
        "minimize_cross_dex_refs_method_ref_weight": 100,
        "minimize_cross_dex_refs_field_ref_weight": 90,
        "minimize_cross_dex_refs_type_ref_weight": 100,
        "minimize_cross_dex_refs_string_ref_weight": 90
    },
    "RegAllocPass" : {
        "live_range_splitting": false
    },
    "string_sort_mode" : "class_order",
    "bytecode_sort_mode" : "class_order"
}
Copy the code

In order to measure the optimization effect, we can use the index Dex information efficiency, and the formula is as follows:

Dex Valid information = Number of define Methods/Number of Reference MethodsCopy the code

If the efficiency of Dex is more than 80%, it is basically qualified.

Use the ReDex to optimize subcontracting and remove debugging messages and line numbers

Next, we will use Redex to further optimize the app-releas-ProGuardWithr8.apk generated in the previous step. (In macOS)

First, we need to enter the command to install the Xcode command line tool

xcode-select --install
Copy the code

2. Then, use Homebrew to install the dependencies used by the Redex project

brew install autoconf automake libtool python3
brew install boost jsoncpp
Copy the code

It is important to note that the February 10, 2020 version of the redex source code requires a boost version of V1.71 or higher, and that you may get a boost version lower than V1.71 when you install boost using Brew Install Boost. Brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew brew I am currently using boost_1_72_0.zip in the Boost V1.7.2 source download address. Redex’s class rearrangement optimization has been mentioned since the in-depth exploration of Android startup optimization, and I was stuck in this step, so I never really completed the class rearrangement optimization.

3. Next, get the source code of ReDex from Github and switch to the directory of ReDex

git clone https://github.com/facebook/redex.git
cd redex
Copy the code

4. Next, use autoconf and make to build ReDex

# If you are using GCC, use gcc-5 autoreconf-IVF &&./configure && make-j4 sudo make installCopy the code

5. Configure the config code of the Redex

In Redex at run time, it is according to Redex/config/default config in the configuration file is the channel passes different optimization items to is added to deal with the APK Dex, We can refer to redex/config/default config this default configuration, the inside of the passes have different configuration items in a specific optimization. In order to optimize the package volume of the App, we added the configuration item in interdex_stripDebuginfo. config to delete the debugInfo and reduce cross-dex calls. The final interdex_stripDebuginfo.config configuration code looks like this:

{
    "redex" : {
        "passes" : [
            "StripDebugInfoPass",
            "InterDexPass",
            "RegAllocPass"
        ]
    },
    "StripDebugInfoPass" : {
        "drop_all_dbg_info" : false,
        "drop_local_variables" : true,
        "drop_line_numbers" : false,
        "drop_src_files" : false,
        "use_whitelist" : false,
        "cls_whitelist" : [],
        "method_whitelist" : [],
        "drop_prologue_end" : true,
        "drop_epilogue_begin" : true,
        "drop_all_dbg_info_if_empty" : true
    },
    "InterDexPass" : {
        "minimize_cross_dex_refs": true,
        "minimize_cross_dex_refs_method_ref_weight": 100,
        "minimize_cross_dex_refs_field_ref_weight": 90,
        "minimize_cross_dex_refs_type_ref_weight": 100,
        "minimize_cross_dex_refs_string_ref_weight": 90
    },
    "RegAllocPass" : {
        "live_range_splitting": false
    },
    "string_sort_mode" : "class_order",
    "bytecode_sort_mode" : "class_order"
}
Copy the code

6. Finally, run the corresponding redex optimization command

Here, we use the Redex command to perform Dex subcontracting optimization and remove debugInfo for app_release -proGuardWithr8.apk obtained in the previous Dex optimization. It uses the greedy local optimal solution to reduce information redundancy caused by cross-dex calls. The command is as follows (note that you may need to add the Android SDK path before redex, because redex uses the ZIPalign tool under the SDK) :

ANDROID_SDK=/Users/quchao/Library/Android/sdk redex --sign -s wan-android-key.jks -a wanandroid -p wanandroid -c ~/Desktop/interdex_stripdebuginfo.config -P app/proguard-rules.pro -o ~/Desktop/app-release-proguardwithr8-stripdebuginfo-interdex.apk ~/Desktop/app-release-proguardwithr8.apk
Copy the code

The meanings of the key parameters of the redex command are as follows:

–sign: signs the generated APK.
-s: configures the signature file of the application.
-a: configures the KEY_alias of the application signature.
-p: configures the key_password of the application signature.
-c: redex Indicates the CONFIG file for Dex processing.
-o: specifies the full path for generating APK.

Using the above redex command, we can re-sign and confuse the optimized APK, wait a while (if your APK has a large number of Dex, this may take a long time), and then generate the optimized APK: App-release – ProGuardWithr8 – StripDebuginfo – Interdex. Apk, as shown in the figure below:

As you can see, our APK size has barely changed because the current APK has only one Dex, and the first Dex is not optimized by default. In order to actually see the optimization effect of Redex, we adopted a new project for the experiment. The project address is as follows:

Redex Optimizes the Apk project address

First, introduce a bunch of open source libraries and try to increase the number of Dex. Then compile directly from assembleDebug. In addition, to clarify the process, you can run export TRACE=2 on the command line to output redex logs. Finally, we enter the following redex command to delete the debugInfo in dex and reduce cross-dex calls, as shown below:

redex --sign -s ReDexSample/keystore/debug.keystore -a androiddebugkey -p android -c redex-test/interdex_stripdebuginfo.config -P ReDexSample/proguard-rules.pro  -o redex-test/strip_output.apk ReDexSample/build/outputs/apk/debug/ReDexSample-debug.apk
Copy the code

Finally, we can see the APK volume comparison before and after as follows:

As you can see, the size of the APK has been reduced from 14.2MB to 12.8MB, which is about 10% optimization, which is pretty significant. In addition, the more Dex your App has, the greater the optimization effect will be.

6. Use XZ Utils for Dex compression

Official document for XZ Utils

XZ Utils is a free general-purpose data compression software with high compression rates. Like 7-Zip, XZ Utils is a successor to LZMA Utils, using the LZMA/LZMA2 algorithm internally. LZMA offers high compression ratios and fast decompression, making it ideal for embedded applications. The main functions of LZMA are as follows:

1) Compression speed: 3 MB/s on a 3 GHz dual-core CPU.
2) Decompression speed: 20-50 MB/s on a modern 3 GHz CPU (Intel, AMD, ARM). 5-15 MB/s on simple 1 GHz RISC-CPUS (ARM, MIPS, PowerPC).
3) Smaller memory requirements for decompression: 8-32 KB + DictionarySize
4) Code size for decompression: 2-8 KB (depending on speed optimization).

Relative to a typical compressed file, the output of XZ Utils is 30% smaller than gzip and 15% smaller than bzip2. The App of FaceBook uses Dex compression, and it places the files compressed by Dex in assets directory, as shown in the picture below:

Let’s first look at classes.dex in the figure above, which contains only the classes to be used for startup. This will buy time for decompression of the dex compressed file secondary.

In addition, under the secondary.dex.jar.xzs file, we notice that there is a series of secondary -x.ex.jar.xzs.tmp~. Meta files, which hold the mapping metadata information of each dex file before compression. We’ll need it when the app first starts decompression.

Although classes.dex takes time to decompress the dex compressed file for the first time, because the file is too large, the decompression time may be 3 to 5 seconds on low-end machines.

In addition, when the Dex is very large, the installation time of the application will increase. If the compression of Dex is also used, the first generation of ODEX may take more than 1 minute. To get around that, Facebook used the Oatmeal tool to make its own ODEX file based on the ODEX file format. Under normal flow, the system will use fork child process to process dex2OAT.

But you might want to make dex2oat the oatmeal by acting as an agent without the time it takes to fork the process. If you had a 10MB Dex, you could reduce the time to 100ms. On Android 5.0, it takes more than 10 seconds to generate an ODEX, and on Android 8.0, it takes about 1 second to use speed mode. But every Android version of ODEX has a different format, and you need to have the oatmeal in different versions. So you might want to put the oatmeal in the box first.

7. Three-party library processing

In the actual development process, we will use a variety of three-party libraries. Especially when the project gets bigger and there’s a lot of developers, so there’s a lot of tripartite libraries that you bring in, for example, someone brings in Fresco, which you might not be familiar with, you bring in Glide, and someone else who might bring in Picasso, which he’s familiar with, Therefore, there may be multiple three-party SDKS with the same functionality in a project, which is certainly true in large projects. Therefore, when we do code slim down, we need to unify the three libraries, such as the image loading library, network library, database and other basic libraries to unify, remove redundant libraries.

At the same time, when choosing third-party SDKS, we can take the package size as one of the selection indicators, we should try to choose those smaller libraries to achieve the same function. For example, for image loading, Picasso, Glide, and Fresco all work, but when you introduce Fresco, the package size increases a lot, whereas Picasso only increases by less than 100KB, so different SDKS will have different effects on the package size. In this case, we can use the AS plugin Android Methods Count. Once installed, it will automatically display the number of Methods you introduced in the build.gradle file.

Finally, if we were to introduce a three-party library, we could introduce only the required part of the code, rather than the entire package. Many libraries have well-designed code structures, such as Fresco, which strip image loading functions such as WebP and GIF into a single library. If we only need The Webp functionality of Fresco, we can remove all libraries other than WebP, so that the three libraries you introduce are small and the package size is reduced. As shown in the figure below, we can keep only the WebP functionality of Fresco and remove all other dependencies.

If you introduce a three-party library that has not been structured, you will need to modify the source code to extract only the functionality you need.

8. Remove useless code

There are two common problems with removing useless code:

1) Business code only increases.
2) Too many codes dare not delete.

Here is a good way to determine exactly which classes users will not use in an online environment. We can do this with AOP. For activities, it’s very simple. We just add statistics to the onCreate of each Activity, and then when it’s online, if the Activity is counted, it’s still in use. For classes that are not activities, we can use AOP to cut their constructors. If a class is used, its constructor will be called. For example, here is the code that uses AspectJ to create a constructor aspect for a class in a package:

@After("execution(org.jay.launchstarter.Task.new(..) ") public void newObject(JoinPoint point) { LogHelper.i(" new " + point.getTarget().getClass().getSimpleName()); }Copy the code

Where new represents the tangent constructor, the parentheses.. Matches all construction parameters. In addition, you can also use the Coverage plugin directly to analyze unwanted code online, but remember to change the server name to your own when registering the report data.

Finally, we can also use Simian tools or Lint offline to scan for duplicate code.

Use Lint to detect invalid code

Step: Click Analyze -> Run Inspection by Name -> unused declaration -> Moudule 'app' -> OKCopy the code

Avoid Java Access methods

What is the Access method?

In order to provide the inner class and its outer class with direct access to each other’s private members without violating encapsulation requirements, the Java compiler automatically generates static access$XXX methods for package visibility during compilation and instead calls the corresponding Access methods where access to each other’s private members is needed.

Avoid ways to generate access methods

There are two main ways to avoid creating access methods:

1) In the development process, it is necessary to pay attention to appropriate adjustments in the case of possible access methods, such as removing private and changing to package visibility.
Use ASM to delete the generated Access method at compile time.

Because the optimization effect is not very obvious, I will not introduce it here. The specific implementation details can be seen in the watermelon video apK Skinny Java Access method deletion. In addition, the access-marking function is also provided in ReDex to remove the Access method in the code, and, ReDex also has the function of Type-Erasure, which has the same optimization effect as access-marking. It can not only reduce the package size, but also improve the startup speed of App.

10. Use code optimization plug-ins in ByteX Gradle plugin platform

If you want to get rid of access during the build phase of your project, I recommend using ByteX’s Access_inline plug-in instead. In addition to Access_inlie, there are four useful Gradle plugins in ByteX to help you reduce the Dex file size, as shown below:

Inline constant field during compilation: const_inline
Remove redundant assignment code during compilation: field_assign_opt
Remove Log code during compilation: method_call_opt
Getter-setter-inline-plugin = getter-setter-inline-plugin = getter-setter-inline-plugin

11, summary

Review the various Dex optimization methods we used above, among which, many optimization items use ReDex. For ReDex, there are five powerful functions it provides, as shown below:

1) Interdex: class rearrangement and file rearrangement, Dex subcontracting optimization. For class rearrangement and file rearrangement, Google introduced Dexlayout in Android 8.0, which is a library for analyzing dex files and reordering them according to configuration files. Similar to ReDex, Dexlayout allows programs to have better memory access patterns by improving file locations by grouping some dex files that are often accessed together, saving RAM and shortening startup time. Unlike ReDex, it uses runtime configuration information to reorder parts of the Dex file. Therefore, dexLayout will be integrated into dex2OAT devices for compilation only after the application is running and when the system is idle for maintenance.
2) “Oatmeal” : You can directly generate an Odex file.
3) StripDebugInfo: Deletes debugging information in the Dex.
4) Access-marking module in the source code: delete the Java Access method.
5) Type – Erasure module in source code: type erasure.

As you can see, ReDex is very powerful, and if you can deeply understand the implementation of the functional modules in the ReDex source code, you will have a very strong technical capital.

Recently, the Douyin Android team has integrated some of the above modules into ByteX in the form of Gradle Transform + ASM. It is suggested that you can develop your own Gradle plug-in directly on this bytecode plug-in development platform.

Finally, there are some code optimizations, such as minimizing the use of enums during development, which can reduce the size of an enum by about 1.0 to 1.4 KB.

Android Package size Optimization

Reference links:

Android Performance Analysis and Optimization chapter 10 App slim optimization

2, Geek Time Android Development master class volume optimization

3, “Android Performance Optimization Best Practices” chapter 7 installation package size optimization

4, the android – arscblamer

5, Android SVG to VectorDrawable

6. App weight-loss best practices

Use the Simian tool to scan for duplicate code

8 FontZip.

9, the Android – Iconics

10, iconfont

Use of IconFont on Android

12, TinyPngPlugin

13. Dynamically deliver the application of so library in Android APK installation package slimming

14, Apktool Install Instructions

15, nimbledroid

16, the android – classyshark

17. Android confounds from beginner to master

18. APK Expansion Files

19. Write amplification

D8 Shrinker and R8 Shrinker

New Dex compiler D8 for Android and new obfuscation tool R8

22, Comparison of ProGuard vs. R8: October 2019 Edition

23, Alipay App construction optimization analysis: Android package size extreme compression

24, Redex

Redex preliminary study and Interdex: Andorid cold start optimization

26. Dalvik executable file format

27, InterDex. CPP greedy algorithm part

28, crossdexRefminimizer. CPP cross-dex call optimization

How do I use NimbleDroid to optimize android app performance

30, XZ Embedded

31, oatmeal,

32, SoLoader

33, buck

34, the android native – library – merging – demo

35. Redex optimized Demo

36, android – the chunk – utils

37, ResourceUsageAnalyzer. Java

38. Knowledge summary of Android installation package

Install package minus 1M– wechat Android resource obfuscation packaging tool

40, AndResGuard

41. Principle of Android APK signature

42, watermelon video APK skinny Java Access method delete

Android Pack size Optimization