I. Introduction
Since 2016, hot repair technology began to be popular in the Android world. It takes the classloader class loading mechanism as the core, which can fix online bugs without releasing a new version and enable the online version to carry out full or incremental updates.
There are two common ideas:
- Class loading scheme, namely DEX piling. The scheme is mainly based on Tencent, including Tinker of wechat and Amigo of Ele. me;
- The low-level substitution, that is, modify the substitution ArtMethod. The scheme is mainly based on Ali AndFix, etc.
This paper mainly introduces the first scheme.
1.1 ART and Dalvik
- Dex: Full Dalvik Executable Format, compressed products processed by a number of.class files that can eventually be executed in the Android runtime environment. It is suitable for systems with limited memory and processor speed.
- Dalvik: A Java virtual machine designed for Android by Google. Support conversion to. Dex format Java program run. DVM uses the CMS garbage collector by default.
- ART: Android Runtime, introduced in Android 4.4, as the default Android Runtime in Android 5.0 and later. Android Runtime: Android Runtime (ART) and Dalvik Both ART and Dalvik are compatible runtimes running Dex bytecode, so ART is backward compatible with applications developed by Dalvik.
- AOT: ART precompiles bytecode to machine language at application installation. This mechanism is called Ahead-of-time (AOT) precompilation. After doing this, the application will be slower to install, but the execution will be more efficient and start up faster.
1.2 dexopt与dexaot
- Dexopt: After the DEX file is loaded to the Dalvik VM, the DEX file is verified and Optimized to obtain odex(Optimized dex). The odex file just uses some optimized opcodes for the dex file.
- Dex2oat: Dex or odex files are AOT precompiled, resulting in OAT (actually ELF files) executable files (machine code). (Compared with oDEX optimization, it takes longer to convert unoptimized DEX to OAT)
1.3 Comparison between ART and Dalvik
- In Dalvik, application operation requires interpretation and execution, and commonly used hot-spot codes convert bytecode to machine code by just-in-time compiler (JIT), resulting in low operation efficiency. In an ART environment, bytecodes are precompiled (AOT) into machine code at installation time, making installation slower but more efficient.
- ART takes up more space than Dalvik (bytecode becomes machine code), “space for time”.
- Precompilation can also significantly improve battery life because the application is not compiled repeatedly each time it is run, thus reducing CPU usage and power consumption.
2 this
2.1 Android running process
When an Android program is compiled, it compiles a.class file from a.java file, and then packages the.class file as a.dex file. When the Android program is running, the Android Dalvik/ART VIRTUAL machine loads the. Dex file from which the. Class file is retrieved and uses it in memory.
2.2 Class loading tool ClassLoader
Any Java program consists of one or more class files that need to be loaded into the JVM through Java’s classloading mechanism when the program is running. Instead of loading all classes at once when a Java program starts, the base classes that are guaranteed to run are loaded into the JVM first, and the rest of the classes are loaded at a later time. The advantage of this is to save memory overhead, time to load again, which is also a manifestation of Java dynamic.
These classes are loaded by a ClassLoader. Each Class object has a ClassLoader field inside it to identify which ClassLoader it is loaded by. Android ClassLoader minor changes to Java ClassLoader.
class Class<T> {...private transientClassLoader classLoader; . }Copy the codeThere are four common Android class loaders:
- BootClassLoader: Load class bytecode files in Android Framework layer (similar to Java BootstrapClassLoader)
- PathClassLoader: load only Apk class bytecode files installed in the Android system. It is the default class loader used by Android. (Java-like AppClassLoader)
- DexClassLoader: can load the dex/jar/ APK /zip file of the specified directory (similar to the Custom ClassLoader in Java), more flexible than the PathClassLoader, is the focus of hot repair;
- BaseDexClassLoader: The parent class of PathClassLoader and DexClassLoader
The e (TAG, "Activity. The class by:" + Activity. Class. GetClassLoader () + "load"); Log.e(TAG, "mainActivity.class" from: "+ getClassLoader() +" load "); 1 / / output: Activity class by: Java. Lang. BootClassLoader @ b1202a1 loading MainActivity. Class by: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.enjoy.enjoyfix-1/base.apk"],nativeLibraryDirectories=[/data/app/com.enjoy.enjoyfix-1/lib/x86, / system/lib/vendor/lib]]] loadingCopy the codeThe relationship between them is as follows:
public class DexClassLoader extends BaseDexClassLoader {
public DexClassLoader(String dexPath, String optimizedDirectory, String librarySearchPath, ClassLoader parent) {
super(dexPath, newFile(optimizedDirectory), librarySearchPath, parent); }}public class PathClassLoader extends BaseDexClassLoader {
public PathClassLoader(String dexPath, ClassLoader parent) {
super(dexPath, null.null, parent);
}
public PathClassLoader(String dexPath, String librarySearchPath, ClassLoader parent){
super(dexPath, null, librarySearchPath, parent); }}Copy the codeThe only difference between PathClassLoader and DexClassLoader is that the constructor of the parent class is called: DexClassLoader passes an additional optimizedDirectory argument and creates it as a File object and passes it to super, while PathClassLoader passes it directly to NULL. So both can load the specified dex, as well as classes.dex in JAR, ZIP, and APK
PathClassLoader pathClassLoader = new PathClassLoader("/sdcard/xx.dex", getClassLoader());
File dexOutputDir = context.getCodeCacheDir();
DexClassLoader dexClassLoader = new DexClassLoader("/sdcard/xx.dex",dexOutputDir.getAbsolutePath(), null,getClassLoader());
Copy the codeOptimizedDirectory is the output directory of dexopt (odex). DexClassLoader can load not only dex files, but also dex files in JAR, APK, and ZIP files. Jar, APK, and ZIP are actually some compressed formats. To get the dex file in the compressed package, you need to decompress it. Therefore, DexClassLoader will specify a directory to decompress when calling the parent class constructor. When the PathClassLoader is created, the directory is null, which means no dexopt. No, the default path for null optimizedDirectory is /data/dalvik-cache.
In the API 26 source code, DexClassLoader optimizedDirectory is marked as deprecated and the implementation becomes identical to that of PathClassLoader:
public DexClassLoader(String dexPath, String optimizedDirectory, String librarySearchPath, ClassLoader parent) {
super(dexPath, null, librarySearchPath, parent);
}
Copy the code2.3 Parental delegation mechanism
You can see that creating a ClassLoader requires receiving a ClassLoader parent parameter. The purpose of this parent is to implement parent delegation for class loading. That is, when a class loader receives a request to load a class, it first delegates the loading task to the parent class loader, recursively. If the parent class loader can complete the class loading task, it returns successfully. Only if the parent class loader is unable to complete the load task, do the load itself.
protectedClass<? > loadClass(String name,boolean resolve) throws ClassNotFoundException{
// Check if the class is loaded
Class c = findLoadedClass(name);
if (c == null) {
long t0 = System.nanoTime();
try {
if(parent ! =null) {
// If parent is not null, the parent loadClass is called to load
c = parent.loadClass(name, false);
} else {
// If parent is null, BootClassLoader is called for loadingc = findBootstrapClassOrNull(name); }}catch (ClassNotFoundException e) {
}
if (c == null) {
// If you can't find it, look it up yourself
longt1 = System.nanoTime(); c = findClass(name); }}return c;
}
Copy the codeC = findBootstrapClassOrNull(name); When parent is null, classes loaded by the BootClassLoader can also be loaded. But actually, the implementation in Android is :(Java is different)
private Class findBootstrapClassOrNull(String name)
{
return null;
}
Copy the code2.4 Three mechanisms of class loaders (Constraints)
The parent delegate mechanism is actually a top-down load with caching, and this mechanism also determines some of its characteristics:
Delegate: A class load is handed over to the parent class loader, which cannot load it itself.
Visibility: The subclass loader sees all classes loaded by the parent class loader, but the parent class loader does not see classes loaded by the subclass loader.
Singleness: A class is loaded only once, and the subclass loader does not reload classes that have been loaded by the parent class loader.
2.5 findClass
You can see that when all the parent ClassLoaders fail to load a Class, they call their own findClass method. FindClass is defined in a ClassLoader as:
protectedClass<? > findClass(String name)throws ClassNotFoundException {
throw new ClassNotFoundException(name);
}
Copy the codeLoadClass and findClass can be overridden by any ClassLoader subclass. Generally, if you don’t want to use parent delegates, override loadClass to change its implementation. Overwriting findClass means defining how to find a Class if both parent classloaders can’t find it. Our PathClassLoader is responsible for loading its own classes in programs like MainActivity, using the parent to delegate the parent ClassLoader to load activities in the Framework. PathClassLoader does not override loadClass, so we can see how findClass is implemented in PathClassLoader.
public BaseDexClassLoader(String dexPath, File optimizedDirectory,String librarySearchPath, ClassLoader parent) {
super(parent);
this.pathList = new DexPathList(this, dexPath, librarySearchPath,
optimizedDirectory);
}
@Override
protectedClass<? > findClass(String name)throws ClassNotFoundException {
List<Throwable> suppressedExceptions = new ArrayList<Throwable>();
// Find the specified class
Class c = pathList.findClass(name, suppressedExceptions);
if (c == null) {
ClassNotFoundException cnfe = new ClassNotFoundException("Didn't find class \"" + name + "\" on path: " + pathList);
for (Throwable t : suppressedExceptions) {
cnfe.addSuppressed(t);
}
throw cnfe;
}
return c;
}
Copy the codeThe implementation is simple, looking up the class from the pathList. Continue to see DexPathList:
String librarySearchPath, File optimizedDirectory) { //......... Add (dexPath) // makeDexElements will go to List<File>.add(dexPath) and use DexFile to load the dex File Element array this.dexElements = makeDexElements(splitDexPath(dexPath), optimizedDirectory, suppressedExceptions, definingContext); / /...Copy the code}
Public Class findClass(String name, List suppressed) {// Retrieve DexFile from element representing Dex for (Element element: dexElements) { DexFile dex = element.dexFile; if (dex ! = null) {// Find class clazz = dex.loadClassBinaryName(name, definingContext, suppressed); if (clazz ! = null) { return clazz; } } } if (dexElementsSuppressedExceptions ! = null) { suppressed.addAll(Arrays.asList(dexElementsSuppressedExceptions)); } return null; }
As you can see, the findClass() method of the BaseDexClassLoader actually gets the class from the findClass() method of the DexPathList, The DexPathList object happens to have been created in the previous BaseDexClassLoader constructor, which parses the path to the dex file and stores the dex files in this.dexElements. The DexPathList class calls' makeDexElements() 'through the constructor to get the' Element 'set' makeDexElements() ': Private static Element[] makeDexElements(ArrayList<File> files, File optimizedDirectory, ArrayList<I // 1. Elements = new ArrayList<Element>(); For (File File: files) {ZipFile zip = null; DexFile dex = null; String name = file.getName(); . // if (name.endswith (DEX_SUFFIX)) {dex = loadDexFile(file, optimizedDirectory); } else {zip = file;} else {zip = file; dex = loadDexFile(file, optimizedDirectory); }... // 3. Wrap the dex file or zip file as an Element object and add it to the Element collection. = null) || (dex ! = null)) { elements.add(new Element(file, false, zip, dex)); Return elements. ToArray (new Element[elements. Size ()]); }Copy the codeAs you can see, the constructor of DexPathList wraps program files (dex, APK, JAR, zip) into Element objects and adds them to the Element collection. Android class loaders (both PathClassLoader and DexClassLoader) only recognize dex files when loading files, and loadDexFile() is the core method for loading dex files. He can extract dex from JAR, APK, zip.
Take a look back at the findClass method in PathClassLoader:
Class c = pathList.findClass(name, suppressedExceptions);
Copy the codeSo you see the findClass() method of DexPathList. As follows:
public Class findClass(String name, List<Throwable> suppressed) {
// Iterate through the dex and Element resource queried from dexPath
for (Element element : dexElements) {
DexFile dex = element.dexFile;
// If the current Element is a dex file Element
if(dex ! =null) {
/ / using DexFile loadClassBinaryName class is loaded
Class clazz = dex.loadClassBinaryName(name, definingContext, suppressed);
if(clazz ! =null) {
returnclazz; }}}if(dexElementsSuppressedExceptions ! =null) {
suppressed.addAll(Arrays.asList(dexElementsSuppressedExceptions));
}
return null;
}
Copy the codeIterates through the Element array and returns the class if it finds a class with the same class name as name, or null if it does not.
Through the above analysis, we found that the entire class loading process is:
- Class loader
BaseDexClassLoaderFirst thedexFile parsing intopathListtodexElementsinside - Load the class from
dexElementsGo inside and see which onedexIf you have this class in there, load it and generate itclassobject
Three hot repair
After the class loading process, the hot repair principle is to put the dex file in the first place of the dexElements array, so that when loading the class, the dex file in the patch package will be found first, and after loading the class, it will not look for the dex file. The class with the same name in the original APK file will no longer be used, so as to achieve the purpose of repair:
The Element array in the PathClassLoader is: [patch.dex, class. dex, classes2.dex]. If there is a key. class in patch.dex and classes2.dex, during class search, the loop obtains the DexFile in dexElements, and returns the Key. It does not matter whether dexfiles in subsequent elements can be loaded into key.class. So a hot fix implementation can create a separate fix. Dex file for the buggy class, and then download fix. Dex from the server and save it to a path when the program starts. We then insert the Element object into the dexElements array header in the pathList of our program’s classloader, PathClassLoader. In this way, when loading a class with a Bug, the fix class in fix.dex will be loaded first to solve the Bug.
There is more than one way to hotfix, and there may be other issues that need to be addressed (e.g., reflection compatibility) if the hotfix is to be fully implemented. Another common form of plug-ins are APK and dex files.
Dex Packing Tool (D8)
How to generate the updated dex file?
The Android SDK provides dex packaging tool D8, which can be found in Android Build Tool 28.0.1 and later:
For normal Java files, directly javac into a class file, you can directly use D8 to compile into a dex file:
./d8 XXX.class
Copy the codeIf you want to update the file in APK format, you can package the updated Module/Lib as APK directly in Android Studio.
Code implementation
Dex replacement:
// Make the substitution in Application
public class MApplication extends Application {
@Override
public void onCreate(a) {
super.onCreate();
//dex is loaded as a plug-indexPlugin(); }.../** * dex is loaded as a plug-in */
private void dexPlugin(a){
// Plug-in package file
File file = new File("/sdcard/hotfix.dex");
if(! file.exists()) { Log.i("MApplication"."Plugin hotfix does not exist");
return;
}
try {
// Get the pathList field of BaseDexClassLoader
// private final DexPathList pathList;
Field pathListField = BaseDexClassLoader.class.getDeclaredField("pathList");
// Break the wrapper and set it to callable
pathListField.setAccessible(true);
// Get the pathList object of the current ClassLoader
Object pathListObj = pathListField.get(getClassLoader());
// Get the bytecode file of the current ClassLoader's pathList object (DexPathList)Class<? > dexPathListClass = pathListObj.getClass();// Get the dexElements field of the DexPathList
// private final Element[];
Field dexElementsField = dexPathListClass.getDeclaredField("dexElements");
// Break the wrapper and set it to callable
dexElementsField.setAccessible(true);
// Create a ClassLoader using the plug-in
DexClassLoader pathClassLoader = new DexClassLoader(file.getPath(), getCacheDir().getAbsolutePath(), null, getClassLoader());
// Get the plugin's DexClassLoader's pathList object
Object newPathListObj = pathListField.get(pathClassLoader);
// Get the dexElements variable of the plugin's pathList object
Object newDexElementsObj = dexElementsField.get(newPathListObj);
// Get the dexElements variable of the current pathList object
Object dexElementsObj=dexElementsField.get(pathListObj);
int oldLength = Array.getLength(dexElementsObj);
int newLength = Array.getLength(newDexElementsObj);
// Create a dexElements object
Object concatDexElementsObject = Array.newInstance(dexElementsObj.getClass().getComponentType(), oldLength + newLength);
// Add a new dex to the dexElement
for (int i = 0; i < newLength; i++) {
Array.set(concatDexElementsObject, i, Array.get(newDexElementsObj, i));
}
// Add the previous dex to the dexElement
for (int i = 0; i < oldLength; i++) {
Array.set(concatDexElementsObject, newLength + i, Array.get(dexElementsObj, i));
}
// Set the generated object to the pathList object of the current ClassLoader
dexElementsField.set(pathListObj, concatDexElementsObject);
} catch(Exception e) { e.printStackTrace(); }}Copy the codeApk:
// ApK is loaded as a plug-in
private void apkPlugin(a) {
// Plug-in package file
File file = new File("/sdcard/hotfix.apk");
if(! file.exists()) { Log.i("MApplication"."Plugin hotfix does not exist");
return;
}
try {
// Get the pathList field of BaseDexClassLoader
// private final DexPathList pathList;
Field pathListField = BaseDexClassLoader.class.getDeclaredField("pathList");
// Break the wrapper and set it to callable
pathListField.setAccessible(true);
// Get the pathList object of the current ClassLoader
Object pathListObj = pathListField.get(getClassLoader());
// Get the bytecode file of the current ClassLoader's pathList object (DexPathList)Class<? > dexPathListClass = pathListObj.getClass();// Get the dexElements field of the DexPathList
// private final Element[];
Field dexElementsField = dexPathListClass.getDeclaredField("dexElements");
// Break the wrapper and set it to callable
dexElementsField.setAccessible(true);
// Create a ClassLoader using the plug-in
DexClassLoader pathClassLoader = new DexClassLoader(file.getPath(), getCacheDir().getAbsolutePath(), null, getClassLoader());
// Get the plugin's DexClassLoader's pathList object
Object newPathListObj = pathListField.get(pathClassLoader);
// Get the dexElements variable of the plugin's pathList object
Object newDexElementsObj = dexElementsField.get(newPathListObj);
// Set the plug-in's dexElements object to the current ClassLoader's pathList object
dexElementsField.set(pathListObj, newDexElementsObj);
} catch(Exception e) { e.printStackTrace(); }}Copy the codeSome extensions
-
The apK file is usually selected by replacing the entire dexElements array instead of inserting the preceding values. However, in reality, hot update may only update some classes or resource files. If apK full replacement is used, it will be very heavy. Then incremental replacement, that is, dex file, is a good way.
-
The updated file is generally placed on the server to be downloaded by the client before inserting the value;
-
The so library is implemented in Android code by calling the System.loadLibrary function. A dynamically registered native method calls JNI_OnLoad method and will re-complete the mapping. Therefore, whether we can complete the new method mapping from Java layer native method to native layer patch by loading the original SO library first and then loading the patch SO library? In this way, patch real-time repair of dynamic registration native method is completed.
-
Update mode of resource file: load APK, reflection calls addAssetPath method of AssetManager.
Reference article:
Android Runtime (ART) and Dalvik
Android hot repair core principle, ClassLoader class loading
Understand class loaders in Java from a fundamental level
Super detailed Java ClassLoader details
Android hot repair implementation and principle