This article introduces a tool for piling Dex, and explains the problems and solutions encountered when directly modifying Dalvik bytecode and Dex files

Bytedance Terminal Technology — Li Yan

background

In offline scenarios, we often need to insert some detection code in APK to record the time spent on method calls or to add some function to print logs. The current general practice is to modify the class bytecode at compile time. For example, byteX provides a convenient framework for modifying the class.

However, compile-time modification is not flexible enough for the compiled APK to be pinned or modified. As a result, many business parties need to configure independent Jenkins packaging to trigger the test of further progress. Nearly half the time of an automated test task is spent in the packaging process.

To address this pain point, we developed a tool, DexInjector, that directly targets APK(DEX) peg. It is used to collect logs and performance data and inject third-party tools to avoid secondary packaging and save test time.

This scheme has been used in log bypass, network data capture, third-party library injection, user information injection, daily debugging and so on.

The tool currently implements:

  • Method Insert pile in front
  • Method Post insertion
  • Initialize the pile insertion

Technical scheme investigation

Investigate the existing bytecode modification schemes on the market.

smali

Smali and Baksmali tools can be used to convert dex files into smALI grammar files that can be read easily. However, smALI tools parse smALI bytecodes by syntax. If a new code is inserted into registers and other operations, there is no way to achieve structured operations.

redex

Redex supports the piling function by configuring piling before the method and implementing pass. However, functions are limited and it is complicated to use, and some fb custom codes are inserted after implementation. However, Redex provides a set of powerful bytecode modification ability, which will be improved based on the bytecode modification ability of Redex in subsequent versions. Github.com/facebook/re…

dexter

Android.googlesource.com/platform/to…

Dexter tool is a tool developed by Google similar to Dexdump, but its internal implementation of dex file structure and bytecode instructions a complete set of operation API, lightweight and simple, bytecode operation can achieve ASM experience.

In summary, Dexter is used to operate dex.

The project design

demand

According to the requirements of performance prevention and traffic statistics, calls to other methods are inserted before and after the method body of a method. Network traffic statistics, for example, require okhttp3. RealCall. GetResponseWithInterceptorChain method of internal opening to insert a method to get the detailed data request request.

Response getResponseWithInterceptorChain() throws IOException { com.netflow.inject.hookRealCall(this); List<Interceptor> interceptors = new ArrayList<>(); interceptors.addAll(client.interceptors()); / /... Return chain.proceed(originalRequest); }Copy the code

Dex plugging pile

The basic flow

  1. Dex File Analysis

First, analyze the format of Dex file and serialize it into various data structures. The structure of Dex file can refer to official documents

Dalvik executable file format

  1. Bytecode parsing

Parsing binary bytecode into a processable data structure in the code segment

  1. Bytecode construction

Bytecode insertion can be completed by constructing bytecode instructions according to bytecode specifications and inserting them into the sequence of existing bytecode.

  1. Bytecode serialization

Recalculate the Index from the modified Dex structure, and serialize each data Section to the Dex file format.

The functional requirements

Staking supports the ability to insert a static method call before and after the body of a method.

  1. Insert pile in front of method body

If the method being inserted is an instance method, the first parameter of the method is this, and subsequent parameters are the same as those of the method being inserted. If the method is static, the definition of the method being inserted must be the same as the type and number of parameters of the method being inserted. For example:

Public class Tracer{private void MethodA(int a,int b){} private void MethodA(int a,int b){} Private static void MethodB(int a,int b){}} public class Hooker{// Insert method private static void TestHookA(Tracer This_,int a,int b){} private static void TestHookB(int a,int b){} //////// after insertion ///////// public class Tracer{private void MethodA(int a,int b){ Hooker.TestHookA(this,a,b); / /... } private static void MethodB(int a,int b){ Hooker.TestHookB(a,b); / /... }}Copy the code

  1. Insert a pile behind the method body

The return value of the inserted method must be of the same type as the returned value of the inserted method.

Pay attention to the processing of parameters. Insert methods should comply with the following rules:

Method name (this, the method argument being inserted, return value type)

For example:

Public class Tracer{// Insert method private void MethodA(int a, int b){//...... } private String MethodB(int a, int b){ //...... return str; } } public class Hooker{ private static void TestHookA(Tracer this_,int a,int b){} private static String TestHookB(Tracer this_,int a,int b,String return_val){// the value of return_val is the true return value of the original method. }} / / / / / / / / insert / / / / / / / / / after public class Tracer {private void MethodA (int a, int b) {/ /... Hooker.TestHookA(this, a, b); } private String MethodB(int a, int b){ //...... return Hooker.TestHookB(this, a, b, str); }}Copy the code

  1. Initialize the pile insertion

This function will parse the Application class defined in the Application node in androidmanifest.xml.

Insert code into the OnCreate or attachBaseContext method, depending on the configuration. If the OnCreate and attachBaseContext methods are not defined, the staking tool generates them.

Common Problem Handling

Due to the limitations of format and instruction of Dex, some rules of Dex and Dalvik instruction should be followed in the process of modification and insertion of bytecode. The following describes some problems and solutions encountered in direct operation of Dex.

Method number processing

When the amount of code increases, a DEX file can only contain 65535 methods and method references due to the design defects of Google in the early years, and the insertion itself will inevitably introduce new methods and method references. At some point, as is common in most apps, a dex file in an APP gets too close to 65K to insert a new method call.

One solution is to merge the whole Dex together and then split it, which will destroy some optimization of the original Dex and need to realize the calculation of application relations between classes. Therefore, a lightweight solution is adopted here because of a large amount of calculation.

Dex unpacking

Solution 1:

Adding –set-max-idx-number at compile time forces the compiler to try not to fill up the dex, but this scheme may not work, and if the APK is processed by a tool like Redex, the dex may also fill up.

Solution 2: Dex split logic

If the number of methods remaining in the current dex is not enough to insert new methods, part of the existing dex classes will be removed and added to an additional dex.

For example, if the maindex list and its referenced classes are added to the first dex, the number of methods will not reach 65535. If the number of methods exceeds 65535, there will be Too many classes in — “main-dex” -list error. The compiler then fills the first dex with classes that have smaller reference relationships. These classes are the ones we want to break down.

The target is mainly to find the classes that are not called in this dex, filter out the references of all classes by traversing the locations of all method calls, attribute references and class references, and filter out the classes that are not called and split them into other dex.

Main logic:

  1. Determine whether the number of methods of dex can continue the pile insertion. If the pile insertion cannot be carried out, dex segmentation logic is required
  2. Iterate over the parameters of each method of each class, recording the type
  3. Iterate over the properties of each class, recording the type
  4. Bytecode instructions are iterated over each method, recording the referenced type through method calls, attribute references, and type-override instructions

Source.android.com/devices/tec…

An instruction with the bytecode format of 22C 21C 31C 35C 3RC has a reference to a method or property of a class in its last operand, which can be retrieved from the class used by the method.

  1. Will allinterface annotationKeep it in the original dex
  2. The remaining classes are those that are not used in the dex and can be split out without affecting the execution of the dex.
  3. Package the unused classes separately into an extra dex. For example, if there are four existing dex, create a new dex to save them.

This will free up some method space for the insertion of dex.

Dex combined

  1. Split DEX merge

After the separation of Dex, Dex is divided into two parts. We need to combine the separated Dex into one Dex and attach it to the last Dex.

As shown above, classes.split.dex, classes3.split.dex,…… Classes9.split. dex will be merged into the same dex as classes11.dex

  1. Dex combination of pile insertion

The code called by the piling method is not generally packaged into APK, so you need to merge the code into APK. The dex to be inserted is directly merged into the last dex. If the last dex cannot be merged, a new dex is created and merged into it.

String Jumbo processing

There are two instructions in Dalvik bytecode to read a string from the constant pool into a register, Const – string vAA, string [@ BBBB] (https://my.oschina.net/u/205605) and const string/jumbo vAA. String [@ BBBBBBBB] (https://my.oschina.net/u/2326784), the first instruction only support access to the range of 0-0 XFFFF string, due to we insert the new method calls, will add a string (name of the class, the method name), In many cases, the total number of strings exceeds 65535. Because the Dex format requires that a utF-16 code point value be used to sort the string content, a reorder is performed after the insertion of a new string, which changes the original string index. Causes the const-string instruction to access a string with an index higher than 0xFFFF, causing the VM to execute abnormally.

Pile tools to do the processing, after completion of the plug pile will scan all const – string vAA, string [@ BBBB] (https://my.oschina.net/u/205605) instruction, If the accessed string index exceeds 65535, the const-string is forcibly changed to the const-string/jumbo directive.

processes

The target method is confused

In most cases, the target method has a high probability of being confused, so we need to find the confused objective function based on the mapping file and then insert the pile.

The inserted dex uses classes from the original APK

In many cases, the insertion method invocation to the dex we inserted may use the classes provided by the original APK. Since the classes in the original APK have experienced confusion, the direct invocation by the name before the confusion will lead to the exception that the class, method and attribute cannot be found.

Use the mapping file to confuse the class name, method name, and attribute name in the inserted dex, and change the caller’s strong line to the confused name.

Class deleted, method inlined/deleted

  1. Priority should be given to adding obfuscations to the original APK compilation process.
  2. If the logical association between the invoked class and the original APK is not very large, it is recommended to rename the used class package name and type it into dex together. In this way, it will appear that the APK has the same class, but the package name is inconsistent. The inserted dex only calls its own integrated class, so that the confusion of this class is not concerned.
  3. In many cases is the need to use to the original apk class that cannot be addressed by renaming the package name, such as through the class of the incoming parameters in the calling method of these classes may be the method to be confusion is deleted, it is possible that be inline or there is no other place to use and be deleted, so as far as possible to avoid calling methods in the calling process.
  4. There are some cases where the get set methods of some properties are inline to access the properties directly

Confusion before:

After the confusion:

To avoid this, try to use property access directly when calling the GET set method.

Such as:

If this get set methods were not inline, so there will be a call property is private and protected to fileld validation is not through, in Java. Lang. IllegalAccessError: Field ‘XXXX’ is inaccessible to class error The solution is to force the called property permission to public. You need to specify in advance which attributes have access permissions to change. These configurations are set in a configuration file, which is explained later.

Class repeat problem handling

In most cases, we will encounter the problem that the inserted Dex and the inserted APK have the same class name, such as calling a common third-party library. The most common encountered is the inserted Dex written in Kotlin, which will have the Kotlin library.

There are two solutions:

  1. Remove duplicate classes in the inserted Dex

When making and inserting Dex, use the function of extracting classes according to the package name of the Dex insertion tool to cut out the required classes to form Dex. At this time, some classes that duplicate APK can be removed, and the inserted Dex uses APK’s own library. In this case, you need to confuse the inserted Dex according to the mapping to invoke it normally.

  1. Renaming conflicting third-party libraries

Calls duplicate classes that call themselves according to the package name. For example, the kotlin package is renamed kotlin_copy, so that its dex calls kotlin_copy. XXXX and the original APK do not conflict.

Bytecode staking

Method Insert pile in front

Add an invoke-static/range {} instruction to the front of the method to pass the parameters of the original method to the hook method

.method public static monitorEvent(Ljava/lang/String; Lorg/json/JSONObject; Lorg/json/JSONObject; Lorg/json/JSONObject;) Registers (); registers (); registers (); p3}, Lcom/bytedance/apm_bypass_tool/monitor/BypassMonitor; ->monitorEvent(Ljava/lang/String; Lorg/json/JSONObject; Lorg/json/JSONObject; Lorg/json/JSONObject;) V const/4 v0, 0x4 ....Copy the code

Method Post insertion

  1. Look for all return instructions and pin them before execution

  1. Return value processing

Since the return value is passed to the hook method as an argument, you need to apply for a register to hold the result of the return value and pass it through.

Except for return-void directives, all return directives come with a return value as follows:

invoke-direct {p2, p0, p1}, Lcom/ss/android/lark/ico$1; -><init>(Lcom/ss/android/lark/ico; Ljava/lang/reflect/Type;) V return-object v4Copy the code

Save the value in register P2 to an extra register, then retrieve the return value of the hook method, and return it

invoke-direct {p2, p0, p1}, Lcom/ss/android/lark/ico$1; -><init>(Lcom/ss/android/lark/ico; Ljava/lang/reflect/Type;) V move-result-object v4 invoke-static {p0, p1, p2, v4}, Lcom/netflow/inject/NetFlowHookReceiver; ->hookCallServerInterceptor_executeCall_end(Lcom/ss/android/lark/ici; Lcom/ss/android/lark/idj; Lcom/ss/android/lark/icy; Lcom/ss/android/lark/idi;) Lcom/ss/android/lark/idi; Move-result-object v5 // You can use v4 return-object v5 if you do not change the return valueCopy the code

Parameter register multiplexing problem

In some cases, the compiler will reuse the parameter register as a general purpose register in order to reuse the register when returning a value. As a result, when we retrieve the parameter after the method, we find that the parameter register has been reused, so we can’t get the value of the parameter correctly.

The parameters introduced in the function are named starting at p0 and increasing in order. For example, a method will use registers v0, v1, P0, p1, and p2. V0 and v1 represent local variable registers. In the case of instance methods, P0 represents a reference to this, and p1 and p2 represent two parameters passed in.

For example, the p1 register was reused to store the return value, which led to the failure of the peg method to obtain the correct P1 parameter

invoke-interface {p1, p2}, Lcom/ss/android/lark/idf; ->a(Lcom/ss/android/lark/idh;) Lcom/ss/android/lark/idj move-result-object p1 return-object p1Copy the code

Solutions:

This problem can be solved by extending the number of registers corresponding to the number of parameters above the original number of registers, for example

A method register layout is as follows: v0 v1 v2 P0 P1 P2 multiplexes the P1 register in the current bytecode. After extending the current register with the same number of parameters, the register layout is as follows: v0 v1 V2 V3 V4 V5 P0 P1 P2 The position of the original bytecode referring to P1 becomes V4. For example, the bytecode becomes v4:  invoke-interface {p1, p2}, Lcom/ss/android/lark/idf; ->a(Lcom/ss/android/lark/idh;) Lcom/ss/ Android /lark/ IDJ move-result-object V4 return-object V4 this prevents parameter registers from being reusedCopy the code

Register expansion problem

In the process of register expansion, the problem of abnormal instruction will be encountered. The main reason is that the number of registers expands too much than 16. The use of registers in the original bytecode can ensure the correct use of registers, and the correct insertion of registers should also be ensured.

In practice, it is uncommon for a method to require more than 16 registers, and quite common for a method to require more than 8 registers, so many instructions are limited to the first 16 registers addressed. Instructions are allowed to reference up to the first 256 registers as far as is reasonably possible. In addition, some instructions have variants that allow more registers, including a pair of catch-all move instructions that address registers in the v0-V65535 range. If the instruction variant cannot be used to address the desired register, the register contents are moved from the original register to the lower register (before the operation) and/or from the lower result register to the higher register (after the operation).

For example, in the directive “move-wide/from16 vAA, vBBBB” :

“Move” is the base operation code, which represents the base operation (move the value of a register).

“Wide” is the name suffix, indicating that the instruction operates on wide (64-bit) data.

“From16” is an opercode suffix that represents a variant with a 16-bit register reference source.

“VAA” is the target register (implied in the operation; In addition, the target parameter is always the first), and the value ranges from v0 to v255.

VBBBB is the source register. The value ranges from V0 to v65535.

For example, when using registers larger than V16, convert move-object vA, vB instructions to move-object/from16 vAA, vBBBB or move-Object /16 vAAAA, vBBBB

Dex pile production

The peg Dex is the extra Dex that we want to insert into APK, which is the code that the peg code calls.

Generate Dex

For example, put the code you want to insert into a separate Gradle Module

After compiling, decompress aar, extract jar package, and convert Java bytecode to dex through d8 command

mkdir resources

./gradlew inject-dex:clean

./gradlew inject-dex:assembleRelease

d8 inject-dex/build/intermediates/aar_main_jar/release/classes.jar --output resources/

mv resources/classes.dex resources/netflow_caller.dex

mv resources/netflow_caller.dex netflow_caller.dex


Copy the code

Scheme 1: extraction pile insertion class

After compiling, some system libraries and third-party libraries that duplicate the original APK will be packaged, so these system libraries or third-party libraries need to be filtered out.

The utility provides a function to extract classes by package name, which can be separated into a dex for classes with a specified package name.

Before extracting:

After the extraction of:

Scenario 2: Rename duplicate third-party libraries

Third party libraries can be renamed using the rename function. The advantage of this over using APK classes is that it can solve the problem of third party library versioning and confusion.

Mars-talk 04 is here!

On the night of February 24th in MARS TALK, we invited the engineers from Volcano Engine APMPlus and Beauty, Online share “APMPlus Java OOM Attribution scheme based on Hprof file” and “Optimization practice based on Mars-APMPlus Performance monitoring tool” and other technical dry goods. Join the event group now and have a chance to get the latest VR all-in-one machine — Pico Neo3!

⏰ Live: 20:00-21:30, Thursday, February 24

💡 Event format: live online

🙋 registration: scan code into the group registration

For the first MARS TALK of the year, we have a big prize for you. In addition to the Pico Neo3, there are logitech M720 Bluetooth mouse, fascia gun and byte peripheral gifts for you to pick up. Don’t miss it!

👉 Click here to learn about APMPlus