GoSSIP_SJTU · 2015/10/01 the assembling
Author: @Love will win
This article is an extended version of “Research on Universal Automatic Unmasking Methods for Android Applications” presented at cloud Summit.
0x00 Background and significance
Android applications compared to the traditional PC application easier to reverse, because after reverse can complete reduction of the Java code or smali intermediate language, has a wealth of high-level semantic information in both, more easy to understand, let the program logic easily exposed to technical ability even before the attacker does not need a high threshold. Therefore, Android application reinforcement protection service arises accordingly. At the beginning, only Party A provided the service, but now large Internet companies have their own reinforcement protection services. Meanwhile, more and more Money-related Android applications, such as banks, have begun to use reinforcement to protect themselves. This market is expanding.
A typical hardening protection service provides the following protections: anti-reverse, anti-tamper, anti-debug, anti-theft, and so on. Although the hardening service cannot avoid and prevent the security problems and vulnerabilities of the application program, it can effectively protect the real logic of the program and the integrity of the application program. However, these features can also be easily exploited by malicious programs, and data show that the proportion of shell malicious programs is increasing with the popularity of hardened protection. On the one hand, malware analysis needs to be unmasked first, and on the other hand, normal applications face increased risk if they are easily unmasked after analysis.
0x01 Study object
The security hardening service provides a DEX security hardening solution and customized security hardening. Custom hardening usually needs to be more closely integrated with development and may involve deeper hardening (native code hardening, etc.), while DEX overall hardening only requires the user to provide a compiled Android application APK. The former is currently lacking in samples and requires in-depth cooperation with reinforcement manufacturers, while the latter is provided by most reinforcement service manufacturers as the most basic free service, so the latter is more widely used. The main research object of this paper is to protect the executable file DEX of the latter Android application, that is, DEX file encryption, aiming to study the general DEX file recovery method. Customized reinforcement services or obfuscation protection for native code are not within the scope of this paper.
0x02 Hardening Service Features
We use an example of a static reverse hardening approach to describe in detail the typical characteristics of a hardening service. This example is a few months ago a reinforcement vendor used the scheme, because the reinforcement services often change decryption algorithm and scheme, so the implementation details do not apply to the current products, or other reinforcement services, but the overall reinforcement ideas and methods and the use of protection means are basically the same.
Usually when we use static tools to analyze a hardened APP, the androidmanifest.xml file will retain all the original information, including defined components, permissions, and so on, and add a new entry point class, usually application.
The DEX code looks like this.
DEX code contains only a few classes and codes. It mainly does some detection work or preparation work, and then loads the original DEX file dynamically by loading a native library. Because of the dynamic loading mechanism, the hardened DEX file does not involve the real code of the original DEX (some hardened files do not adopt the dynamic loading of the full DEX).
IDA is then used to load running native code from the reverse entry point, and often the SO library is also confused and shelled. Methods include destroying ELF headers to make IDA parsing fail, as shown below:
It is obvious from readelf that several fields in the ELF header are problematic.
After the fix, IDA was able to disassemble the SO file as normal. We then start from the entry point and find that decompiling F5 into C code is problematic, with multiple function contents failing to decompile into normal C code. Look directly at the assembly code to see the following flower instruction:
This is our summary of the flower command pattern of the product. It makes it difficult to identify a decompiled function by jumping off the stack, because decompiled usually thinks of a decompressed operation as a function call, when in fact it decompresses the stack, calculates the register value, jumps off the stack, disables the decompiled operation, balances the stack, and then executes a truly useful instruction. So there are only two really useful instructions in the above example.
Real assembly instructions can be extracted by scripting or even manual methods. Extract and then reverse the code to decrypt the JNI_OnLoad function. JNI_OnLoad will decrypt another ELF file from one piece of data before the new ELF file disassembles properly, and the code will then modify the ELF data. First unzip the text side of the new ELF file, extract a key from the text side and then decrypt rotext. Finally, decrypt a real shell program of DEX, like:
The above steps are actually an ELF file shell. The new modified ELF file is the real decryption program for DEX shell. This program does not confuse or shell. After reverse reverse, it finds that it will take a segment of padding data after the original DEX, obtain some parameters required for decryption and decompression, decrypt and decompress the whole segment of padding data, and then get the real original DEX file. Of course ELF also includes some anti-debugging and anti-analysis code, since this is static analysis, we do not need to care about this part of the code, if you use the debugger to attach the process to use dynamic analysis such as dump, you need to consider how to elegant bypass these anti-debugging techniques.
The above example is an example of dynamic loading DEX, although different reinforcement services in many technical details including the decryption algorithm, flower instruction mode, the ELF shell and so on to heaven, but can basically represent the vast majority of using dynamic load DEX way of strengthening the service of the overall decryption release running and the thinking method of static reverse and break it. Let’s take this example for a glimpse. Because the frequent transformation decryption algorithm and reinforcement method is also the first major feature of reinforcement services.
At the same time, in fact, some reinforcement do not use the dynamic loading mechanism of the complete DEX file, but use dynamic self-modification at run time. Under this mechanism, the hardened DEX file will have some accurate information in the original DEX, but the protected part of the code will choose other ways to hide. There are also ways to combine the two. This will be covered in a later case study.
To sum up, a hardened Android application actually hides the real DEX file and adds a lot of protection to prevent it from being easily reversed. It can be seen that if the reverse analysis of the peeling algorithm is purely static, it will be very time-consuming and labor consuming. In addition, different reinforcement services adopt different algorithms, and each of them will frequently change the algorithm and reinforcement technology, so that the pure static reverse peeling method will fail in a short time. At the same time, the hardening service will also take a lot of Android application protection measures in addition to DEX dynamic loading, we will briefly summarize here, not expand, because this part of the content could even be a separate article in detail.
The first category is completeness checking. Includes runtime integrity checks for itself, such as checking the DEX file in memory and checking application certificates to detect repackaging and code insertion. As well as the detection of its own environment, for example, the simulator is detected by checking specific device files, the debugging is detected by ptrace or process status, and the hook specific function prevents the code memory from being read or dumped.
The second category is code obfuscation. Often obfuscations require modifications based on source code or bytecode to make it harder for analysts to understand the semantics of the program. The most common examples include modifying variable names, method names, class names, etc., encrypting constant strings, calling methods using Java reflection, disrupting the program control flow by inserting junk instructions or invalid code, replacing original basic instructions with more complex operations, and interrupting the control flow with JNI methods.
The third category, which we define as anti-analysis or code hiding techniques, is designed to prevent program code from being directly exposed and easily analyzed in a variety of ways. The most common is the DEX integral encryption protection described above, along with dynamic self-modification at run time. Runtime dynamic self-modification mainly decrypts and executes the code when it reaches a specific class or method while the program is running. At the same time, it may modify or modify part of the Dalvik data structure dynamically, making analysis difficult. Other anti-analysis techniques require a little trickery. For example, static analysis tools can be used to fight against bugs or features of parsing, including manifest cheating, APK pseudo-encryption, method hiding in DEX files, and static analysis tools can crash by inserting illegal instructions or non-existent classes.
0x03 Idea of peeling method
In the face of the reinforcement program, there are two methods of shell removal that are popular and commonly used. One is static reverse analysis, which has obvious disadvantages, is difficult and cannot fight transformation algorithms. The other is mainly based on memory dump technique. The disadvantage is that you need to consider various methods of bypass debugging, but also need to face the increasingly developed and new anti-memory dump techniques. For example, tampering with the dex file header to prevent poor search, and dynamically tampering with dalvik data structure to destroy dex files in memory, etc. These countermeasures require a lot of manual repair work after observing the hardened features even after the DEX file is dumped.
Therefore, we propose a general automatic shucking method. Our method is based on dynamic analysis, and it does not need to care about the specific implementation of different reinforcement protection, but also can uniformly bypass various anti-debugging means, and does not need to do a lot of repair work in the late stage.
First of all, the object we unshell is DEX file in the Android application, so we choose to directly modify the source code of Dalvik VIRTUAL Machine in the Android system for plugging. Because all the code in the DEX file needs to be interpreted and executed in the Dalvik VIRTUAL machine, all the real behavior can be exposed in the Dalvik Virtual machine. Dalvik has a number of interpretation modes, among which portable mode is implemented based on C++, while other modes are developed by platform-related assembly language due to optimization. In order to facilitate the implementation of our staked code, once we find that we start to interpret and execute the APP that needs to be unshell, We first (source directory dalvik/vm/interp interp. CPP) will explain mode to portable. One advantage of this is that directly modifying the execution environment makes it more difficult for the packer to detect the presence of unpacking behavior, which is more transparent than, say, debugger attaching. Another advantage of working with the interpreter is that you don’t have to worry about the stage at which the ruggedizer loads and initializes the classes, decrypts the code, etc., and you get the most realistic data and behavior at run time. Plug pile code implements in the Dalvik explain execute each instruction switch (Dalvik/vm/mterp/out/InterpC – portable. CPP), so that we can in the arbitrary enforcement of the instructions for the operation of the shell, the side to run while strengthening program to unlock. Finally, the modification based on the source code can implement real machine deployment, Android native source code can perfectly support all Nexus series phones, there is no need to deal with the detection simulator means of hardening procedures.
The essence of unmasking is to obtain the real behavior of the program, so the staked code is actually to obtain Dalvik data structure in memory to reflect the real code being executed. When the instruction is executed, the structure of Method, which the instruction belongs to, can be obtained directly. And each executed method has the class Object clazz to which the method belongs, and clazz (dalvik/ VM /oo/ object.h) has pDvmDex (Dalvik/VM/dvmdex.h) Object, The pDexFile (dalvik/libdex/ dexfile.h) structure represents the DEX file, that is, after obtaining the current method during execution, CurMethod ->clazz->pDvmDex->pDexFile to get the DEX file structure of this method. This structure contains the memory information of all DEX files when they are interpreted and executed, and the truest DEX can be recovered by parsing this DexFile structure.
0x04 Simple unshell implementation
So far, our first reaction is whether there is a ready-made program that can translate Dalvik bytecode, but with the DexFile structure in memory as the input, and can be implemented directly based on the source code, that is, in C/C++, rather than reading a static DEX file as the input as more static reverse tools. For found in the Android source code itself to provide the DexDump (dalvik/DexDump DexDump. CPP) this tool, directly can satisfy this requirement. We have modified the DexDump code slightly and inserted it into the interpreter as shown below:
Tell him to read the DexFile, and by default execute this code directly in the main Activity of an APP. The main Activity can be retrieved from the androidmanifest.xml file, because the entry point classes in that file are not hidden. We found that this was almost enough to handle most ruggedizes, and we were able to get the real code in the DEX file where the ruggedizer was hidden, as shown below:
However, the disadvantage of this method is also obvious, that is, the output is in the text form of Dalvik bytecode, on the one hand, it cannot be disassembled into Java, on the other hand, the text form is very unsuitable for subsequent analysis of complex programs, our best goal is to get a complete DEX file.
0x05 Perfect peeling implementation
In order to recover the entire DEX file, many other unshell tools will choose to directly read pDexFile->baseAddr or pDvmDex->memMap as the starting address, and directly dump the entire memory size of the file. However, we found that for some reinforcement software, then dump out the code does not contain the real code, this is due to some real information in the DEX file be modified at runtime and mapped to file continuous memory outside of the part, the following figure, a DEX file into the memory, are supposed to be in a continuous memory space, It is then parsed and assigned to the structures required by Dalvik for each dynamic execution, and some of the simple structures should point to contiguous data blocks. However, the hardening program may make some changes, such as tampering with part of the header data, and reallocating discontinuous memory to hold the data data, and directing those index data blocks to the newly allocated data blocks. In this way, if you directly use the dump method, you cannot get a complete DEX file.
We aim to recover the original DEX file in a uniform way, and do not want to have to do subsequent fixes for different shells, as this would be the same dilemma as static reverse hardening algorithms. Therefore, based on the above simple implementation, we have a more perfect implementation scheme, called DEX file reorganization. The process is very simple, that is, during the execution of the program, all Dalvik data structures required by the interpreter are first obtained, which are real data structures in memory to be interpreted and executed, and then these data structures are combined and written back into a new DEX file. As shown in the figure above, even if the memory is discontinuous, we do not need to care about its operation on the original mapped memory. We can directly obtain each discontinuous data and recombine these data into a new DEX file according to certain specifications. The first step is to obtain each Dalvik data structure accurately. In order to ensure the accuracy of obtaining, we adopt the same method as when executing the program in the running interpreter (refer to the dexGetXXXX method in the dexfile. h file), because a DEX file, The same piece of data can be retrieved in many ways, for example, by reading offsets in the file header for a constant string, or by using a stringId list, and so on. Normally these methods should be correct, but hardening programs can do some damage. But it can’t destroy the data that was used to retrieve the data at runtime, because once it’s broken, the program won’t work. The specific acquisition method is shown in the figure below:
We need to iterate through each array (e.g. PStringIds, pProtoIds… , pClassDefs), fetch the Pointers and offsets of each entry, and merge the contents into a class (e.g., stringData, typeList,… , ClassData, Code). Then there are a few things to be aware of when you get the rewrite done. Firstly, we arrange the data blocks by referring to the order of enumeration of map item Type Codes in Dalvik /libdex/ dexfile. h. ParametersOff, stringDataOff, interfacesOff, classDataOff, codeOff, etc. Then for the values in the DexHeader and MapList structures, we need to recalculate and fill them in, instead of taking the original values directly. For some fixed values, such as the file head in the Header, we fill them directly according to the existing knowledge. Finally, we need to take into account the differences between the data expression in memory and some data formats in DEX files. For example, some data items are ULEB128 encoded in files, while they are directly int in memory. In addition, we need to pay attention to the alignment of 4 bytes. And encoded_method_format is field_idx_diff, method_idx_diff instead of simple index, etc. For details, see the official DEX file format document
https://source.android.com/devices/tech/dalvik/dex-format.html
We ignored some data blocks in the reorganization, such as all data structures related to annotations, because these parts are rarely used in real programs, and the structures are extremely complex, so ignoring them will have little impact on the real behavior of analysis programs.
0x06 Experiments and Discoveries
The libdvm module was recompiled and the new libdvm.so was written to the system directory /system/lib/, overwriting the original library files. The objects of our experiment are Android 4.3 for Galaxy Nexus phones and Android 4.4.2 for Nexus 4 phones. We then submitted a simple application and sent it to each online hardening service to get the hardened version of the application and unshell it. Experiments show that the original DEX files can be recovered for almost all hardening programs. Here are some of the findings about the hardening process. The main focus is on the different self-protection methods used for hardening. Some of the results here are DexDump text, because some protection measures are better displayed in this way. Of course, all can be directly restored to DEX files.
The above two examples show that some hardening programs will erase the Magic number to hide the DEX file in memory and invalidate the search method of DEX file. In addition, they will tamper with the size of the header and erase the offset values of various fields in the header. Since the method we use is to recalculate the header, Therefore, the recombinant DEX is not affected by it.
Other hardening programs insert extra classes to disrupt normal decomcompilations, such as this class that has a method that disables dex2jar.
Another shell changes codeOff to a negative value so that the code is mapped out of file memory. Our approach takes the code and writes it back to normal.
There are also shells that override methods, putting code into a new method, decrypting it before execution, and erasing it afterwards. In this case, since our undressing code is staked at each method call, we simply need to adjust the undressing point to the method execution to undress the code.
In addition to the above examples, we also found that some hardening programs will hook the write function in the process space to detect if the written content is specific data (such as the dex file header), so that the write operation will fail, or obtain the memory address space in the mapped DEX file area, so that the write operation will fail. Also, the hardening program will separate the original DEX file into multiple DEX, and modify specific data items such as debug_info_off to incorrect values, which will be dynamically changed back to correct values at run time. There are shells that obfuscate the original program based on bytecode.
(Note: the above examples are not the latest version, do not guarantee that the existing products of the specific hardening program are still consistent with the above examples)
0x07 Discussion and Thinking
First of all, our method still has limitations. First of all, the research object indicates that we only protect DEX file encryption and do not do anti-obfuscation work. Secondly, our method is still based on dynamic analysis, which will face the limitations of dynamic analysis. For example, an encryption code is decrypted only after running, but the method cannot be triggered to execute, and our method cannot decrypt the code of this method. Finally, although it is difficult to be detected by the hardening program with this method, the tool made by this method must have some characteristics in the implementation, which may be used and counterbalanced by the hardening program.
Finally, I’d like to discuss with you some ideas for better Android application hardening. In fact, it is relatively easy to crack the Android platform, but it is not that there are more difficult and more secure hardening solutions, but commercial hardening solutions on mobile platforms need to take into account performance loss and compatibility problems, which is inevitable. At the same time, I think the trend and development of reinforcement protection mainly focus on the following points.
One is that I think Android obfuscation and shell can actually be used together. From an attacker’s point of view, I think brute obfuscation is probably more effective at protecting code logic than shells. But good obfuscation schemes are actually very difficult to design. At present, the domestic reinforcement will almost not do big transformation and confusion to the original code, may be afraid of the modified code will have problems in compatibility. I think this is a point of development. I found that excellent foreign tools will make an issue of this point of deep confusion, such as Dexprotector, which has both shell and confusion. Even if successfully unshell, it still needs to face confusing codes.
In addition, I think the effect of partial reinforcement may be stronger than the overall security. As in the previous example, a method decrypts itself only at run time and re-encrypts or erases itself once it is out of run. This takes advantage of the disadvantage of low coverage of dynamic execution to further protect itself.
The third is that in order to improve the reinforcement effect, the reinforcement process should be changed from the current reinforcement after development to the reinforcement in development as far as possible. There are some good attempts to harden the SDK for this. Use a secure library interface for sensitive operations directly during development. This no matter be in performance go up or the effect can be opposite now integral one size fits all type consolidate to do a qualitative rise. Familiar with the business developer will be very clear that part which is the code needed to protect them, because the logic of a program that really needs to be protected, in fact may only be a fraction, narrowing of the reinforcement range can greatly improve the performance, at the same time, a separate security library files can be targeted protection measures, the effect will be very good, It is also easier to do compatibility testing than the whole APP reinforcement.
Another idea of reinforcement is to use Native code as much as possible, especially the key program logic. The reverse of Native code itself is more difficult than Java, and even more difficult after adding confusion or shell. At the same time, Native code can actually improve the performance, which is a solution of two birds with one stone. Thus, the problem of how to do deep protection for native codes in Android applications can be extended. If sensitive operations are protected by deep confusion protection native codes, the attack cost will be greatly increased.
In the end I think is a trend of strengthening protection as little as possible to use small was catnip to do protection, such as those using static analysis tools of bugs or analytical APK system BUG to do reinforcement in fact meaning is not very big, reinforcing protection should be consider from the whole computer system architecture and reinforcement, and should not focus on a few small skills.