One, foreword

The Ministry of Industry and Information Technology has paid more and more attention to the issue of App claim, and several apps from large factories have been removed from the shelves for rectification:

One of the most critical questions is thisBefore users agree to the privacy agreement, they cannot collect users’ private informationFor example, obtain deviceId and androidId information. In addition, you need to pay attention to frequent and out-of-scope permission applications.

In addition to iterative rectification, from a technical point of view, is there a way to eliminate the non-compliance of privacy calls once and for all?

This is the solution to be introduced in this article. In the early stage, the private method calls are efficiently detected through run-time hook technology.

In the later stage, Gradle Plugin+Transform+ASM is used to hook and replace the privacy method invocation, control the privacy behavior of App and third-party SDK, and completely solve the privacy non-compliance problem.

Second, run-time hook technology

In the early stage of privacy rectification, privacy method calls can be dynamically monitored by uploading APK to Springer platform, and then the platform will install AND run APK, as shown below:

It takes at least 50 minutes to complete the package-upload-test process

As for the real-time monitoring of privacy behavior, the implementation principle is nothing more than using the hook technology at runtime to record method call information.

In theory, we can also use the run-time hook technology to realize the function of quickly detecting private method calls and obtaining call stack offline.

So what are the run-time hooks?

2.1 Xposed

If you are more familiar with the Xposed, and have a root device installed Xposed framework, then directly develop an Xposed module to hook specified method.

Interested in Xposed source analysis can refer to this article: sorry, Xposed really can do whatever you want – eventually · Paoding decoding, the author has a series of Xposed article.

Because my test equipment is root permission, Xposed scheme is not difficult for me, but for ordinary users, there is no root free way?

Some ~

2.2 VirtualXposed

VirtualXposed is based on VirtualApp and Epic in the non-root environment run Xposed module implementation (support 5.0~10.0).

VirtualXposed is actually a support Xposed virtual machine, we put the development of the Xposed module and the corresponding need to hook App installation can achieve hook function.

Due to VirtualApp 2017 closed source to commercial, open source version has many problems, and due to its hook a large number of system functions, so there are many compatibility problems, some App may not open after installation, so if the device at hand just encounter compatibility problems, you can consider changing the phone ~

2.3 epic

In 2014, Ali opened the Dexposed project, which can implement runtime method interception on Dalvik VIRTUAL machine without invasion.

But Android 5.0 began to use ART virtual machine, not support ART Dexposed is history.

After the dimension of the big guy in ART to re-implement the Dexposed, with Dexposed completely the same ability and API, the project address is Epic.

So if you don’t want to fiddle with Xposed or VirtualXposed, as long as the application access epic, you can realize the application of Xposed hook function, meet the needs of running hook.

2.3.1 Epic Principle:

The principle is to modify the entry function of ArtMethod to change the first 8 bytes of the entry function into a section of jump instruction to jump to the function that performs hook operation. The principle is similar to AndFix, the hot repair framework of Ali, as shown in the figure below.

I am Dexposed for a second – on ART runtime Method AOP implementation

2.3.2 Implement a configurable runtime Hook framework based on EPIC

  1. Read configuration:
val inputStream = context.resources.assets.open("privacy_methods.json") val reader = BufferedReader(InputStreamReader(inputStream)) val result = StringBuilder() var line: String? = "" while (reader.readLine().also { line = it } ! = null) { result.append(line) } val configEntity = Gson().fromJson(result.toString(), PrivacyMethod::class.java) configEntity.methods.forEach { hookPrivacyMethod(it) }Copy the code
  1. The JSON configuration is as follows and stored in the assets directory:
{ "methods": [ { "name_regex": "android.app.ActivityManager.getRunningAppProcesses", "message": "Read the current running application process"}, {" name_regex ":" android. Telephony. TelephonyManager. Listen ", "message" : "monitor incoming call information"},... }Copy the code
  1. Hook based on the read configuration

    private fun hookPrivacyMethod(entity: PrivacyMethodData) {
        if (entity.name_regex.isNotEmpty()) {
            val methodName = entity.name_regex.substring(entity.name_regex.lastIndexOf(".") + 1)
            val className = entity.name_regex.substring(0, entity.name_regex.lastIndexOf("."))
            try {
                val lintClass = Class.forName(className)
                DexposedBridge.hookAllMethods(lintClass, methodName, object : XC_MethodHook() {
                    override fun beforeHookedMethod(param: XC_MethodHook.MethodHookParam?) {
                        super.beforeHookedMethod(param)

                        Log.i(TAG, "beforeHookedMethod $className.$methodName")
                        Log.d(TAG, "stack= " + Log.getStackTraceString(Throwable()))
                    }
                })
            } catch (e: Exception) {
                Log.w(TAG, "hookPrivacyMethod:$className.$methodName,e=${e.message}")
            }
        }
    }
  
  
Copy the code
  1. The running effect is as follows:

As shown in the figure, the run-time capability to export the privacy method call stack is basically implemented, supporting methods that require a hook through JSON configuration.

For example, Android 11 only supports 64-bit apps. Therefore, you are advised to use epic only in the DEBUG environment.

Hook technology at compile time

Using Epic only solves the problem of validating private method calls, not the following:

  1. How does the Release environment monitor privacy method calls?
  2. How to control the frequent invocation of privacy methods by third-party SDKS?

Both of these problems can be solved using compile-time hook technology.

When it comes to compile-time hooks, you first need to understand the compilation process

3.1 Compilation Process

We use Android Studio development, using Gradle compilation tools, for APK compilation process everyone should know, as shown below:

The apK compilation process consists of the following big steps: 1. Package the resource file and generate the R.java file 2. Compile the AIDL file into a Java file 3. Compile the Java file into a. Class file 4 using the javac command. 5. Use apkBuilder to package dex files and resource files into APK 6. Apk signature 7.

The fourth step (packaging the class file into a dex file) involves a Gradle Transform process

3.2 to understand the Transform

The Transform schematic diagram is shown below

Taking class files, JAR files, and resource files as input, through a series of transforms,

First there is the custom Transform processing, then there is the system Transform processing, and the last Transform is responsible for generating the dex file.

Relevant source can see TaskManager createPostCompilationTasks method, the compilation process source in this ~

Screenshots just posted a custom Transform source code, the back and the Transform of the system, and the appliesCustomClassTransforms, for example, used to Profile the underlying implementation.

A Transform is associated with a taskFactory. A Transform corresponds to a Gradle Task.

Using the Transform Plugin, we can register a custom Transform into the compilation process to retrieve all the.class files and modify the bytecode using the ASM tool.

Customize Gradle Plugin and register Transform as shown below

class Plugin : Plugin<Project> {

    override fun apply(project: Project) {
  
      if (project.plugins.hasPlugin("com.android.application")) {
          val extension = project.extensions.getByName("android") as AppExtension
          extension.registerTransform(CommonTransform(project))
      }
    }
}
Copy the code

To understand why custom plugins are written this way, you can look at the source code of App compiler plugins AppPlugin

Create AppExtension, name is Android, and finally save it to ExtensionsStorage in a variable called Extensions LinkedHashMap.

The previous eproject. Extensions. GetByName, finally is read from the LinkedHashMap.

So once I have my.class file, how do I modify it? This involves modifying the bytecode scheme selection.

3.3 Bytecode modification framework selection

In addition to ASM, Javaassist is also the mainstream bytecode modification framework.

As the project has high requirements on performance and package volume, it is undoubtedly appropriate to adopt ASM scheme.

3.4 Learning about the ASM Framework

We can get the.class file by custom Transform, and then we can use the ASM tool to process bytecode.

Android looks like a lofty bytecode modification, so learn the right! .

Gradle Plugin + Transform, the framework is basically built by template code. In order to save time and trial and error costs, this article directly refers to Dokit and adopts Booster API as the underlying implementation of the plug-in. Booster shields the differences between apis of different Gradle versions.

Said so much, the most important thing or to see the scheme design ~

Iv. Primary Hook scheme

We can get all of the.class files by custom Transform, and then we can get bytecode for each class and method by ClassVistor and MethodVistor, respectively.

For example, ActivityManager#getRunningAppProcesses, we want to replace it with PrivacyUtil#getRunningAppProcesses, as follows:

The core hook code is as follows:

classNode.methods.forEach { method -> method.instructions? .iterator()? .foreach {insnNode -> if (insnNode is MethodInsnNode) { Replace the if (insnNode desc = = "android/app/ActivityManager getRunningAppProcesses () Ljava/util/List;" && insnNode.name == "getRunningAppProcesses" && insnNode. opCode == opcodes.invokespecial) {// Method instructions replace insnNode.opcode = Opcodes. INVOKESTATIC / / call the class to replace insnNode. Owner = "com/lanshifu/asm_plugin_library/privacy/PrivacyUtil" / / replace the method name Insnnode. name = "getRunningAppProcesses" // Replace insnNode.desc = "com/lanshifu/asm_plugin_library/privacy/PrivacyUtil.getRunningAppProcesses (Landroid/app/ActivityManager;) Ljava/util/List;" }}}}Copy the code

Explanation:

By iterating through each method bytecode instruction, judgment is ActivityManager getRunningAppProcesses this method calls, will replace PrivacyUtil# getRunningAppProcesses calls, involves the bytecode is the basis of operation.

Tip: Why iterate over the bytecode instructions for each method? Because the methods that need hook are system methods and are not packaged into APK, simply iterating through the method name is not found, and the bytecode instructions called in each method must be iterated.

At this point, our primary version of the compile-time privacy method hook function is implemented, but there are several problems:

1. Hard coding is difficult to maintain, and it is troublesome to add hook method;

If PrivacyUtil is Not imported or upgraded, Class Not Found Exception is reported. If PrivacyUtil is Not upgraded, Class Not Found Exception is reported.

3. Development needs to be familiar with ASM bytecode. Every time a new privacy method hook is added, it needs to verify bytecode changes before and after modification, which is very troublesome.

Fifth, the advanced scheme

The key to solving the three problems of the primary solution is to make it “configurable”,

You need to be able to read the hook configuration at compile time. Annotations are appropriate.

The advanced scheme idea is as follows:

  • Use the first Transform to collect annotation information and generate a hook configuration.

  • Replace the privacy method by reading the hook configuration with a second Transform.

5.1 Custom Annotations

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.CLASS)
public @interface AsmMethodReplace {
    Class oriClass();

    String oriMethod() default "";

    int oriAccess() default AsmMethodOpcodes.INVOKESTATIC;
}
Copy the code

The class name, method name, method type (static method/member method) of the method that needs to hook.

5.2 Annotation processing and configuration generation

To replace a method, we need the following configuration:

Original method information (before replacement) : oriClass, oriMethod, oriAccess, oriDesc

TargetMethod information (after replacement) : targetClass, targetMethod, targetAcces, targetDesc

The target method information can be obtained via ClassNode, but the original method information in the AsmMethodReplace annotation is not appropriate, because the oriDesc is more troublesome to write, so the convention here is a rule to use the annotation, and then the oriDesc code to read on the line.

Here are the rules:

  • For hook static methods, the parameters of the annotated method remain the same as the original method
  • In the case of hook member methods, the first argument to the annotated method is a Class object, followed by the same arguments as the original method

OriDesc is then calculated by subtracting the first parameter from targetDesc.

For example: targetDesc = (Landroid/telephony/TelephonyManager;) Ljava/lang/String; OriDesc = Ljava/lang/String;

An 🌰

5.2.1 Example 1: Hook member methods

Suppose you want to replace ActivityManager’s getRunningAppProcesses method

public List<RunningAppProcessInfo> getRunningAppProcesses() { try { return getService().getRunningAppProcesses(); } catch (RemoteException e) { throw e.rethrowFromSystemServer(); }}Copy the code

Since this is a member method, the annotation is written as follows:

@JvmStatic @AsmMethodReplace(oriClass = ActivityManager::class, oriAccess = AsmMethodOpcodes.INVOKEVIRTUAL) fun getRunningAppProcesses(manager: ActivityManager): List<RunningAppProcessInfo? > {//hook processing}Copy the code

5.2.2 Example 2: Hook static methods

Suppose you want to replace settings. System’s getString method

public static String getString(ContentResolver resolver, String name) {
		return getStringForUser(resolver, name, resolver.getUserId());
}
Copy the code

Since this is a static method, the annotation is written as follows:

@JvmStatic @AsmMethodReplace(oriClass = Settings.System::class, oriAccess = AsmMethodOpcodes.INVOKESTATIC) fun getString(resolver: ContentResolver, name: String): String? {/ / processing AndroidId if (Settings. Secure. ANDROID_ID = = name) {} return Settings. The System. Get string (resolver, name)}Copy the code

For details, please refer to the source code at the end of the article.

5.3 the flow chart

The final process as above, should be more clear ~

5.4 Precautions

ASM hook needs to have a trace, must be clear where the bytecode modification, can print log, can save records in the file, if there is a problem can be checked from the hook log.

5.5 summary

The advanced scheme mainly does the following things:

  1. With an annotation processing Transform, custom annotation information is collected at compile time to generate a hook configuration;
  2. Use another Transform to read hook configuration, hook corresponding method;
  3. After privacy method hook, increase cache to solve the problem of SDK frequently reading privacy information;
  4. Before the user has not agreed to the privacy agreement, if the privacy method is called, toast can be prompted and the call stack printed, as shown below, so that the problem is clear at a glance.

Sixth, other

At present, There are some open source libraries for compile-time piling, such as Ele. me open source Lancet, which is based on Gradle Plugin+Transform+ASM.

If you want to learn bytecode staking in depth, I recommend Didi’s open source Dokit, which has many bytecode operations to learn, such as big picture monitoring, network monitoring and so on.

Because Gradle version updates quickly, it is best to build your own compile-time hook infrastructure in your project. In this way, if you have problems, you can solve them easily and improve your bytecode development skills.

Seven,

Starting from the privacy compliance requirements of the Ministry of Industry and Information Technology, this paper roughly introduces the following knowledge points:

  1. Run-time Hook framework introduction and application
  2. Epic usage and principles
  3. Compile-time Hook framework
  4. Introduce the principle and application of Transform from apK compilation process
  5. Compare hook schemes at compile time
  6. Finally, a configurable compile-time method replacement scheme is implemented to completely solve the non-compliance problem of privacy method invocation

In this paper, the difficulty is not very big, mainly is the Gradle plug-ins and skewer bytecode modifications of the entire process, involving technology is basic, finally built a hook framework compilation method, can do a lot of things, based on the framework of the hook after such as slow method detection, all buried points, monitoring thread calls ~

Step by step, in this paper, the source code related reference articles: governance privacy permissions | android black magic Play together in the Android project where bytecode Android client privacy security deal booster

The last post of 2021, happy New Year in advance!