Insert the pile
What is staking? Have you used piling techniques in your development?
Staking is the practice of modifying existing code or generating new code during code compilation.
Which process of compilation is involved in staking?
The role and scene of pile insertion
- Code generation
- Code to monitor
- Code changes
- The code analysis
Java source file mode
Similar to the AndroidAnnotation/APT(Annotation Processing Tool), annotations can be parsed at compile time and new Java files can be generated to reduce manual code input. These code generation scenarios, which generate Java files, intervene at the very beginning of compilation. Typical examples are Greendao and ButterKnife
The ORM mapping database Greendao is used internally in our project. In the build directory there are many files with the *. Java suffix. Build is usually the product of the compilation, and it is obvious that these files are Java files generated by the annotation processor when we build.
The bytecode
For code monitoring, code modification, and code analysis scenarios, bytecode manipulation is generally used. Java bytecode of “. Class “can be manipulated, or Dalvik bytecode of”. Dex “can be manipulated, depending on the staking method we use. Bytecode manipulation is more powerful and has wider application scenarios than Java file manipulation, but it is more complex to use.
Java bytecode
For the Java platform, the Java virtual machine runs Class files that correspond internally to Java bytecode.
Dalvik bytecode
Android is an embedded platform. To optimize performance, Android virtual machines run Dex files. Dex can be understood as the compressed format of Android class developed for mobile devices (limited by the early mobile phone configuration is much lower than PC). The Dx tool in the Android SDK toolkit can package class files into dex. It is loaded into the memory by the PathClassLoader of the Android VM.
Cases that I have experienced around me
The rockets rabbit
Previous projects include Java + Kotlin + Flutter and extensive application of annotation frameworks such as ButterKnife, Dagger and Eventbus, resulting in impressive compilation time. It is often possible to debug code just to add a Log line, but compilation can take 3-5 minutes
Example of incremental compilation using Rocket Rabbit:
We can see in the figure above that we have modified a Java file and a resource file. Incremental compilation restarted running apK in only 10 seconds. Of course, the implementation of this incremental compilation method in addition to the requirements of the peg technology, you also need to understand a lot of other comprehensive knowledge, such as compilation principle, scripting language, implementation cost is not low. However, the more people who can promote and use it in the team, the higher ROI (Return on Investment) will be.
Case Contribution: XianLunKing
Global watermark
Similar to dingding, feishu such office software, have anti-screenshot requirements. Now Party A needs to add global watermarks to all activities and dialogs of the APP, such as the name and id of the employee, etc. Generally speaking, there may be hundreds of pop-ups added to the page of an APP. Okay, so we’re working overtime, but what about activities and dialogs, third-party activities in the app?
With AspectJ’s pegging technology, we took days of work that might otherwise have been error-prone and completed it in just a few hours. Mastering the technology of piling and then thinking of a good Hook point can greatly improve our work efficiency.
Case contribution: YanLongLuo
Comparison of pile insertion schemes
AspectJ
AspectJ’s advantages as an old peg framework are 1 mature and 2 simple to use. The downside of AspectJ, however, is that because it is rules-based, its pointcuts are relatively fixed, reducing the freedom to manipulate and control the development of bytecode files. Also, performance after code injection is an important concern if we are going to implement piling all methods. We want to insert only the code we want to insert, and AspectJ generates some additional wrapping code, which has an impact on performance and package size. AspectJX
Javassist
The Javassist source-level API is easier to use than the actual bytecode operations in ASM. Javassist provides a higher level of abstraction over complex bytecode level operations. The Javassist source-level API requires very little or even any actual knowledge of bytecode, making it easier and faster to implement. Javassist uses reflection, which makes it slower than ASM, which uses Classworking technology at runtime. Javassist
ASM
ASM is more direct and efficient than AspectJ. For more complex cases, however, you may need to use a different Tree API to make more direct changes to Class files, so you need to have some essential Knowledge of Java bytecode. ASM is powerful and flexible, but it can be more difficult to get started than AspectJ. However, it can achieve better performance and is more suitable for large-area pile insertion scenarios. ASM
Insert pile of actual combat
ASM
Let’s start by understanding the three important roles of ASM:
- ClassReader
- ClassVisitor
- ClassWirter
ClassReader
As we have seen above, ASM pegs modify bytecode. There must be a process to read the Class bytecode. So ClassReadr is the reader, which provides methods to read bytecode.
ClassVisitor
The ClassVisitor is the heart of ASM staking, because that is where the staking modification of the bytecode takes place. The ClassVisitor is based on the visitor pattern
ClassWirter
As the name implies, this is the tool that ASM provides to write the content after the bytecode modification. The objects that are written including the objects that are read above can be bytecode arrays or they can wrap a layer of bytecode streams (Strem)
Now that we know about the three core APIS for ASM, let’s use a small case:
ASM API practice
class Printer {
public static void main(String[] args) {
b();
}
private static void b(a) {
long s = System.currentTimeMillis();
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
long e = System.currentTimeMillis();
System.out.println("executed time : " + (e - s) + " :ms"); }}Copy the code
When we are investigating some problems, we may need to make positioning and judgment according to the time-consuming time of a certain method. One or more methods can be written manually as described above. But I need to add hundreds of methods or all the methods in the entire application. It is not practical to manually add them. How can I solve this problem?
Dependency package import
implementation group: 'org.ow2.asm'.name: 'asm-commons'.version: '9.2'
Copy the code
Reads the target byte stream
FileInputStream fis = new FileInputStream("/Users/macbook/Documents/thunder/ASMDemo/app/build/intermediates/javac/debug/classes/com/thunder/asmdemo/zxm31/Printer. class");
ClassReader classReader = new ClassReader(fis);
Copy the code
insert
Let’s start with a piece of pseudocode or unfinished code:
The implementation will be left behind to learn about the trunk flow.
Write out the modified bytecode
FileOutputStream fos = new FileOutputStream("/Users/macbook/Documents/ASMDemo/app/build/intermediates/javac/debug/classes/com/thunder/asmdemo/zxm31/Printer.class");
ClassWriter classWriter = new ClassWriter(classReader, ClassWriter.COMPUTE_FRAMES);
fos.write(classWriter.toByteArray());
fos.flush();
fos.close();
Copy the code
Visitor
We didn’t implement the Accept method for the time being. Here we complete the details:
// Truly change the core of bytecode staking
classReader.accept(new TimeClassVisitor(ASM9, classWriter), ClassReader.EXPAND_FRAMES);
Copy the code
The Accept method accepts an entry from a ClassVisitor:
static class TimeClassVisitor extends ClassVisitor {
public TimeClassVisitor(int api, ClassVisitor classVisitor) {
super(api, classVisitor); }}Copy the code
There are many methods that can be overridden inside the ClassVisitor, and our requirement is that we need to insert methods. Here we implement method-dependent functions.
@Override
public MethodVisitor visitMethod(int access, String name, String descriptor, String signature, String[] exceptions) {
return super.visitMethod(access, name, descriptor, signature, exceptions);
}
Copy the code
VisitMethod will analyze the bytecode and output all the methods in the bytecode to us. Let’s verify:
Method constructs, entry methods, and b methods are pretty straightforward, but the ClassVisitor just provides methods for the elements in the class. How do I insert our own code into the method body?
I notice that the return value from visitMethod is MethodVisitor, there’s nothing wrong with that:
Let’s take a look at the MethodVisitor itself, which provides various apis for doing method exercises, but for our need to do code insertion statistics before and after a method, the Commons package has given us a simpler subclass implementation of AdviceAdapter to look at its inheritance:
Take a look at the methods associated with method
onMethodEnter
onMethodExit
It’s the two empty implementations that we need to override, insert before and after the method, and verify the log print.
That’s what we want. With that in mind, we are now ready to start bytecode staking:
We want to insert at Enter:
long s = System.currentTimeMillis();
Copy the code
In exit you want to insert:
long e = System.currentTimeMillis();
System.out.println("executed time : " + (e - s) + " :ms");
Copy the code
Let’s look at the complete bytecode instruction:
class com/thunder/asmdemo/zxm31/Printer {
// access flags 0x0
<init>()V
ALOAD 0
INVOKESPECIAL java/lang/Object.<init> ()V
RETURN
MAXSTACK = 1
MAXLOCALS = 1
// access flags 0x9
public static main([Ljava/lang/String;)V
// parameter args
INVOKESTATIC com/thunder/asmdemo/zxm31/Printer.b (a)V
RETURN
MAXSTACK = 0
MAXLOCALS = 1
// access flags 0xA
private static b(a)V
TRYCATCHBLOCK L0 L1 L2 java/lang/InterruptedException
INVOKESTATIC java/lang/System.currentTimeMillis (a)J
LSTORE 0
L0
LDC 2000
INVOKESTATIC java/lang/Thread.sleep (J)V
L1
GOTO L3
L2
ASTORE 2
ALOAD 2
INVOKEVIRTUAL java/lang/InterruptedException.printStackTrace (a)V
L3
INVOKESTATIC java/lang/System.currentTimeMillis (a)J
LSTORE 2
GETSTATIC java/lang/System.out : Ljava/io/PrintStream;
NEW java/lang/StringBuilder
DUP
INVOKESPECIAL java/lang/StringBuilder.<init> ()V
LDC "executed time : "INVOKEVIRTUAL java/lang/StringBuilder.append (Ljava/lang/String;) Ljava/lang/StringBuilder; LLOAD2
LLOAD 0
LSUB
INVOKEVIRTUAL java/lang/StringBuilder.append (J)Ljava/lang/StringBuilder;
LDC " :ms"INVOKEVIRTUAL java/lang/StringBuilder.append (Ljava/lang/String;) Ljava/lang/StringBuilder; INVOKEVIRTUAL java/lang/StringBuilder.toString ()Ljava/lang/String; INVOKEVIRTUAL java/io/PrintStream.println (Ljava/lang/String;) V RETURN MAXSTACK =6
MAXLOCALS = 4
}
Copy the code
Looking at the stack of bytecode instructions above, we can vaguely see the contents of the b() method:
There seems to be a sentence
INVOKESTATIC java/lang/System.currentTimeMillis ()J
Call a static method, which we found in visit
Let’s simply analyze the bytecode of the first line
JVM bytecode comparison statements
methodVisitor.visitMethodInsn(INVOKESTATIC, "java/lang/System"."currentTimeMillis"."()J".false);
methodVisitor.visitVarInsn(LSTORE, 2);
Copy the code
If every line of instruction set we’re going to do this. For our bytecode knowledge requirements and ASM API master degree of high requirements, but also slow prone to mistakes and high requirements, is there a simpler way? There are:
ASM Bytecode Viewer
The following article will introduce the installation method, let’s take a look at the use:
With this plugin, we can directly convert the Java&Kotlin source code into the API required by ASM, as shown in the figure above:
@Override
protected void onMethodEnter(a) {
System.out.println("start : ");
visitMethodInsn(INVOKESTATIC, "java/lang/System"."currentTimeMillis"."()J".false);
visitVarInsn(LSTORE, 0);
super.onMethodEnter();
}
@Override
protected void onMethodExit(int opcode) {
System.out.println("end : ");
visitMethodInsn(INVOKESTATIC, "java/lang/System"."currentTimeMillis"."()J".false);
visitVarInsn(LSTORE, 2);
visitFieldInsn(GETSTATIC, "java/lang/System"."out"."Ljava/io/PrintStream;");
visitTypeInsn(NEW, "java/lang/StringBuilder");
visitInsn(DUP);
visitMethodInsn(INVOKESPECIAL, "java/lang/StringBuilder"."<init>"."()V".false);
visitLdcInsn("executed time : ");
visitMethodInsn(INVOKEVIRTUAL, "java/lang/StringBuilder"."append"."(Ljava/lang/String;) Ljava/lang/StringBuilder;".false);
visitVarInsn(LLOAD, 2);
visitVarInsn(LLOAD, 0);
visitInsn(LSUB);
visitMethodInsn(INVOKEVIRTUAL, "java/lang/StringBuilder"."append"."(J)Ljava/lang/StringBuilder;".false);
visitLdcInsn(" :ms");
visitMethodInsn(INVOKEVIRTUAL, "java/lang/StringBuilder"."append"."(Ljava/lang/String;) Ljava/lang/StringBuilder;".false);
visitMethodInsn(INVOKEVIRTUAL, "java/lang/StringBuilder"."toString"."()Ljava/lang/String;".false);
visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream"."println"."(Ljava/lang/String;) V".false);
super.onMethodExit(opcode);
}
Copy the code
Run to view printer.class
Well, we’ve done our job, but there are two small problems.
1. Every method is staked, and there are some meaningless construction methods that we do not need to detect. We do not need to pile at all.
2 method print so many not identified class name and method name, then it is not clear which class which method takes how long,
We use annotations to solve this problem. Let’s first define a annotation as follows:
@Retention(RetentionPolicy.CLASS)
@Target(ElementType.METHOD)
@interface StatisticTime {
}
Copy the code
Mark it on the method we want, and then do we have a method for annotations in the MethondVisitor, which we do:
/** * Visits an annotation of this method. * * @param descriptor the class descriptor of the annotation class. * @param visible {@literal true} if the annotation is visible at runtime. * @return a visitor to visit the annotation values, or {@literal null} if this visitor is not * interested in visiting this annotation. */
public AnnotationVisitor visitAnnotation(final String descriptor, final boolean visible) {
if(mv ! =null) {
return mv.visitAnnotation(descriptor, visible);
}
return null;
}
Copy the code
Print its description:
Define a member variable to indicate whether the method is injected or not;
Now only method B is injected, so we add @derprecated annotation to main as a test. Now the first problem has been solved.
The second problem is also easy to solve, let’s transform the BYtecode information of ASM again (onMethidExit). The name of the class can be obtained directly from the AdviceAdapter of this class. The name of the class can also be obtained through the visit method of the ClassVisitor and passed in through the structure.
ASM Bytecode Outline failure and resolution
Many articles on piling refer to the ASM Bytecode Outline tool. However, the actual measurement has failed in AS(4.2.1), indicating AS follows:
The specific reason may be the compatibility problem of the Android Studio upgrade version. Here is a test of an available bytecode staking tool, recommended as follows:
Displays bytecode for Java or Kotlin classes and ASMified code which will help you in your class generation. The tool also supports Kotlin’s bytecode viewing and generation in addition to Java.
Examples of usage:
1 Preference -> Plugins -> Marketplace find the plugin install restart AS
2 Right-click Dialog -> ASM Bytecode Viewer
3. After waiting for the tool’s conversion time, it can be found on the right panel of AS AS follows:
ASMified below is the snippet of the changed bytecode file and ASM API we need for staking. This tool can simplify the knowledge required for staking development, with little or no knowledge of bytecode we can complete ASM staking work.
GradlePlugin & Transform
We’ve already seen how ASM works and how bytecode staking works, but there seems to be a problem.
We use the Main method to do the staking execution, how to use Android automatically, and the target bytecode file staking is our own to read and write?
This is where GradlePlugin & Transform comes in
GradlePlugin can be defined in three ways:
1 Script Task Mode:
The simplest way to define a Task app is build.gradle
task("hello"){
doLast {
println("-- -- -- -- -- -- -- --")}}Copy the code
Sync Now and pull up the right panel
This approach is used to define simple tasks that do not need to be reused
2 buildSrc:
This method must also require the Module to be named buildSrc, which should be somewhere in between the first and third methods. Use buildSrc to build a custom Gradle plugin
Gradle plugin is more complex than the first one. It is more suitable for external release and reuse scenarios
Gradle plugin custom
Let’s take a look at the definition of the most complex but most widely used GradlePlugin:
Create a Module, either Java or Libray
Define the Gradle directory and add the dependencies required by groovy syntax so that writing groovy file code will be prompted later. Intelli IDEA supports this better. Gradle structure is as follows and script is very simple. Groovy support is provided, remember sync.
plugins {
id 'groovy'
id 'maven'
id 'java'
}
repositories {
mavenCentral()
jcenter()
}
dependencies {
implementation gradleApi()
implementation localGroovy()
}
java {
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
}
Copy the code
If the Java folder is not needed, you can delete it and create groovy and Resource files.
The Groovy folder is where we write our plug-in code, and resource provides external calls
Note that the name of the properties file is the final name of the Apply application, which will be covered later. In addition, it should be noted that in the above picture, TimePlugin can normally jump to timeplugin. groovy under the groovy folder. The error prone here is that some details of package names are not matched.
Let’s do a simple Task build with TimePlugin in Groovy:
class TimePlugin implements Plugin<Project> {
@Override
void apply(Project project) {
project.task("customPlugin") {
doLast {
println("I am plugin ~ !")}}}}Copy the code
In fact, a simple Gradle plug-in has been developed, this time how we use our application?
Upload plugins first, either locally or by uploading local plugin artifacts to a remote public repository or your own private Maven repository. In fact, both rely on the same thing, we use the local way:
uploadArchives {
repositories {
mavenDeployer {
// Deploy to the Maven repository
/ / call way is' com. Thunder. The plugin: timeplugin: 1.0 the SNAPSHOT '
pom.groupId = 'com.thunder.plugin'
pom.version = 1.0 the SNAPSHOT ' '
pom.artifactId = 'timeplugin'
repository(url: uri('.. /.. /repo')) //deploy to the local repository}}}Copy the code
In the Gradle script we add a Task uploaded locally and Sync it to execute the Task. Find the corresponding directory and let’s look at the plug-in product
Well, that’s all the preparatory work. Finally we can apply the plugin in our APP:
Application and validation of custom plug-ins
Build. Gradle = build. Gradle = build. Gradle = build.
Then Sync, at which point there is a high chance of success or the following situation:
This error is not found the local plugin product, remember to check
maven {url uri('.. /.. /repo')}
Copy the code
This path needs to have one level less than uploadArchives because the relative level of root Gradle and plugin Gradle is one level less
The root Gradle dependency is configured and the application in app Gradle is ready
plugins {
id 'com.android.application'
id 'time-plugin'
}
Copy the code
The time-plugin id corresponds to the resouce name in the plugin:
After Sync, we will check whether the task defined in our plugin is in the task on the right side of the APP
It can be found. Let’s do it
At this point, we are done with a simple custom plug-in.
Transform
With the customization and application of the plug-in completed, let’s take a look at Transform
Gradle Transform is a standard Android API that developers can use to modify.class files during the build phase of a project
Plugin Gradle add:
implementation 'com. Android. Tools. Build: gradle: 4.2.2', {
exclude group:'org.ow2.asm'
}
Copy the code
Otherwise, the target Transform will not be found. Note its package name:
Time relation, the processing of transform is basically template code, that is, the processing of its own file directory and the third party. Go straight to the code
import com.android.build.api.transform.*
import com.android.build.gradle.internal.pipeline.TransformManager
import org.apache.commons.io.FileUtils
import org.objectweb.asm.ClassReader
import org.objectweb.asm.ClassVisitor
import org.objectweb.asm.ClassWriter
import org.objectweb.asm.Opcodes
class TimeTransform extends Transform {
@Override
String getName() {
return "TimeTransform"
}
/** * There are two enumerated types * CLASS-> handle Java CLASS files * RESOURCES-> Handle Java RESOURCES * @return */
@Override
Set<QualifiedContent.ContentType> getInputTypes() {
return TransformManager.CONTENT_CLASS
}
/** * specifies the Scope of the Transform to operate on. * 1. EXTERNAL_LIBRARIES only has external libraries * 2. PROJECT only has PROJECT content * 3. PROJECT_LOCAL_DEPS Only has local dependencies of the PROJECT (local JAR) * 4 Provide only local or remote dependencies * 5. SUB_PROJECTS has only subprojects. * 6. SUB_PROJECTS_LOCAL_DEPS has only local dependencies (local JARS) for the subproject. * 7. TESTED_CODE Code tested by the current variable (including dependencies) * @return */
@Override
Set<? super QualifiedContent.Scope> getScopes() {
return TransformManager.SCOPE_FULL_PROJECT
}
/** * whether to incrementally compile * @return */
@Override
boolean isIncremental() {
return false
}
/** ** @param context * @param Inputs Have two types: directory, jar, and output path@Override void transform(Context context, Collection<TransformInput> inputs, Collection<TransformInput> referencedInputs, TransformOutputProvider outputProvider, boolean isIncremental) throws IOException, TransformException, InterruptedException { if (! incremental) {// Delete all OutputProviders without incremental updates
outputProvider.deleteAll()
}
inputs.each { TransformInput input ->
// Traverse the directory
input.directoryInputs.each { DirectoryInput directoryInput ->
handleDirectoryInput(directoryInput, outputProvider)
}
// Traversal jar class introduced by third party (third party is not our pile target)
// input.jarInputs.each { JarInput jarInput ->
// handleJarInput(jarInput, outputProvider)
/ /}}}/** * process the class file */ in the file directory
static void handleDirectoryInput(DirectoryInput directoryInput, TransformOutputProvider outputProvider) {
// Is a directory
if (directoryInput.file.isDirectory()) {
// List all files in the directory (including subfolders and files in subfolders)directoryInput.file.eachFileRecurse { File file -> def name = file.name if (filterClass(name)) { ClassReader classReader = new ClassReader(file.bytes) ClassWriter classWriter = new ClassWriter(classReader, ClassWriter.COMPUTE_MAXS) ClassVisitor classVisitor = new TimeClassVisitor(Opcodes.ASM9, classWriter) classReader.accept(classVisitor, ClassReader.EXPAND_FRAMES) byte[] code = classWriter.toByteArray() FileOutputStream fos = new FileOutputStream(file.parentFile.absolutePath + File.separator + name) fos.write(code) fos.close() } } }// This folder contains our hand-written classes as well as r.class, buildconfig. class, and R$xxx. class
// Get the output directory
def dest = outputProvider.getContentLocation(
directoryInput.name,
directoryInput.contentTypes,
directoryInput.scopes,
Format.DIRECTORY)
// The input path is copied to the output path without manipulating the bytecode
FileUtils.copyDirectory(directoryInput.file, dest)
}
/** * Check if the class file needs to be processed * @param fileName * @return */
static boolean filterClass(String name) {
return (name.endsWith(".class")
&& !name.startsWith("R$")
&& "R.class"! = name &&"BuildConfig.class"! = name) } }Copy the code
Notice that the transform method we need to rewrite is already iterating through the class in our APP. The rest of the code is just the same as the ASM code.
I won’t post them one by one
How do you finally use the defined transform?
class TimePlugin implements Plugin<Project> {
@Override
void apply(Project project) {
def android = project.extensions.findByType(AppExtension)
android.registerTransform(new TimeTransform())
}
}
Copy the code
Two lines of code to register, after registration we update the plug-in product, Sync. Then go to the APP to find the Android environment to try:
Observation log:
Starting from scratch, we are familiar with ASM, GradlePlugin, and Transform to implement a simple bytecode peg, although simple. But no matter how complex ASM + GradlePlugin + Transform piling process and core is like this, but the degree of API mastery is not the same, we will be able to use piling technology and AOP ideas to better help our programming development in the future.
Deprecation of the Transform API and the future of piling technology
The Gradle 7.0 Transform API has been deprecated. The Gradle 7.0 Transform API has been deprecated. We don’t have to worry. Thunder is currently Gradle version 4.8. We have been working on JDK 5 ~ 8 for many years
However, the official website of Oracle has been updated to 17. Another point is that no matter how the pegging technology changes, the core technologies such as annotation, reflection, ClassLoader, bytecode and so on will not change. As long as we master these core technologies well, we will be able to master the new technologies quickly.
conclusion
Piling technology can be like sun Wukong understanding 72 changes. Your imagination is as big as the world behind it.
References & acknowledgements
Juejin. Cn/post / 699772…
www.wanandroid.com/blog/show/3…
www.jianshu.com/p/16ed4d233…
Juejin. Cn/post / 700057…
Github.com/AndroidAdva…
Time.geekbang.org/column/intr…
www.jianshu.com/p/e8433c1eb…
www.jianshu.com/p/9039a3e46…
www.jianshu.com/p/dca3e2c86…
Juejin. Cn/post / 698684…
Mp.weixin.qq.com/s/9vbt73d7n…
Github.com/JetBrains/k…
Blog.csdn.net/bytedancete…
www.jianshu.com/p/6785ddb43…
Tech.youzan.com/you-zan-and…
Note: some pictures in this article are quoted from the Internet, if there is infringement, feel free to contact the author to delete.