preface
Recently, DoKit V3.3.1 has been released, which adds a number of new features and distinguishes between Android X and Android Support in the library name.
For details, see DoKit Android version information
If you are interested, go to the Android reference documentation to upgrade your experience.
Technical background
Zero intrusion of business code has always been DoKit’s bottom line. DoKit is a one-stop terminal development solution. We continue to provide the community with a variety of excellent tools to help them improve their r&d efficiency, while at the same time ensuring that their online code delivery is as good as possible. Fortunately, since the launch of DoKit, we have accumulated more than 10,000 users and have not received any online bugs caused by DoKit integration. So how do we provide powerful tools to our users with zero intrusion of business code? In fact, AOP is indispensable behind this.
DoKit AOP principle
(The following picture is from my DoKit feature share within Didi Group.)
Selection of AOP scheme
There are two major AOP implementations for Android in the community :AspectJ and the AS plug-in +ASM. Earlier versions of DoKit used AspectJ as a solution, but as the DoKit community became more robust and more used, people began to report that AspectJ would clash with AspectJ in their projects due to inconsistent versions, causing compilations to fail. The DoKit team has always been concerned about the user experience of the community, so after a lot of research and community validation, we finally decided to replace the entire AOP solution with AS Plugin+ASM. After several releases of validation, we found that ASM had significantly fewer conflicts during the project integration process than AspectJ, which gave us confidence to optimize the solution in the future. ASM is a lower-level solution that works directly on the JVM bytecode. Therefore, we need to overcome the following two difficulties when using ASM:
1. You should have some knowledge of JVM bytecode (asm.ow2.io for those interested).
2. In order to find the optimal Hook point, we need to understand the library principle of the mainstream third party.
AOP principle
After determining the technology selection, we look at the related principles of ASM. In fact, we can get a general idea of how this works. AS Gradle compiles Java class files, JAR files, and resource files into the first Transform and outputs the raw data to the second Transform. And so on to form a complete link. And ASM is the first red TransformA that acts on this diagram. It will get the raw data from the beginning and then it will do some analysis. And invokes the relevant callback methods for classes, variables, methods, and so on in the JVM bytecode format. In the corresponding callback method we can operate on the relevant bytecode instructions. Such as add, delete and so on. The picture in the middle is a sequence diagram of how it works. Finally, compiling the two together produces a new JVM class file.
AOP Landing scenario
Standing on the shoulders of giants can help us implement relevant features faster and better. Adhering to the concept of not reinventing the wheel, we decided to use Didi Booster as the underlying implementation of the DoKit plug-in after extensive selection of technologies. Booster shields API differences between Gradle versions and is a powerful feature that we strongly recommend for interested students to check out.
To make it easier to understand, let me give you a concrete example. As you can see from the examples in the figure, compiling with the DoKit AOP plug-in is equivalent to writing some code for the user on our own initiative. With this proxy programming model, we can retrieve the user’s object at runtime and modify the object’s properties.
As the figure shows, AOP has been implemented in most of the functionality in DoKit so far.
DoKit AOP scenario landing
Let’s take a closer look at how DoKit performs bytecode manipulation in an elegant way in these landing scenarios.
(All DoKit bytecode operations only apply to Debug packages, so don’t worry about contaminating online code)
(Due to space reasons, I only selected a few features that the community is concerned about for analysis, in fact, the principle of bytecode operation is similar, we need creativity and a lot of reading of the source code of three parties, so as to find the most elegant peg point)
A larger image detection
In fact, the community has a very detailed analysis of the article, I do not have a specific analysis here, we refer to: through ASM to achieve large map monitoring
The function takes
Function time can refer to an article I wrote before: Didi DoKit Android core principle revealed function time
Function switch Configuration
DoKit sets a switch function at compile time for each plug-in function to prevent some bytecode operations from causing compilation failure and runtime bugs in specific scenarios. At the same time, in order to remind users of the status of this function in a more friendly way, we will judge the status of the user at compile time. Gradle. properties or build.gradle configuration information is obtained by DoKit, which is due to bytecode. Let’s take a look at the implementation logic.
DoraemonKitReal has an empty pluginConfig method built in for bytecode instrumentation. A DokitPluginConfig class is then defined to store and read configuration information.
Public class DokitPluginConfig {/** * inject pluginConfig dynamically into DoraemonKitReal#pluginConfig */ public static void inject(Map) config) { //LogHelper.i(TAG, "map====>" + config); SWITCH_DOKIT_PLUGIN = (boolean) config.get("dokitPluginSwitch"); SWITCH_METHOD = (boolean) config.get("methodSwitch"); SWITCH_BIG_IMG = (boolean) config.get("bigImgSwitch"); SWITCH_NETWORK = (boolean) config.get("networkSwitch"); SWITCH_GPS = (boolean) config.get("gpsSwitch"); VALUE_METHOD_STRATEGY = (int) config.get("methodStrategy"); }}Copy the codeInject (map) dokitpluginconfig.inject (map) is dynamically inserted into the pluginConfig method at compile time. This map stores the configuration information at compile time. Let’s look at the code for bytecode manipulation, CommTransformer:
If (className = = "com. Didichuxing. Doraemonkit. DoraemonKitReal") {/ / plug-in configuration klass. The methods? .find { it.name == "pluginConfig" }.let { methodNode -> "${context.projectDir.lastPath()}->insert map to the DoraemonKitReal pluginConfig succeed".println() methodNode?.instructions?.insert(createPluginConfigInsnList()) } } /** * Create pluginConfig code instructions * / private fun createPluginConfigInsnList () : InsnList { //val insnList = InsnList() return with(InsnList()) { //new HashMap add(TypeInsnNode(NEW, "java/util/HashMap")) add(InsnNode(DUP)) add(MethodInsnNode(INVOKESPECIAL, "java/util/HashMap", "<init>", "()V", Add (VarInsnNode(ASTORE, 0)); 0)) add(LdcInsnNode("dokitPluginSwitch")) add(InsnNode(if (DoKitExtUtil.dokitPluginSwitchOpen()) ICONST_1 else ICONST_0)) add( MethodInsnNode( INVOKESTATIC, "java/lang/Boolean", "valueOf", "(Z)Ljava/lang/Boolean;" , false ) ) add( MethodInsnNode( INVOKEINTERFACE, "java/util/Map", "put", "(Ljava/lang/Object; Ljava/lang/Object;) Ljava/lang/Object;" , true ) ) add(InsnNode(POP)) ......... Add (VarInsnNode(ALOAD, 0)) add(MethodInsnNode(INVOKESTATIC, "com/didichuxing/doraemonkit/aop/DokitPluginConfig", "inject", "(Ljava/util/Map;) V", false ) ) this } //return insnList }Copy the codeSince the bytecode instructions are a bit long, I’ll just pick a portion of the code here. First, we use the fully qualified name to find the method we need to operate on in the class during compilation. Then insert the code dynamically through the ASM API. The resulting code is as follows:
private final void pluginConfig() {
HashMap hashMap = new HashMap();
hashMap.put("dokitPluginSwitch", true);
hashMap.put("gpsSwitch", true);
hashMap.put("networkSwitch", true);
hashMap.put("bigImgSwitch", true);
hashMap.put("methodSwitch", true);
hashMap.put("methodStrategy", 0);
DokitPluginConfig.inject(hashMap);
}
Copy the codeIf you are interested, you can use our Demo on Github to see the differences in the pluginConfig method before and after compilation.
Position simulation
As a unicorn enterprise in the travel industry, Didi needs to assist in the development and test simulation of various location information. So this is a tool that we use a lot within the group. Let’s look at the implementation.
At present, there are mainly Autonavi, Tencent, Baidu and several versions of the MAP SDK of Android on the market. At present, DoKit is fully compatible.
System comes with
The latitude and longitude of the system is realized by hook LocationService. For the specific code, refer to LocationHooker. Since this section does not involve bytecode manipulation, I won’t go into details
The three maps
Since we don’t know which SDK will be integrated into the user’s project, we’ll compileOnly (see config. Gradle) :
/ / gold map location compileOnly rootProject. Ext. Dependencies [" amap_location "] / / positioning compileOnly tencent map RootProject. Ext. Dependencies [" tencent_location "] / / baidu map location compileOnly files (' libs/BaiduLBS_Android. Jar)Copy the codeThis avoids introducing unnecessary map SDKS and reduces compilation conflicts. As the SDK calls API of Baidu, Tencent and Autonavi are almost the same, I will take Autonavi as an example for analysis. First, let’s look at how To return latitude and longitude by demo:
private var mapLocationListener = AMapLocationListener { aMapLocation -> val errorCode = aMapLocation.errorCode val errorInfo = aMapLocation.errorInfo Log.i( TAG, "High German positioning ===lat==>" + aMapLocation. Latitude +" LNG ==>" + aMapLocation. Longitude + "errorCode===>" + errorCode +" errorInfo===>" + errorInfo ) } mLocationClient!! .setLocationListener(mapLocationListener)Copy the codeWe can actually get the user’s AMapLocationListener object if we can change the code like this
Public void setLocationListener(AMapLocationListener) {public void setLocationListener(AMapLocationListener) AMapLocationListenerProxy aMapLocationListenerProxy = new AMapLocationListenerProxy(aMapLocationListener); try { if (this.f110b ! = null) { this.f110b.mo19841a((AMapLocationListener) aMapLocationListenerProxy); } } catch (Throwable th) { CoreUtil.m1617a(th, "AMClt", "sLocL"); }}Copy the codeThe AMapLocationListener proxy object is built into DoKit
public class AMapLocationListenerProxy implements AMapLocationListener { AMapLocationListener aMapLocationListener; public AMapLocationListenerProxy(AMapLocationListener aMapLocationListener) { this.aMapLocationListener = aMapLocationListener; } @Override public void onLocationChanged(AMapLocation mapLocation) { if (GpsMockManager.getInstance().isMocking()) { try { mapLocation.setLatitude(GpsMockManager.getInstance().getLatitude()); mapLocation.setLongitude(GpsMockManager.getInstance().getLongitude()); // Force change of p through reflection cause: look at mapLocation. SetErrorCode reflectutils. reflect(mapLocation). Field ("p", 0); mapLocation.setErrorInfo("success"); } catch (Exception e) { e.printStackTrace(); } } if (aMapLocationListener ! = null) { aMapLocationListener.onLocationChanged(mapLocation); }}}Copy the codeSo how does landing into bytecode actually work?
/ / insert related bytecode if Scott map (className = = "com. Amap. API. Location. AMapLocationClient") {klass. The methods? .find { it.name == "setLocationListener" }.let { methodNode -> MethodNode?) instructions?) insert (createAmapLocationInsnList ())}} / / insert the bytecode private fun createAmapLocationInsnList () : InsnList {return with(InsnList()) {// Insert the custom proxy callback class add(TypeInsnNode(NEW, "Com/didichuxing/doraemonkit/aop/AMapLocationListenerProxy"), add (InsnNode (DUP)) / / access the first parameter to the add (VarInsnNode (ALOAD, 1)) add(MethodInsnNode( INVOKESPECIAL, "com/didichuxing/doraemonkit/aop/AMapLocationListenerProxy", "<init>", "(Lcom/amap/api/location/AMapLocationListener;) V", false)) add(VarInsnNode(ASTORE, 1)) this}Copy the codeWe’re going to go through all the class resource files and find the setLocationListener method specified by the fully qualified name. Then we’re going to use the ASM inset method to operate and insert our built-in code where the setLocationListener method starts. So as to achieve the user no perception of the purpose.
The Mock data
As a key feature of DoKit, data Mock has been basically implemented across all platforms (Android, iOS, H5 JS and small programs), and has been widely discussed and highly appraised in the community. So we can focus on that.
Traditional solutions
Let’s start by looking at how data mocks can be performed without using DoKit’s data mocks. Our development and testing often use packet capture tools to view and modify the data returned by the network. First of all, let’s take a look at the problems existing in the existing packet capture scheme:
1) The same interface cannot be operated by multiple people
2) Different scene data cannot be returned for the same interface.
3) The operation of packet capture is very complicated. It needs to be on the same LAN as the mobile phone, and the IP address and port number need to be changed.
DoKit addresses these issues by creating data mocks for all platforms.
In order to achieve this goal AFTER a certain degree of research, I summed up to achieve this goal we have to solve the difficulties.
1) Unify the various network frameworks of Android terminal.
2) Ensure zero intrusion of business code.
3) In order to intercept Ajax requests in H5, we must hook Webview.
Let’s take a look at how DoKit solves these problems on Andoid. (The whole link is still a bit long, please be patient and look down.)
Data Mock(terminal)
This is a simple flowchart of the DoKit data Mock terminal solution at compile time and runtime. Since today’s main focus is on AOP bytecode, let’s take a look at how DoKit is implemented.
1. Unified network request
We all know that Android terminal encapsulation of the three-party network framework there are many, but a careful analysis of the fact that the bottom is basically based on HttpClient(Google abandoned the maintenance of compatibility), HttpUrlConnection, Okhttp(the most used). So we just need to unify the HttpUrlConnection and OkHttp frameworks. After research, OkHttp officially offers a solution to transform HttpUrlConnection into OkHttp requests: ObsoleteUrlFactory.
So we can convert HttpUrlConnection to an OKHTTP request using the following code.
if (protocol.equalsIgnoreCase("http")) {
return new ObsoleteUrlFactory.OkHttpURLConnection(url, OkhttpClientUtil.INSTANCE.getOkhttpClient());
}
if (protocol.equalsIgnoreCase("https")) {
return new ObsoleteUrlFactory.OkHttpsURLConnection(url, OkhttpClientUtil.INSTANCE.getOkhttpClient());
}
Copy the codeOnce you’ve figured out how to convert HttpUrlConnection to OkHttp, the next step is to get the HttpUrlConnection object.
Val url = url (path) // openConnection val urlConnection = url.openconnection () as HttpURLConnection val 'is' = urlConnection.inputStreamCopy the codeThe above code is the standard API for HttpUrlConnection. The urlConnection object is created from url.openConnection(). So we need to change the above code to the following code at compile time.
Val url = (path) / / open the connection url val urlConnection. = HttpUrlConnectionProxyUtil proxy (url. The openConnection ()) as HttpURLConnection / / get the input stream val ` is ` = urlConnection. The inputStreamCopy the codeSo how does it actually fall onto the bytecode? The code is as follows:
private val SHADOW_URL = "com/didichuxing/doraemonkit/aop/urlconnection/HttpUrlConnectionProxyUtil" private val DESC = "(Ljava/net/URLConnection;) Ljava/net/URLConnection;" klass.methods.forEach { method -> method.instructions? .iterator()? .asIterable()? .filterIsInstance(MethodInsnNode::class.java)? .filter { it.opcode == INVOKEVIRTUAL && it.owner == "java/net/URL" && it.name == "openConnection" && it.desc == "()Ljava/net/URLConnection;" }? .forEach { method.instructions.insert(it, MethodInsnNode(INVOKESTATIC, SHADOW_URL, "proxy", DESC, false)) } }Copy the codeThrough these operations above we basically realize the unity of the network framework.
2. Insert interceptor
We all know that the core of OkHttp is its interceptor, so all we need to do is to insert our own built-in interceptor at the head of the interceptor list at project launch so that we can intercept all network requests in the project. Through careful reading of the source code, we see that the initialization of the Okhttp interceptor list is done in the OkHttpClient#Build.
public static final class Builder { Dispatcher dispatcher; @Nullable Proxy proxy; List<Protocol> protocols; List<ConnectionSpec> connectionSpecs; Final List<Interceptor> interceptors = new ArrayList<>(); Final List<Interceptor> networkInterceptors = new ArrayList<>(); EventListener.Factory eventListenerFactory; ProxySelector proxySelector; }Copy the codeThen we need to add our own built-in interceptor to the head of the interceptor list at the end of the OkHttpClient#Build constructor. The code is CommTransformer:
If (className == "okhttp3.OkHttpClient\$Builder") {// Klass.methods? .find { it.name == "<init>" && it.desc == "()V" }.let { zeroConsMethodNode -> zeroConsMethodNode? .instructions? .getMethodExitInsnNodes()? .forEach { zeroConsMethodNode .instructions .insertBefore(it,createOkHttpZeroConsInsnList()) Find {it. Name == "<init>" && it. Desc == "(Lokhttp3/OkHttpClient;) V" }.let { oneConsMethodNode -> oneConsMethodNode? .instructions? .getMethodExitInsnNodes()? .forEach { oneConsMethodNode .instructions .insertBefore(it,createOkHttpOneConsInsnList()) } } }Copy the codeLet’s see what the compiled code looks like.
public Builder() { this.interceptors = new ArrayList(); this.networkInterceptors = new ArrayList(); this.dispatcher = new Dispatcher(); . this.pingInterval = 0; / / compile-time inserted code enclosing interceptors. AddAll (OkHttpHook. GlobalInterceptors); this.networkInterceptors.addAll(OkHttpHook.globalNetworkInterceptors); } Builder(OkHttpClient okHttpClient) { this.interceptors = new ArrayList(); this.networkInterceptors = new ArrayList(); this.dispatcher = okHttpClient.dispatcher; . . / / compile-time inserted code OkHttpHook performOkhttpOneParamBuilderInit (this, okHttpClient); }Copy the codeThere are four interceptors OkHttpHook built into the DoKit SDK
public static void installInterceptor() { if (IS_INSTALL) { return; } try {globalInterceptors.add(new MockInterceptor()); globalInterceptors.add(new LargePictureInterceptor()); globalInterceptors.add(new DoraemonInterceptor()); globalNetworkInterceptors.add(new DoraemonWeakNetworkInterceptor()); IS_INSTALL = true; } catch (Exception e) { e.printStackTrace(); }}Copy the codeAt this point, the terminal network interception function has been completed. It is also the basis for packet capture, data Mock, weak network simulation, and large graph detection. Interested partners can more in-depth understanding of the source code.
The Mock data (js)
Having said that, let’s take a look at how we can intercept JS requests in H5. As you can see, one of the technical requirements for intercepting js requests is WebViewClient#shouldInterceptRequest(check out this method). By convention, we have to hook the WebView first. For example:
mWebView = findViewById<WebView>(R.id.normal_web_view)
initWebView(mWebView)
mWebView.loadUrl(url)
Copy the codeTo load h5, we must call loadUrl. So we need to do something to the webView before loadUrl. Like this:
mWebView = findViewById<WebView>(R.id.normal_web_view)
initWebView(mWebView)
WebViewHook.inject(mWebView);
mWebView.loadUrl(url)
Copy the codeIt doesn’t seem very complicated, but there’s a catch. We need to change the order of the top of the bytecode stack by means of bytecode. So let’s do a little bit of intuition with the code.
klass.methods.forEach { method -> method.instructions? .iterator()? .asIterable()? .filterIsInstance(MethodInsnNode::class.java)? .filter { it.opcode == INVOKEVIRTUAL && it.name == "loadUrl" && it.desc == "(Ljava/lang/String;) V" && isWebViewOwnerNameMatched(it.owner) }? .forEach { method.instructions.insertBefore( it, CreateWebViewInsnList ())}} / * * * * create a webView function instruction set reference: https://www.jianshu.com/p/7d623f441bed * / private fun createWebViewInsnList(): InsnList {return with(InsnList()) {// Copy the top 2 instructions to the instruction set such as aload 2 aload0 aload 2 aload0 add(InsnNode(DUP2)) // throw the top instruction Add (InsnNode(POP)) add(MethodInsnNode(INVOKESTATIC, "com/didichuxing/doraemonkit/aop/WebViewHook", "inject", "(Ljava/lang/Object;) V", false ) ) this } }Copy the codeNote the use of DUP2 in conjunction with the POP directive, the reason for which is explained in the comments. That’s the difficulty of this piece. As you can see, bytecode instructions are very powerful, and you can really do anything you want if you know a lot about bytecode.
So the code compiled by our plugin looks like this:
mWebView = findViewById<WebView>(R.id.normal_web_view)
initWebView(mWebView)
String var3 = this.url;
WebViewHook.inject(mWebView);
mWebView.loadUrl(url)
Copy the codeThere’s an extra line of URL assignment code, but it basically doesn’t affect our functionality and we don’t need to care.
And finally once we’ve got the Webview object we can inject our own WebviewClient. WebViewHook
private static void injectNormal(WebView webView) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
if (!(WebViewCompat.getWebViewClient(webView) instanceof DokitWebViewClient)) {
WebSettings settings = webView.getSettings();
settings.setJavaScriptEnabled(true);
settings.setAllowUniversalAccessFromFileURLs(true);
webView.addJavascriptInterface(new DokitJSI(), "dokitJsi");
webView.setWebViewClient(new DokitWebViewClient(WebViewCompat.getWebViewClient(webView), settings.getUserAgentString()));
}
}
}
Copy the codeWe’ve already said that the entry to the shouldInterceptRequest method doesn’t get the body of the post. After some research, we can actually get the original HTML data stream in this method. So we just need to insert our own JS script into the original HTML data before the Webview starts rendering. In the script, according to the prototype chain principle of JS, We’ll specify prototypes for several core methods of XmlHttpRequest and Fetch, as dokit_js_hook. HTML and dokit_js_vconsole_hook. HTML. Then we inform the terminal of JS request information through jsBridge, and the terminal forwards the request by proxy through OKHTTP, so the whole link goes back to the process of terminal data mock.
The final H5 assistant looks like this:
The business value
The entire link to this data Mock has been analyzed for implementation on Android. This piece does not go into every technical point because of the length, but simply explains the AOP scheme. Welcome interested partners to conduct in-depth communication with me.
conclusion
DoKit has always sought to provide the most convenient and intuitive development experience for developers, and we welcome more people from the community to participate in the construction of DoKit and give us valuable suggestions or PR.
The future of DoKit requires a concerted effort.
Finally, brazen pull a wave of star. Here you are. Click “STAR” before you go. DoKit