This is the 28th day of my participation in Gwen Challenge
This article is part 15 of jakewharton’s series on D8 and R8.
- D8 Library Desugaring
- Originally written by jakewharton
- Translator: Antway
So far in this series of articles, Articles on D8 include Java 8 Language Features deicing, vendor-version-specific bugs for the platform, and methodd-Local optimization for performance. In this article, we’ll cover an upcoming feature in D8 called “Core library desaccharification,” which makes newer apis available on older versions of Android.
Library designs for Java8 apis such as Streams, Optional, and New Time apis were announced at the Google I/O Developer Conference 2019, And announced the first canary release support for Android Studio 4.0 at the 2019 Android DevSummit. This will allow developers to use these features introduced in API 24 and 26 on each version of their application target.
This is also a benefit of the Java library ecosystem. Many libraries have long since moved to Java8, but newer apis are not available to maintain Android compatibility. While not every new API is available, D8 desugaring should allow these libraries to use the API they need most.
1. Not a new feature
Despite the recent hoopla, desugaring the API isn’t actually a new feature of the D8. Since D8 became a usable substitute for dx, it has uncalled the API19 objects.requirenonNULL method. But why this method?
The fixed format of some code causes the Java compiler to synthesize explicit null checks.
class Counter {
final int count = 0;
}
class Main {
void doSomething(Counter counter) {
intcount = counter.count; }}Copy the codeWhen compiled using JKD 8, the doSomething method contains a call to getClass() in its bytecode and returns it directly.
void doSomething(Counter);
Code:
0: aload_1
1: invokevirtual #2 // Method java/lang/Object.getClass:()Ljava/lang/Class;
4: pop
5: iconst_0
6: istore_2
⋮
Copy the codeAs you can see from the bytecode above, the constant 0 is inlined directly into the doSomething method at line 5. So if you pass a null as Counter, you get a null-pointer exception, so you can see that by calling getClass, the program is running properly.
If you recompile this code snippet with JDK 9, the bytecode changes.
void doSomething(Counter);
Code:
0: aload_1
- 1: invokevirtual #2 // Method java/lang/Object.getClass:()Ljava/lang/Class;
+ 1: invokestatic #2 // Method java/util/Objects.requireNonNull:(Ljava/lang/Object;) Ljava/lang/Object;
4: pop
5: iconst_0
6: istore_2
⋮
Copy the codeJdk-8074306 changes the behavior of the Java compiler in this scenario to produce better exceptions. But the Android toolchain doesn’t work properly on JDK9 (and later), so you might be wondering how these calls come about.
The main code is Google’s error-prone compiler and static parser, which works with JDK8 but is built on top of the JDK9 compiler. While error-Prone solves this problem by introducing the off-by-default flag, Retrolambda adds desugaring to the API, which essentially requires D8 to do the same.
Running D8 on Java bytecode (minimum API level less than 19) will call desugaring getClass().
[00016c] Main.doSomething:(LCounter;) V0000: invoke-virtual {v1}, Ljava/lang/Object; .getClass:()Ljava/lang/Class; ⋮Copy the codeObjects.requirenonnull was the only API that D8 was able to design for a long time, and it achieved this through a simple rewrite. But soon, its desugaring capability will have to be expanded to do more.
Java 8 in Kotlin
Unlike the Java compiler, the Kotlin compiler issues references to many apis when generating bytecode for its language features. Data classes are examples of how the compiler generates large amounts of bytecode on your behalf.
data class Duration(val amount: Long.val unit: TimeUnit)
Copy the codeIn version 1.1.60 of Kotlin, when set to compile for Java 8, a Data class hashCode method is referred to some Java 8 APIs.
public int hashCode(a);
Code:
0: aload_0
1: getfield #10 // Field amount:J
4: invokestatic #71 // Method java/lang/Long.hashCode:(J)I
⋮
Copy the codeThe compiler is free to call long.hashcode because we told it that we were targeting Java8.
Typically this is not a problem for Android, as the Kotlin compiler targets Java6 by default. Unfortunately, the community targeted Java8 for its language features, leaving the Kotlin compiler to interact poorly with the decision to specify the target in Kotlin 1.3. As a result, Android developers are starting to see these fledgling NoSuchMethodError exceptions for hashCode calls because they are only available in API 24 and later versions.
Although the behavior of the Kotlin compiler is restored in the Android project, it is still possible for libraries used by the Android project to target Java8 and reference these methods. The D8 team decided to step in and mitigate the problem by decomposing the HashCodeAPI.
Running D8 on Java bytecode (minimum API level less than 24) shows the process of desugaring.
[0003e4] Duration.hashCode:()I
0000: iget-wide v0, v2, LDuration; .amount:J0002: invoke-static {v0, v1}, L$r8$backportedMethods$utility$Long$1$hashCode; I ⋮ hashCode (J)Copy the codeI’m not sure how you want long.hashcode to be desugaring, but I’m guessing it’s not generating a class called $R8 $backportedMethods$Utility $Long$1$hashCode. Unlike objects.requirenonNULL, which was rewritten to getClass() to reduce exceptions, long.hashcode has an implementation that cannot be copied by a simple rewrite.
3. Backporting Methods
In the D8 project, each API has a template implementation that makes it backward compatible.
public final class LongMethods {
public static int hashCode(long l) {
return (int) (l ^ (l >>> 32)); }}Copy the codeThe code for these apis is either written from the Javadoc specification for methods or adapted from libraries such as googleguava. When D8 is built, these templates are automatically transformed into an abstract representation of the method body.
public static CfCode LongMethods_hashCode(a) {
return new CfCode(
/* maxStack = */ 5./* maxLocals = */ 2,
ImmutableList.of(
new CfLoad(ValueType.LONG, 0),
new CfLoad(ValueType.LONG, 0),
new CfConstNumber(32, ValueType.INT),
new CfLogicalBinop(CfLogicalBinop.Opcode.Ushr, NumericType.LONG),
new CfLogicalBinop(CfLogicalBinop.Opcode.Xor, NumericType.LONG),
new CfNumberConversion(NumericType.LONG, NumericType.INT),
new CfReturn(ValueType.INT)));
}
Copy the codeWhen D8 compiles bytecode, it first encounters a call to long.hashcode, which dynamically generates a class using the hashCode method, the body of which is created by calling the factory method. Each long.hashCode call is then overridden to point to the newly generated class.
Class #0 -
Class descriptor : 'L$r8$backportedMethods$utility$Long$1$hashCode; '
Access flags : 0x1401 (PUBLIC ABSTRACT SYNTHETIC)
Superclass : 'Ljava/lang/Object; '
Direct methods -
#0
name : 'hashCode'
type : '(J)I'
access : 0x1009 (PUBLIC STATIC SYNTHETIC)
00044c: |[00044c] $r8$backportedMethods$utility$Long$1$hashCode.hashCode:(J)I
00045c: 1300 2000 |0000: const/16 v0, #int 32
000460: a500 0200 |0002: ushr-long v0, v2, v0
000464: c202 |0004: xor-long/2addr v2, v0
000466: 8423 |0005: long-to-int v3, v2
000468: 0f03 |0006: return v3
Copy the codeThe handling of this process allows the Java8 target data classes to work on Android versions prior to API 24. If you look closely, you might map each Dalvik bytecode back to an abstract representation, and then back to the template source code.
Generating a class for each method may sound excessive, but this ensures that only one implementation of each API needs backporting. When using R8, these composite classes also participate in optimizations, such as method inlining and class merging, ultimately reducing their impact.
D8 can desugar 98 individual apis added to Java7 and Java8 in existing types. But why stop there?
Because it is so easy to add these templates, D8 can also design an additional 58 separate apis from Java9, Java10, and Java11 on top of existing types. This allows the Java library to be targeted at newer versions of Java and still be available on Android.
You can find a full list of apis available for Desugar here. Most of these are already available in AGP3.6.0.
4. Backward compatibility with Types
Java 8 types like Optional, Function, Stream, and LocalDateTime were not added to Android until API 24 and API 26. For a number of reasons, backward compatibility of these methods to ensure that they work at older API levels is more complex than backward compatibility of individual methods.
class Main {
public static void main(String... args) { System.out.println(LocalDateTime.now()); }}Copy the codeLocalDateTime was introduced in API 26 and can only be used directly by apps with the Minimum API 26.
[000240] Main.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System; .out:Ljava/io/PrintStream;0002: invoke-static{}, Ljava/time/LocalDateTime; .now:()Ljava/time/LocalDateTime;0005: move-result-object v0
0006: invoke-virtual {v1, v0}, Ljava/io/PrintStream; .println:(Ljava/lang/Object;) V 0009:return-void
Copy the codeTo enable these types when the minimum API is below 26, Android Gradle Plugin 4.0 or above requires that you enable Core Library desugaring in its DSL.
android {
compileOptions {
coreLibraryDesugaringEnabled true}}Copy the codeRecompilation changes the bytecode to backward compatible types.
[000240] Main.main:([Ljava/lang/String;)V
0000: sget-object v1, Ljava/lang/System; .out:Ljava/io/PrintStream; -0002: invoke-static{}, Ljava/time/LocalDateTime; .now:()Ljava/time/LocalDateTime; +0002: invoke-static{}, Lj$/time/LocalDateTime; .now:()Lj$/time/LocalDateTime;0005: move-result-object v0
0006: invoke-virtual {v1, v0}, Ljava/io/PrintStream; .println:(Ljava/lang/Object;) V 0009:return-void
Copy the codeYou can see that the java.time.localDateTime call is rewritten to j$.time.localDateTime, but the rest of the APK has changed dramatically.
Using the Diffuse Tool, we get an advanced view of the changes.
$ diffuse diff app-min-26.apk app-min-25.apk OLD: app-min-26.apk (signature: V2) NEW: app-min-25.apk (signature: V2) │ compressed │ uncompressed ├ ─ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ─ ─ ─ APK │ old │ new │ diff │ Old │ new │ diff ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ dex 44 KiB 680 B │ │ │ + 43.4 KiB │ arSC │ 524 B │ 520 B │ -4 B │ 384 B │ 384 B │ 0 MANIFEST │ 603 B │ 603 B │ 0 │ 1.2 KiB │ 1.2 KiB │ 0 other │ 229 B │ 229 B │ 0 B │ 95 B │ 95 B │ 0 B ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ the total 2 KiB │ │ 45.4 KiB │ + 43.4 KiB │ KiB 2.6 + 90 KiB 92.6 KiB │ │ │ raw │ unique ├ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ DEX │ old │ new │ diff │ old │ new │ diff ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ the count │ │ │ 2 + 1 │ │ │ strings │ │ │ │ 1005 + 989 16 │ │ + 980 (+ 983-3) 996 types │ │ │ 175 + 7 168 170 │ │ │ 7 + 163 (+ 164-1) classes │ │ │ 88 + 1 87 88 │ │ │ 1 + 87 (+ 87 -2) Methods (methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods, methods,Copy the codeFrom the above conclusion, two conclusions can be drawn:
- our
APKThe size grows43.4 KBThis is entirely due todexThe file caused it. fromdexThere are many new classes, methods, and fields. dexThe number of files increased from one to two, although the total number of methods was nowhere near the limit. These are release versions, so we should get the minimum numberdexFile.
Let’s break each of these down.
4.1 Influence of APK size
Historically, in order to use the Java.time API in applications with a minimum support API level below 26, you needed to use the ThreeTenBP library (or ThreeTenABP). This is a separate repackage of the java.time API in the org.threeten.bp package, requiring all imports to be updated.
D8 performs essentially the same operation, but at the bytecode level. It rewrites the code from calling java.time to j$.time, as shown in bytecode diff above. In order to work with the rewrite, you need to bind the implementation to the application. This is why APK varies greatly.
In this case, the R8 compressed version of APK is used, and R8 also compresses backward compatible code. If compression is disabled, the increase in index size jumps to 180KB, 206 classes, 3272 methods, and 713 fields.
4.2 The second Dex package
The release will cause D8 or R8 to generate the minimum number of dex files required, and this is still the case here. D8 and R8 are responsible for generating dex files for user code and declared libraries. This means that only the primary type will appear in the first dex, which we can confirm by dumping its members.
$ unzip app-min-25.apk classes.dex && \
diffuse members --dex --declared classes.dex
com.example.Main <init>()
com.example.Main main(String[])
Copy the codeWhen D8 or R8 compile code and perform overrides on the J $package, they record the type and API being overridden. This generates a set of shrink rules specific to the backward compatible type. At present (i.e., for AGP4.0.0 – alpha06), these rules in the build/intermediates/desugar_lib_project_keep_rules/release/out / 4, for this case, Contains only localDatetime.now () references.
-keep class j$.time.LocalDateTime {
j$.time.LocalDateTime now(a);
}
Copy the codeAll of the available backward type compatibility processing has been precompiled from OpenJDK source into dex as part of Google’s desugar_jdk_libs. The dex file is downloaded from Google’s Maven repo and then entered into a tool called L8 along with the generated Keep rules. L8 independently shrinks this DEX file using the provided rules to generate the final second DEX file.
The second dex file shrunk by Dumpling L8 shows a set of types and apis that are completely confused except for the localDateTime.now () API that the application is referring to.
$ unzip app-min-25.apk classes2.dex && \
diffuse members --dex classes2.dex | grep -C 6 'LocalDateTime.now'
j$.time.LocalDateTime c(s) → long
j$.time.LocalDateTime compareTo(Object) → int
j$.time.LocalDateTime d(a)- > h j $time. LocalDateTimed(s)- > x j $time. LocalDateTimeequals(Object) → boolean
j$.time.LocalDateTime hashCode(a) → int
j$.time.LocalDateTime now(a)- > LocalDateTime j $time. LocalDateTimetoString(a)- > String j $time. A < init >(k)
j$.time.a a(a)→ k j$.time. A a: k j$.timeb(a)- j $f time. Ac(a) → long
Copy the codeL8 was built specifically to handle this particular DEX file. Prior to this series, R8 was introduced in this article:
… A version of D8 that also performs optimization. It’s not a separate tool or codebase, just the same tool operating in a more advanced mode.
L8 is a version of R8 that optimizes the JDK desugar dex file. It is not a separate tool or code base, just the same tool running in a more advanced mode.
It may not be clear why you need the extra dex explicitly, rather than using the JDK type desugaring like any other library and allowing R8 to handle them as normal. For one thing, Google probably doesn’t want me to talk about it, which in itself should be an indication of why the extra ritual is needed. For more information, you can refer to the OpenJDK source license, especially the latest version. Sorry if that’s not enough information, but I doubt it’s all I can say.
Because you always need at least one second index, you either need to support at least 21 apis or use Legacy Multidex. Most applications should choose the former, or use this feature for another reason, possibly increasing the minimum to 21.
4.3 Backward compatibility methods and types
In addition to backward compatible methods for types that have existed since API1 (such as Long), D8 and R8 will also be compatible with newer methods for these backward compatible types (such as Optional). They use the same templating mechanism as the one detailed earlier, but are only available if your minimum API level is high enough to access the target type, or if you have enabled the core library desugaring.
For Stream and four different optional types, D8 and R8 will back up 18 methods from Java9, 10, and 11.
5. Developer stories
As a developer who wants to write code using these apis, how do you know which APIS are backward compatible? There isn’t a good way to get to know them.
First, with coreLibraryDesugaring enabled, the IDE and Lint will start allowing you to use new types and new apis in support. Running Lint in this case produces no errors, although the minimum API supported is lower than the 26 required for LocalDateTime. However, when the library desugaring is disabled, the NewApi check fails as usual.
Main.java:7: Error: Call requires API level 26 (current min is 25): java.time.LocalDateTime#now [NewApi] System.out.println(LocalDateTime.now()); ~ ~ ~Copy the codeThis ensures that you don’t mistakenly use unsupported types or apis, but it doesn’t help discoverability.
Currently, the best list of backward compatible types is in the Android Studio 4.0 feature list. The lists of compatible apis for existing types are the two lists in this article (1,2). Hopefully, though, these will be easier to spot in the future.
Since the advent of D8 and R8, the backward compatibility of the various apis has been improving. With Android Gradle Plugin 4.0 alphas providing core library desugarization, applications can access base types from Java8, even if their minimum support API level is lower than when they were introduced. This also means that Java libraries can start taking advantage of these types while remaining compatible with Android.
It’s important to remember that even with these shiny new API usability, the JDK and Java apis continue to improve, which is their six-month release pace. While D8 and R8 can help bridge the gap by removing some of the apis from Java9, 10, and 11, the pressure must be maintained to actually ship those apis to the Android framework.