Deeper understanding of Transform

preface

In fact, the Transform API plays a very important role in the packaging process of an Android project, such as the well-known obfuscating processing, class file to dex file processing, are completed through the Transform API. This article mainly focuses on Transform:

1. Transform APIThe use and principle of
2. Bytecode processing frameworkASMUse skills
3. Transform APIUse groping in application engineering

The use and principle of Transform

What is the Transform

Since version 1.5.0-Beta1, the Android Gradle plugin includes a Transform API, which allows third-party plug-ins to process compiled class files before they are converted into dex files. Using the Transform API, we can completely ignore the process of generating and executing tasks. It allows us to focus on how to process the input class file

The use of the Transform

The registration and use of Transform is straightforward, and within our custom plugin, We can through the android. RegisterTransform (theTransform) or android. RegisterTransform (theTransform, dependencies). You can register.

class DemoPlugin: Plugin<Project> {
    override fun apply(target: Project) {
        val android = target.extensions.findByType(BaseExtension::class.java)android? .registerTransform(DemoTransform()) } }Copy the code

And we are a custom Transform inheritance in com. Android. Build. API. The Transform. Transform, we can see the javaDoc, the following code is relatively common Transform process template

class DemoTransform: Transform() {
    /** * transform name */
    override fun getName(a): String = "DemoTransform"

    There are two types of input files we can work with: compiled Java code and resource files (not files under RES but resources within Assests) */
    override fun getInputTypes(a): MutableSet<QualifiedContent.ContentType> = TransformManager.CONTENT_CLASS

    /** * Whether delta is supported * If delta execution is supported, the change input may contain a list of modified/deleted/added files */
    override fun isIncremental(a): Boolean = false

    /** * specifies the scope */
    override fun getScopes(a): MutableSet<in QualifiedContent.Scope> = TransformManager.SCOPE_FULL_PROJECT

    /** * execute the main function of the transform */
    override fun transform(transformInvocation: TransformInvocation?).{ transformInvocation? .inputs? .forEach {// Input source is folder type
          it.directoryInputs.forEach {directoryInput->
              with(directoryInput){
                  // TODO performs bytecode operations on folders
                  val dest = transformInvocation.outputProvider.getContentLocation(
                      name,
                      contentTypes,
                      scopes,
                      Format.DIRECTORY
                  )
                  file.copyTo(dest)
              }
          }

          // Input source is jar package type
          it.jarInputs.forEach { jarInput->
              with(jarInput){
                  // TODO handles Jar files
                  val dest = transformInvocation.outputProvider.getContentLocation(
                      name,
                      contentTypes,
                      scopes,
                      Format.JAR
                  )
                  file.copyTo(dest)
              }
          }
      }
    }
}
Copy the code

Each Transform declares its scope, what it’s working on, what it does, and what it outputs.

scope

The transform #getScopes method lets you declare a custom transform scope. The scopes specified include the following

Function object

Transform#getInputTypes allows us to declare its objects. There are only two types of objects that we can specify

SCOPE_FULL_PROJECT: transformManager.scope_full_project: transformManager.scope_full_project: transformManager.scope_full_project If it is a library-registered transform, we can only specify transformManager.project_only, We can crash at LibraryTaskManager# createTasksForVariantScope see restrictions related to an error code

            Sets.SetView<? super Scope> difference =
                    Sets.difference(transform.getScopes(), TransformManager.PROJECT_ONLY);
            if(! difference.isEmpty()) { String scopes = difference.toString(); globalScope .getAndroidBuilder() .getIssueReporter() .reportError( Type.GENERIC,new EvalIssueException(
                                        String.format(
                                                "Transforms with scopes '%s' cannot be applied to library projects.",
                                                scopes)));
            }
Copy the code

The main object we use is transformManager.content_class

TransformInvocation

We handle our intermediate transformation by implementing the Transform#transform method, and the intermediate information is passed through the TransformInvocation object

public interface TransformInvocation {

    /** * context of the transform */
    @NonNull
    Context getContext(a);

    /** * returns the input source of the transform */
    @NonNull
    Collection<TransformInput> getInputs(a);

    /** * returns the referenced input source */
    @NonNull Collection<TransformInput> getReferencedInputs(a);
    /** * Additional input source */
    @NonNull Collection<SecondaryInput> getSecondaryInputs(a);

    /**
     * 输出源
     */
    @Nullable
    TransformOutputProvider getOutputProvider(a);


    /** * whether to increment */
    boolean isIncremental(a);
}
Copy the code

In terms of input sources, we can roughly divide them into consumption and reference sources and additional input sources

1. A consumerThat’s the kind of object that we need to do the transform operation, and that’s the kind of object that after we process it we have to specify that the output goes to the next level, and we’re going to go throughgetInputs()Gets the input source to consume, and after the transformation, we must also pass the SettingsgetInputTypes()andgetScopes()To specify the output source to be transmitted to the next transform.
2. The reference input source means that we do not perform the transform operation, but may be used when viewing, so we do not need to output to the next level, through overwritegetReferencedScopes()After specifying the scope of our referential input source, we can passTransformInvocation#getReferencedInputs()Gets the referenced input source
3. In addition, we can also define additional input sources for use by the next level, which we rarely use in normal development, but likeProGuardTransformTXT to the next level; The same likeDexMergerTransformIf it is openedmultiDexThe maindexlist. TXT file will be passed to the next level

The principle of the Transform

The execution chain of Transform

Now that we have a rough idea of how it works, let’s see how it works (this source code is based on gradle plugin version 3.3.2). The com.android.application and com.android.library plug-ins of Android inherit from BasePlugin, and their main execution sequence can be divided into three steps

1. The project configuration
2. The extension of the configuration
3. The task of creating

Inside the BaseExtension maintains a transforms collection objects, the android. RegisterTransform (theTransform) is in fact our custom transform instance added to the list of objects. In the 3.3.2 source code, it can also be understood in this way. In BasePlugin#createAndroidTasks, we create the associated build tasks for each variant through VariantManager#createAndroidTasks, Finally through TaskManager# createTasksForVariantScope (application plug-in finally realization method in TaskManager# createPostCompilationTasks, And library plug-in finally realization method in LibraryTaskManager# createTasksForVariantScope) method to obtain BaseExtension maintenance transforms in the object, Transform the corresponding Transform object into a Task via TransformManager#addTransform, registered in the TaskFactory. Here, about the execution process of a series of Transform tasks, we can choose to look at the relevant Transform process in application. Due to the length, we can look at the relevant source code by ourselves. Here’s the transform The task flow is from Desugar->MergeJavaRes-> custom Transform ->MergeClasses->Shrinker ->MultiD Ex ->BundleMultiDex->Dex->ResourcesShrinker->DexSplitter.

TransformManager

The TransformManager manages all the transforms of the project. It maintains streams, a TransformStream collection, inside the TransformManager. Each new Transform consumes the corresponding stream. Then add the processed streams to Streams

public class TransformManager extends FilterableStreamCollection{
    private final List<TransformStream> streams = Lists.newArrayList();
}
Copy the code

We can look at its core method, addTransform

@NonNull
    public <T extends Transform> Optional<TaskProvider<TransformTask>> addTransform(
            @NonNull TaskFactory taskFactory,
            @NonNull TransformVariantScope scope,
            @NonNull T transform,
            @Nullable PreConfigAction preConfigAction,
            @Nullable TaskConfigAction<TransformTask> configAction,
            @Nullable TaskProviderCallback<TransformTask> providerCallback) {

        ...

        List<TransformStream> inputStreams = Lists.newArrayList();
        // The transform task's naming rule definition
        String taskName = scope.getTaskName(getTaskNamePrefix(transform));

        // Get the reference stream
        List<TransformStream> referencedStreams = grabReferencedStreams(transform);

        // Find the input stream and calculate the output stream through the transform
        IntermediateStream outputStream = findTransformStreams(
                transform,
                scope,
                inputStreams,
                taskName,
                scope.getGlobalScope().getBuildDir());

        // The ellipsis code is used to check whether the input stream and reference stream are empty. Theoretically, they cannot be empty. If they are empty, then there is a problem with the transform. transforms.add(transform);// Create the transform task
        return Optional.of(
                taskFactory.register(
                        new TransformTask.CreationAction<>(
                                scope.getFullVariantName(),
                                taskName,
                                transform,
                                inputStreams,
                                referencedStreams,
                                outputStream,
                                recorder),
                        preConfigAction,
                        configAction,
                        providerCallback));
    }
Copy the code

Add a Transform management to TransformManager, and the process can be divided into the following steps

1. Define the transform Task name

static String getTaskNamePrefix(@NonNull Transform transform) {
        StringBuilder sb = new StringBuilder(100);
        sb.append("transform");

        sb.append(
                transform
                        .getInputTypes()
                        .stream()
                        .map(
                                inputType ->
                                        CaseFormat.UPPER_UNDERSCORE.to(
                                                CaseFormat.UPPER_CAMEL, inputType.name()))
                        .sorted() // Keep the order stable.
                        .collect(Collectors.joining("And")));
        sb.append("With");
        StringHelper.appendCapitalized(sb, transform.getName());
        sb.append("For");

        return sb.toString();
    }
Copy the code

From the code above, Transform ${inputtype1. name}And${inputtype2. name}With${transform.name}For${variantName} The corresponding transform Task can also be verified by the generated transform Task

streams

private List<TransformStream> grabReferencedStreams(@NonNull Transform transform) {
        Set<? superScope> requestedScopes = transform.getReferencedScopes(); . List<TransformStream> streamMatches = Lists.newArrayListWithExpectedSize(streams.size()); Set<ContentType> requestedTypes = transform.getInputTypes();for (TransformStream stream : streams) {
            Set<ContentType> availableTypes = stream.getContentTypes();
            Set<? super Scope> availableScopes = stream.getScopes();

            Set<ContentType> commonTypes = Sets.intersection(requestedTypes,
                    availableTypes);
            Set<? super Scope> commonScopes = Sets.intersection(requestedScopes, availableScopes);

            if(! commonTypes.isEmpty() && ! commonScopes.isEmpty()) { streamMatches.add(stream); }}return streamMatches;
    }
Copy the code

3. According to SCOPE and INPUT_TYPE defined in transform, obtain the corresponding consumptive input streams, remove these consumptive input streams in Streams, and retain the streams that cannot match SCOPE and INPUT_TYPE. Build a new output stream and add it to Streams for management

private IntermediateStream findTransformStreams( @NonNull Transform transform, @NonNull TransformVariantScope scope, @NonNull List
       
         inputStreams, @NonNull String taskName, @NonNull File buildDir)
        {

        Set<? superScope> requestedScopes = transform.getScopes(); . Set<ContentType> requestedTypes = transform.getInputTypes();// Get the consumptive input stream
        // Remove the corresponding consumption input streams from Streams
        consumeStreams(requestedScopes, requestedTypes, inputStreams);

        // Create an output stream
        Set<ContentType> outputTypes = transform.getOutputTypes();
        // Create a file-related path for the output stream transformation
        File outRootFolder =
                FileUtils.join(
                        buildDir,
                        StringHelper.toStrings(
                                AndroidProject.FD_INTERMEDIATES,
                                FD_TRANSFORMS,
                                transform.getName(),
                                scope.getDirectorySegments()));

        // Create an output stream
        IntermediateStream outputStream =
                IntermediateStream.builder(
                                project,
                                transform.getName() + "-" + scope.getFullVariantName(),
                                taskName)
                        .addContentTypes(outputTypes)
                        .addScopes(requestedScopes)
                        .setRootLocation(outRootFolder)
                        .build();
        streams.add(outputStream);

        return outputStream;
    }
Copy the code

4. Finally, create the TransformTask and register it with TaskManager

TransformTask

How do WE trigger our Transform#transform method to be executed in the TaskAction corresponding to the TransformTask

void transform(final IncrementalTaskInputs incrementalTaskInputs)
            throws IOException, TransformException, InterruptedException {

        final ReferenceHolder<List<TransformInput>> consumedInputs = ReferenceHolder.empty();
        final ReferenceHolder<List<TransformInput>> referencedInputs = ReferenceHolder.empty();
        final ReferenceHolder<Boolean> isIncremental = ReferenceHolder.empty();
        final ReferenceHolder<Collection<SecondaryInput>> changedSecondaryInputs =
                ReferenceHolder.empty();

        isIncremental.setValue(transform.isIncremental() && incrementalTaskInputs.isIncremental());

        GradleTransformExecution preExecutionInfo =
                GradleTransformExecution.newBuilder()
                        .setType(AnalyticsUtil.getTransformType(transform.getClass()).getNumber())
                        .setIsIncremental(isIncremental.getValue())
                        .build();

        // Some incremental processing, including determining changes in input streams (referential and consumable) in incremental mode. GradleTransformExecution executionInfo = preExecutionInfo.toBuilder().setIsIncremental(isIncremental.getValue()).build(); . transform.transform(new TransformInvocationBuilder(TransformTask.this) .addInputs(consumedInputs.getValue()) .addReferencedInputs(referencedInputs.getValue()) .addSecondaryInputs(changedSecondaryInputs.getValue()) .addOutputProvider( outputStream ! =null
                                                        ? outputStream.asOutput(
                                                                isIncremental.getValue())
                                                        : null)
                                        .setIncrementalMode(isIncremental.getValue())
                                        .build());

                        if(outputStream ! =null) { outputStream.save(); }}Copy the code

We now know the timing, location, and principles of the custom Transform execution. So, now that we have all the compiled bytecode, what do we do with it? So we can look at ASM

The use of the ASM

Common frameworks for handling bytecode include AspectJ, Javasist, and ASM. There are many articles about the selection of the framework on the Internet. From the processing speed and memory occupancy rate, ASM is obviously better than the other two frameworks. This article focuses on the use of ASM.

What is the ASM

ASM is a general-purpose Java bytecode manipulation and analysis framework. It can be used to modify existing classes or to generate classes dynamically directly in binary form. ASM provides some common bytecode conversion and analysis algorithms from which you can build custom complex conversion and code analysis tools. The ASM library provides two apis for generating and transforming compiled classes: the Core API provides event-based class representations, and the Tree API provides object-based representations. Because the event-based API(Core API) does not require a number of objects representing the class to be stored in memory, it is superior to the object-based API(Tree API) in terms of execution speed and memory footprint. However, event-based apis are more difficult to use than object-based apis in terms of usage scenarios, such as when we need to adjust for an object. Since a class can only be managed by one API, we should use the API for each scenario

ASM plug-in

The use of ASM requires a certain learning cost, which can be learned by using ASM Bytecode Outline plug-in. The corresponding plug-in can be found in the plug-in browser of AS

@RouteModule
public class ASMTest {}Copy the code

Transform APIGroping for use in application engineering

The role in component communication

Transform the API has many applications in modular engineering direction, at present in our project in the development of routing framework, through its to do the automatic static registration module, at the same time, considering the uncertainty routing through maintenance agreement document (page routing address maintenance is not timely lead to corresponding development unable to update the corresponding code), We did constant management of routing. Firstly, we collected routing information by scanning the code of the whole project and established the original basic information file of routing in accordance with certain rules. Registered by corresponding to the original information through variant# registerJavaGeneratingTask file generated constants corresponding Java file sinking in the basic task of common component, so that the upper relies on the basic components of the project can use routing through direct call constants. In the case of code isolation of each component, the original information file can be transmitted by the aar of the component, and the corresponding constant table can still be generated by following the above steps, while the problem of class duplication exists, and the combination can be processed by custom Transform

The role of service monitoring

In application engineering, we usually have requirements for network monitoring and application performance detection (including page load time and even the time spent on each method call, which may be warned if the threshold is exceeded), which cannot be embedded in business code and can be handled based on the Transform API. As for burying points, we can also realize the function of automatic burying points through Transform, and transfer as much field information as possible through ASM Core and ASM Tree. Some of these have been realized in our project, and some need to be optimized or realized.

other

Transform Core Api and ASM Tree Api: Transform Core Api and ASM Tree Api

Relevant reference

• ASM User Guide
• Play with bytecode in Android projects
• AOP tools: INTRODUCTION to ASM 3.0
• ASM website