AGP upgrade tour

Lu Kan (XJY2061)

The origin of

Recently, due to the introduction of new tools rely on Android Gradle Plugin (AGP) 4.1 or above version, the project is currently using AGP version 3.5.0, need to upgrade. Considering that some third-party libraries do not yet support the latest AGP version 4.2, we decided to upgrade AGP to 4.1.3, the highest version in 4.1, thus starting the AGP upgrade journey.

According to official documents

The first step, of course, is to read the official Android Gradle plugin version documentation and adapt according to the changes listed in the documentation.

AGP 3.6 adapter

AGP 3.6 introduced the following behavior changes:

By default, native libraries are packaged in uncompressed form

This change enables native library to be packaged in an uncompressed way, which will increase the size of APK and bring limited benefits, and part of the benefits depend on Google Play. If the disadvantages outweigh the benefits after evaluation, you can add the following configuration in Androidmanifest.xml to change the native library to be compressed:

<application
    android:extractNativeLibs="true"
    . >
</application>
Copy the code

AGP 4.0 adapter

AGP 4.0 introduces the following new features:

Dependency metadata

This change will compress and encrypt metadata of application dependencies and store them in APK signature blocks. Google Play will use these dependencies to remind problems, with limited benefits, but will increase the SIZE of APK. For example, if the App is not on Google Play, You can turn this feature off by adding the following configuration to build.gradle:

android {
    dependenciesInfo {
        // Disables dependency metadata when building APKs.
        includeInApk = false
        // Disables dependency metadata when building Android App Bundles.
        includeInBundle = false}}Copy the code

AGP 4.1 adapter

AGP 4.1 introduces the following behavior changes:

Remove the version attribute from the BuildConfig class in the library project

This change removes the VERSION_NAME and VERSION_CODE fields from the Library Module’s BuildConfig class. In general, obtaining the version number in the library module is to obtain the version number of App, and buildconfig. VERSION_NAME and buildConfig. VERSION_CODE in the library module are the version number of the library module itself, and these two fields should not be used at this time. To get the App version number in the library module, use the following code:

private var appVersionName: String = ""
private var appVersionCode: Int = 0

fun getAppVersionName(context: Context): String {
  if (appVersionName.isNotEmpty()) return appVersionName
  return runCatching {
    context.packageManager.getPackageInfo(context.packageName, 0).versionName.also {
      appVersionName = it
    }
  }.getOrDefault("")}fun getAppVersionCode(context: Context): Int {
  if (appVersionCode > 0) return appVersionCode
  return runCatching {
    PackageInfoCompat.getLongVersionCode(
      context.packageManager.getPackageInfo(context.packageName, 0)
    ).toInt().also { appVersionCode = it }
  }.getOrDefault(0)}Copy the code

Problems encountered

Not surprisingly, there are still a number of problems with the adaptation of official documents. These problems are partly caused by behavior changes not explicitly identified in official documents, and partly caused by non-standard practices matching the stricter restrictions of the new AGP. The following are the manifestations, causes and solutions of these problems.

BuildConfig.APPLICATION_IDCan’t find

The buildConfig. APPLICATION_ID field is used in some of our component library modules, and an Unresolved Reference error was detected during compilation.

The value of the buildConfig. APPLICATION_ID field in the library module is the package name of the library module, not the package name of the application, so this field is deprecated as of AGP 3.5 and replaced with the LIBRARY_PACKAGE_NAME field. It was completely removed from AGP 4.0.

We originally used APPLICATION_ID to obtain the App package name for some of the code in the App module. In the later componentized splitting process, when the code in the App module is extracted to the component library, in order to avoid using the package name of the library module as the App package name, we should change the method of obtaining the App package name synchronously. However, it was omitted and no modification was made, resulting in the failure of compiling after AGP upgrade.

To solve this problem, change the library module to use context.getPackagename () to get the App package name.

R and ProGuard Mapping files could not be found

We will back up R and ProGuard Mapping files generated during the build of the release package for later use, but upgrading the backup failed.

This is because starting with AGP 3.6, the path of the two files in the build product changes:

• R.txt:build/intermediates/symbols/${variant.dirName}/R.txt -> build/intermediates/runtime_symbol_list/${variant.name}/R.txt
• mapping.txt:build/outputs/mapping/${variant.dirName}/mapping.txt -> build/outputs/mapping/${variant.name}/mapping.txt

${variable. dirName} = $flavor/$buildType (full/release); ${variable. name} is $flavor${buildType.capitalize()} (for example, fullRelease).

To solve this problem, change the file path in the backup logic to the above new path as follows:

afterEvaluate {
    android.applicationVariants.all { variant ->
        def variantName = variant.name
        def variantCapName = variant.name.capitalize()

        def assembleTask = tasks.findByName("assemble${variantCapName}")
        assembleTask.doLast {
            copy {
                from "${buildDir}/outputs/mapping/${variantName}/mapping.txt"
                from "${buildDir}/intermediates/runtime_symbol_list/${variantName}/R.txt"
                into backPath
            }
        }
    }
}
Copy the code

The fixed resource ID is invalid

In order to avoid the crash caused by the false resource file being found through the resource ID when RemoteView (inflate notification) may occur after the installation of App upgrade overrides, we have fixed the resource ID during the build so that the IDS of some resource files remain unchanged between multiple builds. The resource ids are changed after the upgrade.

The original fixed resource id was implemented afterEvaluate, Use the tasks. The findByName method to obtain the process ${variant. The name. Capitalize ()} Resouces objects (such as processFullReleaseResources) tasks, Then, before AGP 3.5, use the getAaptOptions method. In AGP 3.5, use reflection to obtain aaptOptions attribute objects in task objects. Then add the — stables -ids parameter and the corresponding resource ID profile path value to its additionalParameters property object. However, in AGP 4.1, the processing resource task class no longer has aaptOptions attributes, resulting in fixed failure.

4.1 for AGP, directly set can be replaced with the following android. AaptOptions. AdditionalParameters way of fixed resource id:

afterEvaluate {
    def additionalParams = android.aaptOptions.additionalParameters
    if (additionalParams == null) {
        additionalParams = new ArrayList<>()
        android.aaptOptions.additionalParameters = additionalParams
    }
    def index = additionalParams.indexOf("--stable-ids")
    if (index > - 1) {
        additionalParams.removeAt(index)
        additionalParams.removeAt(index)
    }
    additionalParams.add("--stable-ids")
    additionalParams.add("${your stable ids file path}")}Copy the code

Failed to modify the Manifest file. Procedure

We will modify the androidmanifest.xml file during the build process to add additional information, but the modification fails after the upgrade.

After analyzing the AGP build logs of each version included in this upgrade, AGP 4.1 is found to have added process${varie.name.capitalize ()}ManifestForPackage to the Manifest processing (e.g ProcessFullReleaseManifestForPackage) tasks, The task in the original Manifest processing task process ${variant. The name. Capitalize ()} the Manifest (such as processFullReleaseManifest) after execution, the product is different from the original task 1. The original way to add additional information to the Manifest is after the original Manifest processing task is executed, To perform a custom Manifest processing tasks cmProcess ${variant. The name. Capitalize ()} the Manifest (for example cmProcessFullReleaseManifest), Writes information to the product 2 of the original Manifest processing task. After the upgrade, if two tasks that process the Manifest hit the CACHE (execution status is from-cache), the additional information in the Manifest file in the APK will be the old information written in the previous compilation.

Therefore, the way to write information should be as shown in the figure below, instead writing information to its output file after the new Manifest processing task executes.

The Transform plug-in failed to execute

We added some Transform plugins during the build process. After the upgrade, one of the plugins using ASM for code piling encountered the following error:

Execution failed for task ':app:transformClassesWithxxx'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)
Copy the code

Error when the above can also be Java lang. IllegalArgumentException: Invalid opcode, 169.

To find the exact source of the exception, add the — StackTrace parameter rebuild to locate the exception triggered by Hunter, the third-party library introduced in the plug-in. AGP 3.5 uses ASM 6, and AGP 3.6 uses ASM 7. There is a defect in ASM 7, which causes an exception after the upgrade.

Considering that Hunter only encapsulated the Transform using ASM, and the function implemented by this plug-in was relatively simple, the problem was solved by removing Hunter and re-implementing it.

Cannot change dependencies of dependency configuration

. We used the resolutionStrategy dependencySubstitution switch to implement component library source code, after the upgrade, if cut component library into the source code, click the Run button in the Android Studio build error occurs such as.

Gradlew assembleRelease builds an assembleRelease file. Gradlew assembleRelease builds an assembleRelease file. The only difference between an Android Studio Run build and a command-line build is the addition of a module prefix (:app:assembleRelease) to the tasks you perform. Starting from this distinction, finally found the cause of the problem is in gradle. Open the properties of the org. Gradle. Configureondemand features in the incubation, make gradle configuration tasks related to request the project only, The project source code is not configured when the task is executed in the specified module mode.

Close the org. Gradle. Configureondemand character can solve this problem.

Entry name ‘xxx’ collided

Build after upgrade, an error will occur when executing the package task package${varie.name.capitalize ()} (for example packageFullRelease).

AGP 3.6 introduces the following new features:

New default packaging tool

This feature uses the new packaging tool Zipflinger to build APK when building debug and, starting with AGP 4.1, when building Release.

The error occurred in the task of packaging and generating APK, and is easily associated with the new functionality described above. Use the official documentation provided by adding Android.usenewapkCreator =false to the gradle.properties file to restore the configuration after using the old packaging tool, you can build successfully. However, Java resource files are missing from the generated APK, causing various runtime problems (e.g., OkHttp is missing publicSuffixes.

There are two directions to solve the problem: solving missing Java resource files and solving problems such as build errors. To solve these problems, it is necessary to analyze the causes of the problems first. By debugging the AGP construction process and analyzing the AGP source code, it is found that the implementation class corresponding to the packaging task is PackageApplication, and the main implementation logic is in the parent class PackageAndroidArtifact. The call to write the Android and Java resource files to the APK file is shown below:

The updateSingleEntryJars method writes to the Asset file, and the addFiles method writes to other Android and Java resource files. Use ApkFlinger if it is true, or ApkZFileCreator if it is not. Android. useNewApkCreator The default value is true.

If the old packaging tool ApkZFileCreator is configured, it will use ZFile to read file 3 generated after resource reduction and file 4 generated after obturation, and write the Android and Java resource files in it to the APK file.

The following source code snippet shows the main logic for writing in three steps:

1. createZFileObject that reads the zip file to add each entry in the central Directory toentries 中
2. traverseZFileIn theentriesTo merge compressed resource files into APK files
3. traverseZFileIn theentries, writes uncompressed resource files to APK files

// ApkZFileCreator.java
public void writeZip(
    File zip, @Nullable Function<String, String> transform, @Nullable Predicate<String> isIgnored)
    throws IOException {
  // ...
  try {
    ZFile toMerge = closer.register(ZFile.openReadWrite(zip));
    // ...
    Predicate<String> noMergePredicate =
        v -> ignorePredicate.apply(v) || noCompressPredicate.apply(v);

    this.zip.mergeFrom(toMerge, noMergePredicate);

    for (StoredEntry toMergeEntry : toMerge.entries()) {
      String path = toMergeEntry.getCentralDirectoryHeader().getName();
      if(noCompressPredicate.apply(path) && ! ignorePredicate.apply(path)) {// ...
        try (InputStream ignoredData = toMergeEntry.open()) {
          this.zip.add(path, ignoredData, false); }}}}catch (Throwable t) {
    throw closer.rethrow(t);
  } finally{ closer.close(); }}// ZFile.java
private void readData(a) throws IOException {
  // ...
  readEocd();
  readCentralDirectory();
  // ...
  if(directoryEntry ! =null) {
    // ...
    for (StoredEntry entry : directory.getEntries().values()) {
      // ...
      entries.put(entry.getCentralDirectoryHeader().getName(), mapEntry);
      / /...
    }

    directoryStartOffset = directoryEntry.getStart();
  } else {
    // ...
  }
  // ...
}

public void mergeFrom(ZFile src, Predicate<String> ignoreFilter) throws IOException {
  // ...
  for (StoredEntry fromEntry : src.entries()) {
    if (ignoreFilter.apply(fromEntry.getCentralDirectoryHeader().getName())) {
      continue;
    }
    // ...}}Copy the code

During debugging, I found that there were no Java resource files in entries of ZFile created by reading minified. Jar file. In the front IncrementalSplitterRunnable. In the execute adjustable PackageAndroidArtifact. GetChangedJavaResources change Java resource file, Java resource files can be read using ZipCentralDirectory, indicating that ZFile is defective.

The above problem of missing Java resource files occurred when R8 was turned off. The subsequent R8 test was normal, and the new Demo project test was normal regardless of whether R8 was turned on or not. Therefore, the following conclusions can be drawn:

• Such asZFileAs noted in the comments, it is not a generic ZIP utility class and has strict requirements for the ZIP format and unsupported features; It is limited in some special conditions and may have problems such as missing files to read
• Due to the old packaging tool usedZFileThe generated APk may have problems such as missing Java resource files, which have been officially discarded and should not be used

Now the direction of solving the problem is back to solving the build error of the problem. The new packaging tool ApkFlinger calls the Android or Java resource file as shown in the following figure:

As you can see from ZipArchive. WriteSource, validateName checks the validity of an entry name written to it. If the same name already exists in the central Directory of the current ZIP file, An IllegalStateException is thrown with an error message.

// ZipArchive.java
private void writeSource(@NonNull Source source) throws IOException {
    // ...
    validateName(source);
    // ...
}

private void validateName(@NonNull Source source) {
    byte[] nameBytes = source.getNameBytes();
    String name = source.getName();
    if (nameBytes.length > Ints.USHRT_MAX) {
        throw new IllegalStateException(
                String.format("Name '%s' is more than %d bytes", name, Ints.USHRT_MAX));
    }

    if (cd.contains(name)) {
        throw new IllegalStateException(String.format("Entry name '%s' collided", name)); }}Copy the code

From the source code and debugging results, the cause of the error is generally caused by some non-standard practices that cause the jar file to have the same name as the Android resource file. The two cases we encountered are as follows:

• A third-party library has an asset file in its AAR and the same asset file in its classes.jar
• A third-party library relies on another third-party library’s AAR file as a normal JAR file, resulting in its classes.jarAndroidManifest.xmlfile

After knowing the cause of the problem, run runkjavares. Jar or minified. Jar to locate the corresponding file in the project according to the information in the file (such as package name in Androidmanifest.xml). Make corresponding modification again can.

So file does not have strip

After the upgrade, the so file in the APK generated by the build does not have strip. Use the NM tool 5 in NDK (you can also use the system’s own NM in macOS) to check, and find that the symbol table and debugging information still exist.

After analyzing the build log, it was found that strip${vari.name.capitalize ()}Symbols (such as stripFullReleaseSymbols) had been executed. Then I analyzed the AGP source code and debug the build process. Found the task through StripDebugSymbolsRunnable to strip, so can be seen from the following source code snippets for its main logic:

1. adjustableSymbolStripExecutableFinder.stripToolExecutableFileObtain the strip tool path in NDK
2. If no tool is found, copy so directly to the target location and return
3. Call this tool to strip so and output it to the target location

private class StripDebugSymbolsRunnable @Inject constructor(val params: Params): Runnable {

    override fun run(a) {
        // ...
        val exe =
            params.stripToolFinder.stripToolExecutableFile(params.input, params.abi) {
                UnstrippedLibs.add(params.input.name)
                logger.verbose("$it Packaging it as is.")
                return@stripToolExecutableFile null
            }

        if (exe == null || params.justCopyInput) {
            // ...
            FileUtils.copyFile(params.input, params.output)
            return
        }

        val builder = ProcessInfoBuilder()
        builder.setExecutable(exe)
        // ...
        val result =
            params.processExecutor.execute(
                builder.createProcess(), LoggedProcessOutputHandler(logger)
            )
        // ...
    }
    // ...
}
Copy the code

Therefore, the reason so was not stripped is that the STRIP tool was not found in the NDK. Further analysis of the source code shows SymbolStripExecutableFinder through NdkHandler provides information of the NDK find strip tool path, And NdkHandler through NdkLocator. FindNdkPathImpl function to find the NDK path on top of this and so so can be strip ultimately depends on finding the NDK path. The main logic for searching NDK is as follows:

const val ANDROID_GRADLE_PLUGIN_FIXED_DEFAULT_NDK_VERSION = "21.1.6352462"

private fun findNdkPathImpl(
    userSettings: NdkLocatorKey,
    getNdkSourceProperties: (File) - >SdkSourceProperties? , sdkHandler:SdkHandler?).: NdkLocatorRecord? {
    with(userSettings) {
        // ...
        valrevisionFromNdkVersion = parseRevision(getNdkVersionOrDefault(ndkVersionFromDsl)) ? :return null
        
        // If android.ndkPath value is present then use it.
        if(! ndkPathFromDsl.isNullOrBlank()) {// ...
        }

        // If ndk.dir value is present then use it.
        if(! ndkDirProperty.isNullOrBlank()) {// ...
        }
        // ...
        if(sdkFolder ! =null) {
            // If a folder exists under $SDK/ndk/$ndkVersion then use it.
            val versionedNdkPath = File(File(sdkFolder, FD_NDK_SIDE_BY_SIDE), "$revisionFromNdkVersion")
            val sideBySideRevision = getNdkFolderRevision(versionedNdkPath)
            if(sideBySideRevision ! =null) {
                return NdkLocatorRecord(versionedNdkPath, sideBySideRevision)
            }

            // If $SDK/ndk-bundle exists and matches the requested version then use it.
            val ndkBundlePath = File(sdkFolder, FD_NDK)
            val bundleRevision = getNdkFolderRevision(ndkBundlePath)
            if(bundleRevision ! =null && bundleRevision == revisionFromNdkVersion) {
                return NdkLocatorRecord(ndkBundlePath, bundleRevision)
            }
        }
        // ...}}private fun getNdkVersionOrDefault(ndkVersionFromDsl : String?). =
    if (ndkVersionFromDsl.isNullOrBlank()) {
        // ...
        ANDROID_GRADLE_PLUGIN_FIXED_DEFAULT_NDK_VERSION
    } else {
        ndkVersionFromDsl
    }
Copy the code

The main search process corresponding to the above source snippet is shown below:

Android. ndkPath and Android. ndkVersion are not configured in build.gradle. Properties file does not exist on the packaging machine, so the ndk.dir property does not exist. The NDK version installed on the packaging machine is not the default version 21.1.6352462 specified by AGP, so the NDK path cannot be found.

While the reason has been found, one question remains: why does the strip work properly before the upgrade? To find the answer, let’s look at the way AGP 3.5 looks for NDK. The main logic is as follows:

private fun findNdkPathImpl(
    ndkDirProperty: String? , androidNdkHomeEnvironmentVariable:String? , sdkFolder:File? , ndkVersionFromDsl:String? , getNdkVersionedFolderNames: (File) - >List<String>,
    getNdkSourceProperties: (File) - >SdkSourceProperties?).: File? {
    // ...
    val foundLocations = mutableListOf<Location>()
    if(ndkDirProperty ! =null) {
        foundLocations += Location(NDK_DIR_LOCATION, File(ndkDirProperty))
    }
    if(androidNdkHomeEnvironmentVariable ! =null) {
        foundLocations += Location(
            ANDROID_NDK_HOME_LOCATION,
            File(androidNdkHomeEnvironmentVariable)
        )
    }
    if(sdkFolder ! =null) {
        foundLocations += Location(NDK_BUNDLE_FOLDER_LOCATION, File(sdkFolder, FD_NDK))
    }
    // ...
    if(sdkFolder ! =null) {
        val versionRoot = File(sdkFolder, FD_NDK_SIDE_BY_SIDE)
        foundLocations += getNdkVersionedFolderNames(versionRoot)
            .map { version ->
                Location(
                    NDK_VERSIONED_FOLDER_LOCATION,
                    File(versionRoot, version)
                )
            }
    }
    // ...
    val versionedLocations = foundLocations
        .mapNotNull { location ->
            // ...
        }
        .sortedWith(compareBy({ -it.first.type.ordinal }, { it.second.revision }))
        .asReversed()
    // ...
    val highest = versionedLocations.firstOrNull()

    if (highest == null) {
        // ...
        return null
    }
    // ...
    if(ndkVersionFromDslRevision ! =null) {
        // If the user specified ndk.dir then it must be used. It must also match the version
        // supplied in build.gradle.
        if(ndkDirProperty ! =null) {
            val ndkDirLocation = versionedLocations.find { (location, _) ->
                location.type == NDK_DIR_LOCATION
            }
            if (ndkDirLocation == null) {
                // ...
            } else {
                val (location, version) = ndkDirLocation
                // ...
                return location.ndkRoot
            }
        }

        // If not ndk.dir then take the version that matches the requested NDK version
        val matchingLocations = versionedLocations
            .filter { (_, sourceProperties) ->
                isAcceptableNdkVersion(sourceProperties.revision, ndkVersionFromDslRevision)
            }
            .toList()
        
        if (matchingLocations.isEmpty()) {
            // ...
            return highest.first.ndkRoot
        }
        // ...
        val foundNdkRoot = matchingLocations.first().first.ndkRoot
        // ...
        return foundNdkRoot
    } else {
        // If the user specified ndk.dir then it must be used.
        if(ndkDirProperty ! =null) {
            val ndkDirLocation =
                versionedLocations.find { (location, _) ->
                    location.type == NDK_DIR_LOCATION
                }
            // ...
            val (location, version) = ndkDirLocation
            // ...
            return location.ndkRoot
        }
        // ...
        return highest.first.ndkRoot
    }
}
Copy the code

The corresponding general process is shown in the figure below:

As you can see in AGP 3.5, if the NDK path and version are not configured, the highest version in the NDK directory will be found as long as there is a version in the NDK directory, so there is no problem before the upgrade. AGP 3.6 and 4.0 have the same search logic as AGP 3.5, but add the built-in AGP default version logic if android.ndkVersion is not configured. AGP 3.6 has the default version 20.0.5594570. The default AGP 4.0 version is 21.0.6113669.

Through the above analysis to find the cause of the problem, the solution is ready to come out, in order to have a wider adaptability, you can configure android. NdkVersion to set the NDK version to be consistent with the baler to solve the problem.

summary

This article describes the AGP upgrade process (3.5 to 4.1) and provides the cause analysis and solutions to the problems encountered. While this upgrade was not intended to optimize builds, we were able to speed up builds by about 36% and reduce package sizes by about 5M. Hopefully this article will help readers who need to upgrade successfully complete the upgrade and enjoy the results of the official continuous optimization of the build tool.

At this point, the AGP upgrade journey has come to an end, and our development journey will continue.

The resources

• Android Gradle plugin version description
• Configuration on demand
• AGP source

This article is published from netease Cloud Music big front end team, the article is prohibited to be reproduced in any form without authorization. Grp.music – Fe (at) Corp.Netease.com We recruit front-end, iOS and Android all year long. If you are ready to change your job and you like cloud music, join us!