Author: Liu Tianyu (Modest Wind)

Engineering corruption is a very difficult problem in app iteration. It involves extensive and detailed details and has a relatively “hidden” and indirect impact on R&D efficiency & experience, engineering & product quality, stability, package size and performance. Generally not cause unsustainable obstacles, jumped out but often lead to “pain”, a bit like cavities or wisdom teeth, at some point not not pull, but the difference is that the engineering of corruption by “pulling” once to effect a radical cure very hard, after any “pulling”, need effective sustainable management plan, to form a normalized anticorrosive system.

From the perspective of engineering corruption, it refers to the corruption of code engineering, engineering structure itself and various “elements” (manifest, code, resources, SO and configuration) that comprise APP. In recent years, youku architecture team has been continuously thinking, practicing and governing, and has accumulated some technologies, tools and solutions. Now, one by one, classified and summarized, supplemented by the explanation of relevant field knowledge, sorted into a series of technical articles titled “Firing a gun at engineering corruption”, to share with you. We hope that more students will join us in the long war against engineering corruption.

Series first “nuked engineering corrupt | proguard governance”. This is the second article in a series that focuses on the manifest niche. On engineering corruption, direct firing!

background

The manifest file refers to the androidmanifest.xml file in APK. As an overall information list of APK, it contains a lot of important information, which has a critical impact on app build processing, runtime behavior, app store filtering, etc.

List content & impact

When the contents of the Androidmanifest.xml file change unexpectedly, there are unintended consequences. For example, if the minSdkVersion becomes smaller, users of an earlier OS version upgrade to the latest APK, causing serious user experience problems. TargetSdkVersion increase, OS specific processing of APP runtime changes, not adapted code crash/ functional exceptions; New permissions are introduced, privacy protocols are undeclared, and regulators find them. All of these problems are caused by a “small” configuration value change in the manifest file, and the corruption of the manifest makes such unexpected changes more and more likely. Manifest governance is based on the content management and prevention of Androidmanifest.xml.

Basic knowledge of

This chapter gives you a brief overview of the basics to help you get a “framework” for manifestly. First, take a look at the generation (merging) of the Androidmanifest.xml file.

Merging process

App project, AAR-type subproject project and external dependent AAR module all contain androidmanifest.xml file. During the APK build process, these Androidmanifest.xml files are merged (plus some extra processing) to produce a unique Androidmanifest.xml file, which is compiled and finally placed in the APK root directory.

Merge is performed from low priority to high priority. Horizontal is the priority of different sources; The priority between modules is from high to low, which is the declaration order in APP project. The priorities of build variant, Build Type, and Product flavor decrease gradually. If the product flavor contains multiple dimensions, the order of priority from high to low is specified in flavorDimensions.

Androidmanifest.xml priority & merge order

During the merge process, attributes of the same XML element (generally android: Name attribute value, or element tag) will have merge conflicts. The basic principle is: if the attributes of high priority and low priority exist and are inconsistent, they are considered to be in conflict. Due to the diversity of elements/attributes in the manifest file, the actual rules are more complex, as you can see in the Official Google documentation. In addition to modifying the corresponding androidmanifest.xml file, the merge conflict can be resolved by adding “merge rule markup” in the app project Androidmanifest.xml. In addition, even if there is no conflict, you can do the same when you need to control the contents of the list, as described next.

Merger control

The “tag of merge rules” mentioned above realizes the control of merge results by specifying merge rules for XML nodes and attributes with different granularity. First, we need to add the Tools namespace to the manifest root node:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.myapplication"
    xmlns:tools="http://schemas.android.com/tools">
Copy the code

Then, add the corresponding Tools: attribute to the node as required.

Merge rule tag description

Please refer to the official document for detailed rules of merge control.

The manifest a placeholder

In addition to the above merge control, you can control the attribute values of the nodes in the manifest through the manifest placeholder.

Android {defaultConfig {Manifestplaceholder = [customKey:"customValue",... }... } # androidmanifest.xml <intent-filter... > <data android:scheme="https" android:host="${customKey}" ... / >... </intent-filter> <meta-data android:name="sampleMeta" android:value="${customKey}"/>Copy the code

In addition, there is a default placeholder ${applicationId} that is bound to the applicationId configuration value in the Android DSL. During the build process, all placeholders are replaced with their corresponding values.

Merge decision log

Finally, each node and attribute of Androidmanifest.xml, which manifest file is derived from, and which policy is generated, are recorded in the combined decision log, providing important auxiliary information for problem analysis and troubleshooting. Engineering build file is located in the app/outputs/logs/manifest – merger [- productFlavor] – < buildType > – the report. TXT, sample content is as follows:

activity#com.example.myapplication.MainActivity ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:9-24:20 android:name ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:19-47 intent-filter#action:name:android.intent.action.MAIN+category:name:android.intent.category.LAUNCHER ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:19:13-23:29 action#android.intent.action.MAIN ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:17-69 android:name ADDED from  /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:25-66 category#android.intent.category.LAUNCHER ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:17-77 android:name ADDED from  /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:27-74 ... uses-permission#android.permission.READ_EXTERNAL_STORAGE IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: Com. Example. Libraryaar1 requested WRITE_EXTERNAL_STORAGE MERGED from [com. Youku. Arch: Hound: 2.8.15] / Users/flyeek /. Gradle/caches/transforms - 2 / files - 2.1 / d42ba59a47f7160082879236533c4582 AndroidManifest. XML: 11:5-80 MERGED The from [com. Youku. Arch: Hound: 2.8.15] / Users/flyeek /. Gradle/caches/transforms - 2 / files - 2.1 / d42ba59a47f7160082879236533c4582 AndroidManifest. XML: 11:5-80 uses-permission#android.permission.WRITE_CALL_LOG IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: com.example.libraryaar1 has targetSdkVersion < 16 and requested WRITE_CONTACTSCopy the code

After reading the basic knowledge of this article, the content here, should be able to understand, no longer repeat.

Several interesting configurations

At this point, we have a “structured” overview of the manifest file. Finally, let’s look at some interesting configurations.

package vs applicationId

These two concepts are easy to confuse. From the perspective of the final APK file, the unique identification of APK is the package attribute value of the MANIFEST node in androidmanifest.xml, which is often referred to as “appId” and “app package name”. Directly above:

package vs applicationId

The package value in APP project only affects the construction process. The applicationId value in the Android DSL will eventually replace the package attribute value in androidmanifest.xml as the final APK unique identifier.

Implicit system permissions

In some cases, the manifest file merge process automatically adds additional system permission declarations, which, if left unaddressed, and the APP privacy agreement is not declared, can raise compliance risks. The following table describes how to automatically add permission statements (directly from the official document) :

List of permission claims added during the merge process

For example, the app’s targetSdkVersion is 28 and, as an external dependency, introduces a module that contains targetSdkVersion 14 in androidmanifest.xml (low priority manifest) and declares READ_CONTACTS permission, The final APK manifest file will contain the READ_CALL_LOG permission declaration.

Component export control

Exporting an Android: Exported component is exported by other apps if the android:exported attribute is true. If the Android: Exported value is explicitly set in the listing, it will prevail; If not specified, the implicit rule is: The exported value is true if intent-filter is set, and false otherwise. Many apps use the in-app routing mechanism for components (especially activities). Therefore, intent-filters are set. As a result, components can be exported unexpectedly, causing security risks. This needs special attention and will be covered later.

Note that android:exported values must also be explicitly set when targetSdkVersion is set to 31 (Android12) or above and if the component is intent-filter set. If the exported value is not explicitly set, IDE builds will fail for older versions of Android Studio, builds will succeed for younger versions of Android Studio, but will fail when installed on Devices running Android12 and above.

Governance practices

The basic knowledge of MANIFEST and engineering application have been explained before, and I believe that you have formed a preliminary overall cognition. As the number of engineering modules/code increases, the controllability of the manifest file decreases: whether it is unexpected changes in key configuration values, the introduction of unexpected permissions, or even the accumulation of useless/redundant/risky nodes and attributes. In the struggle against manifest “corruption”, Youku started from the actual needs of the upper layer (such as privacy compliance, security vulnerabilities, online problems), established effective detection capabilities through relevant tools, and formed a daily research and development bayongate mechanism based on this. On the premise of ensuring zero new problems, gradually digest the existing problems.

Global configuration

Some global configurations in the MANIFEST, notably minSdkVersion and targetSdkVersion, have a significant impact on apK installation and runtime behavior. Unintended changes brought online can be disastrous.

Global configuration detection tool, which provides whitelist-based global configuration detection capabilities, including:

  • The whitelist configuration does not exist in the list.
  • The whitelist is configured. The whitelist exists in the list, but the configured values are inconsistent.

If the global configuration is inconsistent with the whitelist, the build process is terminated. The example check result is as follows:

[absent] [uses-feature] Android.hardware. Camera [conflict] [uses-sdk] Attribute values do not agree with the list | -- com. Youku. Arch: testlib: 0.1 the SNAPSHOT # include USES - SDK node module | - project: the library - the aar - 1 to 1. 0 # include USES - | - SDK node module Com. Youku. Arch: testlib2:0.1 the SNAPSHOT # include USES - SDK node module | - [attr] targetSdkVersion # targetSdkVersion attributes values are not consistent | | - [whitelist] 29 | | - 28 | [current] - [attr] minSdkVersion # minSdkVersion attribute values are not consistent | | - [whitelist] 14 | | - [current] 21Copy the code

The global configuration of youku whitelist and the new prevention and control situation are as follows:

Global configuration governance

Through this detection capability and bayonet mechanism, critical global configurations are protected to effectively prevent unexpected changes.

permissions

Permission statement, in the current situation of privacy compliance supervision, needs to be strictly controlled. The “strict” here means neither more nor less, and must be consistent with the APP privacy protocol. In the basics section above, we know that androidmanifest.xml in APK is merged with implicit system permissions, which increases the difficulty of “strict” permissions control.

In this regard, two detection capabilities are developed: the module contains permission list and permission detection.

The module contains a permission list, which lists the uses-permission and permission definition contained in each module, facilitating the locating of permission sources. Example results:

Com. Youku. Android: YPx: 1.20.10.19 | -- [USES - permission] android.. Permission ACCESS_NETWORK_STATE | - [USES - permission] Android. Permission. BLUETOOTH | -- [USES - permission] android. Permission. VIBRATE com. Taobao. The android: ls: 4.10.6.6 | - [uses-permission] android.permission.READ_PHONE_STATE |-- [uses-permission] android.permission.ACCESS_WIFI_STATECopy the code

Permission detection, providing whitelist-based bidirectional detection capabilities:

  • Permissions in the whitelist, not in the list;
  • List of permissions, not in the whitelist.
[excess] [USES - permission]. Android permission. # CALL_PHONE CALL_PHONE permission statement in listing, not in the white list | - project: the app: 1.0 # permission statement, Absent from the app project [USES - permission] android.. Permission ACCESS_NETWORK_STATE ACCESS_NETWORK_STATE # white list, Not in the list | -- com. Youku. Arch: testlib: # 1.0 com. Youku. Arch: testlib module, Containing the permission statement | - com. Youku. Arch: testlib2: # 1.0 com. Youku. Arch: testlib2 module, containing the statementCopy the code

Take it a step further and provide the option to terminate the build process if the test results fail. Through this detection capability and bayonet mechanism, the continuous consistency of permission declaration and APP privacy protocol is guaranteed. Youku’s governance & prevention and control situation is as follows:

Rights Management

Four major components

The four components need to be declared in the manifest file in order to be recognized by the system after apK is installed and at runtime to function properly. In addition, some key behaviors of the four components need to be configured in the list. In the practice of Youku, there are two main problems: missing corresponding classes of components and exporting unnecessary components.

The missing component class refers to the four components declared in the list. The android: Name attribute value corresponds to the Java class and does not exist in APK. The negative effects of missing component classes are as follows:

  • The proGuard generates a useless KEEP rule, which increases the build time (a small keep rule can add up to a lot).
  • Once the component is invoked (started) at runtime, Java exceptions (crash/ functionality unavailable) or security vulnerabilities can be generated. Even for useless components, consider that there are black production organizations that automatically scan components and start them up (spikes appear on crash rate curves).

Exporting unnecessary components (see the definition above) increases the risk of security vulnerabilities at runtime, and Youku has received several related security vulnerabilities. The principles for exporting components are as follows:

  • Unnecessary export and self-developed code. Close the export;
  • Unnecessary export, but two – or three-party code. In the app project manifest file, modify the Android: Exported property to false via the merge rule tag.
  • Need to export, and developed code, for development time debugging. Close the export and converge to the unified R&D debugging toolbox.
  • Need to export, and for self-developed code, for online actual business (external call end, etc.). Disable the export and converge to the unified routing center.
  • Need to export, but for two – or three-party code, for the actual business online (external call, etc.). Add a whitelist.

In response, three detection capabilities have been developed:

  • A component owning module list that lists all four major components and the module that contains this component declaration:
Components that do not exist after the manifest merge are preceded by [delete] # components that are contained by more than two modules, Front will add [duplicate] [duplicate] [activity] com. Example. Myapplication. MainActivity | - project: the app: | - 1.0 Project: library - the aar - 1 to 1. 0 [does] [service] com. Example. Myapplication. FirstService | - project: the app: 1.0 (receiver) Com. Example. Myapplication. FirstReceiver | - project: the library - the aar - 1 to 1. 0 [the provider] com. Example. Myapplication. FirstProvider | - com. Youku. Arch: testlib: 1.0Copy the code
  • Missing component reference detection identifies missing reference component names and which modules declare this component. At the same time, provide options and whitelists, and terminate the build process if the test result fails. The following is an example check result:
[activity] org. Cocos2dx. Javascript. AActivity | -- com. Youku. The android: interactive - engine: 0.2.9 [activity] Com. Ali. Lv. HLActivity | -- com. Ali. Phone. Wt: n - build: 10.2.3.592 [activity] com. Youku. PC. Debug. DActivity | - Com. Youku. Android: YKPChannel: 2.14.1.28 [service] com. Youku. Feed. Utils. FAService | -- com. Youku. The android: FBase: 1.5.20.8Copy the code
  • Export component detection identifies the export component and which modules declare it. At the same time, provide options and whitelists, and terminate the build process if the test result fails. For behavior changes of the Target31 more secure export component, a “No implicit export” configuration item is specifically provided, which ignores the whitelist and adds identifiable flags to the analysis results. The following is an example check result:
# White list components' names will be prefix with [ignored]; If disable Implicit Export is enabled, for implicit export components, Before the name plus [implicit] [activity] com. Youku. App. NPageActivity | -- com. Youku. Android: YoukuHPage: 1.9.43.8 [ignored] [activity] Com. Ali. MIPreviewActivity | -- com. Ali: m - image - the selector: 10.1.6.190 [implicit] [activity] com. Youku. Fbiz. RPageActivity | - com. Youku. Android: fbizSDK: 1.0.2.48Copy the code

In the governance practice of Youku, considering the impact on the r&d students of each business, the inventory problems are added to the whitelist in a centralized manner, and then a unified cleaning action is launched at the appropriate time. With iteration, in addition to effective interception of new issues, there are also some “natural fixes” for stock issues. The overall situation is as follows:

Governance of the four major components

In addition, youku’s current targetSdkVersion is 30, and target31 adaptation will be carried out next year. There are 161 implicit exported components, accounting for about 50% of all exported components. All these need to be solved by then. Under the current tools and bayonet system, we believe that the rectification of this problem will become easy and controllable.

Governance landscape

So far, the manifest list, a more comprehensive and effective anti-corruption capacity construction and governance. Finally, a panorama is given:

Manifest Governance Panorama

What else can be done

Manifest contains limited content. Therefore, the above governance should cover most of the problems, but there are still some low-probability edge cases that can be identified and resolved in advance by using the same approach. For example, the scheme definition of multiple activities is repeated, resulting in a selection popup when an activity is launched implicitly.

It is still difficult to fight against engineering corruption, and it is a long way to go.

【 References 】

  • [Google] App Manifest Overview:developer.android.com/guide/topic…
  • [Google] the Manage the manifest files:developer.android.com/studio/buil…

Pay attention to [Alibaba mobile technology] wechat public number, every week 3 mobile technology practice & dry goods to give you thinking!