Android’s new generation of multi-channel packaging magic

The Gradle Plugin itself provides a multi-channel packaging strategy: First, add a channel information placeholder to androidmanifest.xml:

<meta-data 

android:name="InstallChannel" android:value="${InstallChannel}" />

Copy the code

Then, add channel information using the productFlavors TAB provided by Gradle Plugin:

productFlavors{

    "YingYongBao"{

        manifestPlaceholders = [InstallChannel : "YingYongBao"]

    }

{" 360"

        manifestPlaceholders = [InstallChannel : "360"]

    }

}

Copy the code

This way, when Gradle builds multi-channel packages, it replaces the placeholders in Androidmanifest.xml with different channel information. In code, we can read the channel information in androidmanifest.xml directly.

However, there are some disadvantages to this approach:

1. Having to redo the build process every time a channel package is generated is inefficient and only suitable for scenarios with fewer channels.
2. Gradle generates a different BuildConfig. Java class for each channel package that records channel information, resulting in a different CRC value for DEX for each channel package. In general, this has no effect. However, if you use wechat’s Tinker hot patch scheme, you need to make different patches for different channels, which is totally unacceptable. Tinker generates differential patches by comparing the base package APK with the new package APK, and then combining the patches with the base package APK to create the new package APK. This requires that the base package DEX used for generating differential patches and the base package DEX used for synthesing new packages are exactly the same, that is, the DEX files of each base channel package are exactly the same, otherwise the synthesis will fail.)

Before further studying V1 and V2 signatures, it is necessary to learn some basic knowledge about signatures.

A digital signature and a digital certificate are paired and cannot be separated. A digital signature is used to verify data integrity, and a digital certificate is used to ensure the security of public keys.

To understand the concept of digital signature, you must understand data encryption, transmission, and verification processes. In general, to achieve reliable data communication, the following two problems need to be solved:

1. Make sure the source of the data is its real sender.
2. Ensure that data is not tampered with during transmission or can be detected in time if tampered with.

Digital signature is born to solve these two problems. First, the data sender needs to apply for a public and private key pair and hand the public key to the data receiver. If the data sender needs to send data to the receiver, it first generates a digital signature based on the original data and then sends the original data and digital signature to the receiver. The digital signature is calculated in the following two steps:

1. Computes a data digest of the sent data
2. The extracted data digest is encrypted with a private key

Thus, the data receiver receives a message containing two pieces of content:

1. Raw data content
2. Additional digital signature

The receiver then verifies the authenticity of the data by taking the following steps:

1. The same summarization algorithm is used to calculate the data summarization of the original data.
2. Decrypt the digital signature with the pre-obtained public key.
3. If yes, the data is not tampered with. Otherwise, the data is dirty.

Because only the sender has the private key, no one else can forge a digital signature. In this way, the reliable transmission of data is ensured through digital signature. To sum up, a digital signature is a number string that can only be produced by the sender and cannot be forged by others. This number string is also an effective proof of the authenticity of the data sent by the sender.

That’s a good idea, but the whole process assumes that the receiver can get the sender’s public key correctly. If the receiver’s public key is tampered with, the bad guy is treated as the good guy, while the data sent by the real sender is treated as dirty. So how can you guarantee the security of the public key? This depends on the digital certificate to solve.

A digital certificate is a certificate issued to an applicant by a certificate center (CA) that has credibility. The certificate includes the issuing authority, certificate validity period, public key of the applicant, information about the applicant, algorithm used for digital signature, and digital signature of the certificate content.

It can be seen that digital certificates also use digital signature technology. The signature is the public key of the data sender and some other certificate information. The data sender then sends a message containing three parts:

1. Raw data content
2. Additional digital signature
3. Applied digital certificate.

After receiving the data, the receiver first decodes the sender’s public key from the CA’s public key. Then the verification process is exactly the same as above.

So,Digital certificates mainly solve the security issue of public keys.

Therefore, the entire signing and verification process, including digital certificates, looks like the following figure:

By default, APK uses the V1 signature. After decompressing APK, in the meta-INF directory, you can see three files: manifest. MF, cert. SF, and cert. RSA. They’re all products of the V1 signature.

Among them,MANIFEST.MFThe file content is as follows:



It records the Base64 encoding of the data digest of all raw files in APK, and the data digest algorithm isSHA1.

CERT.SFThe file content is as follows:



SHA1-Digest-Manifest-Main-AttributesThe master property is loggedMANIFEST.MFBase64 encoding of the data digest for all main attributes of the file.SHA1-Digest-ManifestIt records the wholeMANIFEST.MFBase64 encoding of the data digest of the file.

The rest of the common attributes andMANIFEST.MFThe Base64 encoding of the data digest of the corresponding data block is recorded respectively. Such as:CERT.SFThe sha1-digest corresponding to skin_drawable_btm_line.xml in the file is the Base64 encoding of the data Digest for the following.

Name: res/drawable/skin_drawable_btm_line.xml

SHA1-Digest: JqJbk6/AsWZMcGVehCXb33Cdtrk=

\r\n

Copy the code

Note here that the last line newline is essential and needs to be evaluated.

The cert. RSA file contains the digital signature of the cert. SF file and the developer’s digital certificate. RSA is an asymmetric encryption algorithm for computing digital signatures.

For details about the V1 signature process, seeSignApk.java, the whole signature process is as follows:



The final product of the whole signature mechanism is manifest. MF, cert. SF and cert. RSA files.

OK, understood V1 signature and verification process. Let’s see, how does V1 signature protect APK files from tampering? First, if the saboteur modifies any file in APK, the Base64 encoding of the data digest of the tampered file will be inconsistent with the recorded value of the manifest.mf file, resulting in a validation failure. Secondly, if the saboteur modifies the Base64 value of the corresponding file in the manifest.mf file at the same time, the Base64 value of the corresponding data block in the manifest.mf file will be inconsistent with the value recorded in the cert.sf file, resulting in verification failure. Finally, if the saboteur goes further and changes the Base64 value of the corresponding file in the cert. SF file, the digital signature of cert. SF will be inconsistent with the signature recorded in cert. RSA, and the verification will fail. Is it possible to forge cert.sf’s digital signature? Theoretically impossible, because the saboteur doesn’t have the developer’s private key. Is it possible for the saboteur to re-sign with his private key and digital certificate?

In summary, any changes to APK files will fail at installation unless APK is re-signed. However, apKS with the same package name and different signatures cannot be installed at the same time.

According to the previous V1 signature and verification mechanism, the V1 signature checks only all the compressed files in the first part and ignores the following two parts. Therefore, as long as the channel information is written to the last two pieces of content, V1 verification can be passed, and the annotation field of EOCD is undoubtedly the best choice.

The advantage of adding magic numbers here is that it is easy to read data from back to front and locate channel information. Therefore, reading channel information includes the following steps:

1. Locate magic number
2. Read forward two bytes to determine the length LEN of the channel information
3. Read LEN bytes forward, and that’s the channel information.

Through the hexadecimal editor, you can view the APK (little endian mode) after adding channel information, as follows:

6C 74 6C 6F 76 75 7A 68 is magic number, 04 00 means channel information length is 4, 6C 65 6F 6E is channel information Leon. 0E 00 is the APK comment length, which is exactly 15.

The whole plan is clear, but inFind the EOCD data blockThere is a problem with this step. If the APK itself has no annotations, the last 22 bytes are EOCD. But how do you determine the starting position of the EOCD if the APK already contains comment fields? Here we refer to the system V2 signature scheme to determine the location of EOCD. The whole calculation process is shown in the figure below:

The whole scheme is introduced, the biggest advantage of this scheme is: no need to decompress APK, no need to re-sign, only need to copy APK, add channel information in the comment field. Each channel package takes only a few seconds, which is ideal for APK with many channels.

However, after Android7.0, V2 signature was added, which will verify the data digest of the entire APK, making the channel packaging scheme invalid. So if you want to continue using the above solution, you need to turn off the V2 signing option in Gradle Plugin and disable V2 signing.

Introduced from the previous V1 signature, it can be known that V1 has two drawbacks:

1. MANIFEST.MFThe data summary is calculated based on the original uncompressed file. Therefore, you need to decompress the original file before verification. Decompression is definitely time consuming.
2. V1 signature only verifies files in the first part of APK, and lacks integrity verification of APK. Therefore, after signing, we can modify APK files, such as byte alignment with zipalign, and still install normally.

Based on these two points, Google proposed V2 signature to solve the above two problems:

1. V2 signature is a data summary calculation for APK itself. There is no operation to decompress APK, which reduces the verification time.
2. V2 signatures are validated against the entire APK (excluding the signature block itself), so any changes to APK (including adding comments and zipalign bytes) cannot be validated against V2 signatures.

For the first point, there’s some lab data (Nexus 6P, Android 7.1.1) for reference.

It can be seen that the installation speed of V2 signature on APK is greatly improved.

APK signature blocks are located before the central directory and after the file data. V2 signature also modifies the offset of the central directory in the EOCD to make the signed APK conform to the ZIP structure.

The specific structure of APK signature block is shown as follows:



The first is the 8-byte signature block size, which does not include 8 bytes of the field itself; The second is the id-value sequence, which is a 4-byte ID and the corresponding data. Then another 8-byte signature block size, equal to the initial 8-byte signature block size; Finally, the signature block magic number of 16 bytes.

Among them,ID for0x7109871aValue is V2 signature block data.

For details about how to generate V2 signature blocks, seeApkSignerV2, the overall structure and process are shown in the figure below:

Firstly, according to multiple signature algorithms, the data abstracts of the entire APK are calculated to form the APK data abstracts set in the upper left corner. Next, assemble the data digest, digital certificates, and additional attributes from the leftmost column to form an “MF” file similar to the V1 signature (column 2, first line); Secondly, using the same private key and different signature algorithms, the digital signature of “MF” file is calculated to form “SF” file similar to V1 signature (the second row of the second column). Then, assemble the mF-like file, sF-like file, and developer public key in the second column into a v2 signature block signed through a single keystore (the first line of the third column). Finally, the complete V2 signature block is assembled from the signature blocks of multiple keystores (Android allows multiple keystores to sign apK).

The above process is tedious. In short, a single keystore signature block consists of three main parts, the three data blocks in the second column above:Similar to MF file,Similar to SF fileandDeveloper public key, its structure is shown in the figure below:

In addition, Google has also optimized the algorithm for computing the summary of the data in parallel, as shown below:

The calculation of the data summary includes the following steps:

1. First of all, the file content block, central directory and EOCD in the APK are divided into some small pieces according to the size of 1MB.
2. Then, a summary of the data for each small chunk is computed, with the base data being 0xa5 + block byte length + block content.
3. Finally, the overall data digest is calculated, and the base data is 0x5A + the number of data blocks + the summary content of each data block.

In this way, the data digest of each data block can be computed in parallel, speeding up the speed of V2 signature and verification.

In short: first check V2, no or do not know V2, check V1.

Here is another problem: APK signature, only V2 signature, no V1 signature is ok? On trial, this situation compiles and runs correctly on Android 7.0. In 7.0, however, there is no signature error because you do not know V2 and do not have V1 signature.

OK, after making clear the verification selection of V1 and V2 signatures on Android platform, let’s take a look at the specific verification process of V2 signatures (PackageManagerService.java -> PackageParser.java -> ApkSignatureSchemeV2Verifier.java), as shown below:



The strongest signature algorithm is generated based on the data summarization algorithm used by the algorithm, for example, SHA512 > SHA256.

The verification is successful if at least one signature block corresponding to the keystore is found and all signature blocks are successfully verified according to the preceding procedure.

How does V2 signature protect APK from tampering? First, if the saboteur modifies any part of the APK file (except the signature block itself), the APK data digest will be inconsistent with the data digest recorded in the “MF” data block, resulting in a validation failure. Secondly, if the saboteur modifies the data digest in the “MF” data block at the same time, the digital signature of the “MF” data block will be inconsistent with the digital signature recorded in the “SF” data block, resulting in a verification failure. Then, using the developer’s public key to decrypt the digital signature in the “SF” block will fail if the cracker uses its own private key to encrypt it; Finally, further, if the cracker even replaces the developer’s public key, the public key in the digital certificate will fail to verify the public key in the signature block, which is exactly what digital certificates do.

In summary, any changes to APK will fail at installation unless APK is re-signed. However, apKS with the same package name and different signatures cannot be installed at the same time.

At this point, V2 signatures have been introduced. But in the last step of “data summary check”, there is a hidden point, I wonder if anyone found it? Because, the data summary in V2 signature block is calculated against the file content block of APK, central directory and EOCD. However, after writing the signature block, the central directory offset in the EOCD was changed. Therefore, when performing the V2 signature verification, the “data digest verification” step should fail. But why does V2 signature check through that?

This is an important question because the multichannel packaging scheme based on V2 signatures that we describe below also modifies the central directory offset of the EOCD.

In fact, it is very simple, the original Android system when verifying APK data digest, first will replace the EOCD central directory offset with the offset of the signature block, and then calculate the data digest. The offset of the signature block is the offset of the central directory before v2 signature. Therefore, the calculated data digest is exactly the same as the one in the “MF” data block. Specific code logic, can reference ApkSignatureSchemeV2Verifier. Java line 416 ~ 420.

The biggest advantage of this scheme is that it supports the addition of any auxiliary information in the V2 check – out  block added on 7.0.

At present, packer-ng-Plugin and Meituan Walle are the main multi-channel packaging tools on the market. The table below is a simple comparison between our ApkChannelPackage and them.

The reason why I support V1 and V2 signature schemes at the same time here is mainly because I worry that the Android platform will strengthen the signature verification mechanism in the future, so V2 multi-channel packaging scheme will not work, so I can switch to V1 signature scheme painless. I will support command line tools as soon as possible.