The latest Tencent cloud technology open class live, Tencent W3C representative, how to become a technical expert from the white?Click here for more details.
The author | li tao edit | township
ApkChannelPackage is a fast multi-channel packaging tool that supports multi-channel packaging based on V1 signature and V2 signature. The plug-in itself automatically detects the signature method used by Apk and selects the appropriate multi-channel packaging method, which is completely transparent to the consumer.
An overview of the
As we all know, due to the current situation of the Domestic Android application distribution market, when we release apps, we generally need to generate multiple channel packages and upload them to different application markets. These channel packages need to contain different channel information, which will be carried when the APP interacts with the background or data is reported. In this way, we can count the number of downloads, users and other key data for each distribution market.
Plain old multi-channel packaging
Since we need to do multi-channel packaging, let’s take a look at the most common multi-channel packaging solutions.
Android Gradle Plugin
The Gradle Plugin itself provides a multi-channel packaging strategy: First, add a channel information placeholder to androidmanifest.xml:
<meta-data
android:name="InstallChannel" android:value="${InstallChannel}" />
Copy the codeThen, add channel information using the productFlavors TAB provided by Gradle Plugin:
productFlavors{
"YingYongBao"{
manifestPlaceholders = [InstallChannel : "YingYongBao"]}"360"{
manifestPlaceholders = [InstallChannel : "360"]}}Copy the codeThis way, when Gradle builds multi-channel packages, it replaces the placeholders in Androidmanifest.xml with different channel information. In code, we can read the channel information in androidmanifest.xml directly.
However, there are some disadvantages to this approach:
1) The construction process has to be re-executed every time a channel package is generated, which is inefficient and only suitable for scenarios with few channels.
2)Gradle generates a different BuildConfig. Java class for each channel package that records channel information, resulting in a different CRC value for DEX for each channel package. In general, this has no effect. However, if you use wechat’s Tinker hot patch scheme, you need to make different patches for different channels, which is totally unacceptable. Tinker generates differential patches by comparing the base package APK with the new package APK, and then combining the patches with the base package APK to create the new package APK. This requires that the base package DEX used for generating differential patches and the base package DEX used for synthesing new packages are exactly the same, that is, the DEX files of each base channel package are exactly the same, otherwise the synthesis will fail.)
ApkTool
ApkTool is a reverse analysis tool that can unpack APK, add code, and repackage it into APK. Therefore, the multi-channel packaging scheme based on ApkTool is divided into the following steps:
Copy a new APK using the ApkTool, decompress the APK (ApkTool d Origine.apk) delete the existing signature information add channel information (You can add channel information in any APK file) Use the ApkTool, Repackaging generates a new APK (apktool B newApkDir) re-signature after testing, this scheme is completely feasible.
Advantages:
There is no need to rebuild the new channel package, just copy the changes. Because the signature is re-signed, both V1 and V2 signatures are supported.
Disadvantages:
ApkTool is unstable. After Gradle Plugin is upgraded, ApkTool of an earlier version fails to decompress APK. When generating a new channel package, it needs to be unpacked, packaged, and signed again, which is relatively time consuming. Test: it takes about 16 minutes to generate 10 channel packages of Penguin e-sports, although much less time than Gradle Plugin. However, if hundreds of channel packages need to be generated at the same time, it will take several hours, which is obviously not suitable for business scenarios with many channels.
Is there a way to add channel information without re-signing it? First let’s look at APK’s signature and verification mechanism.
Data digest, digital signature and digital certificate
Before further studying V1 and V2 signatures, it is necessary to learn some basic knowledge about signatures.
Data in this paper,
Data summarization algorithm is an algorithm that can produce a specific output format. Its principle is to extract some form of information from the original data according to certain operation rules. The extracted information is the message summarization of the original data, also known as data fingerprint. In general, the data summarization algorithm has the following characteristics:
No matter how large the input data is, the length of the calculated data digest is always fixed. For example, the digest calculated by MD5 algorithm has 128 bits. In general (regardless of collisions), as long as the original data is different, the corresponding data digest will not be the same. At the same time, if there is any change to the original data, the data summary will be completely different. That is, the same original data must have the same data abstracts, and different original data must have different data abstracts. Irreversibility, that is, only forward extraction of the data abstract of the original data, but cannot recover the original data from the data abstract. The famous abstract algorithm has RSA MD5 algorithm and SHA algorithm series.
Digital signature and digital certificate
A digital signature and a digital certificate are paired and cannot be separated. A digital signature is used to verify data integrity, and a digital certificate is used to ensure the security of public keys.
To understand the concept of digital signature, you must understand data encryption, transmission, and verification processes. In general, to achieve reliable data communication, the following two problems need to be solved:
1. Make sure the source of the data is the real sender. 2. Ensure that data is not tampered with during transmission or can be detected in time if tampered with.
Digital signature is born to solve these two problems. First, the data sender needs to apply for a public and private key pair and hand the public key to the data receiver. If the data sender needs to send data to the receiver, it first generates a digital signature based on the original data and then sends the original data and digital signature to the receiver. The digital signature is calculated in the following two steps:
Calculate the data digest of the sent data. 2. Encrypt the extracted data digest with a private key
Thus, the data receiver receives a message containing two pieces of content:
1. Original data content 2. Additional digital signature
The receiver then verifies the authenticity of the data by taking the following steps:
- The same summarization algorithm is used to calculate the data summarization of the original data.
- Decrypt the digital signature with the pre-obtained public key.
- If yes, the data is not tampered with. Otherwise, the data is dirty.
Because only the sender has the private key, no one else can forge a digital signature. In this way, the reliable transmission of data is ensured through digital signature. To sum up, a digital signature is a number string that can only be produced by the sender and cannot be forged by others. This number string is also an effective proof of the authenticity of the data sent by the sender.
That’s a good idea, but the whole process assumes that the receiver can get the sender’s public key correctly. If the receiver’s public key is tampered with, the bad guy is treated as the good guy, while the data sent by the real sender is treated as dirty. So how can you guarantee the security of the public key? This depends on the digital certificate to solve.
A digital certificate is a certificate issued to an applicant by a certificate center (CA) that has credibility. The certificate includes the issuing authority, certificate validity period, public key of the applicant, information about the applicant, algorithm used for digital signature, and digital signature of the certificate content.
It can be seen that digital certificates also use digital signature technology. The signature is the public key of the data sender and some other certificate information. The data sender then sends a message containing three parts:
- Raw data content
- Additional digital signature
- Applied digital certificate.
After receiving the data, the receiver first decodes the sender’s public key from the CA’s public key. Then the verification process is exactly the same as above.
So,Digital certificates mainly solve the security issue of public keys.
Therefore, the entire signing and verification process, including digital certificates, looks like the following figure:
V1 signature and multi-channel packaging scheme
V1 Signature Mechanism
By default, APK uses the V1 signature. After decompressing APK, in the meta-INF directory, you can see three files: manifest. MF, cert. SF, and cert. RSA. They’re all products of the V1 signature.
Among them,MANIFEST.MFThe file content is as follows:
It records the Base64 encoding of the data digest of all raw files in APK, and the data digest algorithm isSHA1.
CERT.SFThe file content is as follows:
SHA1-Digest-Manifest-Main-AttributesThe master property is loggedMANIFEST.MFBase64 encoding of the data digest for all main attributes of the file.SHA1-Digest-ManifestIt records the wholeMANIFEST.MFBase64 encoding of the data digest of the file.
The rest of the common attributes andMANIFEST.MFThe Base64 encoding of the data digest of the corresponding data block is recorded respectively. Such as:CERT.SFThe sha1-digest corresponding to skin_drawable_btm_line.xml in the file is the Base64 encoding of the data Digest for the following.
Name: res/drawable/skin_drawable_btm_line.xml
SHA1-Digest: JqJbk6/AsWZMcGVehCXb33Cdtrk=
\r\n
Copy the codeNote here that the last line newline is essential and needs to be evaluated.
The cert. RSA file contains the digital signature of the cert. SF file and the developer’s digital certificate. RSA is an asymmetric encryption algorithm for computing digital signatures.
For details about the V1 signature process, seeSignApk.java, the whole signature process is as follows:
The final product of the whole signature mechanism is manifest. MF, cert. SF and cert. RSA files.
V1 verification process
When APK is installed, the Android system verifies the signature to check whether the APK has been tampered with. The code flow is:PackageManagerService.java -> PackageParser.java.PackageParserClass is responsible for the specific verification of V1 signatures. The whole verification process is shown in the figure below:
APK cannot be installed if any of the verification steps fail.
OK, understood V1 signature and verification process. Let’s see, how does V1 signature protect APK files from tampering? First, if the saboteur modifies any file in APK, the Base64 encoding of the data digest of the tampered file will be inconsistent with the recorded value of the manifest.mf file, resulting in a validation failure. Secondly, if the saboteur modifies the Base64 value of the corresponding file in the manifest.mf file at the same time, the Base64 value of the corresponding data block in the manifest.mf file will be inconsistent with the value recorded in the cert.sf file, resulting in verification failure. Finally, if the saboteur goes further and changes the Base64 value of the corresponding file in the cert. SF file, the digital signature of cert. SF will be inconsistent with the signature recorded in cert. RSA, and the verification will fail. Is it possible to forge cert.sf’s digital signature? Theoretically impossible, because the saboteur doesn’t have the developer’s private key. Is it possible for the saboteur to re-sign with his private key and digital certificate?
In summary, any changes to APK files will fail at installation unless APK is re-signed. However, apKS with the same package name and different signatures cannot be installed at the same time.
APK file structure
As you can see from the V1 signing and verification mechanism above, modifying any files in APK will cause installation failure! Then how to add channel information that? We have to start with the structure of APK.
The APK file is essentially a ZIP compressed package. The ZIP format is fixed and consists of three parts, as shown in the following figure:
The first section is the content block, where all the compressed files are located. Each compressed file has onelocal file header, records the file name, compression algorithm, file size before and after compression, modification time, and CRC32 value.
The second part, called the central directory, contains more than onecentral directory file header(and the first partlocal file headerEach central directory file header mainly records the compression algorithm, annotation information, correspondencelocal file headerOffset, etc., to facilitate rapid positioning of data.
The last part is EOCD, which mainly records the central directory size, offset and ZIP comment information, as shown in the figure below:
According to the previous V1 signature and verification mechanism, the V1 signature checks only all the compressed files in the first part and ignores the following two parts. Therefore, as long as the channel information is written to the last two pieces of content, V1 verification can be passed, and the annotation field of EOCD is undoubtedly the best choice.
Multi-channel packaging scheme based on V1 signature
Since the breakthrough was found, the multi-channel packaging scheme based on V1 signature came into being: channel information was added in the comment field of APK file.
The whole program includes the following steps:
- Copy the APK
- Find the EOCD data block
- Modify comment length
- Adding Channel Information
- Add the length of channel information
- Add the magic number
The EOCD data block after channel information is added is as follows:
The advantage of adding magic numbers here is that it is easy to read data from back to front and locate channel information. Therefore, reading channel information includes the following steps:
- Locate magic number
- Read forward two bytes to determine the length LEN of the channel information
- Read LEN bytes forward, and that’s the channel information.
Through the hexadecimal editor, you can view the APK (little endian mode) after adding channel information, as follows:
6C 74 6C 6F 76 75 7A 68 is magic number, 04 00 means channel information length is 4, 6C 65 6F 6E is channel information Leon. 0E 00 is the APK comment length, which is exactly 15.
While the overall scheme is clear, there is a problem with finding EOCD blocks. If the APK itself has no annotations, the last 22 bytes are EOCD. But how do you determine the starting position of the EOCD if the APK already contains comment fields? Here we refer to the system V2 signature scheme to determine the location of EOCD. The whole calculation process is shown in the figure below:
The whole scheme is introduced, the biggest advantage of this scheme is: no need to decompress APK, no need to re-sign, only need to copy APK, add channel information in the comment field. Each channel package takes only a few seconds, which is ideal for APK with many channels.
However, after Android7.0, V2 signature was added, which will verify the data digest of the entire APK, making the channel packaging scheme invalid. So if you want to continue using the above solution, you need to turn off the V2 signing option in Gradle Plugin and disable V2 signing.
V2 signature and multi-channel packaging scheme
Why is a V2 signature required
Introduced from the previous V1 signature, it can be known that V1 has two drawbacks:
1) The data summary in manifest.mf is calculated based on the original uncompressed file. Therefore, you need to decompress the original file before verification. Decompression is definitely time consuming. 2) V1 signature only verifies files in the first part of APK, and lacks integrity verification of APK. Therefore, after signing, we can modify APK files, such as byte alignment with zipalign, and still install normally.
Based on these two points, Google proposed V2 signature to solve the above two problems:
- V2 signature is a data summary calculation for APK itself. There is no operation to decompress APK, which reduces the verification time.
- V2 signatures are validated against the entire APK (excluding the signature block itself), so any changes to APK (including adding comments and zipalign bytes) cannot be validated against V2 signatures.
For the first point, there’s some lab data (Nexus 6P, Android 7.1.1) for reference.
| Comparison of APK installation time | Take the average time of 5 times (seconds) |
|---|---|
| V1 signed APK | 11.64 |
| V2 signed APK | 4.42 |
It can be seen that the installation speed of V2 signature on APK is greatly improved.
V2 Signature Mechanism
Unlike V1, V2 signatures generate a signature block that is inserted into APK. Therefore, the APK structure after V2 signature is shown as follows:
APK signature blocks are located before the central directory and after the file data. V2 signature also modifies the offset of the central directory in the EOCD to make the signed APK conform to the ZIP structure.
The specific structure of APK signature block is shown as follows:
The first is the 8-byte signature block size, which does not include 8 bytes of the field itself; The second is the id-value sequence, which is a 4-byte ID and the corresponding data. Then another 8-byte signature block size, equal to the initial 8-byte signature block size; Finally, the signature block magic number of 16 bytes.
Among them,The Value corresponding to the ID 0x7109871A is V2 signature block data.
V2 signature block generation can refer to ApkSignerV2. The overall structure and process are as follows:
Firstly, according to multiple signature algorithms, the data abstracts of the entire APK are calculated to form the APK data abstracts set in the upper left corner.
Next, assemble the data digest, digital certificates, and additional attributes from the leftmost column to form an “MF” file similar to the V1 signature (column 2, first line);
Secondly, using the same private key and different signature algorithms, the digital signature of “MF” file is calculated to form “SF” file similar to V1 signature (the second row of the second column).
And then, let’s take the second columnSimilar to MF file,Similar to SF fileandDeveloper public keyAssemble the v2 signature block (column 3, line 1) signed through a single keystore.
Finally, the complete V2 signature block is assembled from the signature blocks of multiple keystores (Android allows multiple keystores to sign apK).
The above process is tedious. In short, a single keystore signature block consists of three main parts, the three data blocks in the second column above:Similar to MF file,Similar to SF fileandDeveloper public key, its structure is shown in the figure below:
In addition, Google has also optimized the algorithm for computing the summary of the data in parallel, as shown below:
The calculation of the data summary includes the following steps:
First of all, the file content block, central directory and EOCD in the APK are divided into some small pieces according to the size of 1MB. Then, a summary of the data for each small chunk is computed, with the base data being 0xa5 + block byte length + block content. Finally, the overall data digest is calculated, and the base data is 0x5A + the number of data blocks + the summary content of each data block. In this way, the data digest of each data block can be computed in parallel, speeding up the speed of V2 signature and verification.
V2 Verification Process
By default, V1 and V2 signatures are enabled on Android Gradle Plugin2.2. Cert. SF files that contain both V1 and V2 signatures have a special main attribute, as shown in the following figure:
This property forces APK to go through the V2 validation process (above 7.0) to take full advantage of V2 signatures (faster and more sophisticated validation mechanisms).
Therefore, the verification process of APK containing both V1 and V2 signatures is as follows:
In short: first check V2, no or do not know V2, check V1.
Here is another problem: APK signature, only V2 signature, no V1 signature is ok? On trial, this situation compiles and runs correctly on Android 7.0. In 7.0, however, there is no signature error because you do not know V2 and do not have V1 signature.
OK, after making clear the verification selection of V1 and V2 signatures on Android platform, let’s take a look at the specific verification process of V2 signatures (PackageManagerService.java -> PackageParser.java-> ApkSignatureSchemeV2Verifier.java), as shown below:
The strongest signature algorithm is generated based on the data summarization algorithm used by the algorithm, for example, SHA512 > SHA256.
The verification is successful if at least one signature block corresponding to the keystore is found and all signature blocks are successfully verified according to the preceding procedure.
How does V2 signature protect APK from tampering?
First, if the saboteur modifies any part of the APK file (except the signature block itself), the APK data digest will be inconsistent with the data digest recorded in the “MF” data block, resulting in a validation failure.
Secondly, if the saboteur modifies the data digest in the “MF” data block at the same time, the digital signature of the “MF” data block will be inconsistent with the digital signature recorded in the “SF” data block, resulting in a verification failure.
Then, using the developer’s public key to decrypt the digital signature in the “SF” block will fail if the cracker uses its own private key to encrypt it;
Finally, further, if the cracker even replaces the developer’s public key, the public key in the digital certificate will fail to verify the public key in the signature block, which is exactly what digital certificates do.
In summary, any changes to APK will fail at installation unless APK is re-signed. However, apKS with the same package name and different signatures cannot be installed at the same time.
At this point, V2 signatures have been introduced. But in the last step of “data summary check”, there is a hidden point, I wonder if anyone found it?
Because, the data summary in V2 signature block is calculated against the file content block of APK, central directory and EOCD. However, after writing the signature block, the central directory offset in the EOCD was changed. Therefore, when performing the V2 signature verification, the “data digest verification” step should fail. But why does V2 signature check through that?
This is an important question because the multichannel packaging scheme based on V2 signatures that we describe below also modifies the central directory offset of the EOCD.
In fact, it is very simple, the original Android system when verifying APK data digest, first will replace the EOCD central directory offset with the offset of the signature block, and then calculate the data digest. The offset of the signature block is the offset of the central directory before v2 signature. Therefore, the calculated data digest is exactly the same as the one in the “MF” data block. Specific code logic, can reference ApkSignatureSchemeV2Verifier. Java line 416 ~ 420.
Multi-channel packaging scheme based on V2 signature
In the V2 signature verification process in the previous section, there is an important detail: The Android system only pays attention to the V2 signature block whose ID is 0x7109871A and ignores other id-values. In addition, the V2 signature only protects the APK itself, not the signature block.
Therefore, the multi-channel packaging scheme based on V2 signature came into being: an ID-value was added to the APK signature block to store channel information.
The whole program includes the following steps:
- Find the EOCD block of APK
- Find the APK signature block
- Obtain an existing id-value Pair
- Example Add an id-value containing channel information
- Generates a new signature block based on all the id-values
- Change the offset of the EOCD central directory (as described above: Changing the offset of the EOCD central directory does not cause digest verification failure)
- APK with channel information is generated by replacing old signature block with new signature block
In fact, we can add any auxiliary information to the APK signature block except channel information.
Through the hexadecimal editor, you can view the APK (little endian mode) after adding channel information, as follows:
6C 65 6F 6E is our channel information Leon. The first four bytes: FF 55 11 88 is the ID we added, and the next eight bytes: 08 00 00 00 00 00 00 00 00 00 00 is the length of our id-value, which is exactly 8.
After the introduction of the whole scheme, the biggest advantage of this scheme is that it supports V2 signature added on 7.0, and has all the advantages of V1 scheme at the same time.
Strong verification of multi-channel packages
So how do you ensure that the channel packages generated by these solutions will be correctly installed on all Android platforms?
It turns out that Google provides a tool that supports both V1 and V2 signature and verification: APksig. It includes an apksigner command line and an Apksig class library. The former is the command line tool under the Android SDK build-tools. It is with the help of apkSIG that we carry out strong verification of channel packages, which can ensure that channel packages are verified between APK Minsdk and the highest version. Refer to verifyapk.java for detailed code
Comparison of multi-channel packaging tools
At present, packer-ng-Plugin and Meituan Walle are the main multi-channel packaging tools on the market. The table below is a simple comparison between our ApkChannelPackage and them.
| Comparison of multi-channel packaging tools | ApkChannelPackage | packer-ng-plugin | Walle |
|---|---|---|---|
| V1 signature scheme | support | support | Does not support |
| V2 Signature Scheme | support | Does not support | support |
| APK with comments | support | Does not support | Does not support |
| Generate channel packages based on existing APK | support | Does not support | Does not support |
| Command line tool | Does not support | support | support |
| Strong check | support | Does not support | support |
The reason why I support V1 and V2 signature schemes at the same time here is mainly because I worry that the Android platform will strengthen the signature verification mechanism in the future, so V2 multi-channel packaging scheme will not work, so I can switch to V1 signature scheme painless. I will support command line tools as soon as possible.
ApkChannelPackage plug-in access
For details about the access process, see the APKChannelPackage plug-in access document.