preface

HTTP protocol development has been more than 20 years of history, the whole development trend is mainly in two directions: efficiency and security. In terms of efficiency, from HTTP1.0 requesting one connection at a time, to HTTP1.1 connection multiplexing, to SPDY/HTTP2 multiplexing, to QUIC/HTTP3 udP-based transport, it is becoming more efficient in terms of efficiency. Security aspects, from HTTP plaintext, HTTP2 to enforce TLSv1.2, to QUIC/HTTP3 to enforce TLSv1.3, more and more attention to the security of data transmission. In short, the DEVELOPMENT of the HTTP protocol has been user-friendly, but not so developer-friendly.

Packet capture is one of the required skills of every programmer, especially in interface debugging and program reverse has a broad use. However, as more and more communication protocols use encrypted HTTPS and the use of HTTPS becomes mandatory at the system level, packet capture seems to become more and more difficult.

This blog focuses on the Android platform, HTTPS packet capture common problems and solutions. If you want to do a good job, you have to do a good job. The blog uses HttpCanary as a packet capture tool to explain. For more information on HttpCanary, see:

Click on my GitHub address to get it for free!

Caught the principle

Almost all network data is captured using the man-in-the-middle (MITM) method, including well-known packet capture tools such as Fiddler and Charles. HttpCanary also uses the man-in-the-middle method to capture packets.

From the above schematic diagram, it can be seen that the core problems of packet capture are mainly two:

  • How does MITM Server masquerade as a real Server
  • How does the MITM Client masquerade as a real Client

First, to become a real Server, MITM Server must be able to issue a public key certificate to the specified domain name and pass the security verification of the system. For example, if a Client sends a network request to www.baidu.com/, the MITM Server must send a public key certificate of the domain name www.baidu.com to the Client and have a private key matching the public key.

The MITM Server extracts the domain name www.baidu.com from the first SSL/TLS handshake package Client Hello and uses the built-in CA certificate to create a public key certificate and private key for the domain name www.baidu.com. The created public key certificate is sent to the Client during the SSL/TLS handshake. After the Client receives the public key certificate, the system verifies the certificate to determine whether it is a certificate held by Baidu. However, it is clear that the certificate is forged by a packet capture tool. To ensure that the system considers the public key certificate to be valid when verifying the public key certificate, you need to manually install the CA certificate embedded in the packet capture application to act as a real certificate issuer (CA), that is, to wash the certificate. This is why the CA certificate must be installed before HTTPS packet capture.

Second problem, MITM Client masquerades as Client. Since the server does not validate the Client (for the most part), this problem generally does not exist. For example, the Server does not care whether the Client is a Chrome browser or an Internet Explorer browser, or an Android App or an iOS App. Of course, the Server can also verify the Client, which will be discussed later.

/ Install the CA certificate /

The BUILT-IN CA certificate of a packet capture application must be installed in the system. The Android system divides CA certificates into two types: user CA certificates and system CA certificates. Gu Mingsi discussed, the user CA certificate is installed by the user, the system CA certificate is built by the system, obviously the latter is more real and effective.

The system CA certificate is stored in the /etc/security. cacerts/ directory. The name of the CA certificate is the first four digits of the Md5 value of the CA certificate subjectDN and the extension is.0, for example, 00673b5b.0. For security reasons, the system CA certificate can be added or deleted only when you have Root permission.

For non-root Android devices, users can only install the user CA certificate.

Either system CA certificate or user CA certificate can be viewed in Settings -> System Security -> Encryption and Credentials -> Trusted credentials:

/ Android7.0 user CA restrictions /

Android no longer trusts user CA certificates as of 7.0 (targetSdkVersion >= 24, targetSdkVersion < 24 will be trusted even if the system is 7.0+). This means that even if the user CA certificate is installed, on Android 7.0+ machines, the HTTPS package of the application targetSdkVersion >= 24 will not be caught.

For example, the packet capture tool uses the built-in CA certificate to create a public key certificate for the www.baidu.com domain name and sends it to the Client. When the system verifies the certificate, it finds that the certificate is issued by the user CA certificate. Sorry… So what if we get around this limitation? The following four methods are known (ignore for systems below 7.0).

Configuration networkSecurityConfig

If we want to catch our own App, we just need to configure networkSecurityConfig in AndroidManifest:

<? The XML version = "1.0" encoding = "utf-8"? ><manifest ... > <application android:networkSecurityConfig="@xml/network_security_config" ... >... </application></manifest> <? The XML version = "1.0" encoding = "utf-8"? ><network-security-config> <base-config cleartextTrafficPermitted="true"> <trust-anchors> <certificates src="system" /> <certificates src="user" /> </trust-anchors> </base-config></network-security-config>Copy the code

In this way, App trusts the user’S CA certificate and allows the system to pass the verification of the user’s CA certificate. For more information, see

The Network security configuration:

Developer.android.com/training/ar…

Lower targetSdkVersion < 24

If you want to capture an App package, you can find a historical version, just need its targetSdkVersion < 24. However, with GooglePlay starting to restrict targetSdkVersion, which now requires it to be >=26 and must be >=28 after August 1, 2019, the domestic app market is gradually responding to this restriction. The targetSdkVersion of most apps will be greater than 24, which means it’s getting harder to grab HTTPS packages.

Parallel space packet capture

What if we want to catch packages for applications with targetSdkVersion >= 24? We can use parallel space or VirtualApp to save the country. Multi-open applications such as parallel Spaces and VirtualApp can be used as host systems to run other applications. If the targetSdkVersion of parallel Spaces and VirtualApp is < 24, then the problem is solved.

Here, I recommend using parallel Spaces, which are more stable than some open source VirtualApps. Note that the targetSdkVersion < 24 is available below version 4.0.8625 in parallel space. Of course, HttpCanary is set up to install parallel Spaces directly.

Installation to the system CA certificate directory

This is the perfect solution for Root machines. If the CA certificate is installed in the system CA certificate directory, the fake CA certificate is truly whitewashed. Since the system CA certificate format is a special.0 format, we must export the CA certificate built into the packet capture tool in this format. HttpCanary provides this export option directly.

Operation path: Settings -> SSL Certificate Settings -> Export HttpCanary root certificate -> System Trusted(.0).

HttpCanary v2.8.0 does not provide a valid certificate name for HttpCanary v2.8.0.

Once you have exported the.0 certificate, you can use MT manager to copy the.0 file to /etc/security/cacerts./ or you can use ADB remount and push (for a quick mention, don’t look for this directory in sdcard).

/ Firefox certificate installation /

Firefox implements its own CA certificate management. Firefox does not recognize either system CA certificates or user CA certificates. In this case, we need to import the CA certificate into Firefox in a special way, otherwise Firefox web browsing will not work.

HttpCanary V2.8.0 provides Firefox certificate import options. Go to Settings -> SSL Certificate Settings -> Add HttpCanary root certificate to Firefox:

Click the copy button in the upper right corner to copy the URL to the paste board, then leave the page as it is, open Firefox paste and enter the copied URL.

After the download certificate dialog box appears, be sure to manually tick: Trust for website and Trust for email users. And then you can make sure.

/ Public key certificate fixed /

Certificate Pinning is the true public key Certificate on the Server side built into the Client side. During an HTTPS request, the public key certificate sent from the Server to the Client must be consistent with the built-in public key certificate of the Client.

In this case, the public key certificate created by the MITM Server is inconsistent with the built-in public key certificate on the Client. Therefore, the MITM Server cannot pretend to be a real Server. In this case, packet capture is an App network error. Known applications, such as Ele. me, use certificate fixation.

In addition, some servers use self-licensing documents (certificates not issued by a real CA issuer), in which case the App must use a fixed certificate when requesting it.

In general, a public key certificate (in. CRT or. Cer format) is built into the App and then added to the certificate when creating the TrustManager. Many applications also disguise or encrypt the built-in public key certificate to prevent reverse extraction, such as Ele. me disguised as PNG. Of course, there is no need to disguise or encrypt the public key certificate.

Fixed certificate to capture package is a very troublesome obstacle, but we always have a way around, is a bit of trouble.

JustTrustMe Decryption certificate fixed

Xposed and Magisk have the corresponding module, used to crack the certificate fixed, to achieve normal packet capture. The cracking principle is roughly as follows: Hook to create SSLContext and other methods related to TrustManager to remove the fixed certificate.

Fixed certificate cracking based on VirtualApp Hook mechanism

Xposed and Magisk need to brush and other special processing, but if you don’t want to brush, we can also add Hook code in VirtualApp, and then use VirtualApp to open the target application for packet capture. Of course, there are developers who have implemented this functionality. See:

Case 1:

Github.com/PAGalaxyLab…

Case 2:

Github.com/rk700/CertU…

However, there is something wrong with the CertUnpinning plug-in code here that needs to be fixed.

Import a real public key certificate and private key

If the Client has a fixed public key certificate, MITM Server must hold the real public key certificate and the matching private key. If the developer has a real server’s public key certificate and private key (such as baidu’s public key certificate and private key), if so, you can import it into HttpCanary and do normal packet capture.

Choose Settings > SSL Certificate Settings > Manage SSL Import Certificates, switch to the server, and import the public key certificate and private key. P12 and. BKS files are supported.

/ Two-way authentication /

The SSL/TLS protocol provides bidirectional authentication. That is, both the Client and the Server need to verify the authenticity of the Client. Such cases are rare, but there are still some applications that enable two-way authentication. For example, part of the interface of the anonymous social app Soul uses two-way authentication. Packets cannot be captured in HTTPS requests that use bidirectional authentication.

The principle of bidirectional authentication

First, bidirectional authentication requires the support of the Server. The Client must have a built-in public key certificate and private key. During the SSL/TLS handshake, the Server requests a certificate from the Client. The Client must send the built-in public key certificate to the Server, and the Server verifies the authenticity of the public key certificate.

Note that the built-in public key certificate is different from the public key certificate fixed in point 5 above. The built-in public key certificate + private key for bidirectional authentication is an additional set, which is different from the built-in public key certificate of certificate fixed.

If a Client uses both certificate fixation and bidirectional authentication, the Client must have a built-in public key certificate, a public key certificate, and a private key. The first set is the same as the public key certificate on the Server. It is used to verify whether the public key certificate on the Client is the same as the certificate sent by the Server. During the second SSL/TLS handshake, the public key certificate is sent to the Server for signature verification, that is, bidirectional authentication.

The public key certificate and private key used for bidirectional authentication represent the identity of the Client, so they are private and stored in. P12 or. BKS files + keys. As it is built into the Client, the stored key is generally written into the Client code. In order to prevent decompression, some apps will write the key into the SO library, such as S anonymous social App, but there is a way to extract it as long as it exists in the Client.

Bidirectional authentication packet capture

Here, S anonymous social App is taken as an example to explain how to capture the HTTPS package of the App that uses bidirectional authentication. If the server uses Nginx and bidirectional authentication is enabled, a 400 Bad Request error will occur during packet capture, as follows:

Some servers may not return 404 and simply fail the request. Next, see how to configure bidirectional authentication capture using HttpCanary.

First, decompress APK and extract. P12 or. BKS files. Binary files are usually stored in raw or Assets directories.

Import the client.p12 file into the phone, then go to HttpCanary Settings -> SSL Certificate Settings -> Manage SSL Import Certificates, switch to the client (since it needs to be allocated to MITM Client), and import the.p12 file.

The public key certificate and private key for bidirectional authentication are protected by keys, so you need to enter the password:

Generally, the key can be extracted from APK in reverse order. Details are omitted here. After entering the key, you need to enter the mapping domain name. Wildcard * is used to map all related domain names:

The import is as follows:

To see the details, click on certificate Details. The client.p12 file contains the public key certificate and private key for two-way authentication.

After the configuration is complete, capture packets again to check the effect.

Two Bad Requests for bidirectional authentication were successful.

/ SSL renegotiation /

Some servers may enable SSL renegotiation, that is, the server asks for a re-handshake when the REQUEST is sent after the SSL/TLS handshake succeeds. Such cases are rare, but not excluded, with known applications such as the 10000 community using SSL renegotiation.

The Android operating system has limited support for SSL renegotiation. Therefore, packet capture of some versions of the Android operating system fails, indicating a network exception. SslSocket fully supports SSL renegotiation up to Android 8.1, but SSLEngine does not, and HttpCanary uses SSLEngine to resolve SSL/TLS. In Android 8.1 and above, SSLEngine and SslSocket are implemented in the same way, so SSL renegotiation is supported.

Therefore, if the server uses SSL for renegotiation, use the system 8.1 or later to capture packets.

/ Non-HTTP packet capture /

If HttpCanary still fails to capture packets, you are most likely not using HTTP. For example, wechat chat and live video broadcast do not use HTTP. In this case, other Packet Capture tools are needed, such as Packet Capture, which directly resolves TCP/UDP. However, non-HTTP packets cannot be resolved even if they are captured. Because most of them are binary and not text.

Summary /

Packet capture is a technical job, which requires a general understanding of network protocols. Students who are interested in packet capture can refer to TCP, UDP, SSL/TLS, HTTP and other related materials.

HttpCanary is a professional HTTP capture tool that has been dedicated to HTTP for 30 years (overblown), but does not currently support QUIC/HTTP3 until QUIC/HTTP3 becomes available.