Some time ago, chatting with QQ group friends accidentally talked about the topic of catching bags. Packet capture can be said to be a programmer’s daily development and debugging of an important means, can help us clarify the data transmission between the client and the server, in order to dump the fault.

In the past, network requests were based on HTTP protocol, and packet capture was a very simple matter. However, in recent years, HTTP has been gradually phased out, and almost all web requests have been switched to HTTPS, which complicates things.

“HTTPS is impossible to capture packets, otherwise how can you ensure the security of HTTPS transmission, after all, so many companies use this protocol to transmit important data?

This is actually an interesting topic, HTTPS is really very secure. But at the same time, HTTPS does capture packets, and there is no conflict between the two.

Considering that there are still many friends in this area still some do not know, I am going to write two articles to talk about HTTPS packet capture related knowledge. This article will teach you how to capture HTTPS requests on Android phones. The next article will explain the principle, let’s analyze, why so secure HTTPS protocol can still be captured?

Use the packet capture tool

To capture network requests, you must first select a packet capture tool.

There are many professional package capture workers. According to my observation, most developers in China prefer to use Charles to capture packages. But I personally prefer Fiddler, and we use Fiddler when we need to grab bags at work. Fiddler is compatible with Microsoft’s internal log analysis tools, and Fiddler’s authors work at Microsoft.

In this article, I will use Fiddler as an example. Of course, if you are used to using Charles, there is no problem, but there may be some differences in the operation of the tool, and the principle is completely the same.

The first step is to install Fiddler on your computer. It’s completely free to download.

www.telerik.com/fiddler/fid…

Once installed, you can log in and use it. It will automatically grab all network request packets from your current computer.

But if we want to grab network requests from our phones, we need to do a little extra configuration.

Click View -> Preferences -> Connections from the top Fiddler toolbar and you’ll see something like this:



There are two important points to note here. One is the port number. The default value is 8866.

Allow remote computers to connect Allow remote computers to connect Allow remote computers to connect

Check the second option, click SAVE, and the configuration on the computer side is complete.

Next we need to do some simple configuration on the mobile end.

Make sure your phone is on the same LOCAL area network as the computer you’re using to capture your bag.

Then modify the advanced option of the current Wifi connection of the mobile phone, change the proxy type to manual, change the proxy host name to the IP address of the computer, and change the proxy port to 8866, as shown below:

Click Save.

With that done, you can actually use Fiddler to capture network requests from your phone. You can try the following address on your phone’s browser:

guolin.tech/api/china/

You should see something like this:

Then check Fiddler and you will find that the network request packet on your phone has been successfully captured by Fiddler (sometimes Fiddler displays too much information about the packet, so you can use Ctrl+X to clear the information) :



You can see all the details of the network request in Fiddler, including the request header, the response header, the response body, and so on.

Packet capture tool will be behind the network communication details all moved to the table, so that when the client and server encounter a joint adjustment problem, in the end is the client pot or the server pot, look at the captured package will be all clear.

This is the most traditional use of the packet capture tool. However, this use is no longer effective because fewer and fewer network requests still use HTTP. Most network requests are HTTPS.

Capture HTTP requests

HTTPS is a network protocol that encrypts data transmission. Data is transmitted after encryption rather than in plain text.

This protocol ensures user data security, but it is a vexing thing for packet capture. Because the data was encrypted, and the packets we caught were ciphertext messages, there was no way to locate the problem.

For example, try accessing Bing in a mobile browser and viewing the package information captured in Fiddler, as shown below:



As you can see, Fiddler captures network requests to bing, but it doesn’t decrypt the actual transmission, which doesn’t help us analyze the problem.

So how do we capture the network packet of HTTPS request? Don’t worry, Fiddler supports this feature, so follow me through.

To enable HTTPS packet capture in Fiddler, click View -> Preferences -> HTTPS from the top toolbar.

On the HTTPS Settings page, click Trust Root Certificate to install the certificate, and then select Capture HTTPS Traffic, as shown below:



Click SAVE so you can grab the HTTPS request package on your computer.

However, we still can’t catch the HTTPS request package on the phone. You can try bing again on the phone and you will see the following screen:

This kind of error is basically caused by the certificate. In the next article, I will analyze the cause of this error in detail. In this article, we will solve it first.

Visit the following address in your mobile browser:

ipv4.fiddler:8866/

You’ll see a page built into Fiddler:

Click on FiddlerRoot Certificate to download and install Fiddler’s mobile phone certificate.

After the installation is complete, visit Bing again and you will find that no errors will be reported. Instead, the content of the web page will display normally:

Then looking at Fiddler, you can see that the network package requesting the bing home page has also been successfully captured, and this time it’s not ciphertext, but decrypted data:



HTTPS packet capture problem solved!

Capture Packets of Android applications

In this way, HTTPS packet capture does not seem to be a difficult task.

Yes, but there’s one more detail I need you to note. The above scheme only works for web requests in the browser. If you want to capture web requests from other applications, you still can’t catch them.

To verify this, let’s create a new application and write the simplest network request code to see if we can catch the network request it’s making.

The whole program is very simple. We add a button to MainActivity, and when the button is clicked, it initiates a network request. The code is as follows:

class MainActivity : AppCompatActivity() {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        val button: Button = findViewById(R.id.button)
        button.setOnClickListener {
        	sendRequest()
        }
    }

    private fun sendRequest() {
        thread {
            var connection: HttpURLConnection? = null
            try {
                val response = StringBuilder()
                val url = URL("https://www.bing.com")
                connection = url.openConnection() as HttpURLConnection
                val input = connection.inputStream
                val reader = BufferedReader(InputStreamReader(input))
                reader.use {
                    reader.forEachLine {
                        response.append(it)
                    }
                }
                Log.d("TAG", response.toString())
            } catch (e: Exception) {
                e.printStackTrace()
            } finally {
                connection?.disconnect()
            }
        }
    }

}

Copy the code

Yeah, that’s all there is to it. But don’t forget that we also have to declare network permissions in androidmanifest.xml:

<? The XML version = "1.0" encoding = "utf-8"? > <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.capturetest"> <uses-permission android: /> ... </manifest>Copy the code

Ok, now let’s run the program and see what happens. Clicking a button on the screen makes a web request to bing’s home page, and then watches the packets in Fiddler:



As you can see, we are not able to successfully catch and parse the HTTPS request package information as we did in the browser.

Why is that? This is because of a security update to Android in OS 7.0. As of Android 7.0, HTTPS requests cannot be captured only with the certificate of the capture tool installed on the phone, and a network security configuration must be added to the application code.

This upgrade makes every application more secure, because capturing HTTPS packets is really a risky behavior, and all encrypted transmitted data is displayed in clear text. Of course, this is a valid reason to capture packages for debugging purposes, but you should only capture and debug your own programs. If you can capture HTTPS requests from all apps as long as you have a certificate installed on the phone, the security of these apps will undoubtedly be greatly reduced.

That’s why this security update is available in Android 7.0. By default, you can’t capture HTTPS requests for individual apps. If you want to capture HTTPS requests for your own App, you can.

Create a network_security_config. XML file in the res/ XML directory and add the following configuration:

<? The XML version = "1.0" encoding = "utf-8"? > <network-security-config> <base-config cleartextTrafficPermitted="true"> <trust-anchors> <certificates src="user"/> <certificates src="system"/> </trust-anchors> </base-config> </network-security-config>Copy the code

The need in the AndroidManifest. XML configuration android: networkSecurityConfig attributes to make the configuration to take effect:

<application
    ...
    android:networkSecurityConfig="@xml/network_security_config">
    ...
</application>

Copy the code

This allows us to capture HTTPS network requests made by the current application.

Rerun the program and let’s try it again. The result looks like this:



It worked as we expected.

One last little puzzle

That’s about the end of this article.

Since Android 7.0 requires you to add network security configuration to your application in order to capture HTTPS requests, why did you manage to capture HTTPS requests in the first place with nothing configured in the browser?

This question puzzled me for a long time until I finally solved it when I joined the Microsoft Edge project.

Edge is a Chromium-based browser, as is Chrome, as are many of the major browsers. The answer has always been in the Chromium source, but I never had the courage to look at it before.

Let’s take a look at the Androidmanifest.xml file in the Chromium source code, and part of the code is shown below:



As you can see, the source of Chromium also joined a android: networkSecurityConfig configuration, so we continue to go in to see what is configured inside:



Isn’t this exactly what we configured in the Demo?

It turns out that the reason browsers don’t need to do any extra configuration to capture HTTPS requests is because it’s already configured in the Chromium source code and all browsers based on the Chromium kernel automatically have this capability.

If you want to see the Chromium source online, you can go to this address:

source.chromium.org/

Well, that’s all for this article. I believe that after reading this article, you will be on the usual network development and debugging work to produce a certain help.

Having solved the problem of how to use it, the next step is to understand the principle. In the next article, we will discuss why HTTPS, the supposedly secure protocol, can still be captured. I’ll see you next time.

On the other hand, if you want to learn about Kotlin and the latest on Android, check out my new book, Line 1, 3rd Edition.

Pay attention to my technical public account “Guo Lin”, there are high-quality technical articles pushed every working day.