In the previous article, WIREshark was used to debug HTTP requests and responses. The whole process of TCP connection and disconnection is described in detail. This article attempts to capture TLS packets using Wireshark and takes a look at the process of HTTPS requests and responses. 🌞

Too lazy to see the full text of the TLS process directly to the bottom of the picture explanation

Curl and Wireshark will be installed using curl and Wireshark. 😝

HTTPS is based on TLS. Wireshark cannot be used to decrypt information encrypted at the TLS layer without the target private key. Wireshark cannot be used to decrypt information encrypted at the TLS layer.

Curl curl curl curl curl curl curl

Then enter TLS to filter the request, and the suspected IP address is the target website

Enter the IP address, verify that it is the destination website httpbin, and then try to view the returned data information, because the RETURNED JSON data is not visible with TLS encryption

TLS uses the encryption algorithm of symmetric key generated by Diffie – Hermann key exchange. Therefore, a series of necessary information for key generation can be obtained before key decryption data can be generated.

Before you worry about how to solve this problem, first of all, let’s review a few knowledge points, generally understand the encryption algorithm and CA digital certificate and TLS encryption process:

A few essential points 📖

Encryption algorithm 🔒

In ancient times, symmetric encryption was used. This encryption and decryption key is the same, so it is extremely insecure. Once one of the keys is leaked, it will lead to the decryption of the encrypted file

Asymmetric encryption was invented later. The logic of this encryption mode is that the public key is used for encryption and the private key is used for decryption. One party uses the public key of the other party for encryption, transmits the ciphertext to the other party, and the other party uses the private key to decrypt the ciphertext. However, this encryption and decryption efficiency is low, and then derived from the hybrid encryption, which is a combination of asymmetric encryption and symmetric encryption.

Hybrid encryption first uses asymmetric encryption to generate symmetric keys, and then uses the symmetric keys for secure data transmission, which makes encryption and decryption much more efficient. Diffie Hermann, used in TLS, is one of them

As for the Defy-Hermann key exchange algorithm, it is briefly mentioned here that this algorithm has several characteristics:

  1. A key can be synthesized but cannot be decomposed
  2. The synthesized key can be further synthesized
  3. The resultant key is independent of the resultant order
  4. It is essentially a key generation algorithm

The algorithm’s encryption flow looks like this:

  • A and B want to exchange and generate symmetric keys
  • A sends P to B
  • A combines PSA with P and its private key SA
  • B uses P and its own private key SB to compose PSB
  • Both sides exchange PSA and PSB
  • A Use the private key SA and PSB to synthesize PSBSA
  • B Use private key SB and PSA to synthesize PSASB

In this way, the symmetric key PSASB and PSBSA are synthesized, and the two parties can use the symmetric key to encrypt the transmitted data 🔑

⚠️ The problem is that A sends P to B, how can B confirm that P is really from A, instead of tampering with P under man-in-the middle attack? This requires CA authentication. As long as there is A digital signature authenticated by CA and CA’s public key can verify that P is from A.

Digital certificate 📄

So here’s the CA certificate and briefly how it works:

If A website A wants to obtain A digital certificate, the process is as follows:

  • A Sends the website information and the public key PA to the CA certification authority
  • The CA authenticates the website information and encrypts it with its private key SCA to generate A digital signature, which is then sent back to A
  • If the digital signature received by USER A contains user A’s public key PA, other users can use the public key of the CA to authenticate and decrypt the digital signature and obtain the public key PA of the digital signature
  • Then other people can be sure that PA really came from A

In this way, A’s public key PA can be considered to be indeed from A, since it is authenticated and endorsed by A digital certificate issued by the CA. 🥂

⚠️ Another question is who will authenticate the CA public key? What if the CA’s public key is also maliciously replaced? So the CA of the CA is required to authenticate, so who is the CA of the CA’s public key authenticated? According to this logic, there is a root CA at the end, which is used to do the final authentication.

TLS encryption process 👷

Finally, let’s review the encryption process of TLS

TLS is established on the basis of TCP. Therefore, three TCP handshakes are required to establish a TCP connection, and then TLS is established

  1. Client Hello
    1. Client Hello packet: The Client supports different encryption algorithms. Therefore, the Client needs to send a Cipher Suite supported by the Client to the server and generate a random number to save the packet on the Client and send the packet to the service
  2. Server Hello
    1. ServerCertificate packet: After receiving the Hello message from the Client, the server sends a CA-authenticated digital certificate to the Client to identify the server. In addition, a random number is generated and stored on the server and sent to the Client
    2. Server Hello Done packet: Indicates that the Server declares the end of phase 1 handshake negotiation
    3. Optional: Certificate Request message: If necessary, the client is required to send a Certificate for identity authentication
    4. Optional: Server Key Exchange message: If the information provided by the CA certificate is insufficient, the Server sends supplementary information
  3. Client Finish
    1. Client Key Exchange packet: The Client receives and authenticates the CA digital certificate, and decrypts the CA public Key to obtain the server public Key. Client Key Exchange packets contain a random number, which is called pre-master Key /secret. A notification that subsequent messages are sent using a mutually agreed encryption method and key; There is also a negotiated HASH algorithm that computes the HASH value of all previous information content to provide server-side verification. The information is encrypted and sent to the server through the server’s public key
    2. ClientCipherSpec message: The packet notifies the server that the negotiated encryption algorithm is used to calculate the symmetric key for subsequent communication (that is, two random numbers and a third pre-master key/secret are used to calculate a symmetric key session key/secret).
    3. Finished packet: This packet contains the verification values of all packets connected to the Finished port, which are encrypted using the server public key
    4. Optional: ClientCertificate packet: If requested by the server, the client sends a CA digital certificate
    5. Optional: CertificateVerify packet: If the server requires a CA digital certificate, the server uses the HASH algorithm to compute a summary of the information sent from the server
  4. Server Finish
    1. The server decrypts the Finished packet sent by the client using the server private key
    2. ClientCipherSpec packet: a packet notifies the server that the session key/secret is calculated using a negotiated encryption algorithm for subsequent encrypted communication
    3. Finished packet: indicates that the TLS connection is established successfully
  5. After the TLS handshake succeeds, the communication is encrypted using the symmetric session key/secret

Above, we see that the two generated random numbers are calculated together with the pre-master key to generate the session key, which is realized by the Defy-Hermann key exchange algorithm mentioned above. After receiving the CA digital certificate, the client obtains the public key of the server for encryption, which also complies with the CA authentication process mentioned in the preceding section. Server private key decryption The client uses the server public key for encryption, as described in the asymmetric key section above. Therefore, TLS is a secure transmission technology that uses a variety of authentication and encryption. In addition, the purpose of using HASH computing, which was not mentioned above, is to prevent data tampering, which is the same principle as downloading software on the Internet and calculating MD5 verification.

⚠️ if there are incorrect places, everyone please point out in the comment area thank you

Finally, packet capture verifies the above encryption process 📦

As mentioned above, wireshark cannot see the decrypted data by capturing TLS packets directly. Therefore, we need to use some means to obtain the decrypted data

So how to decrypt data, there are articles can see jimshaver.net/2015/02/11/…

It turns out that Firefox and Chrome both support logging the symmetric session key used to encrypt TLS traffic to a file. You can then point Wireshark at said file and presto! decrypted TLS traffic.

The idea is that the browser will use an environment variable called SSLKEYLOGFILE that has already been set. Obtain the random number, preMasterSecret, and MasterSecret generated by each HTTPS connection from the client or server and save them in the file specified by the environment variable.

Configure the environment variable 🎀

To get started, here on Windows, we configure environment variables:

The configuration is complete, so that the browser writes the key information to the file at the specified path

How to set environment variables for macOS/Linux

Configuration wireshark 🦈

Open the Wireshark, and click The Configuration information to specify the pre-master key path for TLS

With that done, let’s try opening Chrome and accessing an HTTPS url:

The wireshark displays the decrypted data on the baidu homepage

Start packet capture analysis TLS handshake process 🤝

Then open the terminal and send an HTTPS request using curl

The wireshark uses httpbin as an example to filter HTTP.

Right click follow

This is the whole process of TCP establishment, TLS handshake, HTTP request response and TCP disconnection. Let’s analyze one by one:

As mentioned in the previous TCP article, we will not mention it here. Let’s look at TLS:

Client Hello

In the Client Hello phase, the Client sends a random number and all Cipher Suites supported by the Cipher Suites Client to the server

Server Hello

In the Server Hello phase, the Server sends a random number and a Cipher Suite to the client

The Server then sends the CA digital certificate and Server Key Exchange and Hello Done messages to the client to complete the first phase of the handshake:

Here is the certificate:

The Server Key Exchange negotiates an encryption algorithm:

This is Server Hello Done:

Client Finish

The Client sends Client Key Exchange, Change Cipher Spec, and Finished packets

Finished Verify Data Specifies the Finished Verify Data for all packets connected to this connection. The information is encrypted using the public key provided by the server

The client is ready to switch to symmetric key encryption

Server Finish

Finally, the Server returns a Change Cipher Spec and Server Finish

The server is ready to switch to symmetric key encryption

TLS handshake successful

In wireshark, the HTTP request response packet is displayed:

Photo explanation 🖼

Finally, I drew a simplified version of the brain map to make it easier to understand:

⚠️ if there are incorrect places, everyone please point out in the comment area thank you

Please pay attention to my subscription number, push technical articles about JS irregularly, only talk about technology not gossip 😊