HyperText Transfer Protocol (HTTP) is used to Transfer information between Web browsers and Web servers. It is at the application layer in TCP/IP. TCP/IP is divided into four layers: application layer, transmission layer, network layer and data link layer.

The purpose of layering is to decouple and dynamically replace intra-layer protocols

Each layer contains:

Application layer: communication activities (FTP, DNS, HTTP) when providing application services to users

Transport layer: data transfer between two computers in a network connection (TCP, UDP)

Network layer: Processes the packets flowing on the network and transmits them to the other party (IP) through the transmission path.

Data link layer: hardware-related network cards, device drivers, and so on

However, HTTP also has the following obvious disadvantages:

  1. Communications use clear text and can be eavesdropped
  2. The identity of the communicating party is not verified, so it is possible to encounter camouflage
  3. The integrity of the packet cannot be proved, so it may be tampered with

Here’s where HTTPS comes in. S in HTTPS stands for SSL or TLS, which adds a security layer for data encryption, decryption, and identity authentication on the basis of HTTP

  • HTTP + Encryption + Authentication + Integrity Protection = HTTPS

Encryption related preparatory knowledge: symmetric encryption and asymmetric encryption.

  1. Symmetric encryption: Encrypt and decrypt data using the same key. This encryption method is characterized by high speed, common symmetric encryption algorithm AES;
  2. Asymmetric encryption: Encryption and decryption use different keys that form a unique pair, called a public key and a private key. Data encrypted with a public key must be decrypted with a private key, and data encrypted with a private key must be decrypted with a public key. Generally speaking, the private key is kept by itself, and the public key is disclosed to others (generally, the public key will not appear alone, but will be written into the certificate), so that others can use their public key to encrypt data and send it to themselves, so that only themselves can decrypt the data. This encryption mode is characterized by low speed and high CPU overhead. RSA is a common asymmetric encryption algorithm.

CA certificates:

A CA certificate is a digital certificate issued by the Certification Authority (CA). It contains information about the e-visa authority, public key user information, public key, signature and validity period. Public key The public key of the server refers to the public key of the server. The public key of the server refers to the hash function used to calculate the summary of the public plaintext information, and the CA private key is used to encrypt the summary. The encrypted ciphertext is the signature. That is: certificate = public key + signature + applicant and issuer information. The client supports decryption signatures because the CA’s public key is preset in the operating system (the signature is encrypted using the CA’s private key).

With this in mind, you can take a look at how HTTPS achieves secure authentication.

HTTPS unidirectional authentication

Let’s take a look at the one-way authentication process:

(2) The server goes to the CA organization to apply for a CA certificate. As mentioned above, the certificate contains the server’s public key and signature. The CA certificate is sent to the client

(3) The client reads the plaintext information of the CA certificate, uses the same hash function to calculate a hash digest (hash purpose: to prevent the content from being modified), and then decrypts the signature with the CA’s public key (because the signature is encrypted with the CA’s private key), and compares the information digest in the certificate. If yes, the certificate is trusted and the server public key is extracted

(4) The client generates a random number (key F), encrypts this random number with the server B_ public key, and sends it to the server.

(5) The server uses its own B_ private key to decrypt the ciphertext, and obtains the key F

(6) The server and client use this key F to communicate in the subsequent communication process. Instead of asymmetric encryption, this is symmetric encryption from the start

HTTPS bidirectional authentication

The principle of bidirectional authentication is similar to that of unidirectional authentication. The unidirectional authentication client needs to authenticate the server. In bidirectional authentication, the server authenticates the client

(2) The server goes to the CA organization to apply for a CA certificate. As mentioned above, the certificate contains the server’s public key and signature. The CA certificate is sent to the client

(3) The client reads the plaintext information of the CA certificate, uses the same hash function to calculate a hash digest (hash purpose: to prevent the content from being modified), and then decrypts the signature with the CA’s public key (because the signature is encrypted with the CA’s private key), and compares the information digest in the certificate. If yes, the certificate is trusted and the server public key is extracted

(4) The client sends its own client certificate to the server, which contains the public key of the client: C_ public key

(5) The client sends the supported symmetric encryption scheme to the server for selection

(6) After the server chooses the encryption scheme, it uses the C_ public key obtained just now to encrypt the selected encryption scheme

(7) the client uses its own C_ private key to decrypt the selected encryption scheme, the client generates a random number (key F), and uses the server B_ public key to encrypt this random number to form ciphertext, which is sent to the server.

(8) The server and client use this key F to communicate in the subsequent communication process. Instead of asymmetric encryption, this is symmetric encryption from the start

Ps: Client certificates for bidirectional authentication can be self-signed certificates such as those generated by OpenSSL, including client. CRT and client.key. These two parts can be integrated into a P12 certificate.

Summary of HTTPS basic ideas

HTTPS uses the combination of symmetric encryption and asymmetric encryption to ensure secure data transmission. In short, the asymmetric encryption algorithm is used to generate, confirm, and exchange the final communication key, and then the final communication key is used for symmetric encryption in subsequent communication. The reason why asymmetric encryption is not complete is that asymmetric encryption requires a large amount of computation and affects communication efficiency.

Caught the principle

Even if HTTPS is secure, packets can be captured. The common packet grabber includes Charles, Fildder, etc.

The HTTPS packet capture method is used as a middleman to disguise the client as the server and the server as the client. In a nutshell:

  • Intercepts THE HTTPS request from the client and sends the REQUEST to the server disguised as a middleman client
  • The receiving server returns and sends the data content to the client disguised as a man-in-the-middle server with its own certificate.

The specific process is shown in the figure below:

Anti-packet capture policy

To prevent man-in-the-middle attacks, you can use the technology of SSL-pinning to reverse capture packets. The point of the man-in-the-middle attack can be found by forging a fake server certificate to the client, which the client mistakenly thinks is real. The solution is that the client also preset a server certificate, compared to know the true or false.

There are two ways of SSL-pinning: Certificate pinning and Public Key pinning.

  • Certificate lock need to be in the client code built-in accept only specify the domain name certificate, and do not accept the operating system or browser built-in CA root corresponds to any certificate, by the way of authorization and guarantee the uniqueness of the APP and the server communication and security, so the client and the server (for example, the API gateway) is the communication between can guarantee absolutely safe. However, all certificates issued by CA have validity issues. The disadvantage is that the certificate needs to be re-built into the APP after the certificate is renewed.
  • Public key lock The public key is extracted from the certificate and embedded in the client. The client compares the public key value with the server to verify the connection. When you create a certificate key, the public key remains unchanged (that is, the key pair remains unchanged) before and after the certificate renewal period, avoiding certificate validity period problems. This method is recommended.

Break through SSL-Pinning capture packages

In the opposite bound, one mountain is taller than the other. If you have a built-in certificate or public key, there will always be a function to compare the verification, so you can control the return of the function and let the verification pass. Here’s a classic twist on slL-pinning: Xposed+justTrustme module. This program is the use of JustTrustMe this Xposed module, what it does is to all kinds of known HTTP request library for verification certificate API Hook, so that whether it is a trusted certificate, verification results are returned to the normal state, so as to realize the effect of bypassing the certificate check. For more tips on breaking ssl-pinning, see: What to do when you can’t catch an APP request package when you’re writing a crawler. 【 译 文 】

Wei Wei learns Python