Cabbage Java self study room covers core knowledge

1. HTTPS

Hypertext Transfer Protocol Over Secure Socket Layer (HTTPS) is a Secure HTTP channel. In short, IT is the Secure version of HTTP.

That is, add SSL layer to HTTP, and SECURE Sockets Layer (SSL) is the basis of HTTPS security. Therefore, SSL is required for details of encryption.

HTTPS: it is a URI scheme (abstract identifier system) with the same syntax as HTTP: system. For secure HTTP data transfer. The URL indicates that it uses HTTPS, but HTTPS has a different default port than HTTP and an encryption/authentication layer (between HTTP and TCP). Originally developed by Netscape, the system provides authentication and encrypted communications, and is now widely used for security-sensitive communications on the World Wide Web, such as transactions and payments.

2. Introduction of SSL

SSL (Secure Socket Layer) was developed by Netscape to Secure data transmission over the Internet. SSL has three versions: SSL1.0, SSL2.0, and SSL3.0. It has been widely used for authentication and encrypted data transfer between Web browsers and servers.

SSL uses data encryption, authentication, and message integrity authentication mechanisms to ensure the security of data transmission on the network. SSL supports various application layer protocols. Because SSL sits between the application layer and the transport layer, it provides security for any application-layer protocol based on reliable connections such as TCP.

The SSL protocol is divided into two layers:

  • SSL Record Protocol (SSL Record Protocol) : Based on reliable transport protocols (such as TCP), it supports data encapsulation, compression, and encryption for high-level protocols.
  • SSL Handshake Protocol: Based on the SSL recording Protocol, it is used for identity authentication, encryption algorithm negotiation, and encryption key exchange between communication parties before data transmission.

There are three SSL security mechanisms:

  • Authentication mechanism:

Certificate-based authentication of server and client using digital signature method, where client authentication is optional.

  • Confidentiality of data transmission:

Symmetric key algorithm is used to encrypt the transmitted data.

  • Message integrity verification:

MAC algorithm is used to verify message integrity during message transmission.

3. Introduction of TLS

Transport Layer Security (Transport Layer Security), used to provide confidentiality and data integrity between two applications.

TLS 1.0 is a new protocol developed by the Internet Engineering Task Force (IETF). It is based on the SSL 3.0 protocol specification and is the subsequent version of SSL 3.0, which can be understood as SSL 3.1. It’s written to the RFC. This protocol consists of two layers: TLS Record and TLS Handshake. The lower layer is the TLS recording protocol, which sits on top of some reliable transport protocol, such as TCP.

4. Introduction of HMAC

Keyed-hash message Authentication code (HMAC), also known as HMAC.

Message authentication code (MAC) is a message authentication code (MAC) generated by a special calculation. It uses a password hash function and an encryption key to ensure data integrity and authenticate a message. It is defined in RFC2104 with the mathematical formula:

Among them:

  1. H for password hash function (such as MD5 or SHA-1)
  2. K is the secret key.
  3. M is the message to be authenticated
  4. K’ is another secret key derived from the original key K (Padding zero to the right if K is shorter than the hash function’s input block size; Hashes K if it is larger than the block size.)
  5. | | concatenated
  6. Student: ⊕ means XOR.
  7. Opad is an external fill (0x5C5C5C… 5c5C, a hex constant)
  8. IPad is internally filled (0x363636… 3636, a hexadecimal constant)

5. Similarities and differences between TLS and SSL

5.1. Main services provided by SSL/TLS protocols

  1. Authenticate users and servers to ensure that data is sent to the correct clients and servers;
  2. Encrypt data to prevent it from being stolen;
  3. Maintain data integrity and ensure that data is not changed during transmission.

5.2. Main differences between SSL and TLS protocols

  1. Version: The TLS record format is the same as the SSL record format, but the value of the version number is different. Version 1.0 of TLS uses the version number SSLv3.1.
  2. Packet identification code: SSLv3.0 and TLS have different MAC algorithms and MAC calculation ranges. TLS uses the HMAC algorithm defined by RFC-2104. SSLv3.0 uses a similar algorithm. The difference is that SSLv3.0 uses a join operation between the fill byte and the key, while HMAC uses an xOR operation. But they are equally safe.
  3. Pseudo-random functions: TLS uses a pseudo-random function called PRF to extend the key into blocks of data, which is a more secure way.
  4. Alarm code: TLS supports almost all SSLv3.0 alarm codes, and many additional alarm codes are defined in TLS. For example, decryption_failed, record overflow, unknown CA (Unknown_CA), and access_denied.
  5. Ciphertext and client certificates: SSLv3.0 differs slightly from TLS in that TLS does not support Fortezza key exchange, encryption algorithms, and client certificates.
  6. Certificate_verify and FINISHED messages: SSLv3.0 and TLS Use slightly different inputs to calculate MD5 and SHA-1 hash codes using CERTIFicate_Verify and FINISHED messages, but the security is similar.
  7. Encryption computing: TLS and SSLv3.0 compute master secret values in a different way.
  8. Padding: Padding bytes to be added before user data is encrypted. In SSL, the length of the filled data must be a minimum integer multiple of the length of the ciphertext block. In TLS, the length of the filled data can be any integer multiple of the length of the ciphertext block (but the maximum length of the filled data is 255 bytes), which prevents attacks based on packet length analysis.

5.3. Major enhancements to TLS

The main goals of TLS are to make SSL more secure and to make the specification of the protocol more precise and complete. TLS builds on SSL V3.0 with the following enhancements:

  1. A more secure MAC algorithm
  2. A tighter alert
  3. A clearer definition of “gray zone” norms

5.4. TLS for security improvement

  1. Use key hashing for message authentication: TLS uses key hashing for Message Authentication Code (HMAC), which ensures that records cannot be changed when they are transmitted over an open network such as the Internet. SSLv3.0 also provides keyed message authentication, but HMAC is more secure than the (message authentication code) MAC functionality used by SSLv3.0.
  2. Enhanced pseudo-random capability (PRF) : PRF generates key data. In TLS, HMAC defines PRF. PRF uses two hash algorithms to ensure its security. If either algorithm is exposed, the data remains safe as long as the second algorithm is not.
  3. Improved completed message validation: BOTH TLS and SSLv3.0 provide completed messages to both endpoints that authenticate the messages exchanged without being changed. However, TLS bases this completed message on PRF and HMAC values, which is also more secure than SSLv3.0.
  4. Consistent certificate handling: Unlike SSLv3.0, TLS attempts to specify the type of certificate that must be exchanged between TLS.
  5. Specific alert messages: TLS provides additional specific and additional alerts to indicate problems detected by either session endpoint. TLS also logs when certain alerts should be sent.

6. Basic operation process of SSL

SSL sits between the application layer and the transport layer and provides security for any application-layer protocol based on reliable connections such as TCP.

The SSL protocol itself has two layers:

  1. At the upper layer, SSL Handshake Protocol, SSL Change Cipher Spec Protocol, and SSL Alert Protocol are used.
  2. The bottom layer is SSL Record Protocol.

  • The SSL handshake protocol is an important component of the SSL protocol. It is used to negotiate the encryption suite (encryption algorithm, key exchange algorithm, and MAC algorithm) used in communication, securely exchange keys between the server and clients, and implement identity authentication between the server and client.
  • SSL Password change protocol: The client and server use the password change protocol to notify the peer. Subsequent packets are protected and transmitted using the newly negotiated encryption suite and key.
  • SSL warning protocol: Used to report alarm information to the peer end, including the severity and description of the alarm.
  • SSL recording protocol: blocks upper-layer data (such as SSL handshake, SSL password change, SSL warning, and application-layer protocol packets), calculates and adds MAC values, encrypts the data, and transmits processed record blocks to the peer end.

6.1. Basic operation process

  1. The client requests and verifies the public key from the server.
  2. The two parties negotiate to generate a “dialogue key”.
  3. The two sides use a “conversation key” to encrypt communication.

The first two are known as the handshake stage.

6.2. Detailed process of handshake stage

The “handshake phase” involves four communications, all of which are in clear text.

Asymmetric encryption is usually used for key exchange because of its slow speed. The two parties negotiate a key through the public key algorithm and then communicate with each other through symmetric encryption. Of course, to ensure data integrity, HMAC must be processed before encryption.

TLS handshake takes place after TCP connection is established. The handshake is actually a negotiation to negotiate some parameters and information required by the encryption and decryption protocol. The TLS handshake can be one-way authentication or two-way authentication. One-way authentication means that the server sends the certificate to the client, and the client verifies the validity of the certificate on the server. Common HTTPS websites such as Baidu, Sina, and Google are used for one-way authentication. Two-way verification is not only the client will verify the validity of the server, while the server will also verify the validity of the client, such as bank online banking login, alipay login transaction.

By default, SSL authentication is performed only on the server and is optional on the client. Here’s the flow chart:

6.2.1. The Client sends a request (Client Hello)

Clients (such as browsers) support different encryption and decryption algorithms. However, the same encryption and decryption algorithm must be used during TLS protocol transmission to ensure normal encryption and decryption of data. During the TLS handshake phase, the client first informs the server of the encryption algorithms it supports. Therefore, the client needs to send the list of locally supported Cipher Suites to the server. In addition, the client also needs to generate a random number, which on the one hand needs to be saved in the client, and on the other hand needs to be transmitted to the server. The random number of the client needs to be combined with the random number generated by the server to generate Master Secret.

To sum up, in this step, the client provides the following information to the server:

  1. Supported protocol Version (SSL Version), such as TLS 1.0.
  2. A client-generated random number (Client Random1) that is later used to generate the “conversation key”.
  3. Support Ciphers supported by the client, for example, RSA public-key encryption and DH encryption.
  4. Supported compression algorithms.
  5. Some SSL/TLS extensions (sever_name, etc.) supported.
  • Asymmetric encryption algorithm RSA: RSA was conceived in 1977 by Ron Riverst, Adi Shamir, and Leonard Adleman, all of whom were working at THE Massachusetts Institute of Technology, RSA is the initials of their surnames.

The security of RSA algorithm is based on the difficulty of factor decomposition of large numbers. As people have not found an effective method to factor decomposition of large numbers, RSA is still safe at present. RSA algorithm can be used for encryption and signature, and is the most widely used public key algorithm at present.

  • Diffie-hellman (DH) : a secure key exchange protocol, which allows two parties to create a key over an insecure channel without any prior information of the other party. This key can be used as a symmetric key to encrypt communication contents in subsequent communication.

  • SNI (Server Name Indication) Under the TLS protocol, the client can tell the host name of the server it is connecting to at the beginning of the handshake process. This allows the server to bind multiple certificates to the same IP address and TCP port number. And thus allows multiple secure HTTPS sites to be served on the same IP address, which is the same concept as virtual hosting.

6.2.2. Server Hello

From Server Hello to Server Done, some server-side implementations send each message individually, and some server-side implementations send each message together. Sever Hello and Server Done are headers with no content.

After receiving the Client Hello, the server needs to send its certificate to the Client. This certificate is a form of authentication for the server. For example, the client receives a data from someone claiming to be www.alipay.com, but how can it prove that the other party is legitimate Alipay? That’s what the certificate does. Alipay’s certificate proves that it’s Alipay and not TenPay. Certificates are e-certificates that need to be applied for and issued by a dedicated digital Certificate Authority (CA) after a very strict review. A private key and a public key are generated when a certificate is issued. The private key is kept by the server itself and cannot be disclosed. The public key is attached to the information of the certificate and can be made public. The certificate itself also comes with an electronic signature of the certificate, which verifies the integrity and authenticity of the certificate and prevents the certificate from being changed. In addition, the certificate has an expiration date.

If the certificate sent by the Server does not provide sufficient information (certificate public Key), the Server can also send a Server Key Exchange to the client. In addition, the Server also needs to verify the client for important confidential data to ensure that the data is transmitted to a secure and legitimate client. The server can send a Cerficate Request message to the client to Request the client to send a certificate to verify the client’s validity. For example, financial institutions tend to allow only authenticated customers to connect to their networks, and will provide a USB key containing a client certificate to a regular customer.

Just like the client, the server needs to generate a random number and send it to the client. Both clients and servers need to use these two random numbers to generate Master Secret.

Finally, the Server sends a Server Hello Done message to the client, indicating that the Server Hello message ends.

To sum up, at this step, the server responds with the following:

  1. Verify the version of the encrypted communication protocol used, such as TLS 1.0. If the browser does not match the version supported by the server, the server turns off encrypted communication.
  2. A server-generated Random number (Sever Random) that is later used to generate the “conversation key”.
  3. Confirm the encryption method used, such as RSA public key encryption.
  4. Server Certificate.
  5. Some SSL/TLS extensions supported.

This is one-way authentication. In case of two-way authentication, the Certificate Request needs to be sent before Sever Hello Done, indicating that the client needs to provide the Certificate. The content is as follows:

  1. The type of certificate that the client should provide
  2. List of certificates accepted by the server

6.2.3. Client Key Exchange

Client Key Exchange

Generate a random number. If RSA encryption is used, obtain the public key in the certificate and generate a random number. If the DH algorithm is used, the dual-shot DH encryption parameters and rules are used to generate random numbers.

In addition, the client uses change Cipher Spec to notify the server to use encrypted packets to transmit data. Before this, all information is in plaintext. Therefore, I can view the information through the Wireshark. At this stage, the client will use RSA algorithm to generate a 48-byte premaster secret, which contains the version number of TLS on the client, and will immediately stop sending data if there is an intermediary who intends to lower the TLS version.

The premaster secret is a secret key. Once intercepted, the final master secret can be calculated by combining the pseudo-random number transmitted in plaintext. Therefore, the key is encrypted through the public key certificate and sent to the server. The server decrypts the pre-primary key through the private key. In this case, both the client and server have the same pseudo-random number and pre-primary key.

Certificate Verify

If the Server needs to authenticate the client, after receiving the Server Hello message, the client needs to send the client certificate to the Server for the Server to authenticate the client.

The client then needs to check the server’s certificate, and if the certificate is not issued by a trusted authority, or if the domain name in the certificate is inconsistent with the actual domain name, or if the certificate has expired, a warning is displayed to the visitor, who can choose whether to continue the communication. If there is nothing wrong with the certificate, the client retrieves the server’s public key from the server certificate. Then, send the following three pieces of information to the server:

  1. A random number (Pre Master Secret). The random number is encrypted with the server’s public key to prevent eavesdropping.
  2. Change Chiper Spec, indicating that subsequent messages will be sent using an agreed encryption method and key.
  3. Notification indicating that the client handshake is over. This item is also the hash value of all the previously sent content for the server to verify.

The random number of the first item above is the third random number that occurs in the entire handshake phase. It is the result of the client using some encryption algorithms (e.g. RSA (Diffie-Hellman) generates a 48-byte Key called a PreMaster Secret, or PreMaster Key in many materials.

Dog250 explains why it is necessary to use three random numbers to generate “session keys” :

“Both the client and the server need random numbers so that the generated key is not the same every time. Because SSL certificates are static, it is necessary to introduce a randomness factor to ensure the randomness of the negotiated keys. For the RSA key exchange algorithm, the pre-master-key itself is a random number, plus the randomization in the Hello message, three random numbers through a key exporter finally exported a symmetric key. The existence of Pre Master is that SSL protocol does not trust each host to generate a completely random random number. If the random number is not random, pre Master Secret may be guessed, so it is not appropriate to only apply Pre Master Secret as the key. Therefore, it is necessary to introduce a new random factor, so it is not easy to guess the key generated by the client and server together with pre Master secret. One pseudo-random may not be random at all, but three pseudo-random will be very close to random. Every increase of one degree of freedom, the randomness will not increase by one.”

Change Cipher Spec

Change Cipher Spec is an independent protocol, which is a byte of data in a packet. It is used to inform the server that the client has switched to the Cipher Suite and is ready to use the Cipher Suite to encrypt data and transmit data.

After the transfer of the Change Chiper Spec, the client uses the previously negotiated encryption suite and Session Secret to encrypt a piece of Finish data and send it to the server. This data is used to verify the encryption and decryption channel just established by handshake before formal transfer of application data.

Encrypted Handshake Message

Encrypted Handshake Message: Generates Session Secret using determined encryption algorithm through the first two random numbers and the last random number (Pre Master Secret), encrypting the Message with the conversation Secret.

In the case of bidirectional authentication, the client also sends Certificate and Certificate Verify packets. Certificate is the client’s Certificate. Certificate Verify is the client’s signature information for all handshakes. The server verifies the certificate and information, and terminates the SSL/TLS connection if the certificate and information are incorrect.

6.2.4. Final response from the server (New Session Ticket)

New Session Ticket

In this packet, the server sends a new session ticket to the client, and notifies the client to use encryption to transmit data.

Session ticket is equivalent to a session in a common website. When the browser sends a request for the second time or the connection is disconnected due to an intermediate network reason, the session ID is carried in the Hello phase of the client, and the client is considered to be accessed by the same person instead of the certificate exchange phase. Reuse of this session ID is also an optimization point, which will be discussed at the end. Similarly, this data is encrypted, and the server can decrypt it to complete the handshake quickly. The field contains not only session ticket but also the lifetime of the ticket, such as the expiration time and length.

After receiving the PreMaster encrypted data from the client, the server uses the private key to decrypt the encrypted data and verify the data. Session Secret will also be generated in the same way as the client. After all is ready, A ChangeCipherSpec is sent to the client to inform it that it has switched to the negotiated ciphersuite state and is ready to encrypt data using ciphersuite and Session Secret. The server then uses Session Secret to encrypt a Finish message to the client to verify that the encryption and decryption channel established by the handshake was successful.

If the client and server can encrypt and decrypt the Finish message and verify the message correctly, the handshake channel has been established. Then, both parties can use the Session Secret generated above to encrypt and transmit data.

After receiving the encrypted packet from the client, the server decrypts and verifies the packet. If the packet succeeds, a session ticket is generated (for details, see the following). Then respond:

  1. Server transfer Change Chiper Spec
  2. The first Encrypted Handshake Message on the server

Change Chiper Spec

Change Chiper Spec notifies the client of the Change Chiper Spec sent by the client.

Encrypted Handshake Message

Encrypted Handshake Message has the same meaning as the first packet sent by the client and is used for packet authentication by the client.

6.3. Summary of operation process

After the TCP connection is established, the SSL client (which is also a TCP client) initiates the handshake by sending a ClientHello message containing its list of implementable algorithms and other required messages. The SSL server responds with a ServerHello, It determines the algorithm needed for this communication, and then sends its certificate (which contains its identity and its public key). After receiving this message, the Client generates a secret message, encrypts it with the public key of the SSL server, and sends it. After the SSL server decrypts it with its own private key, the session key negotiation succeeds, and both parties can use the same session key to communicate.

After the client determines the key and succeeds in authentication, the entire handshake process is basically complete. Subsequent packets are encrypted using the negotiated key.

Steps for HTTPS communication:

  1. The client sends packets for SSL communication. The packet contains the specified VERSION of SSL supported by the client and the list of encryption components (such as encryption algorithm and key length).
  2. The server replies with an SSL version and an encryption component. The server’s encryption component content is filtered from the received client encryption component.
  3. The server sends a packet containing a public key certificate.
  4. The server sends a packet to notify the client that the initial SSL handshake negotiation is complete.
  5. After the first SSL handshake, the client sends a packet as a response. The packet contains a random password string called pre-master secret used in communication encryption. The password string has been encrypted using the server’s public key.
  6. The client sends a packet and prompts the server to encrypt subsequent packets using the pre-master secret key.
  7. The client sent a Finished packet. Procedure The packet contains the overall checksum of all packets so far connected. Whether the handshake negotiation can be successfully completed depends on whether the server can decrypt the packet correctly.
  8. The server also sends a Change Cipher Spec packet.
  9. The server also sent a Finished packet.
  10. After exchanging Finished packets between the server and client, the SSL connection is established.
  11. Application layer protocol communication, that is, sending HTTP responses.
  12. Finally, the client disconnects. When the link is disconnected, the close_nofify packet is sent.

7. Session Hold (Session ID and Session Ticket)

I have to mention session ticket and session ID in detail. The simple understanding is that the session ID is a session id. If the session ID is carried in the next access, it is considered that the same person accesses the server. The session ID may not be identified because it is provided by the server for the first time. The session ID is the main information stored by the server, consumes server memory resources, and is difficult to expand. Redis or memcache is used to store and reuse Session ids.

Therefore, session ticket came into being. This string contains some information used in the session at that time, which can be quickly reused after decryption (RFC5077 explains this in detail). Session ticket is stored in the client. The server encrypts the session status using a ticket key that it knows and sends the session status to the client. The client saves the ticket and sends the session status to the server during the next connection. The problem with this mode is that all servers in the cluster use the same ticket key. Therefore, the rotation of the key and the compatibility between the old and new keys need to be considered.

Session cache configuration: ssl_session_cachessL_session_ticket_key

session ID

The Session Identifier is the Session ID generated during the TLS handshake. The server can store the negotiated Session ID, or the browser can store the Session ID and carry it with it in subsequent ClientHello handshakes. If the server can find a match, it can complete a quick handshake.

session ticket

The Session Identifier mechanism has some drawbacks:

  1. In load balancing, Session information is not synchronized between multiple machines. If two requests from the client do not land on the same machine, the matching information cannot be found.
  2. The server stores the information corresponding to the Session ID. The expiration time cannot be controlled. If the Session ID is too short, it takes up a lot of resources on the server.

Session Ticket can solve these problems. Session Ticket is the Session information encrypted with a security key known only by the server and saved in the browser. A browser with a Session Ticket in ClientHello can complete a quick handshake if the server successfully decrypts it.

8. CA certification bodies and certificates

Certificate Authority (CA) plays an important role in HTTPS. CA is the core implementation organization of PKI and a major component of Public Key Infrastructure (PKI). In a broad sense, certification center should also include certificate application Registration Authority RA (Registration Authority), it is the digital certificate application Registration, certificate issuance management organization. How does the client verify that the CA authority is valid? In fact, the root certificate of the CA center is embedded in the system browser, and the root certificate is the trusted authority, as shown in the figure:

8.1. Main responsibilities of CA

CA is an authoritative, trusted and impartial third-party organization that guarantees e-commerce, online banking, and Internet environment health.

  1. Generally speaking, CA center is responsible for the issuance and management of certificates.
  2. Verify and identify the identity of the certificate applicant, examine the credit degree of the certificate applicant, the purpose of applying for the certificate, the true reliability of the identity and other issues to ensure the accuracy of the binding of the certificate and the identity;
  3. Ensure the quality and security of the asymmetric key used by the CA to sign the certificate;
  4. Manage certificate information, certificate serial number and CA identification, ensure the uniqueness of certificate body identification, and prevent the repetition of certificate body name. Determine and check the validity period of certificates in the use of certificates, ensure that no expired or invalid certificates are used, ensure the safety of online transactions, publish and maintain the list of invalid certificates (CRL).

8.2. Work of certification bodies

The certification authority generates a pair of secret keys, public and private.

  • Public-key: a Public certificate is a legal document issued by a CA center. It can be distributed on the Internet and can be easily obtained by anyone. The extension of a public key certificate file is a difference in usage. The extension includes but is not limited to CRT, cer, key, der, and pem. A PEM file may contain a public key file and a private key file. The public key contains information such as the domain name issued to, company name, encryption algorithm, organization, and validity period.

  • Private key: the private key is usually called the so-called private key. The private key is produced at the same time when the CSR file is generated. The suffix is usually.

  • CSR (Cerificate Signing Request) The certificate request file is sent to the CA center for generating public keys. When the file is generated, basic information such as country code, company name, province and city, and administrator email address are filled in. The file can be generated online or through OpenSSL, for example:

openssl req -new -nodes -newkey rsa:2048 -keyout xxx.key-out xxx.csr
Copy the code

The generation of public key certificate includes two parts:

  • A digital signature
  • Server Public Key

The generation process of digital signature is as follows:

  1. Server public key, digital digest algorithm, digital fingerprint generation;
  2. The generated digital fingerprint is encrypted with the private key of the authentication authority to generate a digital signature.

8.3. How do I Verify the CA Certificate on a Client

The server sends the public key certificate to the client, and the client verifies the certificate to ensure the validity of the public key.

  1. The client takes out the public key of the authentication authority built in the mobile phone in advance.
  2. Using the public key of the certification authority to decrypt the digital signature in the public key certificate to obtain the digital fingerprint;
  3. The client digests the server public key of public key certificate to generate digital fingerprint.
  4. Compare the digital fingerprint generated by the client itself (Step 3) with the digital fingerprint decrypted (step 2). If they are consistent, the public key certificate is verified and the server can be trusted. If they are not, the certificate is wrong and may have been tampered with, prompting the browser to fail to establish an HTTPS connection. In addition, the system also verifies the validity time and domain name of the CA certificate.