1. OkHttp principle overview

OkHttp is an open source networking framework from Square that works with Retrofit, RxJava, or coroutines. OkHttp supports both synchronous and asynchronous requests. The synchronous request corresponds to RealCall, and the asynchronous request corresponds to AsyncCall, which is an inner class of RealCall. RealCall and AsyncCall can be understood as synchronous request operations and asynchronous request operations.

When a synchronous request is made using the Execute () method of a RealCall, it is placed on the request Dispatcher’s synchronous request action queue and then executed directly with the Dispatcher’s executed() method.

When an asynchronous request is made using RealCall’s enqueue() method, RealCall creates an AsyncCall and passes it to the Enqueue () method of the Dispatcher. The Dispatcher will place the asynchronous request on the asynchronous request operation queue and then submit the asynchronous request to the thread pool for execution. When the asynchronous request is executed, RealCall will initiate the request through the chain of interceptors. The interceptors in the chain process the request in the following order: Custom interceptor – Retry and Redirection interceptor – header fill interceptor – Cache interceptor – Connection interceptor – Custom network interceptor – Data exchange interceptor

The implementation of Dispatcher is relatively simple, mainly to make some judgment of the number of requests, for example, the default maximum number of requests from the same host is 5, and the maximum number of asynchronous requests at the same time is 64. When the value exceeds these two values, the asynchronous requests will not be immediately submitted to the thread pool.

OkHttp allows us to customize interceptors, which are the interceptors that execute first, and web interceptors, which process requests only after the connection is established, as well as web interceptors, which do not process WebSocket connections.

The RealInterceptorChain has a proceed() method. After each interceptor pair has finished processing its logic, it calls the proceed() method for the next interceptor to process the request.

In retry and redirection interceptors, the proceed() method is executed in a while loop and is wrapped in a try-catch block so that the retry and redirection interceptors can handle route and IO exceptions when other interceptors throw them. RetryAndFollowUpInterceptor redirection of logic is mainly in retry and redirect the interceptor followUpRequest () method, in this method will be according to the different response status code build redirect requests, such as a status code of 407, A request containing the authentication challenge is returned.

The next interceptor for retry and redirection interceptors is the BridgeInterceptor, which is responsible for filling in some request headers, such as taking the host component for the requested address and putting it in the host header. When the BridgeInterceptor completes the header that is required by default, it hands the request to the CacheInterceptor for processing.

OkHttp does not Cache requests and responses by default. If you want to Cache requests and responses, you need to create a Cache with the Cache directory and the Cache size. In the Cache, a DiskLruCachec is used. The Cache is set to OkHttpClient to Cache the response data. By default, the Cache interceptor only caches requests for methods such as GET and HEAD. If you want to modify methods such as POST or PUT, you need to customize the Cache interceptor.

The connection mechanism of OkHttp starts from the Intercept () method of the ConnectInterceptor. The connection mechanism can be divided into HTTP connection mechanism, HTTPS connection mechanism, and HTTP/2 connection mechanism.

The Intercept () method of the ConnectInterceptor calls RealCall’s initExchange() method to reuse or establish new connections. In the initExchange() method, ExchangeFinder’s find() method is called to find reusable connections or create new ones, The find() method calls the connection lookup method findConnection() indirectly through the findHealthyConnection() method.

In the findConnection() method, the first attempt is to reuse the RealCall’s existing connection, and when the RealCall has no connection, that is, the Connection field is empty, the attempt is to retrieve the connection from the connection pool. If there are no connections in the pool either, a new connection, RealConnection, is created and RealConnection’s connect() method is called to establish the connection. After the findConnection() method returns RealConnection, the find() method calls RealConnection’s newCodec() method to get and return ExchangeCodec, a data codec. RealCall’s initExchange() method gets Exchange Dec and uses it to create a data Exchange.

The core logic of RealConnection’s connect() method is executed in a while loop. If a tunnel is needed, the connectTunnel() method is called to pass through the client and server data. Otherwise, call the connectSocket() method to establish a connection with the server Socket, and then call the establishProtocol() method to establish a protocol, which involves the logic of HTTPS and HTTPS/2 connections.

In the connectSocket() method of RealConnection, a Socket is first created with the SocketFactory, Platform’s connectSocket() method is then called to establish a connection with the server Socket, and the input and output streams of the server Socket are initialized with their source and sink fields. After the connection is established, RealCall’s initExchange() method returns RealConnection to the ConnectInterceptor, The ConnectInterceptor then passes the request to the next interceptor, CallServerInterceptor.

In RealConnection establishProtocol(), it first determines whether the current request has the corresponding SSLSocketFactory, that is, whether the current request is an HTTPS request, if not, If H2_PRIOR_KNOWLEDGE exists in the protocol list, call startHttp2() and send an Upgrade message to check whether the server supports HTTP/2. The second thing the establishProtocol() method does is call connecTls() to start the SSL/TLS handshake. The third thing establishProtocol() does is to check if the RealConnection protocol is HTTP/2, and if so, call startHttp2() to establish an HTTP/2 connection.

Two things are done in the startHttp2() method, one is to create an Http2Connection, and the other is to call the start() method of Http2Connection to send a prelude message, SETTINGS frame, and WINDOW_UPGRADE frame. The Http2Connection start() method writes the preface message and the SETTING frame using the Http2Writer connectionPreface() and Settings () methods. The Http2Writer windowUpdate() method is then called to send a WINDOW_UPDATE frame, and the stream read task ReaderRunnable is queued for execution.

In the connectTls() method, an SSLSocket is created by calling SSLSocketFactory’s createSocket() method, The SSLSocket ConnectionSpec is then obtained using the configureSecureSocket() method of ConnectionSpecSelector. If the ConnectionSpec supports TLS extensions, Call Platform’s configureTlsExtensions() method to configure the TLS extension, and then call SSLSocket’s startHandshake() method to begin the TLS handshake. Then, HostNameVerifier is used to verify the validity of the host certificate and CertificatePinner is used to verify the validity of the host certificate. The input and output streams in the SSLSocket that successfully connects are used to initialize source and sink, and the afterHandshake() method of the Platform is called to end the handshake.

When the CallServerInterceptor receives a request, it uses Exchange to write the request header and body. Exchange writes the request information through the output stream provided by the Socket, and reads the response information through the input stream. When the CallServerInterceptor finishes reading the response, it passes it upward until it returns it to the place where the request originated.

2. OkHttp basic usage

The next version of OkHttp is 4.9.0, and the demo code is written in Kotlin.

First add the dependency.

Then create OkHttpClient and Request, and create a RealCall using OkHttpClient’s newCall() method. RealCall’s execute() method is then used to initiate a synchronous request, or a callback is passed into RealCall’s enQueue () method to initiate an asynchronous request.

3. Request information Request

Request contains information related to the Request, such as the Request method and the Request address and Request information. Let’s look at the use of these fields in Request.

3.1.1 Unified Resource Locator HttpUrl

Request converts the Request address we pass into the URL () function into an HttpUrl object. HttpUrl contains the protocol scheme, login information (username and password), host address, port number, query path, query parameters, and fragment identifier.

1. The agreement

When using a protocol scheme name such as HTTP: or HTTPS to access resources, specify the protocol type and add a colon (:) to the end.

2. Login information (authentication)

It is optional to specify a user name and password as the login information (authentication) necessary to obtain resources from the server.

3. The host

The host component identifies the host machine on the Internet that can access resources, such as www.xxx.com or 192.168.1.66.

4. The port number

The port component identifies the network port on which the server is listening, and the default port is 80 for HTTP, which uses the UNDERLYING TCP protocol.

5. Query the path

The local name of the resource on the server, separated from the previous URL component by a slash (/). The syntax of the path component is dependent on the server’s schema.

The path component describes where the resource is located on the server, similar to a hierarchical file system path, such as /goods/ Details.

6. Query parameters

For example, the database service can narrow the request resource scope by providing query parameters, passing in the page number and page size query list http://www.xxx.com/?page=1&pageNum=20.

7. Fragment identifiers

Fragment represents the name of a part of the resource. This field is not sent to the server, but is used internally by the client, separated from the rest of the URL by a hash sign (#).

3.1.2 Header field Headers

Header stores the HTTP Header. The Headers field has only one namesAndValues field of the type Array

. For example, the namesAndValues of addHeader(a, 1) is [A, 1].

HTTP request and response packets must contain an HTTP header, which provides information for the client and server to process the request and response respectively. An HTTP packet consists of the method, URI, HTTP version, and HTTP header fields.

3.1.3 RequestBody

RequestBody is an abstract class with the following three methods.

  1. ContentType contentType()

    Such as application/x – WWW – form – urlencoded;

  2. ContentLength contentLength()

  3. WriteTo ()

    Write the requested content to the Sink provided by Okio;

    There are also four extension methods for creating a RequestBody in xxx.toRequestBody(), such as map.toString ().torequestBody ().

3.1.4 label

We can tag the request with the tag() method and then do different things in the interceptor depending on the tag bar.

val request = Request.Builder()
    .url(...)
    .tag("666")
    .build()
Copy the code

In Retrofit, we use @Tag annotations, like this one.

@POST("app/login")
suspend fun login(
  @Query("account") phone: String.@Query("password") password: String.@Tag tag: String
) : BaseResponse<User>
Copy the code

In a custom interceptor, you can then retrieve the tag based on its type.

override fun intercept(chain: Interceptor.Chain): Response {
    val request = chain.request()
    val tag = request.tag(String::class.java)
    Log.e("intercept"."tag: ${tag}")
    return chain.proceed(request)
}
Copy the code

4. OkHttp request distribution mechanism

4.1 Requesting a Call Operation

After we create the Request Request, we create a RealCall object using OkHttpClient’s newCall() method and then invoke execute() to initiate a synchronous Request or enQueue () to initiate an asynchronous Request.

RealCall implements the Call interface, and is the only implementation class of this interface. RealCall is a bridge between the OkHttp application and the network layer, and this class exposes primitives of the high-level application layer. Connect, request, response, and flow. You can also think of RealCall as a synchronous request operation, while RealCall’s inner class AsyncCall is an asynchronous request operation.

The following are implementations of the two methods used in RealCall comparisons: execute() and enqueue().

1. Execute ()

When we call RealCall’s execute() method to initiate a synchronous request, an illegal status exception will be thrown if the request has already been executed, so catch the exception when making a synchronous request.

Execute () calls AsyncTimeout’s Enter () method to close the Socket or stream if the request has not been executed. AsyncTimeout has an internal WatchDog class that inherits Thread. AsyncTimeout blocks and wakes up the WatchDog Thread with Object.wait()/notify().

AsyncTimeout has two internal classes, SocketAsyncTimeout and StreamTimeout. When a request times out, the timeOut() method of AsyncTimeout is called. In SocketAsyncTimeout, The implementation of the timeOut() method is to close the Socket, while the implementation in StreamTimeout is to close the stream.

Executed () of the Dispatcher is called after the execute() method of RealCall calls the Enter () method to queue synchronous requests. Then call getResponseWithInterceptorChain () method to get response, after get the response, will let the Dispatcher removed the request from the synchronous operation in the queue.

2. Enqueue ()

RealCall’s execute() method creates an asynchronous request operation AsyncCall and hands it off to the Dispatcher.

AsyncCall implements the Runnable interface. When the Dispatcher receives AsyncCall, it adds AsyncCall to readyAsyncCalls. Then call your own promoteAndExecute() method.

After adding AsyncCall to the asynchronous request queue, the Dispatcher determines when to execute the asynchronous request, and when to execute the request task, it will submit it to the thread pool ExecutorService.

As with synchronous requests, the first thing to do in AsyncCall’s Run () method is to make AsyncTimeout enter the timeout logic, and then get the response with the interceptor chain.

The AsyncCall run() method calls the onResposne() Callback we set up when no exception is encountered, and the onFailure() method is called if an exception is encountered.

Whether the asynchronous request succeeds or fails, RealCall finally calls the Dispatcher’s Finished () method to remove the request from the running asynchronous request queue runningAsyncCalls.

The Dispatcher enqueue() method is implemented as follows.

4.2 Request the Dispatcher

The request Dispatcher doesn’t do much other than maintain three queues and a thread pool: asynchronous pending, asynchronous running, and synchronous running.

The Dispatcher enqueue() method first adds AsyncCall to the queue of pending requests and then finds other requests from the queue of pending and run requests that have the same host address as the current request. AsyncCall’s callsPerHost field. CallsPerHost represents the number of executed requests for the host address currently requested. Each time a request is executed for the same host address, the value of callsPerHost is incremented by 1. We can change the value of maxRequestsPerHost in Dispatcher if multiple requests are frequently made in our application and we do not request multiple host addresses. MaxRequetsPerHost Indicates the maximum number of concurrent requests for a host address at a certain time.

okHttpClient.dispatcher.maxRequestsPerHost = 10
Copy the code

MaxRequestsPerHost defaults to 5. If the number of requests for the host address does not exceed the maximum value, the Dispatcher will traverse the asynchronous request queue. The Dispatcher determines whether the number of asynchronous requests running exceeds the maximum number of maxRequests allowed for concurrent requests, which is 64 by default and can be modified if the number of asynchronous requests does not exceed the maximum number of requests for host addresses. The pending request is submitted to the thread pool for execution.

When synchronous or asynchronous request execution, RealCall is called getResponseWithInterceptorChain () method by request, in getResponseWithInterceptorChain () method, We first create a list of interceptors, then add interceptors to the list, then create a chain of interceptors using interceptors, RealInterceptorChain, and then call the proceed() method of the chain of interceptors.

OkHttp retry and redirection mechanism

1. Retry mechanism

Retry and redirect the interceptor is responsible for the request fails retry and redirect, in RetryAndFollowUpInterceptor intercept () method in the code is performed in a while, only when the retry condition there was, the request will be cut off, The interceptor does not set an upper limit on the number of retries. The maximum number of retries is 20. If you have special requirements, you can customize a retry interceptor and a redirection interceptor.

In the Intercept () method of retrying and redirecting interceptors, the Recover () method is called when a RouteException or IO exception is encountered while the request is being processed in a subsequent interceptor to determine whether to retry or not, and an exception is thrown if the request is not retried.

If any of the following conditions is met, no retry is performed.

  • The value of retryOnConnectionFailure for OkHttpClient is false

  • The request body cannot be sent again

    If the following conditions are met, the request body cannot be sent again.

    • Encountered in the process of request execution IO exception (not including Http2Connection thrown ConnectionShutdownException)

    • RequestIsOneShot () returns true, which defaults to false unless we override it ourselves.)

  • A fatal exception

    • Protocol exception ProtocalException

    • Socket timeout exception SocketTimeoutException

    • Certificate verification is abnormal. Procedure

    • SSL for abnormal SSLPeerUnverifiedException client-side validation

  • There are no more routes to retry

    More routes are likely to be retried only if the following two things happen

    • The OkHttpClient proxy is set up

    • The DNS server returns multiple IP addresses

private fun recover(a): Boolean {
    // The application layer forbids retries
    if(! client.retryOnConnectionFailure)return false

    // The request body cannot be sent again
    if (requestSendStarted && requestIsOneShot(e, userRequest)) return false

    // Fatal exception
    if(! isRecoverable(e, requestSendStarted))return false

    // There are no more routes to retry
    if(! call.retryAfterFailure())return false

    // Retry with a new connection and the same route selector
    return true
  }
Copy the code

RealCall enterNetworkInterceptorExchange () method is used to initialize a ExchangeFinder, ExchangeFinder role is to find a reusable connection, The implementation of ExchangeFinder will be covered later.

When ExchangeFinder is initialized, the Request is passed on to other interceptors for processing, and if an IO or route exception is encountered during this process, the rocover() method is called to determine whether the Request is recovered, and if not, an exception is thrown.

2. Redirection mechanism

If other interceptor when processing the current request without exception is thrown, then RetryAndFollowUpInterceptor intercept () method will determine whether a response (priorResponse) is empty, if it is not empty, The followUpRequest() method will be called to retrieve the redirection request after the new Response is created.

The followUpRequest() method builds a redirection request based on different response status codes. If the status code is 407 and the protocol is HTTP, it returns a request containing an authentication challenge that was obtained using an Authenticator. Authenticator has a authenticate() method and defaults to an empty implementation of NONE. If we want to replace it, we can call the Authenticator () method when creating OkHttpClient to replace the default empty implementation.

In addition to NONE, another implementation JavaNetAutheitcator is provided in Authenticator, with the corresponding static variable authenticator.javA_net_authenticator.

In JavaNetAuthenticator’s authenticate() method, the list of challenges in the response is retrieved, The Challenge list is generated after www-authenticate and proxy-Authenticate response headers are parsed.

3. Process the 3XX redirection status code

When the response status code is 300, 301, 302, 303, 307, 308, the followUpRequest() method will call buildRedirectRequest() to build the redirection request, The 3XX redirect status code either tells the client to use an alternate location to access the resource the client is interested in, or provides an alternate response instead of the content of the resource.

When a resource is moved, the server can send a redirection status code and an optional Location header to tell the client that the resource has been removed and where to find it now, so that the client can retrieve the resource at the new Location without disturbing the user.

6.OkHttp header padding mechanism

Retry and Redirection The interceptor only does what it needs to do when it encounters an exception or needs to be redirected during the request. When it receives the request, it passes it directly through the interceptor chain to the next interceptor, the BridgeInterceptor.

The BridgeInterceptor is called the first build interceptor because the information we set for the Request is missing part of the header. In this case, the BridgeInterceptor puts the missing header into the Request. Here is the header field that BridgeInterceptor adds to the request.

  • Content-type: indicates the media Type of the entity body
  • Content-length: size of entity body (in bytes)
  • Transfer-encoding: Specifies the transmission mode of the packet body
  • Host: indicates the server that requests resources
  • Connection: hop – by – hop header, Connection management
  • Accept-encoding: Indicates the preferred content Encoding
  • Cookie: local cache
  • User-agent: indicates information about HTTP client programs
class BridgeInterceptor(private val cookieJar: CookieJar) : Interceptor {

  @Throws(IOException::class)
  override fun intercept(chain: Interceptor.Chain): Response {

    val userRequest = chain.request()

    // Create a new request
    val requestBuilder = userRequest.newBuilder()

    val body = userRequest.body
    if(body ! =null) {
      // Content type
      val contentType = body.contentType()
      if(contentType ! =null) {
        requestBuilder.header("Content-Type", contentType.toString())
      }

      // Content length
      val contentLength = body.contentLength()
      if(contentLength ! = -1L) {
        requestBuilder.header("Content-Length", contentLength.toString())
        requestBuilder.removeHeader("Transfer-Encoding")}else {
        requestBuilder.header("Transfer-Encoding"."chunked")
        requestBuilder.removeHeader("Content-Length")}}// Pull the host component from the URL
    if (userRequest.header("Host") = =null) {
      requestBuilder.header("Host", userRequest.url.toHostHeader())
    }

    // jump to head
    if (userRequest.header("Connection") = =null) {
      requestBuilder.header("Connection"."Keep-Alive")}// Content encoding
    var transparentGzip = false
    if (userRequest.header("Accept-Encoding") = =null && userRequest.header("Range") = =null) {
      transparentGzip = true
      requestBuilder.header("Accept-Encoding"."gzip")}// Local cache
    val cookies = cookieJar.loadForRequest(userRequest.url)
    if (cookies.isNotEmpty()) {
      requestBuilder.header("Cookie", cookieHeader(cookies))
    }

    // User agent
    if (userRequest.header("User-Agent") = =null) {
      requestBuilder.header("User-Agent", userAgent)
    }

    val networkResponse = chain.proceed(requestBuilder.build())

    // ...
    }

    return responseBuilder.build()
  }

  // ...
}
Copy the code

Now let’s look at what these heads do.

1. Content-type: indicates the media Type of the entity body

Content-Type: text/html; charset-UTF-8

The header field content-type specifies the media Type of the object in the entity body. The field value is assigned as Type /subtype, such as image/ JPEG.

2. Content-length: specifies the size of the entity body

The content-Length header field indicates the size (in bytes) of the entity body. You cannot use the content-Length header field when encoding the entity body.

3. Transfer-encoding: Specifies the transmission mode of the packet body

Transfer-Encoding: chunked

The header field transfer-encoding specifies the Encoding mode used to transmit the main body of the packet. The HTTP/1.1 transmission Encoding mode is only valid for block transmission.

4. Host: indicates the server that requests resources

Host: www.xxx.com

The header field Host tells the server the Internet Host name and port number of the requested resource. The Host header field is a header field that must be included in the request according to the HTTP/1.1 specification.

5. Connection

HTTP allows a string of HTTP intermediate entities (proxies, caches, and so on) between the client and the ultimate source server. HTTP packets can be forwarded to the source server hop by hop from the client through these intermediate devices.

In some cases, two adjacent HTTP applications apply a set of options to the Connection they share, and the Connection header field has a comma-separated list of link labels that specify options for this Connection that will not be propagated to other connections. For example, Connection:close indicates the Connection that must be closed after the next packet is sent.

The Connection header can hold three different types of tags.

  • The HTTP header field name, which lists the headers that are only related to this connection;
  • Any label value that describes a nonstandard option for this connection;
  • Close: indicates that the persistent connection should be closed after the operation is complete.

In the BridgeInterceptor, when we don’t have a Connection header, the BridgeInterceptor passes a Connection header with a keep-alive value to open persistent connections, which we’ll talk about later.

6. Cookie

Two header fields related to cookies.

  • Set-cookie: Indicates the Cookie information used to start the state management
  • Cookie: Cookie information received by the server

Cookie: status=enable

The header field Cookie will tell the server that when the client wants to obtain HTTP state management support, it will include the Cookie received from the server in the request. When multiple cookies are received, they can also be sent in the form of multiple cookies.

In BridgeInterceptor, the cookie-related implementation is the CookieJar interface, which defaults to an empty implementation class. If we want to pass cookies to the server, We can pass in our own implementation by calling cookieJar() when we create OkHttpClient.

7. User-agent: information about HTTP client programs

The header field user-Agent will convey information such as the browser that creates the request and the name of the User Agent to the server. When the web crawler initiates the request, the email address of the crawler author may be added in the field. If the request passes through the Agent, the name of the proxy server may also be added in the middle.

In the BridgeInterceptor, when we don’t set user-Agent, the default UserAgent is okhttp: version number, which is user-Agent: okhttp:4.9.0.

OkHttp caching mechanism

When the BridgeInterceptor puts the header to the server into the Request, it hands the Request to the CacheInterceptor.

5.1 Handling Procedure of HTTP Cache

The basic caching of an HTTP GET packet consists of seven steps: receiving, parsing, querying, freshness detection, response creation, sending, and log creation.

Receiving is the caching of incoming request packets read from the network. The cache parses packets and extracts urls and headers. A query is a cache to see if a local copy is available, and if not, to get a copy and save it locally. Freshness detection is caching to see if the cached copy is fresh enough, and if not, to ask the server if there are new resources. Creating a response is when the cache constructs a response message from the new header and the cached body. Sending is when the cache sends the response to the client over the network. Creating a log means that the cache can create a log file entry describing the transaction.

CacheInterceptor generally follows these seven steps to handle caching, with some refinement.

The Cache Control header, cache-control, plays a major role in OkHttp’s Cache mechanism. You can manipulate the Cache mechanism by specifying cache-control instructions, which are optional and separated from each other, as shown in the following example.

Cache-Control: private, max-age=0, no-cache
Copy the code

5.2 Obtaining cache information

RealCall getResponseWithInterceptorChain () method, when creating CacheInterceptor will give CacheInterceptor cache of OkHttpClient field assignment, Null by default. If we want to use caching, we set the cache() method when creating OkHttpClient initialization, as shown below.

/** * Maximum value of network cache data (bytes) */
const val MAX_SIZE_NETWORK_CACHE = 50 * 1024 * 1024L

private fun initOkHttpClient(a) {
  valnetworkCacheDirectory = File(cacheDir? .absolutePath +"networkCache")

    if(! networkCacheDirectory.exists()) { networkCacheDirectory.mkdir() }val cache = Cache(networkCacheDirectory, MAX_SIZE_NETWORK_CACHE)

    okHttpClient = OkHttpClient.Builder()
        .cache(cache)
        .build()
}
Copy the code

It’s important to note that the CacheInterceptor only caches requests for methods that GET resources, such as GET and HEAD. It doesn’t cache requests and responses that modify resources, such as POST and PUT.

In the CacheInterceptor intercept() method, candidate caches are first fetched via cache.get (), whereas in the Cache get() method, the key is first fetched based on the requested address, The key of the Cache Snapshot is the URL value after md5 processing, and the Snapshot is the DiskLruCache value in the Cache. The Snapshot contains the input flow of the cached file.

When the get() method gets a snapshot, it creates an Entry from the snapshot’s input stream. In the Entry constructor, information about cached requests and responses is read from the input stream, and the input stream is closed after reading.

After creating an Entry, cache.get () determines whether the request address and method in the Cache match the current request and returns a response if they do, or closes the response body and returns NULL if they don’t.

5.3 Cache Policy CacheStrategy

After getting the candidate cache response, the CacheInterceptor produces a CacheStrategy using the compute() method of the cache policy factory, An important method in CacheStrategy is the isCacheable() that determines whether the current request and response are cached.

1. Cacheable response status code

In the isCacheable() method of CacheStrategy, the response’s status code is first determined to be “cacheable status code.”

To simplify the isCacheable() activity diagram, I call the following state codes “cacheable state codes”;

  • 200 OK
  • 203 Not Authoritative Information
  • 204 No Content
  • 300 Multiple Choices
  • 301 Moved Permanently
  • 308 Permanent Redirect
  • 404 Not Found
  • 405 Method Not Allowed
  • 410 Gone
  • 414 Request-URI Too Large
  • 501 Not Implemented
2. Cache judgment of the temporary redirection status code

When the response’s status code is 302 or 307, the isCacheable() method determines whether to return false (not cached) based on the response’s Expires header and cache-control header.

The Expires header allows the server to specify an absolute date, and if that date has already passed, the document is not “fresh.”

5.4 Obtaining a Response

When you create a CacheStrategy by calling the compute() method on the CacheInterceptor, if you have the onlyIfCached directive in CacheControl, The cacheResponse field in CacheStrategy is also empty.

When the onlyIfCached directive is present in CacheControl, it means that no other interceptor is used to get the response. In this case, the CacheInterceptor returns an empty response.

When the request is fresh (age less than fresh), the networkRequest field in CacheStrategy is empty, and the CacheInterceptor returns the response in the cache.

When a request is no longer fresh, the CacheInterceptor gets the response via ConnectInterceptor and CallServerInterceptor.

5.5 Saving response

Upon receiving the response, the CacheInterceptor determines whether the cached response is empty. If it is not, and the status code is 304 (unmodified), it replaces the cache in LruCache with a new response.

If the Cache response is empty, the response is saved to disk through cache.put (). After saving, if the request method is PATCH, PUT, DELETE and MOVE, the response is removed from the Cache.

8. OkHttp connection establishment mechanism

Having looked at the caching mechanism, let’s take a look at the ConnectInterceptor from OkHttp, which is responsible for establishing connections.

The ConnectInterceptor intercept() method does nothing but call RealCall’s initExchange() method to establish a connection.

In RealCall’s initExchange() method, exchangeFinder.find () is used to find reusable connections or create new ones, The ExchangeFinder.find() method returns a data codec, ExchangeCodec. ExchangeCodec encodes HTTP requests and decodes HTTP responses. Codec stands for coder-decoder.

Once RealCall gets an Exchange Dec, it creates a data Exchange with Exchange DEC, The next interceptor, the CallServerInterceptor, uses Exchange to write request messages and get response messages.

ExchangeFinder’s find() method calls its core findConnection() method. Before we look at the findConnection() method implementation, let’s learn a little about HTTP connections.

class ExchangeFinder(
  / /...
) {

  // ...

  fun find(client: OkHttpClient, 
           chain: RealInterceptorChain): ExchangeCodec {
    try {
      val resultConnection = findHealthyConnection(
        // ...
      )
      return resultConnection.newCodec(client, chain)
    } // ... 
  }

  @Throws(IOException::class)
  private fun findHealthyConnection(
    // ...
  ): RealConnection {
    while (true) {
      val candidate = findConnection(
        // ...
      )

      // ...}}}Copy the code

6.1 HTTP Connection Management

The HTTP specification clearly explains HTTP packets, but not much about HTTP connections, which are the file channels through which HTTP packets are transmitted. In order to better understand the problems that may be encountered in network programming, HTTP application developers need to understand the ins and outs of HTTP connections and how to use them.

Almost all HTTP traffic in the world is carried by TCP/IP, a common packet-switched network layer used by computers and network devices around the world.

A client application can open a TCP/IP connection to a server application that may be running anywhere in the world, and once the connection is established, messages exchanged between the client and server computers are never lost, corrupted, or out of order.

1. TCP/IP communication flow

When the TCP/IP protocol family is used to communicate with each other on the network, the sender goes from the application layer to the bottom, and the receiver goes from the link layer to the top.

Take HTTP as an example. First, the client as the sender sends an HTTP request to view a Web page at the application layer (HTTP protocol).

The sender splits the HTTP packets received from the application layer at the transport layer, marks the serial number and port number on each packet and forwards the packets to the network layer. The server at the receiving end receives the data at the link layer and sends the data to the upper layer in sequence until it reaches the application layer.

That is, when transmitting data between layers, the sending end types the header information of the layer after each layer, and the receiving end removes the corresponding headers during data transmission between layers. This method of wrapping data information is called encapsulate.

2. TCP socket programming

Operating systems provide some tools for manipulating TCP connections. The following are some of the main interfaces provided by the Socket API, which was originally developed for Unix operating systems but is now available in almost all operating systems and languages.

  • Socket () : creates a new, unnamed, unassociated socket;
  • Bind () : assigns a local port number and interface to the Socket;
  • Listen () : identifies a local Socket so that it can legally receive connections.
  • Accept () : Wait for someone to establish a connection to a local port;
  • Connect () : creates a connection between the local Socket and the remote host and port.
  • Read () : Attempts to read n characters from the socket into the buffer;
  • Write () : attempts to write n bytes from the buffer to the socket;
  • Close () : the TCP connection is closed completely.
  • Shutdown () : disables only the input or output of the TCP connection.

The Socket API allows users to create TCP endpoints and data structures, connect these endpoints to TCP endpoints on remote servers, and read and write data streams.

6.2 Releasing the Connection

With this knowledge of HTTP connections behind us, let’s look at the implementation of ExchangeFinder’s findConnection() method.

The findConnection() method does roughly three things. First, it tries to reuse existing RealCall connections. If there are no existing connections, it tries to retrieve connection reuse from the connection pool. Creates a new connection and returns it to the CallServerInterceptor.

In the findConnection() method, it first looks at whether to release the connection to the current RealCall. ExchangeFInder determines whether the Connection field of the RealCall is empty. If not, Indicates that the request was called and the connection was successfully established. ExchangeFinder determines if noNewExchanges in RealCall’s connection is true, which indicates that no new data exchange can be created. The default is false.

When a request or response has a Connection header with the value close, the value of noNewExchanges in Connection is changed to true. Since Connection:close means not reusing a Connection, if you forgot what the Connection header does, go back to the header interceptor in section 4.

When the connection’s noNewExchanges value is true, or the host and port number of the currently requested address are not the same as those of the host and port number in the connection, ExchangeFinder will call RealCall releaseConnectionNoevents () method attempts to release the connection, if if the connection is not released, it returns the connection, or closing the corresponding Socket connection.

A RealCall connection is of type RealConnection. A RealConnection maintains a Call list. Whenever a RealCall multiplexes this connection, RealConnection will add it to the list.

The operation of releasing the connection is to check whether there is a current RealCall in the Call list of the RealConnection. If there is a current RealCall, the current RealCall is removed from the list. Then the connection is released. Returns the current Call connection to the CallServerInterceptor.

6.3 Obtaining a Connection from a Connection Pool

When the RealCall connection is released, ExchangeFinder attempts to retrieve the connection from the RealConnectionPool, The two most important members of the RealConnectionPool are keepAliveDuration and Connection.

KeepAliveDuration specifies the duration of a keepAliveDuration. By default, the duration of a connection is 5 minutes. ConcurrentLinkedQueue specifies the duration of a connection.

Each time a connection is established, the pool initiates a connection cleanup task, which is run by the TaskRunner or, in DiskLruCache, the TaskRunner cleans the cache.

When a connection cannot be obtained from the connection pool for the first time, ExchangeFinder tries to select other available routes using the RouteSelector RouteSelector, passes these routes to the connection pool, attempts to obtain the connection again, and returns the connection when it is obtained.

6.4 Creating a New Connection

When two attempts from a connection pool connection fail, ExchangeFinder creates a new connection, RealConnection, then calls its connect() method and returns the connection.

class ExchangeFinder(
  private val connectionPool: RealConnectionPool,
  internal val address: Address,
  private val call: RealCall,
  private val eventListener: EventListener
) {
  @Throws(IOException::class)
  private fun findConnection(
    // ...
  ): RealConnection {
    // ..

    // Try to reuse an existing connection in RealCall
    val callConnection = call.connection 
    if(callConnection ! =null) {
      var toClose: Socket? = null
      synchronized(callConnection) {
        if(callConnection.noNewExchanges || ! sameHostAndPort(callConnection.route().address.url)) { toClose = call.releaseConnectionNoEvents() } }if(call.connection ! =null) {
        check(toClose == null)
        return callConnection
      }

      // ...
    }

    // Try to get a connection from the connection pool
    if (connectionPool.callAcquirePooledConnection(address, call, null.false)) {
      val result = call.connection!!
      eventListener.connectionAcquired(call, result)
      return result
    }

    // ...

    Create a new connection
    val newConnection = RealConnection(connectionPool, route)
    call.connectionToCancel = newConnection
    try {
      newConnection.connect(
          // ...)}finally {
      call.connectionToCancel = null
    }
    call.client.routeDatabase.connected(newConnection.route())

    // ...

    // Return the connection
    return newConnection
  }

  // ...

}
Copy the code

6.5 connect the Socket

In RealConnection’s connect() method, it first determines whether the current connection is connected, that is, whether the connect() method has been called, and if so, it throws an illegal status exception.

If no connection has been made, the request is determined to use HTTPS. If yes, the request is connected to the tunnel, and if not, the connectSocket() method is called to connect to the Socket.

We’ll talk about connection tunneling later when we talk about HTTPS, but let’s look at the implementation of the connectSocket() method.

In RealConnection’s connectSocket() method, the proxy method is first determined. If the proxy method is DIRECT or HTTP proxy, the Socket factory is used to create the Socket. Otherwise, use Socket(proxy) to create sockets.

After the Socket is created, RealConnection calls Platform’s connectSocket() method to connect to the Socket and initializes the Source and Sink used to exchange data with the server.

Platform’s connectSocket() method calls the Socket’s Connect () method.

6.6 Establishing a Protocol

Once the Socket is created, RealConnection’s Connect () method calls establishProtocol() to establish the protocol.

In the establishProtocol() method, if HTTP is used, it determines whether HTTP/2 (rfc_7540_34) is started based on prior knowledge, which means that the client knows that the server supports HTTP/2, No upgrade request is required. If HTTP/2 is not initiated prior, set the protocol to HTTP/1.1.

OkHttpClient’s default protocols are HTTP/1.1 and HTTP/2. If we already know that the server supports plaintext HTTP/2, we can change the protocol to the following.

val client = OkHttpClient.Builder()
    .protocols(mutableListOf(Protocol.H2_PRIOR_KNOWLEDGE))
    .build()
Copy the code

If the request is HTTP, the establishProtocol() method will call the connectTls() method to connect to TLS, or initiate the HTTP/2.0 request if the HTTP version is HTTP/2.0.

9. HTTPS connection establishment mechanism

Before we look at the implementation of the connectTls() method, let’s look at some of the basics of HTTPS. If you already know, you can skip this section and start with section 8.2.

9.1 HTTPS Basics

In HTTP mode, search or access requests are transmitted in plaintext and pass through middleman channels such as proxy servers, routers, WiFi hotspots, and service operators, making it possible for “middlemen” to obtain data and tamper with data.

However, upgrading from HTTP to HTTPS is not as simple as enabling Web servers to support HTTPS, but also requires consideration of CDN, load balancing, reverse proxy servers, which devices to deploy certificates and private keys, and involves changes in network architecture and application architecture.

9.1.1 Man-in-the-middle Attack

Now let’s take a look at what man-in-the-middle attacks are. There are two types of man-in-the-middle attacks: passive and active.

A middleman is an invisible hand in the communication between the client and the server. The client and the server are unaware of the existence of the middleman and have no way to defend against it.

1. Passive aggression

It is more and more popular for mobile devices, and the charge of mobile data is very expensive, so many users will choose to use WiFi, especially in the outdoor, users try to use free WiFi.

Many attackers offer free WiFi, and once connected to a malicious WiFi network, users have no privacy. Provide WiFI network an attacker can intercept all the HTTP traffic, and HTTP traffic is clear, the attacker can know the user’s password, bank card information, and browsing habits, there is no any analysis needed to get the user privacy, and users don’t know your information has been leaked, this attack method is also called passive attack.

2. Be aggressive

When many users browse a web page, they often find that an advertisement pops up on the page, and this advertisement has nothing to do with the web page they visit. This kind of attack is mainly sent by ISP (Internet Service Provider), and users cannot defend against it.

When users visit a website, they must pass through an ISP. In order to obtain advertising fees, THE ISP inserts a piece of HTML code into the response, which leads to the attack. This attack is called active attack, that is, the attacker knows the existence of the attack.

More seriously, isPs or attackers insert malicious JavaScript scripts into the page, which can have worse consequences once the script runs on the client side, such as XSS attacks (Cross Site Scripting).

9.1.2 Handshake Layer and Encryption Layer

HTTPS (TLS/SSL protocol) is cleverly designed and consists of two layers: the handshake layer and the encryption layer. The handshake layer is on top of the encryption layer and provides the information needed for encryption (key blocks).

For an HTTPS request, HTTP messages are not transmitted to the encryption layer until the handshake is complete. Once the handshake is complete, all HTTP messages at the application layer are transmitted to the key layer for encryption.

1. Shake hands layer

The client and server exchange some information, such as protocol version number, random number, and password suite (combination of cryptography algorithms). After negotiation, the server determines the password suite to be used for this connection, which must be agreed by both parties.

After the client uses the certificate sent by the server to confirm its identity, the two parties start key negotiation. Finally, the two parties negotiate the primary key, primary key, and key block. With the key block, the confidentiality and integrity of subsequent application layer data can be protected.

2. Encryption layer

The encryption layer has the key block provided by the handshake layer to protect confidentiality and integrity. The logic of the encryption layer is relatively simple, but before the handshake layer is completed, the client and server need to go through multiple rounds to complete the handshake, which is the reason why TLS/SSL protocol is slow.

The following is the flow chart of TLS protocols using RSA cipher suite and DHE_RSA cipher suite respectively.

9.1.3 handshake

Handshake refers to the key steps and concepts of the handshake between the client and server before data is transmitted to each other. Data encryption and integrity can be processed only after the agreement is reached.

1. Certification

Before exchanging keys, the client must authenticate the identity of the server. Otherwise, man-in-the-middle attacks may occur. The server entity cannot prove itself, so it needs to be authenticated by the CA.

The technical solution of authentication is the signed digital certificate. The certificate describes the digital signature algorithm adopted by the CA. After obtaining the certificate, the client uses the corresponding signature algorithm to verify the certificate.

2. Password suite negotiation

Password suite is one of the most important concepts in TLS/SSL. To understand the password suite is equivalent to understanding the TLS/SSL protocol. The client and server need to negotiate a password suite that is agreed by both sides. The password suite determines the encryption algorithm, HMAC algorithm, and key negotiation algorithm used to connect the client and server.

Cipher suite of negotiation process is similar to customer purchase process, the customer (client) in front of the shop (server) to the businessman to tell your needs, budget, dealer stores after understanding the needs of customers, recommend products to users according to the customer and the circumstances, only when both sides are satisfied, the transaction to complete.

For TLS/SSl protocol, the next step can be carried out only after the password suite is negotiated.

HTTP does not have a handshake process. To complete an HTTP interaction, the client and server need only one request/response to complete.

In an HTTP interaction, the client and the server need multiple interactions to complete, and the interaction process is negotiation. The lactation client tells the server the password suite it supports, and the server chooses a password suite supported by both sides.

The following figure shows the composition of a password suite.

9.1.4 encryption

Compared with the handshake layer, the processing of the encryption layer is relatively simple. The handshake layer negotiates the algorithm required by the encryption layer and the corresponding key block of the algorithm. The encryption layer performs the encryption operation and integrity protection.

In TLS/SSL, there are three common encryption modes: stream password encryption, packet encryption, and AEAD.

9.1.5 TLS/SSL Handshake Protocol

The values of Security Paramters in THE TLS recording protocol are filled in by the TLS/SSL handshake protocol. The corresponding values are negotiated by the client and server, and are unique.

For a full handshake session, the client and server go back and forth several times to negotiate encryption parameters.

The most associated with encryption parameters is the password suite. The client and server will list the supported password suite and then select a password suite supported by both sides to negotiate all encryption parameters based on the password suite. The most important encryption parameter is the master secret.

Before I go into the process, there are a couple of points.

The handshake protocol consists of many sub-messages, and for a full handshake, the client and server typically go back and forth two times to complete the handshake.

ChangeCipherSpec is not part of the handshake protocol and can be understood as a child message of the handshake protocol.

The asterisk (*) indicates whether the corresponding submessage is sent or not, depending on the cipher suite; for example, the ServerKeyExchange submessage does not appear in the RSA cipher suite.

In HTTPS, both the server and client can provide certificates for identity verification.

The following is a complete TLS/SSL handshake process.

The main steps of the handshake protocol are as follows:

They exchange Hello sub-messages with each other. The messages exchange random values and the supported password suite list, negotiate the password suite and the corresponding algorithm, and check whether the session can be recovered.

Exchange certificates and cryptographic information, allowing server and client to verify each other’s identity;

Exchange the necessary cryptographic parameters, and the client and server get a consistent primary key;

Generate the master key by preparing the master key and server/client random value;

The handshake protocol provides encryption parameters (mainly password blocks) to the TLS record layer protocol;

The client and server verify the Finished messages of each other to prevent the Finished handshake messages from being tampered.

9.1.6 extension

With extensions, both client and server sides can gain additional capabilities without updating the TLS/SSL protocol.

In RFC 5246, only some conceptual frameworks and design specifications are defined for extensions. The detailed definition of extensions is defined in RFC 6066. Each extension is registered and managed by IANA.

The extension works as follows:

  • The Client sends multiple extensions to the server according to its own requirements. The extension list message is contained in the Client Hello message.
  • The Server parses the extensions in the Client Hello message according to the RFC definition and returns the same type of extensions in the Server Hello message.

9.1.7 Session Recovery Based on Session Ticket

SessionTicket solves the disadvantages of Session ID Session recovery and is a better Session recovery mode. The SessionTicket processing standard is defined in RFC 5077. In TLS/SSL protocol, SessionTicket recovers sessions using TLS extensions. The implementation of the SessionTicket extension is defined in RFC 4507.

Session tickets are especially good if you have the following problems.

Session ID The Session information is stored on the server, which consumes a large amount of memory for large HTTPS websites.

HTTPS providers want the lifetime of the session message to be longer and use short handshakes as much as possible.

The HTTPS website provider expects Session information to be accessible across hosts, but Session ID Session recovery does not.

Embedded servers do not have much memory to store session information.

9.1.8 SessionTicket Interaction Process

SessionTicket From an application perspective, the mechanism is simple. The server encrypts session information and sends it to the client in the form of a ticket. The server does not store session information.

The client receives the ticket and stores it in memory. If the client wants to resume the session, it sends the ticket to the server at the next connection. After the server decrypts the ticket, if it is confirmed, the session can be resumed, which completes a short handshake.

SessionTicket has two changes relative to the Session ID:

  • Session information is saved by the client

  • Session information needs to be decrypted by the server

    The client does not participate in the decryption process, only responsible for storage and transmission;

SessionTicket can be implemented in many different ways, as described below.

1. Perform a complete handshake based on the SessionTIcket

If the server is expected to support SessionTicket session recovery for a new connection, the Client Hello message contains an empty SessionTicket TLS extension.

If the Server supports SessionTicket session recovery, the Server Hello message of the Server must also contain an empty SessionTicket TLS extension.

The server encrypts session information, generates a ticket, and sends the ticket in the Newssession Ticket sub-message, which is an independent sub-message of the handshake protocol. Since it is a full handshake, other sub-messages are processed as well.

After receiving the Newssession ticket message, the client stores the ticket for future use.

2. Perform a short handshake based on SessionTicket

The session recovery process based on SessionTicket is as follows.

  1. The Client stores a ticket. If the session is resumed, the Client Hello message contains a non-empty SessionTicket TLS extension.
  2. After receiving a non-empty ticket, the Server performs decryption verification on the ticket. If the ticket can be recovered, the Server sends an empty SessionTicket TLS extension in the Server Hello message.
  3. Since it is a short handshake, sub-messages such as Certificate and ServerKeyChange are not sent. Next, a Newssession ticket sub-message is sent to update the ticket, which also has a validity period.
  4. The client and server verify the Finished message, indicating that the handshake is complete and the session is successfully restored.

9.2 Connecting a Tunnel

1. The tunnel

A tunnel is an HTTP application that blindly forwards raw data between two connections after an establishment. An HTTP tunnel is usually used to forward non-HTTP data over one or more HTTP connections without snooping data.

A common use of HTTP tunneling is to carry SSL (encrypted Secure Sockets Layer) traffic over an HTTP connection so that SSL traffic can pass through a firewall that only allows Web traffic.

The HTTP/SSL tunnel receives an HTTP request to establish an output connection to the sum of the destination ports, and then tunnels encrypted SSL traffic over the HTTP channel so that it can be blindly forwarded to the destination server.

2. connectTunnel()

RealConnection’s connect() method first determines whether the current connection is connected, that is, whether the connect() method has been called, and if so, throws an illegal status exception.

If there is no connection, then determine whether the URL uses HTTPS scheme, if yes, then connect to the tunnel.

After RealConnection calls connectTunnel(), connectTunnel() calls connectSocket() and createTunnel() to create the Socket and tunnel.

9.3 Creating a Tunnel

In the createTunnel() method of RealConnection, Http1ExchangeCodec is created, used to write the request line, flushed the buffer, and read the response header.

If the status code is 200, the server sends the ServerHello message after the client sends the ClientHello message.

If the status code is 407, proxy authentication is required. In this case, Authenticator’s Authenticate () method is used to create and return an authentication request.

4 Obtain connection specifications

The connectTls() method was mentioned earlier, so let’s look at the implementation of this method.

In the connectTls() method, an SSLSocket factory is first used to create SSLSocket. SSLSocket is a secure socket that uses TLS/SSL protocols and adds a layer of security over basic network transport protocols such as TCP. The implementation of SSLSocket will not be covered in this article.

After SSLSocket is created, the connectTls() method calls ConnectionSpecSelector’s create ConnectionSpec, ConnectionSpec, ConnectionSpec includes the password suite, TLS versions, and whether TLS extensions are supported.

The default connection specifications in OkHttpClient are MODERN_TLS and CLEARTEXT. MODERN_TLS supports TLS extensions with TLS versions 1.2 and 1.3. The list of cipher suites corresponds to the array APPROVED_CIPHER_SUITES, which is similar to the cipher suites used by Chrome 72. CLEARTEDX indicates that it is plain text, does not support TLS extension, and does not have TLS version and password suite.

If we want to change the cipher suite or the TLS version to use, all we need to do is set it up with the connectionSpecs() method when creating OkHttpClient.

When the URL scheme is HTTPS, the connection specification is MODENR_TLS.

9.5 the TLS expand

1. ALPN extension

ALPN (Application Layer Protocol Negotiation), HTTP has two versions, namely HTTP/1.1 and HTTP/2. When a user enters a URL in the browser, when the browser connects to the server, It is not known whether the server supports HTTP/2.

To ask the Server if it supports a particular application layer protocol, the ALPN extension appears. The Client sends the extension in a Client Hello message, and responds to it in a Server Hello message once the Server supports HTTP/2. This allows HTTP/2 to be used uniformly on both the client and server sides.

2. Configure TLS extensions

RealConnection’s connectTls() method, upon receiving the ConnectionSpec, determines whether the TLS extension is supported by the ConnectionSpec. If so, Call the configureTlsExtentions() method for a specific Platform to configure the TLS extension.

For example, Android10Platform calls the configureTlsExtrentions() method of Android10SocketAdapter, Android10SocketAdapter SSLSocket is set to useSessionTicket (useSessionTickets) and then enable the ALPN extension.

After configuring the TLS extension, RealConnection initiates the TLS/SSL handshake by calling the startHandshake() method of SSLSocket. SSLSocket is an abstract class, The default startHandshake () implementation in ConscryptFileDescriptorSocket, The ConscryptFileDescriptor eventually calls the SSL_do_handshake() method provided by OpenSSL.

9.6 Certificate Locking

After calling SSLSocket’s startHandshake() method, RealConnection creates a Handshake object that contains the peer certificate, which for our client is the server-side certificate.

Create Handshake, RealConnection will use certificate lock CertificatePinner to check whether the peer certificate matches the PIN we set. If so, initialize the Source and Sink used for data exchange.

CertificatePinner locks the certificate that the client trusts through its public key. In the Check () method of CertificatePinner, The peer certificate (PeerCerfcertificates) is converted to an X509 certificate (X509Certificate) and determines whether the hash of the peer certificate’s public key is the same as the hash of Pin. The default password is not locked. If you want to lock the peer certificate, do as follows:

First configure an incorrect hash value.

val certificatePinner = CertificatePinner.Builder()
        .add(
            "publicobject.com"."sha256/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
        .build()

val client = OkHttpClient.Builder()
                .certificatePinner(certificatePinner)
                .build()
Copy the code

You will then see information containing the public key hash value of the server side certificate, as shown below (if the certificate is configured on the server side).

javax.net.ssl.SSLPeerUnverifiedException: Certificate pinning failure!
   Peer certificate chain:
     sha256/afwiKY3RxoMmLkuRW1l7QsPZTJPwDS2pdDROQjXw8ig=: CN=publicobject.com, OU=PositiveSSL
     sha256/klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=: CN=COMODO RSA Secure Server CA
     sha256/grX4Ta9HpZx6tSHkmCrvpApTQGo67CYDnvprLg5yRME=: CN=COMODO RSA Certification Authority
     sha256/lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=: CN=AddTrust External CA Root
   Pinned certificates for publicobject.com:
     sha256/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
   at okhttp3.CertificatePinner.check(CertificatePinner.java)
   at okhttp3.Connection.upgradeToTls(Connection.java)
   at okhttp3.Connection.connect(Connection.java)
   at okhttp3.Connection.connectAndSetOwner(Connection.java)
Copy the code

You can then add these hashes to a CertificatePinner as pins.

val certificatePinner = CertificatePinner.Builder()
    .add("publicobject.com"."sha256/afwiKY3RxoMmLkuRW1l7QsPZTJPwDS2pdDROQjXw8ig=")
    .add("publicobject.com"."sha256/klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=")
    .add("publicobject.com"."sha256/grX4Ta9HpZx6tSHkmCrvpApTQGo67CYDnvprLg5yRME=")
    .add("publicobject.com"."sha256/lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=")
    .build()
Copy the code

10. HTTP/2 connection establishment mechanism

10.1 HTTP/2 Basics

HTTP/2 is used to address HTTP/1 performance issues. The following concepts are added to HTTP/2:

  • Binary protocol
  • multiplexing
  • Flow control function
  • Data flow priority
  • The first compression
  • Server push

HTTP/2 has many similarities with HTTPS in that they encapsulate standard HTTP messages in a special format before they are sent and unpack the response upon receipt, so while clients and servers need to understand the details of sending and receiving messages, upper-layer applications don’t discriminate between versions. Because they use similar HTTP concepts.

1. Binary format

One of the main differences between HTTP/1 and HTTP/2 is that HTTP/2 is a binary, datagrams based protocol, while HTTP/1 is entirely text-based. Text-based protocols are easy for humans to read, but difficult for machines to parse.

With a text-based protocol, the request is sent and the response is received before the next request begins.

HTTP/2 is a fully binary protocol. HTTP messages are sent in clearly defined data frames. All HTTP/2 messages are encoded using chunking, which is standard behavior and does not need to be explicitly set up.

Frames are similar to the TCP datagrams that support HTTP connections in that when all data frames are received, they can be combined into a complete HTTP message.

Binary representations in HTTP/2 are used to send and receive message data, but the messages themselves are similar to previous HTTP/1 messages, and binary frames are typically handled by the underlying client or class library.

2. Multiplexing

HTTP/1 is a synchronous, exclusive request-response protocol. The client sends an HTTP/1 message, and the server returns an HTTP/1 response. In order to send and receive more data more quickly, the solution to HTTP/1 is to open multiple connections and merge resources to reduce the number of requests. But this solution introduces additional problems and performance overhead.

HTTP/2 allows multiple requests to be executed simultaneously on a single connection. Each HTTP request or response uses a different stream. By using the Binary Framing Layer, assign each town a stream identifier to support multiple independent requests at the same time. The receiver can combine frames into complete messages.

Frames are the key to sending multiple messages at the same time, and each town has a label indicating which message (flow) it belongs to, so there can be two, three or even hundreds of messages on a single connection at the same time.

Strictly speaking, HTTP/2 requests are not sent simultaneously, because frames need to be sent sequentially over HTTP/TCP connections, and HTTP/1.1 is essentially the same, because while there may appear to be multiple connections, there is usually only one connection at the network layer, so each request is eventually queued from the network layer.

An important difference between HTTP/1 and HTTP/2 sending multiple requests simultaneously is that HTTP/2 connections do not need to block until a response is returned after the request has been sent.

HTTP/2 uses multiple binary frames to send HTTP requests and responses, multiplexed as streams, using a single TCP connection.

HTTP2 is different from HTTP/1 mainly in the level of message sending. At a higher level, the core concepts of HTTP remain unchanged. For example, a request contains a method (e.g. GET), a resource (e.g. Img.png), a header, a body, a status code, a cache and a Cookie. These are consistent with HTTP/1.

3. Streams in HTTP/2

Each flow in HTTP2 acts like a connection in HTTP/1, but there is no reuse of the flow in HTTP/2, and the flow is not completely independent.

A stream is closed after it has transferred the resource, and a new stream is discarded when requesting the resource. A stream is a virtual concept, which is a number marked on each frame, namely the stream iD.

Closing or creating a stream is much less expensive than creating an HTTP/1.1 connection, which involves a three-way handshake and possibly HTTPS protocol negotiation before sending the request.

HTTP/2 connections are more expensive than HTTP/1 connections because of the added prelude messages and SETTINGS frames, but HTTP/2 has a lower flow overhead.

4. HTTP/2 prelude message

Regardless of which method is used to enable an HTTP/2 connection, the first message sent on an HTTP/2 connection must be an HTTP/2 connection prelude. This message is the first message sent by a client on an HTTP/2 connection and is a sequence of 24 8-bit bytes, The sequence, converted to an ASCII string, looks like this:

PRI * HTTP / 2.0 SMCopy the code

The message states that the HTTP request method is PRI, not GET or POST, the requested resource is *, the HTTP version used is HTTP/2.0, and then the request body SM.

The function of the prelude message is to check compatibility and let the client discover in advance that the server does not support HTTP/2. If the server does not support HTTP/2, it cannot be parsed and will reject the message.

If the server supports HTTP/2, it can be inferred from the prelude message that the client supports HTTP/2, in which case the client must send a SETTINGS frame as the first message (which can be empty).

5. SETTINGS frame

The SETTINGS frame is the first frame that the server and client must send (after a prelude message) and contains no data or only key-value pairs.

6. HTTP/2 frame structure

Each HTTP/2 frame consists of a header of fixed length and a payload of variable length, and contains the following four fields:

  • Length of the frame Length

  • The frame Type, Type,

  • Flag bit Flags

  • Reserved Bit Reserved Bit

  • The Stream Identifier

    Used to mark the stream ID to which the frame belongs;

10.2 Http2Connection. Start ()

In the establishProtocol() method, HTTPS is determined, and if HTTPS is used, the connectTls() method is called to connect to TLS, which is what happens in Section 7.

After connecting to TLS, the establishProtocol() method determines whether HTTP/2.0 is used, in which case it calls the startHttp2() method to send the initial frame, and starts reading frames from the peer (server).

In the startHttp2() method, an Http2Connection is first created and then the http2Connection.start () method is called.

In the Http2Connection start() method, Http2Writer’s connectionPreface() method is used to send the preface message, which is then sent to the server by Http2Writer.

object Http2 {

  // Prelude message
  val CONNECTION_PREFACE = HTTP / 2.0 "PRI * \ r \ n \ r \ nSM \ r \ n \ r \ n".encodeUtf8()

  // ...
}
Copy the code

After sending the prelude message, the start() method sends the SETTINGS frame.

After sending the prelude message and SETTINGS frame, the start() method determines whether the value of initialWindowSize is the default value, and if not, sends the difference to the server.

After sending the initial window size change to the server, the start() method creates a new TaskQueue with TaskRunner, and then calls the TaskQueue’s execute() method to put ReaderRunnable into the queue for execution.

ReaderRunnable implements http2Reader.handler and the Function0 interface provided by Kotlin, Function0 -> Unit.

A Task is created in the Execute () method of TaskQueue and the invoke() method of ReaderRunnable is executed in the runOnce() method.

In the Invoke (() method of ReaderRunnavle, the nextFrame() method of Http2Reader is called. In Http2Reader, different methods are called to send data of the corresponding type depending on the frame type. For example, when the frame type is DATA, the ReaderRunnable DATA () method is called to fetch the sent DATA.

The resources

  • Illustrated HTTP
  • UNIX Network Programming Volume 1: Socket Networking apis (version 3)
  • HTTP/2 in Action
  • How the Web Is Connected
  • The Definitive GUIDE to HTTP
  • HTTPS in Plain English
  • Read OkHttp3 source code (ii) : CertificatePinner
  • The use of SSLSocket
  • TLS handshake process (PART 1)