Http Request Process

The index data

DNS query time 3. Socket Connect time 4. TLS connection time 5. Request sending time 6. Response transmission time 7. First packet time 8. Response parsing time 9

The collection of the above indicators, combined with the data visualization tool, can have an intuitive feeling of the TIME consuming and error distribution of Http stages, while providing data support for optimizing business Http requests.

How to get metrics

To obtain indicator data, we need to find the key code that generates indicator data first, and then insert the collection code. How do I insert code if the framework used in the business does not have the source code or is not repackaged? This requires the ability to use aOP-capable tools, provide annotations and configuration files in the previously shared Monitor, and insert related code into specified functions. This implementation can also separate the monitoring code from the business code.

OkHttp framework

OkHttp is the most commonly used Http request framework on Android. The latest version of OkHttp has been upgraded to 4.0.x, and the implementation has been completely replaced by Java with Kotlin. Some of the API usage will also be a little different. Because 4.x devices do not support TLSV1.2 by default, OkHttp 3.13.x and higher versions need to be developed on Android 5.0+(API Level 21+) and Java 1.8. However, OkHttp also created a separate 3.12.x branch to support 4.x devices. The OkHttp version used in this article is version 3.12.3.

OkHttp overall flow

Start by quoting someone else to draw a flow chart (original drawing from)

Request Process analysis

1. Create OkHttpClient

The new okHttpClient.builder () method is called in new OkHttpClient(), and Builder() sets some default values. OkHttpClient() copies the fields of the object generated by okHttpClient.Builder () to the fields of the corresponding OkHttpClient object. SslSocketFactory is not set in the Builder. OkHttp gets the system’s default sslSocketFactory.

public OkHttpClient.Builder(){/** * a dispenser for asynchronous requests, which uses an unlimited thread pool with a lifetime of 60 seconds to execute asynchronous requests * by default, the number of simultaneous asynchronous requests is limited to 64, and the number of simultaneous asynchronous requests per host is limited to 5. * */ dispatcher = new Dispatcher(); // Supported protocol type http1.0/1.1/2,QUIC protocols = DEFAULT_PROTOCOLS; // Support TLS and ClearText connectionSpecs = DEFAULT_CONNECTION_SPECS; EventListenerFactory = EventListener.Factory (eventListener.None); // system-level proxy server proxySelector = proxySelector. GetDefault ();if(proxySelector == null){ proxySelector = new NullProxySelector(); } // Do not use Cookie cookieJar = cookiejar.no_cookies; // Socket factory socketFactory = socketFactory.getdefault (); / / for the HTTPS hostname validation hostnameVerifier = OkHostnameVerifier. INSTANCE; / / used to constraint which certificate is credible, which can be used to fixed certificatePinner = certificatePinner certificate. The DEFAULT; ProxyAuthenticator = authenticator.none; Authenticator = authenticator.none; ConnectionPool = new connectionPool (); connectionPool = new connectionPool (); Inetaddress.getallbyname (hostname) DNS = dns.system; inetaddress.getallbyName (hostname) DNS = dns.system; // Support SSL redirects followSslRedirects =true; // Support redirects followRedirects =true; // Connection failure whether to retry retryOnConnectionFailure =true; CallTimeout = 0; ConnectTimeout = 10_000; // Timeout of socket readreadTimeout = 10_000; //socket writeTimeout time writeTimeout = 10_000; //websocket heartbeat interval pingInterval = 0; } / / get the default sslSocketFactory X509TrustManager trustManager = Util. PlatformTrustManager (); this.sslSocketFactory = newSslSocketFactory(trustManager); this.certificateChainCleaner = CertificateChainCleaner.get(trustManager);Copy the code

2.Request Execution process

As you can see from the above flowchart, whether synchronous or asynchronous requests, Eventually will be called to RealCall. GetResponseWithInterceptorChain (), getResponseWithInterceptorChain () Call again RealInterceptorChain. Proceed (Request Request) method by final Request, let’s analyze the specific code of these two methods.

Response RealCall. GetResponseWithInterceptorChain () throws IOException {/ / assemble all the Interceptors List < Interceptor > interceptors = new ArrayList<>(); / / business since the Interceptors, through OkHttpClient. Builid. AddInterceptor add Interceptors. AddAll (client) Interceptors ()); / / other function interceptors interceptors. Add (retryAndFollowUpInterceptor); interceptors.add(new BridgeInterceptor(client.cookieJar())); interceptors.add(new CacheInterceptor(client.internalCache())); interceptors.add(new ConnectInterceptor(client)); // If notforWebSocket request, add through OkHttpClient. Builid. AddNetworkInterceptor add Interceptorif (!forWebSocket) { interceptors.addAll([client.networkInterceptors](http://client.networkinterceptors/)()); } // Interceptor interceptors.add(new CallServerInterceptor()forWebSocket)); Chain Chain = new RealInterceptorChain(interceptors, null, null, 0); originalRequest, this, eventListener, client.connectTimeoutMillis(), client.readTimeoutMillis(), client.writeTimeoutMillis()); Response Response = chain.proceed(originalRequest); // If the request is cancelledif (retryAndFollowUpInterceptor.isCanceled()) {
    closeQuietly(response);
    throw new IOException("Canceled");
  }
  return response;
}
Copy the code

Then see RealInterceptorChain. Proceed (Request Request) code

public Response proceed(Request request, StreamAllocation streamAllocation, HttpCodec httpCodec, RealConnection connection) throws IOException {/* *index Indicates the position of the current Interceptor in the Interceptor list. If the value exceeds the size of the *Interceptor list, Error * in RealCall getResponseWithInterceptorChain () method calls proceed for the first time the index value of 0 * /if(index >= interceptors.size()) throw new AssertionError(); Calls++; //httpCodec is created in the ConnectInterceptor, which corresponds to a socket connectionif(this.httpCodec ! = null && ! this.connection.supportsUrl(request.url())) { throw new IllegalStateException("network interceptor " + interceptors.get(index - 1) 
        + " must retain the same host and port"); 
  } 
  // If we already have a stream, confirm that this is the only call to chain.proceed(). 
  if(this.httpCodec ! = null && calls > 1) { throw new IllegalStateException("network interceptor " + interceptors.get(index - 1) 
        + " must call proceed() exactly once"); } // Create a new RealInterceptorChain, RealInterceptorChain next = new RealInterceptorChain(interceptors, streamAllocation, httpCodec, connection, index + 1, request, call, eventListener, connectTimeout,readTimeout, writeTimeout); // Interceptor Interceptor = interceptors.get(index); // Execute the current interceptor and pass the newly created RealInterceptorChain, // Interceptor. Intercept calls the new RealInterceptorChain's proceed method, Response Response = interceptor.Intercept (next); // If the current RealInterceptorChain's httpCodec is not empty, make sure the next Interceptor is called only once. HttpCodec is assigned to the ConnectInterceptorif(httpCodec ! = null && index + 1 < interceptors.size() && next.calls ! = 1) { throw new IllegalStateException("network interceptor " + interceptor 
        + " must call proceed() exactly once"); } // Response is nullif (response == null) { 
    throw new NullPointerException("interceptor " + interceptor + " returned null"); } // Response.body is emptyif (response.body() == null) { 
    throw new IllegalStateException( 
        "interceptor " + interceptor + " returned a response with no body"); } // return responsereturn response; 
}
Copy the code

From the code, it can be seen that the Interceptor is the core function class of OkHttp. The Interceptor integrates the actual network request, caching, transparent compression, and other functions. Each function is implemented as one Interceptor. They form an interceptor. Chain of responsibilities, in which each Interceptor can complete the task from Request to Response. The responsibility Chain allows each Interceptor to decide whether and how to complete the task. Completing network requests has also been removed from the RealCall class, simplifying responsibility and logic and making code elegant. The two most important interceptors are ConnectInterceptor and CallServerInterceptor. The main function of a ConnectInterceptor is to find reusable connections in the connection pool. Create a new socket, do a TLS handshake, wrap the socket with Okio, and create HttpCodec. The CallServerInterceptor uses HttpCodec for protocol transmission and resolution. The findConnect process and CallServerInterceptor request process in ConnectInterceptor are analyzed below.

ConnectInterceptor findConnection process analysis

In the ConnectInterceptor, an available RealConnection is created for the request, first from the pool of connections that can be reused, then a new socket is created if none is available, and then HttpCodec is created using RealConnection. Create a link for StreamAllocation RealConnection method call. NewStream () – > StreamAllocation. FindHealthyConnection () – > StreamAllocation. FindConnection (), findConnection () is the main code to create a connection.

private RealConnection findConnection(int connectTimeout, int readTimeout, int writeTimeout, int pingIntervalMillis, boolean connectionRetryEnabled) throws IOException {
    boolean foundPooledConnection = false; RealConnection result = null; Route selectedRoute = null; // Connection releasedConnection; Socket toClose; synchronized (connectionPool) {if (released) throw new IllegalStateException("released");
        if(codec ! = null) throw new IllegalStateException("codec ! = null");
        if (canceled) throw new IOException("Canceled"); / * * * if there are any redirects to the same address, will use have been allocated in RetryAndFollowUpInterceptor StreamAllocation * redirect requests, The connection is not empty, but it is not a valid connection. * **/ releasedConnection = this.connection; /** * If RealConnection already exists, Connection =null and return the socket to be closed * releaseIfNoNewStreams() does nothing when the current request is first executed * */ toClose = releaseIfNoNewStreams(); /** ** this. Connection is not null * this. Connection =null ** */ when the request is first executedif(this.connection ! = null) { // We had an already-allocated connection and it's good. result = this.connection; releasedConnection = null; } /** * reportedAcquired=fasle ** / if (! ReportedAcquired =fasle ** / if (! reportedAcquired) { // If the connection was never reported acquired, don't report it as released! releasedConnection = null; } /** ** Try to fetch ** / from the connection pool if the target RealConnection has not been foundif(result == null) {/** * For non-HTTP /2 protocols, if there is a protocol that does not exceed the RealConnection reuse maximum, * This RealConnection can be reused * If a RealConnection is fetched from the connection pool, * streamallocation.acquire () is called to set connection to the new value * */ internal.instance. get(connectionPool, address, this, null); /** * connection ! Null indicates that an appropriate * RealConnection has been obtained from the connection pool. Set foundPooledConnection =true; * * /if(connection ! = null) { foundPooledConnection =true;
                result = connection;
            } else{ selectedRoute = route; }} /** * Close The socket ** / closeQuietly(toClose); /** * If the current RealConnection needs to be released, call eventListener ** /if(releasedConnection ! = null) { eventListener.connectionReleased(call, releasedConnection); } /** * If a RealConnection is fetched from the pool, call eventListener ** /if(foundPooledConnection) { eventListener.connectionAcquired(call, result); } /** * The current RealConnection can continue to be used or a suitable RealConnection can be found from the connection pool * return this RealConnection ** /if(result ! = null) { // If we found an already-allocated or pooled connection, we're done. return result; } // If we need a route selection, Make one. This is a blocking operation. /** * routeSelector is created in the StreamAllocation constructor * Connection pool research * * */ Boolean newRouteSelection = false; if (selectedRoute == null && (routeSelection == null || ! routeSelection.hasNext())) { newRouteSelection = true; routeSelection = routeSelector.next(); } synchronized (connectionPool) { if (canceled) throw new IOException("Canceled"); if (newRouteSelection) { // Now that we have a set of IP addresses, make another attempt at getting a connection from // the pool. This could match due to connection coalescing. List
      
        routes = routeSelection.getAll(); for (int i = 0, size = routes.size(); i < size; i++) { Route route = routes.get(i); Internal.instance.get(connectionPool, address, this, route); if (connection ! = null) { foundPooledConnection = true; result = connection; this.route = route; break; } } } if (! foundPooledConnection) { if (selectedRoute == null) { selectedRoute = routeSelection.next(); } // Create a connection and assign it to this allocation immediately. This makes it possible // for an asynchronous cancel() to interrupt the handshake we'
      re about to do.
            route = selectedRoute;
            refusedStreamCount = 0;
            result = new RealConnection(connectionPool, selectedRoute);
            acquire(result, false);
        }
    }
    // If we found a pooled connection on the 2nd time around, we're done. / * * * if the connection pool to find the right RealConnection, return * * / if (foundPooledConnection) {eventListener. ConnectionAcquired (call, result); return result; } /** * If not found, ConnectPool * */ / Do TCP + Tls handshakes. This is a blocking operation. result.connect(connectTimeout, readTimeout, writeTimeout, pingIntervalMillis, connectionRetryEnabled, call, eventListener); routeDatabase().connected(result.route()); Socket socket = null; synchronized (connectionPool) { reportedAcquired = true; // Pool the connection. Internal.instance.put(connectionPool, result); // If another multiplexed connection to the same address was created concurrently, then // release this connection and acquire that one. if (result.isMultiplexed()) { socket = Internal.instance.deduplicate(connectionPool, address, this); result = connection; } } closeQuietly(socket); eventListener.connectionAcquired(call, result); return result; }Copy the code

4. CallServerInterceptor routine analysis

public Response intercept(Chain chain) throws IOException { RealInterceptorChain realChain = (RealInterceptorChain) chain; HttpCodec = realchain.httpStream (); // HttpCodec = realchain.httpStream (); StreamAllocation streamAllocation = realChain.streamAllocation(); /** * the connection used by the request, Generated in ConncetInterceptor * Connections may be selected or recreated from ConnectPool **/ RealConnection Connection = (RealConnection) realChain.connection(); Request request = realChain.request(); long sentRequestMillis = System.currentTimeMillis(); realChain.eventListener().requestHeadersStart(realChain.call()); / * * * by using Okio httpCodec parcel socket write requests head * * / httpCodec writeRequestHeaders (request); realChain.eventListener().requestHeadersEnd(realChain.call(), request); Response.Builder responseBuilder = null; /** * if the request has a request body, send body **/if(HttpMethod.permitsRequestBody(request.method()) && request.body() ! = null) {the request body. / * * * request header is Expect: 100 - continue to read the response headers. * httpCodec readResponseHeaders method reads to a status code of 100, the * * / returns nullif ("100-continue".equalsIgnoreCase(request.header("Expect"))) {
      httpCodec.flushRequest();
      realChain.eventListener().responseHeadersStart(realChain.call());
      responseBuilder = httpCodec.readResponseHeaders(true); } /** * If the body part of the request header has an Expect: 100-continue Expect: 100-continue, but the server does not return a status code of 100 and is not Http/2, close the current connection **/if (responseBuilder == null) {
      realChain.eventListener().requestBodyStart(realChain.call());
      long contentLength = request.body().contentLength();
      CountingSink requestBodyOut =
          new CountingSink(httpCodec.createRequestBody(request, contentLength));
      BufferedSink bufferedRequestBody = Okio.buffer(requestBodyOut);
      request.body().writeTo(bufferedRequestBody);
      bufferedRequestBody.close();
      realChain.eventListener()
          .requestBodyEnd(realChain.call(), requestBodyOut.successfulCount);
    } else if(! connection.isMultiplexed()) { streamAllocation.noNewStreams(); } } httpCodec.finishRequest(); /** * Normally, read headers **/if (responseBuilder == null) {
    realChain.eventListener().responseHeadersStart(realChain.call());
    responseBuilder = httpCodec.readResponseHeaders(false);
  }
  Response response = responseBuilder
      .request(request)
      .handshake(streamAllocation.connection().handshake())
      .sentRequestAtMillis(sentRequestMillis)
      .receivedResponseAtMillis(System.currentTimeMillis())
      .build();
  int code = response.code();
  if (code == 100) {
    responseBuilder = httpCodec.readResponseHeaders(false); response = responseBuilder .request(request) .handshake(streamAllocation.connection().handshake()) .sentRequestAtMillis(sentRequestMillis) .receivedResponseAtMillis(System.currentTimeMillis()) .build(); code = response.code(); **/ realchain-.eventListener ().responseHeadersend (realchain-.call (), response); /** * if yesforWebSocket, and code == 101 returns an empty response * others return a RealResponseBody object **/if (forWebSocket && code == 101) {
    // Connection is upgrading, but we need to ensure interceptors see a non-null response body.
    response = response.newBuilder()
        .body(Util.EMPTY_RESPONSE)
        .build();
  } else{/** * will * no response content * chunk content, The response body is wrapped as a RealResponseBody object ** */ response = response.newBuilder() .body(httpCodec.openResponseBody(response)) .build(); } /** * server shutdown requires connection **/if ("close".equalsIgnoreCase(response.request().header("Connection"))
      || "close".equalsIgnoreCase(response.header("Connection"))) { streamAllocation.noNewStreams(); } /** * code is 204/205, contentLength() is greater than 0, throws protocol error **/if ((code == 204 || code == 205) && response.body().contentLength() > 0) {
    throw new ProtocolException(
        "HTTP " + code + " had non-zero Content-Length: " + response.body().contentLength());
  }
  return response;
}
Copy the code

When the CallServerInterceptor intercepts, it returns a Response that reads headers, but does not read the body. The OkHttp network request code is finished here. The following parseReponse frameworks are in the higher level. For example, Retrofit is done by calling GsonConvter or other Convertor in servicemethod. toResponse(catchingBody) in the okHttpCall.parsereponse () method.

Obtain specific implementation of indicators

For Http request time, anomaly, data size, status code acquisition, directly use the previous implementation of MAOP, intercept OkHttpClient.Builder build method to add statistics Interceptor,DNSLookUp time, connection time, SSL time, You can collect it directly by setting eventListener.Factory. The parsing time needs to be collected by intercepting the parseReponse method of the upper-layer framework. The first packet time needs to be implemented by intercepting the OkHttp read request data. OKHttpClient finally calls the CallServerInterceptor, and the key code is the time to read readResponseHeaders.

MAOP implementation

Use the MAOP functionality provided earlier to add, Intercept OkHttpClient builder method and Http1Codec readHeaderLine method and okhttp3 internal. Http2. Http2Stream takeResponseHeaders method of configuration

Add the statistics Interceptor and EventListenerFactory to the build() method of the Builder that intercepts OkHttpClient

First packet time through: It is considered that the header time when the first read response header is returned, Intercept okhttp3. Internal. Http1. Http1Code. The method and okhttp3 readHeaderLine. Internal. Http2. Http2Stream. TakeResponseHeaders computed first time

Retrofit Parse takes time to collect

AOP configuration files to add to retrofit2. OKHttp. ParseResponse interceptor configuration

Method returns the data associated with the process

In summary, this scheme can basically achieve the acquisition of Http basic indicators, but some details need to be improved.