OKHttp parsing caching mechanism CacheInterceptor

I wrote an article about the browser HTTP caching mechanism: the Browser HTTP caching mechanism

Unlike browsers, clients need a network framework to implement caching, and today we’ll take a look at Okhttp caching.

How to Use Cache (CacheControl)

OKhttp provides a selection of cache policies that are uniformly configured through CacheControl, which you can see through the constructor.

  private CacheControl(boolean noCache, boolean noStore, int maxAgeSeconds, int sMaxAgeSeconds,
      boolean isPrivate, boolean isPublic, boolean mustRevalidate, int maxStaleSeconds,
      int minFreshSeconds, boolean onlyIfCached, boolean noTransform, boolean immutable,
      @Nullable String headerValue) {}Copy the code

Example Command output:

Request request = new Request.Builder()
                .url(httpUrl)
                // Cache policy, enforce cache
                .cacheControl(CacheControl.FORCE_CACHE)
                .build();
Copy the code

When caching is enabled, we can see the following files in the cache directory:

Let’s take a look at some of the documents:

Journal file

59 b20da873a6f4c405e64ad3e29a24e9. 0 file

59 b20da873a6f4c405e64ad3e29a24e9. 1

You can see some of the header information about the network request, and some of the body information about the response, and that’s the cache file and that’s the main mechanism for caching;

CacheInterceptor

OKhttp cache implementation is done by CacheInterceptor, located in RetryAndFollowUpInterceptor and BridgeInterceptor, in real network connection, network requests before the interceptor.

  @Override public Response intercept(Chain chain) throws IOException {
    // 1. Get standby Response from cacheResponse cacheCandidate = cache ! =null
        ? cache.get(chain.request())
        : null;

    long now = System.currentTimeMillis();
     // 2. Obtain the current network Request Request object and cache Response object according to the standby Response, cache policy, Request, etc
    CacheStrategy strategy = new CacheStrategy.Factory(now, chain.request(), cacheCandidate).get();
    Request networkRequest = strategy.networkRequest;
    Response cacheResponse = strategy.cacheResponse;

    //3. Track the current cache hit count and network request count for recording
    if(cache ! =null) {
      cache.trackResponse(strategy);
    }
    
    // 4. If the standby cache is unavailable, disable the standby cache
    if(cacheCandidate ! =null && cacheResponse == null) {
      closeQuietly(cacheCandidate.body()); // The cache candidate wasn't applicable. Close it.
    }
    
    Build Response (Code 504) instead of using network request and caching result is invalid
    // If we're forbidden from using the network and the cache is insufficient, fail.
    if (networkRequest == null && cacheResponse == null) {
      return new Response.Builder()
          .request(chain.request())
          .protocol(Protocol.HTTP_1_1)
          .code(504)
          .message("Unsatisfiable Request (only-if-cached)")
          .body(Util.EMPTY_RESPONSE)
          .sentRequestAtMillis(-1L)
          .receivedResponseAtMillis(System.currentTimeMillis())
          .build();
    }

    // 6. If the network is not allowed and the cache is available, the cache result is returned directly.
    // If we don't need the network, we're done.
    if (networkRequest == null) {
      return cacheResponse.newBuilder()
          .cacheResponse(stripBody(cacheResponse))
          .build();
    }

    // 7. Start the follow-up interceptor to start the actual network request;
    Response networkResponse = null;
    try {
      networkResponse = chain.proceed(networkRequest);
    } finally {
      // If we're crashing on I/O or otherwise, don't leak the cache body.
      if (networkResponse == null&& cacheCandidate ! =null) { closeQuietly(cacheCandidate.body()); }}If the cloud resource has not changed, i.e. the code is 304, the cache result is directly used
    if(cacheResponse ! =null) {
      if (networkResponse.code() == HTTP_NOT_MODIFIED) {
        Response response = cacheResponse.newBuilder()
            .headers(combine(cacheResponse.headers(), networkResponse.headers()))
            .sentRequestAtMillis(networkResponse.sentRequestAtMillis())
            .receivedResponseAtMillis(networkResponse.receivedResponseAtMillis())
            .cacheResponse(stripBody(cacheResponse))
            .networkResponse(stripBody(networkResponse))
            .build();
        networkResponse.body().close();
          
        // 9. The response entity has not changed and the cache needs to be updated to the latest response result
        // Update the cache after combining headers but before stripping the
        // Content-Encoding header (as performed by initContentStream()).
        cache.trackConditionalCacheHit();
        cache.update(cacheResponse, response);
        return response;
      } else{ closeQuietly(cacheResponse.body()); }}// 10. Build the final Response and store it in the cache if possible.
    Response response = networkResponse.newBuilder()
        .cacheResponse(stripBody(cacheResponse))
        .networkResponse(stripBody(networkResponse))
        .build();

    if(cache ! =null) {
      if (HttpHeaders.hasBody(response) && CacheStrategy.isCacheable(response, networkRequest)) {
        // Offer this request to the cache.
        // 11. Write the network request to the cache and return a writable Response. Write the cache to the cache file at read time.
        CacheRequest cacheRequest = cache.put(response);
        return cacheWritingResponse(cacheRequest, response);
      }
      //12. Some of the following HTTP request methods are invalid when caching and need to delete cached data
      if (HttpMethod.invalidatesCache(networkRequest.method())) {
        try {
          cache.remove(networkRequest);
        } catch (IOException ignored) {
          // The cache cannot be written.}}}return response;
  }
  
  public static boolean invalidatesCache(String method) {
    return method.equals("POST")
        || method.equals("PATCH")
        || method.equals("PUT")
        || method.equals("DELETE")
        || method.equals("MOVE");     // WebDAV
  }
Copy the code

The overall distance is as follows:

1. Get standby Response from cache first;
2. Obtain the current network Request Request object and cache Response object according to the standby Response, cache policy and Request mentioned above.
3. Track the current cache hit times and network request times for recording;
4. If the standby cache is unavailable, disable the standby cache.
5. Disallow network requests and cache results are invalid, build Response directly (Code 504);
6. If the network is not allowed and the cache is available, the cache result is returned directly.
7. Start executing the follow-up interceptor and start the actual network request;
8. If the cloud resource has not changed, i.e. the code is 304, the cache result is directly used;
9. The cloud response entity has not changed, so the cache needs to be updated to the latest response result.
10. Build the final Response and store it in the cache if possible;
11. Write the network request to the cache, and return a writable Response, and write the cache to the cache file at the time of reading;
12. Some of the following HTTP request methods are invalid when caching and need to remove cached data

Cache (read)

In the first step above, we see that there is a member variable of InternalCache type Cache. Let’s look at how to get the Cache from the Cache.

  @Nullable Response get(Request request) {
    //1. Generate a unique key based on the Url;
    String key = key(request.url());
    DiskLruCache.Snapshot snapshot;
    Entry entry;
    try {
      //2. Obtain the cache snapshot from the cache
      snapshot = cache.get(key);
      if (snapshot == null) {
        return null; }}catch (IOException e) {
      // Give up because the cache cannot be read.
      return null;
    }
    METADATA (url, request method, protocol, certificate, return code, etc.);
    try {
      entry = new Entry(snapshot.getSource(ENTRY_METADATA));
    } catch (IOException e) {
      Util.closeQuietly(snapshot);
      return null;
    }
    // 4. Build response based on cached data
    Response response = entry.response(snapshot);
    if(! entry.matches(request, response)) { Util.closeQuietly(response.body());return null;
    }
    return response;
  }
Copy the code

The cache data here uses a DiskLruCache. This time we will not go into the implementation of DiskLruCache.

1. The cached key generates a unique identifier based on the URL (generated after md5 encryption of the URL and hexadecimal conversion)
2. Obtain the cache Snapshot from cache(DiskLruCache). Here are the specific functions of the three classes
   • An Entry is a data Entry that can be read from the cache or written to the cache.
   • DiskLruCache.Snapshot Cache Snapshot, which provides the operation of reading and writing an entry.
   • DiskLruCache.Editor A cache Editor used to edit data inside Entry, including write and read operations.
3. Create an Entry from a Cache file, which is used to read headers from the Cache file, similar to the following:

   * { * http://google.com/foo * GET * 2 * Accept-Language: fr-CA * Accept-Charset: Utf-8 * HTTP/1.1 200 OK * 3 * content-type: image/ PNG * Content-Length: 100 * cache-control: max-age=600 *} utF-8 * HTTP/1.1 200 OK * 3 * content-type: image/ PNG * Content-Length: 100 * cache-control: max-age=600 *}Copy the code

4. Build a Response object based on reading the cache data that has been read.

CacheStrategy

The required data has been read from the cache file above, but the decision to use caching, and how our configured caching policy takes effect, is governed by CacheStrategy. Let’s take a look at how you build a CacheStrategy

CacheStrategy strategy = new CacheStrategy.Factory(now, chain.request(), cacheCandidate).get();
Request networkRequest = strategy.networkRequest;
Response cacheResponse = strategy.cacheResponse;
Copy the code

The Factory object is constructed to read cache-relevant Response header data from the cached Response.

 public Factory(long nowMillis, Request request, Response cacheResponse) {
      this.nowMillis = nowMillis;
      this.request = request;
      this.cacheResponse = cacheResponse;

      if(cacheResponse ! =null) {
        this.sentRequestMillis = cacheResponse.sentRequestAtMillis();
        this.receivedResponseMillis = cacheResponse.receivedResponseAtMillis();
        Headers headers = cacheResponse.headers();
        for (int i = 0, size = headers.size(); i < size; i++) {
          String fieldName = headers.name(i);
          String value = headers.value(i);
          if ("Date".equalsIgnoreCase(fieldName)) {
            servedDate = HttpDate.parse(value);
            servedDateString = value;
          } else if ("Expires".equalsIgnoreCase(fieldName)) {
            expires = HttpDate.parse(value);
          } else if ("Last-Modified".equalsIgnoreCase(fieldName)) {
            lastModified = HttpDate.parse(value);
            lastModifiedString = value;
          } else if ("ETag".equalsIgnoreCase(fieldName)) {
            etag = value;
          } else if ("Age".equalsIgnoreCase(fieldName)) {
            ageSeconds = HttpHeaders.parseSeconds(value, -1); }}}}Copy the code

The following logic is key to the effectiveness of caching mechanisms and caching policies.

    public CacheStrategy get(a) {
      CacheStrategy candidate = getCandidate();
        If the network is not allowed and the cache is not available, return 504
      if(candidate.networkRequest ! =null && request.cacheControl().onlyIfCached()) {
        // We're forbidden from using the network and the cache is insufficient.
        return new CacheStrategy(null.null);
      }
      return candidate;
    }
Copy the code

    private CacheStrategy getCandidate(a) {
      // There is no cached data, return directly
      if (cacheResponse == null) {
        return new CacheStrategy(request, null);
      }
      // HTTPS request, no handshake data, no cached data
      // Drop the cached response if it's missing a required handshake.
      if (request.isHttps() && cacheResponse.handshake() == null) {
        return new CacheStrategy(request, null);
      }
      // Determine whether the current cache result is available, according to the response code and response header to determine which code can be cached can refer to the code
      // If there is a "no-store" in the request header or response header, no cache is used
      if(! isCacheable(cacheResponse, request)) {return new CacheStrategy(request, null);
      }
        
      If no cache is configured, or If "if-modified-since" or "if-none-match" exists in the request header, no cache is used.
      CacheControl requestCaching = request.cacheControl();
      if (requestCaching.noCache() || hasConditions(request)) {
        return new CacheStrategy(request, null);
      }
      // 
      CacheControl responseCaching = cacheResponse.cacheControl();
      // The current age of the cache response, which is how long the cache exists
      long ageMillis = cacheResponseAge();
      // How long does the cache expire
      long freshMillis = computeFreshnessLifetime();
      if(requestCaching.maxAgeSeconds() ! = -1) {
        freshMillis = Math.min(freshMillis, SECONDS.toMillis(requestCaching.maxAgeSeconds()));
      }
      // Minimum cache, which requires the cache server to return the cached data in the min-fresh time
      long minFreshMillis = 0;
      if(requestCaching.minFreshSeconds() ! = -1) {
        minFreshMillis = SECONDS.toMillis(requestCaching.minFreshSeconds());
      }
      // maxStaleMillis indicates that the client is willing to accept a response beyond its freshness lifecycle
      long maxStaleMillis = 0;
      if(! responseCaching.mustRevalidate() && requestCaching.maxStaleSeconds() ! = -1) {
        maxStaleMillis = SECONDS.toMillis(requestCaching.maxStaleSeconds());
      }
      //ageMillis + minFreshMillis < freshMillis + maxStaleMillis
      // Indicates that the current cache exists for less than the cache validity period and returns cached data.
      if(! responseCaching.noCache() && ageMillis + minFreshMillis < freshMillis + maxStaleMillis) { Response.Builder builder = cacheResponse.newBuilder();if (ageMillis + minFreshMillis >= freshMillis) {
          builder.addHeader("Warning"."110 HttpURLConnection \"Response is stale\"");
        }
        long oneDayMillis = 24 * 60 * 60 * 1000L;
        if (ageMillis > oneDayMillis && isFreshnessLifetimeHeuristic()) {
          builder.addHeader("Warning"."113 HttpURLConnection \"Heuristic expiration\"");
        }
        return new CacheStrategy(null, builder.build());
      }

      // Check whether the conditions of the current cached response header are met. If so, the cache Reponse is returned. Otherwise, the network request is normal.
      String conditionName;
      String conditionValue;
      if(etag ! =null) {
        conditionName = "If-None-Match";
        conditionValue = etag;
      } else if(lastModified ! =null) {
        conditionName = "If-Modified-Since";
        conditionValue = lastModifiedString;
      } else if(servedDate ! =null) {
        conditionName = "If-Modified-Since";
        conditionValue = servedDateString;
      } else {
        return new CacheStrategy(request, null); // No condition! Make a regular request.
      }

      Headers.Builder conditionalRequestHeaders = request.headers().newBuilder();
      Internal.instance.addLenient(conditionalRequestHeaders, conditionName, conditionValue);

      Request conditionalRequest = request.newBuilder()
          .headers(conditionalRequestHeaders.build())
          .build();
      return new CacheStrategy(conditionalRequest, cacheResponse);
    }

Copy the code

According to the request and cache RePONse objects, the user configured cache policy and the cache policy information related to the response header, a series of judgments are made to determine whether to use cache data, and a cache policy object is constructed. For the cache header information involved, please refer to the description at the beginning of this chapter. It mainly involves the following logic

1. “No-store” : If “no-store” exists in the request header or response header, no cache is used
2. “No-cache” : No cache is used If no-cache is configured in the cache or If “if-modified-since” or “if-none-match” exists in the request header.
3. Stale stale stale stale stale stale stale stale stale ageMillis + minFreshMillis < freshMillis + maxStaleMillis
4. “If-none-match”, “ETag”, “if-modified-since” : When it comes to server cache, determine whether the server should return 304. See the browser cache logic for this piece of logic.

Cache handling

When Wang builds the cache policy object from the cache configuration policy, the subsequent logic goes back to CacheInterceptor:

5. Disallow network request and cache result is invalid, build Response directly (code 504); 6. If the network is not allowed and the cache is available, the cache result is directly returned. 7. Start executing subsequent interceptors and start actual network requests; 8. Judge if the cloud resource has not changed, i.e. the code is 304, and directly use the cache result; 9. The cloud response entity has not changed, so the cache needs to be updated as the latest response result; 10. Build the final Response and store it in the cache if possible. 11. Write the network request to the cache and return a writable Response. Write the cache to the cache file when reading. 12. When some of the HTTP request method cache is invalid (POST, a MOVE that PUT, PATCH, DELETE), need to DELETE the cached data.Copy the code

Cache write logic

  @Nullable CacheRequest put(Response response) {
    String requestMethod = response.request().method();
    // Some invalid cache data needs to be deleted
    if (HttpMethod.invalidatesCache(response.request().method())) {
      try {
        remove(response.request());
      } catch (IOException ignored) {
        // The cache cannot be written.
      }
      return null;
    }
    // Non-get request, not written to cache
    if(! requestMethod.equals("GET")) {
      return null;
    }
    // If the mutable header contains an asterisk, it is not cached
    if (HttpHeaders.hasVaryAll(response)) {
      return null;
    }
    // The actual write cache operation
    Entry entry = new Entry(response);
    DiskLruCache.Editor editor = null;
    try {
      editor = cache.edit(key(response.request().url()));
      if (editor == null) {
        return null;
      }
      entry.writeTo(editor);
      return new CacheRequestImpl(editor);
    } catch (IOException e) {
      abortQuietly(editor);
      return null; }}Copy the code