Okhttp’s caching mechanism

Caching mechanism over Http

Mandatory cache

The data transmitted through HTTP is stored in the cache database. Mandatory cache means that if the data in the cache database is still valid, the data is directly obtained from the cache database without sending requests to the server through HTTP. There are two important fields for controlling invalidation:

Expires

The expiration date, which is determined by the server. When retrieving data from the cached database, determine whether to use the cached data directly by comparing the current time to Expires. However, it is worth noting that there is a delay between the server and the client, there is no uniform time standard, so as Http protocol development, the opportunity to use less and less

Cache-Control

Literally means cache control, but actually means cache properties. Similar to scopes in Java, caches in HTTP fall into the following types:

  1. Public: indicates that data can be stored, including private information such as passwords, and can be accessed by all with low security
  2. Private: Stored in the user’s private cache, accessible only by the user (default)
  3. No-cache can be cached only after authentication is established between the client and server (for comparison caching)
  4. No-store indicates that the request and response information will not be cached
  5. Max-age Indicates that the returned data is expired or invalid

Compared to the cache

Check with the server’s cache before using. Determines whether to use the status code returned by the server

304: Use comparison cache data

200: Use the latest data of the server

Determine the field

Etag

A unique identifier for a resource, similar to a person’s ID number. Once the content of the resource is changed, the Etag will change. The client sends a request in the format of if-none-match +Etag. After receiving the request, the server compares it with the cached Etag

Last-Modified

The last Modified time is determined by the server. The client uses if-modified-since + when sending a request. If the amount of data stored on the client is less than or equal to this time, the resource has not changed

To put into a nutshell :

Okhttp’s caching mechanism

It can be seen that the cache mechanism of HTTP itself is relatively simple and cannot meet the actual requirements. Okhttp is more complex. First the conclusion:

  1. Caching is based on file storage
  2. Internal maintenance of cache cleaning threads based on the LRU algorithm

Okhttp reads the cache flow

Okhttp stores the caching process

The source code parsing

CacheControl

Rules for specifying the cache

Public final class CacheControl {// indicates that this is a CacheControl that uses network authentication first and only after the authentication is passed. NoCache public static final CacheControl FORCE_NETWORK = new Builder().noCache().build(); // this is a first-use cache control, OnlyIfCached and maxStale Max values public static final CacheControl FORCE_CACHE = new Builder().onlyifcached () .maxStale(Integer.MAX_VALUE, TimeUnit.SECONDS) .build(); Private final Boolean noCache; private final Boolean noCache; private final boolean noStore; private final int maxAgeSeconds; private final int sMaxAgeSeconds; private final boolean isPrivate; private final boolean isPublic; private final boolean mustRevalidate; private final int maxStaleSeconds; private final int minFreshSeconds; private final boolean onlyIfCached; private final boolean noTransform; Public static CacheControl parse(Headers Headers) {// parses fields in the header file to obtain the CacheControl class. }}Copy the code

CacheStrategy

It is used to determine whether to use cached data

public final class CacheStrategy { public Factory(long nowMillis, Request request, Response cacheResponse) { this.nowMillis = nowMillis; // Network request and cache response this.request = request; this.cacheResponse = cacheResponse; if (cacheResponse ! Headers Headers = cacheresponse.headers (); for (int i = 0, size = headers.size(); i < size; I ++) {String fieldName = headers (I); String value = headers.value(i); if ("Date".equalsIgnoreCase(fieldName)) { servedDate = HttpDate.parse(value); servedDateString = value; } else if ("Expires".equalsIgnoreCase(fieldName)) { expires = HttpDate.parse(value); } else if ("Last-Modified".equalsIgnoreCase(fieldName)) { lastModified = HttpDate.parse(value); lastModifiedString = value; } else if ("ETag".equalsIgnoreCase(fieldName)) { etag = value; } else if ("Age".equalsIgnoreCase(fieldName)) { ageSeconds = HeaderParser.parseSeconds(value, -1); } else if (OkHeaders.SENT_MILLIS.equalsIgnoreCase(fieldName)) { sentRequestMillis = Long.parseLong(value); } else if (OkHeaders.RECEIVED_MILLIS.equalsIgnoreCase(fieldName)) { receivedResponseMillis = Long.parseLong(value); }}}} public CacheStrategy get() {CacheStrategy candidate = getCandidate(); if (candidate.networkRequest ! = null && request.cachecontrol ().onlyifcached ()) {// If the network request of the cache policy is not empty, but only cache is used, the cache policy with both are empty is returned. return new CacheStrategy(null, null); } return candidate; } /** Returns a strategy to use assuming the request can use the network. */ private CacheStrategy getCandidate() { // // If (cacheResponse == null) {return new CacheStrategy(request, null); // No cached response. } // Drop the cached response if it's missing a required handshake. If (request.ishttps () && Cacheresponse.Handshake () == NULL) {return new CacheStrategy(request, null); if (request.ishttps () && Cacheresponse.Handshake () == null) {return new CacheStrategy(request, null); } // If this response shouldn't have been stored, it should never be used // as a response source. This check should be redundant as long as the // persistence store is Well - pour and the rules are constant. // If the response to the request cannot be cached, then the policy without cached response is returned if (! isCacheable(cacheResponse, request)) { return new CacheStrategy(request, null); } // Obtain CacheControl information in the request header. CacheControl requestCaching = request.Cachecontrol (); If the CacheControl information in the request header is not cached, It returns no cache response strategy if (requestCaching. NoCache () | | hasConditions (request)) {return new CacheStrategy (request, null); } // Get the response age long ageMillis = cacheResponseAge(); // computeFreshnessLifetime(); long freshMillis = computeFreshnessLifetime(); / / if the request has a maximum duration, the smaller value as if the last refresh response time (requestCaching. MaxAgeSeconds ()! = -1) { freshMillis = Math.min(freshMillis, SECONDS.toMillis(requestCaching.maxAgeSeconds())); } // If the request has a minimum refresh time, use it as the minimum refresh time; if (requestCaching.minFreshSeconds() ! = -1) { minFreshMillis = SECONDS.toMillis(requestCaching.minFreshSeconds()); } // Max expiration time long maxStaleMillis = 0; / / cache the response headers CacheControl information CacheControl responseCaching = cacheResponse. CacheControl (); // If the cached response does not require revalidation and the request has a maximum expiration time, use the maximum expiration time of the request as the maximum expiration time if (! responseCaching.mustRevalidate() && requestCaching.maxStaleSeconds() ! = -1) { maxStaleMillis = SECONDS.toMillis(requestCaching.maxStaleSeconds()); } // If caching is supported and duration + minimum refresh time < last refresh time + maximum validation time can be cached if (! responseCaching.noCache() && ageMillis + minFreshMillis < freshMillis + maxStaleMillis) { Response.Builder builder = cacheResponse.newBuilder(); if (ageMillis + minFreshMillis >= freshMillis) { builder.addHeader("Warning", "110 HttpURLConnection \\"Response is stale\\""); } long oneDayMillis = 24 * 60 * 60 * 1000L; if (ageMillis > oneDayMillis && isFreshnessLifetimeHeuristic()) { builder.addHeader("Warning", "113 HttpURLConnection \\"Heuristic expiration\\""); } return new CacheStrategy(null, builder.build()); } // Create a new conditional Request, add if-none-match, The information such as the if-modified-since Request. Builder conditionalRequestBuilder = Request. NewBuilder (); if (etag ! = null) { conditionalRequestBuilder.header("If-None-Match", etag); } else if (lastModified ! = null) { conditionalRequestBuilder.header("If-Modified-Since", lastModifiedString); } else if (servedDate ! = null) { conditionalRequestBuilder.header("If-Modified-Since", servedDateString); } Request conditionalRequest = conditionalRequestBuilder.build(); / / according to whether there is an If - None - Match, the if-modified-since information, return different caching policy return hasConditions (conditionalRequest)? new CacheStrategy(conditionalRequest, cacheResponse) : new CacheStrategy(conditionalRequest, null); } /** * Returns true if the request contains conditions that save the server from sending a response * that the client has locally. When a request is enqueued with its own conditions, the built-in * response cache won't be used. */ private static boolean hasConditions(Request request) { return request.header("If-Modified-Since") ! = null || request.header("If-None-Match") ! = null; }}Copy the code

Cache

Open to the outside of the cache class, similar to the database can add, delete, change and check

  1. Add to the cache
CacheRequest put(Response response) { String requestMethod = response.request().method(); / / if the request is "POST", "PUT", "PATCH", "PROPPATCH", "the REPORT" is to remove the cache if (HttpMethod. InvalidatesCache (response. The request () method ())) { try { remove(response.request()); } catch (IOException ignored) { } return null; } // Only GET requests are cached. requestMethod.equals("GET")) { return null; } if (HttpHeaders. HasVaryAll (response)) {return null; } // Build response into an Entry object. Entry = new Entry(response); DiskLruCache.Editor editor = null; Editor = cache.edit(key(response.request().url()))); if (editor == null) { return null; } // Write entry to the cache. WriteTo (editor); IO Sink object return new CacheRequestImpl(Editor); } catch (IOException e) { abortQuietly(editor); return null; }}Copy the code
  1. Find the cache
String key = key(request.url()); String key = key(request.url()); DiskLruCache.Snapshot snapshot; Entry entry; Try {// Obtain the corresponding snapshot based on the key snapshot = cache.get(key); if (snapshot == null) { return null; } } catch (IOException e) { return null; } try {// Create an Entry object and get Sink Entry from snapshot.getSource() = new Entry(snapshot.getsource (ENTRY_METADATA)); } catch (IOException e) { Util.closeQuietly(snapshot); return null; } // Generate respson from entry and response, obtain request body from okio.buffer, then encapsulate various request information response response = entry.response(snapshot); if (! Entry. Matches (request, response)) {// Matches request with response. Util.closeQuietly(response.body()); return null; } return response; }Copy the code
  1. Update the cache
Void update(Response cached, Response network) {// Create an Entry with Respon Entry = new Entry(network); DiskLruCache.Snapshot DiskLruCache.Snapshot Snapshot = ((CacheResponseBody) cached.body()).snapshot; DiskLruCache.Editor editor = null; Try {/ / get DiskLruCache. The Snapshot. Edit the object editor = the Snapshot. Edit (); // Returns null if snapshot is not current. if (editor ! = null) {// Write entry into the editor. WriteTo (editor); editor.commit(); } } catch (IOException e) { abortQuietly(editor); }}Copy the code
  1. Delete the cache

The main body is located in DiskLruCache

Void remove(Request Request) throws IOException {// Delete cache cache.remove(key(request.url())) using the key converted from the URL. }Copy the code
  1. writeTo ok.io
 public void writeTo(DiskLruCache.Editor editor) throws IOException {
      BufferedSink sink = Okio.buffer(editor.newSink(ENTRY_METADATA));

      sink.writeUtf8(url)
          .writeByte('\\n');
      sink.writeUtf8(requestMethod)
          .writeByte('\\n');
      sink.writeDecimalLong(varyHeaders.size())
          .writeByte('\\n');
      for (int i = 0, size = varyHeaders.size(); i < size; i++) {
        sink.writeUtf8(varyHeaders.name(i))
            .writeUtf8(": ")
            .writeUtf8(varyHeaders.value(i))
            .writeByte('\\n');
      }

      sink.writeUtf8(new StatusLine(protocol, code, message).toString())
          .writeByte('\\n');
      sink.writeDecimalLong(responseHeaders.size() + 2)
          .writeByte('\\n');
      for (int i = 0, size = responseHeaders.size(); i < size; i++) {
        sink.writeUtf8(responseHeaders.name(i))
            .writeUtf8(": ")
            .writeUtf8(responseHeaders.value(i))
            .writeByte('\\n');
      }
      sink.writeUtf8(SENT_MILLIS)
          .writeUtf8(": ")
          .writeDecimalLong(sentRequestMillis)
          .writeByte('\\n');
      sink.writeUtf8(RECEIVED_MILLIS)
          .writeUtf8(": ")
          .writeDecimalLong(receivedResponseMillis)
          .writeByte('\\n');

      if (isHttps()) {
        sink.writeByte('\\n');
        sink.writeUtf8(handshake.cipherSuite().javaName())
            .writeByte('\\n');
        writeCertList(sink, handshake.peerCertificates());
        writeCertList(sink, handshake.localCertificates());
        sink.writeUtf8(handshake.tlsVersion().javaName()).writeByte('\\n');
      }
      sink.close();
    }

Copy the code

DiskLruCache

The cache feature class for real storage (file format) uses LinkedHashedMap based. You can see that it mainly includes three important inner classes in addition to some key methods.

  1. Entry

An entity class for storing cached data, one URL for each entity, and a Snapshot object in Entry

private final class Entry { final String key; /** Lengths of this entry's files. */ final long[] lengths; final File[] cleanFiles; final File[] dirtyFiles; /** True if this entry has ever been published. */ boolean readable; /** The ongoing edit or null if this entry is not being edited. */ Editor currentEditor; /** The sequence number of the most recently committed edit to this entry. */ long sequenceNumber; Entry(String key) { this.key = key; lengths = new long[valueCount]; cleanFiles = new File[valueCount]; dirtyFiles = new File[valueCount]; // The names are repetitive so re-use the same builder to avoid allocations. StringBuilder fileBuilder = new StringBuilder(key).append('.'); int truncateTo = fileBuilder.length(); for (int i = 0; i < valueCount; i++) { fileBuilder.append(i); cleanFiles[i] = new File(directory, fileBuilder.toString()); fileBuilder.append(".tmp"); dirtyFiles[i] = new File(directory, fileBuilder.toString()); fileBuilder.setLength(truncateTo); } } /** Set lengths using decimal numbers like "10123". */ void setLengths(String[] strings) throws IOException { if (strings.length ! = valueCount) { throw invalidLengths(strings); } try { for (int i = 0; i < strings.length; i++) { lengths[i] = Long.parseLong(strings[i]); } } catch (NumberFormatException e) { throw invalidLengths(strings); } } /** Append space-prefixed lengths to {@code writer}. */ void writeLengths(BufferedSink writer) throws IOException { for (long length : lengths) { writer.writeByte(' ').writeDecimalLong(length); } } private IOException invalidLengths(String[] strings) throws IOException { throw new IOException("unexpected journal line: " + Arrays.toString(strings)); } /** * Returns a snapshot of this entry. This opens all streams eagerly to guarantee that we see a * single published snapshot. If we opened streams lazily then the streams could come from * different edits. */ Snapshot snapshot() { if (! Thread.holdsLock(DiskLruCache.this)) throw new AssertionError(); Source[] sources = new Source[valueCount]; long[] lengths = this.lengths.clone(); // Defensive copy since these can be zeroed out. try { for (int i = 0; i < valueCount; i++) { sources[i] = fileSystem.source(cleanFiles[i]); } return new Snapshot(key, sequenceNumber, sources, lengths); } catch (FileNotFoundException e) { // A file must have been deleted manually! for (int i = 0; i < valueCount; i++) { if (sources[i] ! = null) { Util.closeQuietly(sources[i]); } else { break; } } // Since the entry is no longer valid, remove it so the metadata is accurate (i.e. the cache // size.) try { removeEntry(this); } catch (IOException ignored) { } return null; }}}Copy the code
  1. Snapshot
public final class Snapshot implements Closeable { private final String key; private final long sequenceNumber; private final Source[] sources; private final long[] lengths; Snapshot(String key, long sequenceNumber, Source[] sources, long[] lengths) { this.key = key; this.sequenceNumber = sequenceNumber; this.sources = sources; this.lengths = lengths; } public String key() { return key; } /** * Returns an editor for this snapshot's entry, or null if either the entry has changed since * this snapshot was created or if another edit is in progress. */ public @Nullable Editor edit() throws IOException { return DiskLruCache.this.edit(key, sequenceNumber); } /** Returns the unbuffered stream with the value for {@code index}. */ public Source getSource(int index) { return sources[index]; } /** Returns the byte length of the value for {@code index}. */ public long getLength(int index) { return lengths[index]; } public void close() { for (Source in : sources) { Util.closeQuietly(in); }}}Copy the code
  1. Editor

In the initialization of the Editor, you pass in an Editor, which is the class that edits entry

public final class Editor { final Entry entry; final boolean[] written; private boolean done; Editor(Entry entry) { this.entry = entry; this.written = (entry.readable) ? null : new boolean[valueCount]; } /** * Prevents this editor from completing normally. This is necessary either when the edit causes * an I/O error, or if the target entry is evicted while this editor is active. In either case * we delete the editor's created files and  prevent new files from being created. Note that once * an editor has been detached it is possible for another editor to  edit the entry. */ void detach() { if (entry.currentEditor == this) { for (int i = 0; i < valueCount; i++) { try { fileSystem.delete(entry.dirtyFiles[i]); } catch (IOException e) { // This file is potentially leaked. Not much we can do about that. } } entry.currentEditor = null; } } /** * Returns an unbuffered input stream to read the last committed value, or null if no value has * been committed. */ public Source newSource(int index) { synchronized (DiskLruCache.this) { if (done) { throw new IllegalStateException(); } if (! entry.readable || entry.currentEditor ! = this) { return null; } try { return fileSystem.source(entry.cleanFiles[index]); } catch (FileNotFoundException e) { return null; } } } /** * Returns a new unbuffered output stream to write the value at {@code index}. If the underlying * output stream encounters errors when writing to the filesystem, this edit will be aborted * when {@link #commit} is called. The returned output stream does not throw IOExceptions. */ public Sink newSink(int index) { synchronized (DiskLruCache.this) { if (done) { throw new IllegalStateException(); } if (entry.currentEditor ! = this) { return Okio.blackhole(); } if (! entry.readable) { written[index] = true; } File dirtyFile = entry.dirtyFiles[index]; Sink sink; try { sink = fileSystem.sink(dirtyFile); } catch (FileNotFoundException e) { return Okio.blackhole(); } return new FaultHidingSink(sink) { @Override protected void onException(IOException e) { synchronized (DiskLruCache.this) { detach(); }}}; } } /** * Commits this edit so it is visible to readers. This releases the edit lock so another edit * may be started on  the same key. */ public void commit() throws IOException { synchronized (DiskLruCache.this) { if (done) { throw new IllegalStateException(); } if (entry.currentEditor == this) { completeEdit(this, true); } done = true; } } /** * Aborts this edit. This releases the edit lock so another edit may be started on the same * key. */ public void  abort() throws IOException { synchronized (DiskLruCache.this) { if (done) { throw new IllegalStateException(); } if (entry.currentEditor == this) { completeEdit(this, false); } done = true; } } public void abortUnlessCommitted() { synchronized (DiskLruCache.this) { if (! done && entry.currentEditor == this) { try { completeEdit(this, false); } catch (IOException ignored) { } } } } }Copy the code
  1. delete
boolean removeEntry(Entry entry) throws IOException { if (entry.currentEditor ! = null) { entry.currentEditor.detach(); // Prevent the edit from completing normally. } for (int i = 0; i < valueCount; i++) { fileSystem.delete(entry.cleanFiles[i]); size -= entry.lengths[i]; entry.lengths[i] = 0; } redundantOpCount++; journalWriter.writeUtf8(REMOVE).writeByte(' ').writeUtf8(entry.key).writeByte('\\n'); lruEntries.remove(entry.key); if (journalRebuildRequired()) { executor.execute(cleanupRunnable); } return true; }Copy the code