OkHttp is a powerful HTTP client that is included in Android. Okio is a low-level IO library that complices java.io and java.nio, making it much easier to access, store, and process data. Here’s its website: square.github. IO /okio/. It started out as a component of OKHttp, but now it can be used on its own to solve IO problems.

ByteString and Buffer

Okio is built around these two types, which integrate a lot of functionality into a simple API:

ByteString is an immutable sequence of bytes. String is based on characters, and ByteString, like String’s cousin, can easily treat binary data as some value. This class is very clever: it knows how to encode and decode itself in hexadecimal, Base64, and UTF-8.

Buffers are mutable sequences of bytes. As with ArrayList, there is no need to set the buffer size beforehand. Read and write buffers as queues: write data to the end and then read from the queue head. There is no need to manage read location, range, or capacity.

Internally, ByteString and Buffer do some clever things to save CPU and memory. If you encode a UTF-8 string as a ByteString, it caches a reference to that string so that it doesn’t have to do anything to decode it later.

Buffers are implemented as a linked list of segments. When you move data from one buffer to another, it reassigns ownership of segments rather than copying data across buffers. This approach is particularly useful for multithreaded programs: threads associated with network requests can exchange data with worker threads without any duplication or redundant operations.

The Source and Sink

One elegant design in java.io is how streams are layered to handle transformations such as encryption and compression. Okio also has its own stream types: Source and Sink, similar to Java’s Inputstream and Outputstream respectively, but with some key differences:

• Timeout: The flow provides access to the underlying I/O Timeout mechanism. Unlike java.io’s socket stream, both the read() and write() methods give timeout mechanisms.
• Simple implementation: Source declares only three methods: read(), close(), and timeout(). There are no performance degradation issues like available() or single-byte reads.
• Ease of use: Although only three methods need to be implemented in source and sink, callers can implement the Bufferedsource and Bufferedsink interfaces, which provide a rich API for everything you need.
• There is no intuitive difference between byte streams and character streams: they are both data. You can read and write in bytes, UTF-8 strings, big-endian 32-bit integers, little-endian short integers, whatever you want; No more InputStreamReader needed!
• Simple to test: The Buffer class implements both BufferedSource and BufferedSink, so the test code is straightforward.

Sources and Sinks interact with InputStream and OutputStream respectively. You can treat any Source as an InputStream, and you can treat any InputStream as a Source. The same is true for sinks and OutputStreams.

Example use of Okio

This is its Maven-style dependency:

<dependency>
    <groupId>com.squareup.okio</groupId>
    <artifactId>okio</artifactId>
    <version>2.9.0</version>
</dependency>
Copy the code

1. Read text line by line

public void readLines(File file) throws IOException {
    try (Source fileSource = Okio.source(file);
         BufferedSource bufferedSource = Okio.buffer(fileSource)) {
        while (true) {
            String line = bufferedSource.readUtf8Line();
            if (line == null) break; System.out.println(line); }}}Copy the code

The readUtf8Line() API reads all data up to the next delimiter \n, \r\n, or the end of the file. It returns the data as a string and omits the delimiter at the end. This method returns an empty string when an empty line is encountered. It returns NULL if there is no more data to read, so it is OK to use for instead of while(true), which makes the program more compact:

public void readLines(File file) throws IOException {
    try (BufferedSource source = Okio.buffer(Okio.source(file))) {
        for(String line; (line = source.readUtf8Line()) ! =null;) { System.out.println(line); }}}Copy the code

2. Write the string to a text file

Above we used Source and BufferedSource to read the file. When writing files, we use a Sink and a BufferedSink. They have something in common: more powerful apis and higher performance.

public void writeToFile(File file) throws IOException {
    try (Sink fileSink = Okio.sink(file);
         BufferedSink bufferedSink = Okio.buffer(fileSink)) {
        bufferedSink.writeUtf8("Hello");
        bufferedSink.writeUtf8("\n");
        bufferedSink.writeAll(Okio.source(new File("my.txt"))); }}Copy the code

3. Utf-8 encoding

In the above API, you can see that Okio is very fond of UTF-8. Early computer systems encountered many incompatible character encodings: ISO-8859-1, ASCII, EBCDIC, etc. Writing software that supports multiple character sets sucks, we don’t even have emojis! Today, we are fortunate that utF-8 has been standardized all over the world, and few other character sets are used in legacy systems.

If you need other character sets, you can use readString() and writeString(). These methods require passing in parameters that specify the character set. Otherwise, you might accidentally create data that can only be read by the local computer, and most programs should just use methods like writeUtf8().

Although we use UTF-8 whenever we read or write strings in I/O, Java strings use the outdated character encoding UTF-16 when they are in memory. This is the wrong encoding because it uses 16-bit characters for most characters, but some characters are inappropriate. In particular, most emojis use two Java characters. This is problematic because String.length() returns a surprising result: the number of UTF-16 characters instead of the font’s original number of characters:

String s1 = "Cafe \ uD83C \ uDF69";
String s2 = "Cafe \ uD83C \ uDF69";
System.out.println(s.length());
System.out.println(s2.length());
Copy the code

In most cases, Okio lets you ignore these issues and focus on the data. But when you need them, there are handy apis for handling low-level UTF-8 strings. Use utf8.size () to count the number of bytes needed to encode the string to UTF-8 (but not actually encode it once). This is handy when dealing with fixed length prefixes such as in protocol buffers.

Using BufferedSource. ReadUtf8CodePoint () reads a Codepoint, and makes the BufferedSink. WriteUtf8CodePoint () writes a Codepoint.

Serialization and deserialization

Okio loves testing. The library itself has been rigorously tested, and one pattern we have found very useful is the “gold value” test, which verifies that the current program can safely decode data encoded using earlier versions of the program.

We’ll illustrate this by encoding values using Java serialization. Although we must deny that Java serialization is a poor coding system, and most programs would prefer some other format like JSON or Protobuf! Anyway, this is a method that takes an object, serializes it, and returns the result as ByteString:

private ByteString serialize(Object o) throws IOException {
  Buffer buffer = new Buffer();
  try (ObjectOutputStream objectOut = new ObjectOutputStream(buffer.outputStream())) {
    objectOut.writeObject(o);
  }
  return buffer.readByteString();
}
Copy the code

Instead of using Java’s ByteArrayOutputstream, you get the output stream object from Buffer and write it to Buffer via ObjectOutputStream. When you write to Buffer, It always writes to the end of the buffer. Finally, a ByteString object is read from the buffer through the buffer object’s readByteString(), which reads from the buffer’s header. The readByteString() method can specify the number of bytes to read, or the entire contents if not specified.

We serialize an object using the method above and output the resulting ByteString as base64:

Point point = new Point(8.15);
ByteString pointBytes = serialize(point);
System.out.println(pointBytes.base64());
Copy the code

rO0ABXNyAA5qYXZhLmF3dC5Qb2ludLbEinI0fsgmAgACSQABeEkAAXl4cAAAAAgAAAAP
Copy the code

Okio calls this string Golden Value. Next, we try to deserialize the string (Golden Value) as a Point object, first returning to ByteString:

public static void main(String[] args) throws Exception {
    Point point = new Point(8.15);
    ByteString pointBytes = new App().serialize(point);
    String base64 = pointBytes.base64();
    System.out.println(base64);

    ByteString byteString = ByteString.decodeBase64(base64);
    Point other = (Point) new App().deserialize(byteString);
    System.out.println(other.equals(point)); // true
}

private Object deserialize(ByteString byteString) throws IOException, ClassNotFoundException {
    Buffer buffer = new Buffer();
    buffer.write(byteString);
    try (ObjectInputStream objectIn = new ObjectInputStream(buffer.inputStream())) {
        Object result = objectIn.readObject();
        if(objectIn.read() ! = -1) throw new IOException("Unconsumed bytes in stream");
        returnresult; }}private ByteString serialize(Object o) throws IOException {
    Buffer buffer = new Buffer();
    try (ObjectOutputStream objectOut = new ObjectOutputStream(buffer.outputStream())) {
        objectOut.writeObject(o);
    }
    return buffer.readByteString();
}
Copy the code

This allows us to change the way objects are serialized without breaking compatibility.

One significant difference between this serialization and Java native serialization is that GodenValue is compatible between different clients (as long as the serialization and deserialization classes are the same). What does that mean? For example, if I serialize a User object using Okio on the PC, the GodenValue string generated by the User object, you can also deserialize the User object on the phone.

5. Write the byte stream to the file

Encoding binaries is no different from encoding text files. Okio uses the same BufferedSink and BufferedSource bytes. This is handy for binary formats that contain both byte and character data. Writing binary data is more dangerous than writing text, because if you make an error, it is often difficult to diagnose. To avoid such an error, note the following:

• Width of each field: the number of bytes. Okio has no mechanism for releasing some bytes. If you need to, you need to shift and mask the bytes yourself before writing them.
• Byte order for each field: All multi-byte fields have terminations: the order of bytes is from highest to lowest (big byte big endian) or lowest to highest (little endian). Okio’s methods for sorting small bytes all have the suffix Le; Methods without suffixes are sorted by large bytes by default.
• Signed and unsigned: Java has no unsigned base type (except char!) Therefore, this situation is often encountered at the application layer. For ease of use, Okio’s writeByte() and writeShort() methods can accept int. You can just pass an unsigned byte like 255, and Okio will do the right thing.

methods	The width of the	Byte ordering	value	Encoded value
writeByte	1		3	03
writeShort	2	big	3	00 03
writeInt	4	big	3	00 00 00 '03
writeLong	8	big	3	00 00 00 00 00 03
writeShortLe	2	little	3	03 00
writeIntLe	4	little	3	03 00 00 00
writeLongLe	8	little	3	03 00 00 00 00 00 00 00
writeByte	1		Byte.MAX_VALUE	7f
writeShort	2	big	Short.MAX_VALUE	7f ff
writeInt	4	big	Int.MAX_VALUE	7f ff ff ff
writeLong	8	big	Long.MAX_VALUE	7f ff ff ff ff ff ff ff
writeShortLe	2	little	Short.MAX_VALUE	ff 7f
writeIntLe	4	little	Int.MAX_VALUE	ff ff ff 7f
writeLongLe	8	little	Long.MAX_VALUE	ff ff ff ff ff ff ff 7f

The following example code encodes a file in the BMP file format:

void encode(Bitmap bitmap, BufferedSink sink) throws IOException {
  int height = bitmap.height();
  int width = bitmap.width();

  int bytesPerPixel = 3;
  int rowByteCountWithoutPadding = (bytesPerPixel * width);
  int rowByteCount = ((rowByteCountWithoutPadding + 3) / 4) * 4;
  int pixelDataSize = rowByteCount * height;
  int bmpHeaderSize = 14;
  int dibHeaderSize = 40;

  // BMP Header
  sink.writeUtf8("BM"); // ID.
  sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.
  sink.writeShortLe(0); // Unused.
  sink.writeShortLe(0); // Unused.
  sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.

  // DIB Header
  sink.writeIntLe(dibHeaderSize);
  sink.writeIntLe(width);
  sink.writeIntLe(height);
  sink.writeShortLe(1);  // Color plane count.
  sink.writeShortLe(bytesPerPixel * Byte.SIZE);
  sink.writeIntLe(0);    // No compression.
  sink.writeIntLe(16);   // Size of bitmap data including padding.
  sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).
  sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).
  sink.writeIntLe(0);    // Palette color count.
  sink.writeIntLe(0);    // 0 important colors.

  // Pixel data.
  for (int y = height - 1; y >= 0; y--) {
    for (int x = 0; x < width; x++) {
      sink.writeByte(bitmap.blue(x, y));
      sink.writeByte(bitmap.green(x, y));
      sink.writeByte(bitmap.red(x, y));
    }

    // Padding for 4-byte alignment.
    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {
      sink.writeByte(0); }}}Copy the code

The code writes binary data to the file in BMP format, which generates a BMP image file. BMP format requires each line to start with 4 bytes, so the code adds a lot of zeros for byte alignment.

The format for encoding other binaries is very similar. Some points to note:

• Writing tests with Golden Values makes debugging easier by confirming the expected results of the program.
• useUtf8.size()Method to calculate the length of the encoded string in bytes. This islength-prefixedFormat is essential.
• useFloat.floatToIntBits()andDouble.doubleToLongBits()To encode floating point values.

6. Use Socket for communication

Sending and receiving data over the network is a bit like reading and writing files. Okio uses BufferedSink to encode the output and BufferedSource to decode the input. Like files, network protocols can be text, binary, or a mixture of both. But there are also some substantial differences between networks and file systems.

When you have a file object, you can only choose to read or write, but the network can read and write at the same time! In some protocols, this problem is handled in rotation: write requests, read responses, and repeat. You can implement this protocol with a single thread. In other protocols, you can read and write simultaneously. Usually you need a dedicated thread to read the data. For writing data, you can use dedicated threads or synchronized so that multiple threads can share a Sink. Okio’s streams are not safe to use in concurrent situations.

For Okio’s Sinks buffer, flush() must be manually called to transfer data to minimize I/O operations. Typically, message-oriented agreements refresh after each message. Note that Okio refreshes automatically when buffered data exceeds a certain threshold. But this is just to save memory and cannot be relied upon for protocol interaction.

Okio. Socket is used to establish a connection. When you create a server or client using the socket, you can use okio.source (socket) to read data and use okio.sink (socket) to write data. These apis also work with SSLSocket.

To cancel the socket connection in any thread, the socket.close () method can be called, which will cause sources and sinks to throw IOException immediately and fail. Okio allows you to set timeout limits for all socket operations, but you don’t need to call socket methods to set timeout limits: Source and Sink provide timeout interfaces, and the API remains valid even if the flow is decorated.

Okio official Demo wrote a simple Socket proxy service to sample complete network interaction, the following is part of the code intercept:

private void handleSocket(final Socket fromSocket) {
    try {
        final BufferedSource fromSource = Okio.buffer(Okio.source(fromSocket));
        final BufferedSink fromSink = Okio.buffer(Okio.sink(fromSocket));
        / /...
        / /...
    }  catch(IOException e) { ..... }}Copy the code

It can be seen that the way of creating sources and sinks through Socket is the same as that through files. In both cases, the source or Sink object corresponding to the Socket is got through okio.source () and then the corresponding decorative buffer object is obtained through okio.buffer (). In Okio, once you create a Source or Sink for a Socket object, you can no longer use InputStream or OutputStream.

Buffer buffer = new Buffer();
for (long byteCount; (byteCount = source.read(buffer, 8192L)) != -1;) { sink.write(buffer, byteCount); sink.flush(); }Copy the code

If you don’t need to flush() every time you write, you can replace the two lines in the for loop with one line bufferedsink.writeAll (source).

You’ll notice that in the read() method 8192 is passed as the number of bytes to read. Any number can be passed here, but Okio prefers 8 kib because that’s the maximum Okio can handle in a single system call. Most of the time application code doesn’t have to deal with such limitations!

int addressType = fromSource.readByte() & 0xff;
int port = fromSource.readShort() & 0xffff;
Copy the code

Okio uses signed types such as byte and short, but generally protocols require unsigned values, and the preferred way to convert signed values to unsigned values in Java is through the bitwise and & operator. Here is a list of byte, short, and integer conversions:

Type	Signed Range	Unsigned Range	Signed to Unsigned
byte	– 128… 127	0… 255	int u = s & 0xff;
short	– 32768… 32767	0… 65535	int u = s & 0xffff;
int	– 2147483648… 2147483647	0…4,294,967,295	long u = s & 0xffffffffL;

There is no basic type in Java that can represent an unsigned long.

7, the hash

Hashing functions are widely used, such as HTTPS certificates, Git submissions, BitTorrent integrity checks, and blockchain blocks, to name a few. Using hashing well can improve application performance, privacy, security, and simplicity. Each cryptographic hash function takes a variable length byte input stream and generates a fixed length string value, called a hash value. Hash functions have the following important properties:

• Deterministic: Every input always produces the same output.
• Unity: Each output byte string has the same probability. It is difficult to find or create different input pairs that produce the same output. Collision.
• Irreversible: Knowing the output does not help you find the input.
• Easy to understand: Hashing is implemented and well understood in many environments.

Okio supports some common hash functions:

• MD5:128-bit (16-byte) encrypted hash. It is both unsafe and outdated because of its low reverse cost! This hash is provided because it is popular and convenient to use on less secure systems.
• Sha-1:160 bits (20 bytes) encrypted hash. Recent studies have shown that it is possible to create SHA-1 collisions. Consider upgrading from SHA-1 to SHA-256.
• Sha-256:256 bits (32 bytes) encrypted hash. Sha-256 is widely understood and costly to reverse. This is the hash that most systems should use.
• Sha-512:512 bits (64 bytes) encrypted hash. Reverse operation is expensive.

Okio can generate encrypted hashes from ByteString:

ByteString byteString = readByteString(new File("README.md"));
System.out.println(" md5: " + byteString.md5().hex());
System.out.println(" sha1: " + byteString.sha1().hex());
System.out.println("sha256: " + byteString.sha256().hex());
System.out.println("sha512: " + byteString.sha512().hex());
Copy the code

Generated from Buffer:

Buffer buffer = readBuffer(new File("README.md"));
System.out.println(" md5: " + buffer.md5().hex());
System.out.println(" sha1: " + buffer.sha1().hex());
System.out.println("sha256: " + buffer.sha256().hex());
System.out.println("sha512: " + buffer.sha512().hex());
Copy the code

Get the hash value from the Source input stream:

try (HashingSink hashingSink = HashingSink.sha256(Okio.blackhole());
     BufferedSource source = Okio.buffer(Okio.source(file))) {
  source.readAll(hashingSink);
  System.out.println("sha256: " + hashingSink.hash().hex());
}
Copy the code

Get the hash value from the Sink output stream:

try (HashingSink hashingSink = HashingSink.sha256(Okio.blackhole());
     BufferedSink sink = Okio.buffer(hashingSink);
     Source source = Okio.source(file)) {
  sink.writeAll(source);
  sink.close(); // Emit anything buffered.
  System.out.println("sha256: " + hashingSink.hash().hex());
}
Copy the code

Okio also supports HMAC (Hash message Authentication code), which combines a secret key value with a hash value. Applications can use HMAC for data integrity and authentication:

ByteString secret = ByteString.decodeHex("7065616e7574627574746572");
System.out.println("hmacSha256: " + byteString.hmacSha256(secret).hex());
Copy the code

You can generate hMacs from ByteString, Buffer, HashingSource, and HashingSink. Note that Okio does not implement HMAC for MD5. Okio using Java Java. Security. MessageDigest for encryption hash and javax.mail crypto. Mac HMAC is generated.

8. Encryption and decryption

Use okio. cipherSink(Sink,Cipher) or okio. cipherSource(Source,Cipher) to encrypt or decrypt the Stream using the block encryption algorithm. The caller is responsible for initializing the encryption or decryption password with the algorithm, the key, and additional parameters specific to the algorithm, such as the initialization vector. The following example shows a typical use of AES encryption, where the key and iv arguments should both be 16 bytes long:

void encryptAes(ByteString bytes, File file, byte[] key, byte[] iv)
    throws GeneralSecurityException, IOException {
  Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
  cipher.init(Cipher.ENCRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));
  try(BufferedSink sink = Okio.buffer(Okio.cipherSink(Okio.sink(file), cipher))) { sink.write(bytes); }}ByteString decryptAesToByteString(File file, byte[] key, byte[] iv)
    throws GeneralSecurityException, IOException {
  Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
  cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));
  try (BufferedSource source = Okio.buffer(Okio.cipherSource(Okio.source(file), cipher))) {
    returnsource.readByteString(); }}Copy the code

The above is part of the translation of OKio official documents, English better words can refer to the official document: OKio Reference, OKio implementation details wait until the source code analysis of the following article to discuss in detail.