In our business scenario, we need to use the client to collect pictures, upload them to the server, and then identify the picture information. In order to improve the performance of the program, we need to ensure the speed of the image upload server, and ensure the quality of the image used for recognition. The whole optimization includes two aspects:

  1. Camera photography optimization: including camera parameters selection, preview, start speed and photo quality, etc.
  2. Image compression optimization: Compress based on the image taken and the image selected from the album, control the image size and size.

In this article, we’ll focus on image compression optimization, and then we’ll cover how to package and optimize Android cameras. This project is mainly based on the image compression API of Android, combining the advantages of Luban and Compressor, and providing an interface for user-defined compression policies. The main purpose of the project is to unify the realization of the picture compression box library, the integration of two commonly used image compression algorithms, so that you can integrate the image compression function into their own projects at a lower cost.

1. Basic knowledge of picture compression

For general business scenarios, Glide helps us deal with the size of the loaded images when we show them. But before uploading the collected pictures to the server, we need to compress the pictures in order to save traffic.

On Android, there are three types of compression provided by default: mass compression and two-size compression, proximity sampling, and bilinear sampling. The following is a brief introduction to the three compression methods are used:

1.1 Mass compression

The so-called mass compression is the following line of code, which is a Bitmap method. Once we have the Bitmap, we can use this method to achieve quality compression. It is usually the last step in all of our compression methods.

/ / android. Graphics. Bitmap
compress(CompressFormat format, int quality, OutputStream stream)
Copy the code

The method accepts three parameters, with the following meanings:

  1. Format: enumeration, with three optionsJPEG.PNG 和 WEBP, represents the format of the picture;
  2. Quality: indicates the quality of the image[0100]Between, indicates the quality of the picture. The larger the picture, the higher the quality of the picture.
  3. Stream: an output stream, usually a stream of files from which we compress the output

1.2 Adjacent sampling

Adjacent sampling is based on the near point interpolation algorithm, which replaces the surrounding pixels with pixels. The core code for adjacent sampling consists of the following three lines,

BitmapFactory.Options options = new BitmapFactory.Options();
options.inSampleSize = 1;
Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.blue_red, options);
Copy the code

Adjacent to the sampling core lies the calculation of inSampleSize. It is usually the first step in the compression algorithm we use. We can set inSampleSize to get the sample of the original image instead of loading the original image into memory to prevent OOM. Standard postures are as follows:

    // Get the size of the original image
    BitmapFactory.Options options = new BitmapFactory.Options();
    options.inJustDecodeBounds = true;
    options.inSampleSize = 1;
    BitmapFactory.decodeStream(srcImg.open(), null, options);
    this.srcWidth = options.outWidth;
    this.srcHeight = options.outHeight;

    // The image is loaded into memory
    options.inJustDecodeBounds = false;
    options.inSampleSize = calInSampleSize();
    Bitmap bitmap = BitmapFactory.decodeStream(srcImg.open(), null, options);
Copy the code

There are two main steps, each of which means:

  1. Set Options firstinJustDecodeBoundsTrue to load the image to get the size of the image. At this time, the picture will not be loaded into the memory, so it will not cause OOM. Meanwhile, we can get the size information of the original picture through Options.
  2. Calculate an inSampleSize based on the size information from the image in the previous step, then set the inJustDecodeBounds to false to load the sampled image into memory.

A quick note about inSampleSize: For example, if inSampleSize is 4, the width and height of the compressed image will be 1/4 of the original, and the number of pixels will be 1/16 of the original. InSampleSize usually selects an exponent of 2. If it’s not an exponent of 2, the internal calculation will be closer to an exponent of 2. So, in practice, we avoid the uncertainty caused by internal calculations by explicitly specifying an index of inSampleSize of 2.

1.3 Bilinear sampling

Proximity sampling can effectively control the size of images, but it has several problems. For example, when I need to reduce the width of the image to about 1200, if the original image had a width of 3200, I can only reduce it to 1600 by setting inSampleSize to set the sampling rate to 2. At this point the size of the picture is larger than our requirements. That is, adjacent sampling does not allow for more precise control over the size of the image. If you need more precise control over the image size, bilinear compression is used.

Bilinear sampling adopts bilinear interpolation algorithm. Compared with neighboring sampling, it is simple and crude to select one pixel point to replace other pixels. Bilinear sampling refers to the value of 2×2 points around the corresponding position of the source pixel, and takes the corresponding weight according to the relative position to obtain the target image through calculation.

It’s also relatively easy to use on Android,

Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.blue_red);
Matrix matrix = new Matrix();
matrix.setScale(0.5 f.0.5 f);
Bitmap sclaedBitmap = Bitmap.createBitmap(bitmap, 0.0, bitmap.getWidth()/2, bitmap.getHeight()/2, matrix, true);
Copy the code

CreateBitmap () is applied to the resulting Bitmap and passed into Matrix to specify the scale of image size. The Bitmap returned by this method is the result of bilinear compression.

1.4 Summary of image compression algorithm

In actual use, we usually combine three compression methods. The general steps are as follows:

  1. Use adjacent sampling to sample the original image and control the image to a size slightly larger than the target size to prevent OOM;
  2. Bilinear sampling is used to compress the size of the image and control the size of the image to the size of the target.
  3. The image Bitmap obtained after the above two steps is qualitatively compressed and output to disk.

Of course, Android image coding is essentially done by the Skia library, so in addition to using Android’s own libraries for compression, we can also call external libraries for compression. In order to pursue higher compression efficiency, we usually process images in the Native layer, which will involve JNI knowledge. I have introduced the general idea of JNI invocation on Android platform in the previous article “Summary of using JNI in Android”, interested students can refer to it.

Github is an open source image compression library

Currently, there are two main picture compression frameworks on Github: Luban and Compressor. The number of stars is also relatively high, one at 9K and the other at 4K. However, the two image compression libraries have their own advantages and disadvantages. Here’s a table to summarize:

The framework advantages disadvantages
Luban It is said to be based on wechat image compression inverse algorithm 1. It is only suitable for general picture display scenes and cannot accurately compress the size of pictures; 2. Internal encapsulation AsyncTaks to carry out asynchronous picture compression, RxJava support is not good.
Compressor 1. The size of the picture can be compressed; 2. Support RxJava. 1. The size compression scenario is limited, if there are special requirements, you need to manually modify the source code; 2. There was a problem in the calculation of image compression sampling, resulting in the size of the sampled image always smaller than the size we specified

The chart above has been summarized in great detail. So, based on the strengths and weaknesses of the two libraries above, we are going to develop a new image compression framework. It satisfies the following functions:

  1. Support for RxJava: Like using Compressor, we can specify the thread to compress the image and the thread to listen to the result.
  2. Support for Luban compression algorithm: The core part of Luban compression algorithm is only inSampleSize calculation, so we can easily integrate it into our new library. Luban was added to make our library suitable for general image presentation scenarios. The user does not need to specify the size of the image, which is easy to use.
  3. Support the Compressor compression algorithm and specify more parameters: The Compressor compression algorithm is the sum of the three compression algorithms mentioned above. However, it provides only one scenario when the aspect ratio to be compressed is inconsistent with that of the original image. This will be explained in more detail when we introduce our framework below. Of course, you can actively calculate an aspect ratio before calling the framework’s methods, but you’ll need to actively go through the first stage of image compression, painstakingly.
  4. Provide an interface for user – defined compression algorithms: We want to design a library that allows user – defined compression strategies. When you want to replace the image compression algorithm, you can change the policy directly through a method of the chain call. That is, we want to enable users to replace the image compression algorithm in the project at the lowest cost.

3. Overall project architecture

The following is the overall architecture of our image compression framework, here we only enumerate some of the core code. The Compress here is the starting point for our chained call, which we can use to specify the basic parameters for image compression. Then, when we use its strategy() method, the method will enter the image compression strategy. At this point, we continue to chain call the custom method of compression strategy, and set the parameters of each compression strategy individually:

All of the compression strategies here inherit from the AbstractStrategy base class, which provides two default implementations of Luban and Compressor. The CompressListener and CacheNameFactory interfaces are used to listen for the image compression progress and the name of a custom compressed image, respectively. The following three are image-related utility classes that users can invoke to implement their own compression strategies.

4, the use of

First, add the address of my Maven repository to my project Gradle:

maven { url "https://dl.bintray.com/easymark/Android" }
Copy the code

Then, in your project’s dependencies, add the library’s dependencies:

Implementation 'me.shouheng.com pressor: compressor: 0.0.1'Copy the code

You can then use it in your project. You can see how the Sample project is used. However, here is a brief explanation of some of its apis.

4.1 Use of Luban

The following is an example of the use of the Luban compression strategy, which is similar to the use of the Luban library. In addition to Luban’s library, we added a copy option, which indicates whether to copy the original image to the specified directory if the image is not compressed because it is smaller than the specified size. Because, for example, when you use a callback to get a compressed image, if you follow the logic of the Luban library, you will get the original image, so you need to make extra judgments. Therefore, we have added this Boolean parameter that allows you to specify that the original file should be copied so that you do not need to determine whether the original image is the original image in the callback.

    // Specify Context and the File to Compress in the with() method of Compress
    val luban = Compress.with(this, file)
        If you don't use RxJava, you can use it to handle compressed results
        .setCompressListener(object : CompressListener{
            override fun onStart(a) {
                LogUtils.d(Thread.currentThread().toString())
                Toast.makeText(this@MainActivity."Compress Start", Toast.LENGTH_SHORT).show()
            }

            override fun onSuccess(result: File?).{ LogUtils.d(Thread.currentThread().toString()) displayResult(result? .absolutePath) Toast.makeText(this@MainActivity."Compress Success : $result", Toast.LENGTH_SHORT).show()
            }

            override fun onError(throwable: Throwable?). {
                LogUtils.d(Thread.currentThread().toString())
                Toast.makeText(this@MainActivity."Compress the Error:$throwable", Toast.LENGTH_SHORT).show()
            }
        })
        // Compressed image name factory method, used to specify the file name of the compressed result
        .setCacheNameFactory { System.currentTimeMillis().toString() }
        // Image quality
        .setQuality(80)
        // The image compression policy is Luban
        .strategy(Strategies.luban())
        // If the image is less than or equal to 100K, it will not be compressed
        .setIgnoreSize(100, copy)

        // Once you have a Luban instance as above, there are two ways to start image compression
        // Boot mode 1: use RxJava for processing
        val d = luban.asFlowable()
            .subscribeOn(Schedulers.io())
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe { displayResult(it.absolutePath) }
    
        // Start mode 2: start directly. In this case, the encapsulated AsyncTask is compressed. The compressed result can only be processed in the above callback
        luban.launch()
Copy the code

4.2 Using Compressor

The following is the basic usage of the compression strategy. Before calling the strategy() method to specify the compression strategy, your task is the same as Luban. Therefore, if you need to change the image compression algorithm, you can directly use the strategy() method to change the strategy, the previous part of the logic does not change, therefore, you can reduce the cost of changing the compression strategy.

    val compressor = Compress.with(this, file)
        .setQuality(60)
        .setTargetDir("")
        .setCompressListener(object : CompressListener {
            override fun onStart(a) {
                LogUtils.d(Thread.currentThread().toString())
                Toast.makeText(this@MainActivity."Compress Start", Toast.LENGTH_SHORT).show()
            }

            override fun onSuccess(result: File?).{ LogUtils.d(Thread.currentThread().toString()) displayResult(result? .absolutePath) Toast.makeText(this@MainActivity."Compress Success : $result", Toast.LENGTH_SHORT).show()
            }

            override fun onError(throwable: Throwable?). {
                LogUtils.d(Thread.currentThread().toString())
                Toast.makeText(this@MainActivity."Compress the Error:$throwable", Toast.LENGTH_SHORT).show()
            }
        })
        .strategy(Strategies.compressor())
        .setMaxHeight(100f)
        .setMaxWidth(100f)
        .setScaleMode(Configuration.SCALE_SMALLER)
        .launch()
Copy the code

Here setMaxHeight(100f) and setMaxWidth(100f) are used to indicate the target size for image compression. How do you calculate the exact size? In the Compressor library you cannot be sure, but in our library you can specify this using the setScaleMode() method. This method accepts an enumeration of integer types that can be in the range of SCALE_LARGER, SCALE_SMALLER, SCALE_WIDTH, and SCALE_HEIGHT, which we’ll explain in more detail. The default compression method is SCALE_LARGER, which is the compression method of the Compressor library. So what do these four parameters mean?

Here we use an example to illustrate, suppose there is an image width 1000, height 500, short writing (W:1000, H:500), setMaxHeight() and setMaxWidth() specified parameters are 100, then, called the size of the target image, The width is 100, the height is 100, and the short form is (W:100, H:100). Then according to the above four compression methods, the final result will be:

  • (W:100, H:50) SCALE_LARGER: Compress the larger of the height and the larger of the length, and the other is self-adaptive, so that the compression result is (W:100, H:50). That is, since the original image has a 2:1 aspect ratio, we need to keep that ratio and then compress it. The target aspect ratio is 1:1. The width of the original image is relatively large, so we choose the width as the base of compression, and reduce the width by 10 times and the height by 10 times. This is the default compression strategy of the Compressor library, which obviously prioritizes the resulting image size. This is fine in normal situations, but when you want to keep the short edge to 100, you can’t do anything about it (you need to pass the parameter after the calculation), so use SCALE_SMALLER instead.
  • SCALE_SMALLER: compresses the larger of the height and length and the other ADAPTS, so the result is (W:200, H:100). That is, you shrink the height by 5 times to reach 100, and then you shrink the width by 5 times to reach 200.
  • SCALE_WIDTH: Compressed width and adaptive height. The result is consistent with SCALE_LARGER.
  • SCALE_HEIGHT: The height is compressed and the width is adaptive, so the result is consistent with SCALE_HEIGHT.

4.3 User-defined Policies

It’s also easy to customize an image compression strategy. You can do this by inheriting from SimpleStrategy or directly from AbstractStrategy:

class MySimpleStrategy: SimpleStrategy() {

    override fun calInSampleSize(a): Int {
        return 2
    }

    fun myLogic(a): MySimpleStrategy {
        return this}}Copy the code

Note that to implement chained calls, the custom compression strategy method needs to return itself.

5, the last

Because in our project, we need to control the short side of the picture to 1200, and the change of the length can only be adapted, and the sampling rate can only be changed by changing the Luban. The side length can only be controlled to a range, which cannot be accurately compressed. Therefore, we thought of Compressor and proposed the compression mode of SCALE_SMALLER. But Luban is not useless, generally used for display image compression, it is more convenient to use. Therefore, we combined the two frameworks in the library, which is not a lot of code. Of course, in order to make our library more functional, so we put forward a custom compression strategy interface, is also used to reduce the replacement cost of compression strategy.

Finally, the project is open source on Github, the address is: github.com/Shouheng88/… Star and Fork are welcome to contribute code or issue to this project 🙂

Later, the author will be on Android camera optimization and JNI operation OpenCV image processing to explain, interested in the author yo 🙂

Thanks for reading