JPEG Basics

JPEG (Joint Photographic Experts Group) is a widely used standard method for lossy compression of Photographic images.

JPG is the most commonly used extension for image files compressed in JPEG format. Other commonly used extensions include.jpeg,.jpe,.jfif, and.jif.

JEPG coding principle

While JEPG files can be encoded in a variety of ways, the most common is using JFIF encoding, which involves the following steps:

  • Color space conversion

    Convert the image from RGB to Y ‘cbCR in different color Spaces. The Y’ component represents the brightness of the pixel, while Cb and Cr represent the “chromatic difference” (blue and red components). Y ‘CBCR color space allows for greater compression without having a significant impact on perceived image quality.

    About “Chromatic aberration”

    “Color” this concept originated in the television industry, the earliest is black and white TV, then need to transport the luminance signal transmit television signals only, namely Y signal can, color TV appeared, people outside of the Y signal increases the two color difference signal to transfer color information, and the aim is for the sake of compatibility with black and white TV set, Because black and white TV only needs to process the Y signal in the signal.

    According to the principle of three primary colors, it is found that the brightness contributed by red, green and blue is different. Green has the highest “brightness”, while blue has the darkest. If the proportion of brightness contributed by red is KR, and the proportion of brightness contributed by blue is KB, then brightness is

    As a rule of thumb, KR=0.299, KB=0.114, then

    The color difference between blue and red is defined as follows

    Finally, the mathematical formula for RGB conversion to YCbCr can be obtained as

  • The sampling

    The human eye is more sensitive to the brightness of the image (Y’ component) in image fineness than the color of the image (Cb and Cr component).

    For the human eye, the change of light and shade in the image is more easily perceived, which is caused by the structure of the human eye. There are two types of photoreceptors in the retina, rods, which sense changes in brightness, and cones, which sense colors. Since rods are much more numerous than cones, we are more likely to perceive light and dark details. Take the picture below

    Just keep the Y prime component

    Just keep the Cb component

    Just keep the Cr component

    Using this property, the Y ‘CBCR color space can be further downsampled, that is, the spatial resolution of Cb and Cr components can be reduced.

    The lower sampling rate is “4:4:4”, indicating that no lower sampling is carried out.

    A downsampling rate of “4:2:2” means a horizontal decrease of 2 times

    A lower sampling rate of “4:2:0” means a two-fold reduction in both horizontal and vertical directions (most commonly used)

    The downsampling rate is usually expressed as a three-part ratio J: A :b, or four parts if transparency is present, which describes the number of luminance and chroma samples in a conceptual region j pixels wide and 2 pixels high.

    • J represents the horizontal sampling rate reference (width of concept area)
    • A represents the chromatic aberration sampling (Cr, Cb) of the first row.
    • B represents the color difference sampling (Cr, Cb) change between the second and first rows

  • Block segmentation

    After downsampling, each channel must be split into 8×8 pixel blocks, with the minimum coding unit (MCU) depending on the downsampling used.

    If the down-sampling rate is “4:4:4”, the size of the minimum coding unit block is 8×8.

    If the down-sampling rate is “4:2:2”, the minimum coding unit block size is 16×8.

    If the down-sampling rate is “4:2:0”, the minimum coding unit block size is 16×16.

    If the channel data cannot be cut into integer multiples of blocks, it is usually filled with a solid color, such as black.

  • Discrete cosine transform

  • quantitative

    The human eye is good at seeing small brightness differences over relatively large areas, but not very good at distinguishing the exact intensity of high-frequency brightness changes. This allows one to greatly reduce the amount of information in the high-frequency component. You simply divide each component in the frequency domain by its constant, and then round (lossy) to the nearest integer.

    This step is irreversible

  • All 8×8 blocks of data are further compressed using lossless algorithms (a variant of Huffman coding).

JEPG compression effect

The picture Quality ([1,100]) Size (bytes) The compression ratio
Highest quality (100) 81447 2.7:1.
High quality (50) 14679 15:1
Medium quality (25) 9407 then
Low quality (10) 4787 His journey
Minimum quality (1) 1523 144:1.

JEPG coding implementation

  • libjpeg

    Widely used C library for reading and writing JPEG image files.

  • libjpeg-turbo

    High-performance JEPG image decomcoder that uses SIMD instructions to accelerate JEPG file compression and decompression on x86, x86-64, Arm, and PowerPC systems, as well as progressive compression on x86, x86-64 systems.

    Libjpeg-turbo is 2-6 times faster than libjpeg on x86 and x86-64 systems, and can be significantly faster than libjpeg on other systems.

Android image decoding

To display an image on Android, the image needs to be decoded into a Bitmap object, which represents a collection of image pixels. The memory occupied by pixels depends on the Bitmap configuration. Currently, Android supports the following configurations:

  • ALPHA_8

    Only transparency channels are stored

  • ARGB_4444

    Each pixel uses 2 bytes of storage

  • ARGB_8888

    Use 4 bytes of storage per pixel (default)

  • HARDWARE

    Special configuration, Bitmap data stored in dedicated graphics memory (Native)

  • RGBA_F16

    Each pixel uses 8 bytes of storage

  • RGB_565

    Each pixel uses 2 bytes of storage, with only RGB channels.

The source code parsing

. Usually, we can call BitmapFactory decodeStream method from the image stream decoding, Java is just a simple layer of entrance, related implementation in the Native BitmapFactory. DoDecode method.

// frameworks/base/libs/hwui/jni/BitmapFactory.cpp
static jobject doDecode(JNIEnv* env, std::unique_ptr<SkStreamRewindable> stream,jobject padding, jobject options, jlong inBitmapHandle,jlong colorSpaceHandle) {
  // ...  
}
Copy the code

1. Initialize related parameters

  • sampleSize

    Sampling rate

  • onlyDecodeSize

    Whether to decode size only

  • prefCodeType

    Preferred color type

  • isHardware

    Whether to store in dedicated image memory

  • isMutable

    If the variable

  • scale

    The zoom factor

  • requireUnpremultiplied

    Whether the color channel does not require a “preloaded” transparent channel

  • javaBitmap

    Reusable Bitmaps

Create a decoder

According to the image format decoded, create different decoders SkCodec.

Image format SkCodec
JPEG SkJpegCodec
WebP SkWebpCodec
Gif SkGifCodec
PNG SkPngCodec

SkCodec is responsible for the core implementation, and SkAndroidCodec is a wrapper class for SkCodec that provides some Android-specific apis. Similarly, SkAndroidCodec also creates different SkAndroidCodec based on the image format.

Image format SkAndroidCodec
JPEG, PNG, Gif SkSampledCodec
WebP SkAndroidCodecadapter

Create a memory allocator

Depending on whether there are reusable bitmaps and whether scaling is required, a different memory Allocator is used.

Allocate pixel memory

Call the skbitmap. tryAllocPixels method to try to allocate the required pixel memory.

  • Java Heap OOM
  • Native Heap OOM
  • The reusable Bitmap used is too small

Five. Perform decoding

Start the encoding operation. Call SkAndroidCodec getAndroidPixels method.

SkCodec::Result SkAndroidCodec::getAndroidPixels(const SkImageInfo& requestInfo,
        void* requestPixels, size_t requestRowBytes, const AndroidOptions* options) {
  // ...
return this->onGetAndroidPixels(requestInfo,requestPixels,requestRowBytes,*options);
}
Copy the code

SkAndroid onGetAndroidPixels method has two, respectively is SkSampledCodec and SkAndroidCodecadapter.

Here we take JPEG image decoding as an example, from the above, it uses SkSampledCodec and SkJpegCodec, SkJpegCodec is the core implementation.

In addition to supporting full decoding using BitmapFactory, Android also supports partial decoding using BitmapRegionDecoder, which is especially useful when working with very large images.

Android JPEG compression

Android has a problem with image compression. IOS displays images of the same size with more detail, which means better compression. For a more detailed discussion of this problem, see github.com/bither/bith…

Skia is an open source 2D rendering engine used by Android. Skia relies on the libjpeg library to decomcode JPEG image files. Optimize_coding, which is TRUE, provides better compression but also consumes more time.

Up to 7.0, Google has set this value to FALSE for compatibility with less powerful devices, 7.0 and above has been set to TRUE.

For more information about optimize_coding being FALSE, see groups.google.com/g/skia-disc…

Below 7.0: androidxref.com/6.0.1_r10/x…

7.0 and above: androidxref.com/7.0.0_r1/xr…

As a result, JPEG image files can be compressed based on libjPEg-Turbo in versions 7.0 and below.

The source code parsing

Image compression can be done by calling the bitmap.press method. Optional configurations are:

  • format

    Compressed image format, JPEG, PNG, WEBP.

  • quality

    Compression quality, available values range from 0 to 100.

Again, the Java layer only provides API entry and implements the bitmap.bitmap_comperss () method in the Native layer.

// framework/base/libs/hwui/jni/Bitmap.cpp
static jboolean Bitmap_compress(JNIEnv* env, jobject clazz, jlong bitmapHandle,jint format, jint quality,jobject jstream, jbyteArray jstorage) {}Copy the code

Create an encoder

Create different encoders based on the image format.

Image format The encoder
JPEG SkJpegEncoder
PNG SkPngEnccoder
WebP SkWebpEncoder

Set coding parameters

Android JPEG decoding is dependent on libjpeg and libjpeg-Turbo.

A series of parameters are set before compression encoding begins.

  • Image size

  • Color type

    Common color types are:

    JCS_EXT_BGRA,           /* blue/green/red/alpha */
JCS_EXT_BGRA,           /* blue/green/red/alpha */
Copy the code

  • The sampling rate

    Currently Android supports “4:2:0” (default), “4:2:2” and “4:4:4”.

  • Best Huffman coding table

    The default value is true, indicating that using the best Huffman encoding table reduces the compression performance but improves the compression efficiency.

    // Tells libjpeg-turbo to compute optimal Huffman coding tables
// for the image. This improves compression at the cost of
// slower encode performance.
fCInfo.optimize_coding = TRUE;
Copy the code

  • The quality of

    This parameter affects the “quantization” step in JPEG encoding

Three. Perform coding

// external/skia/src/imagess/SkImageEncoder.cpp
bool SkEncoder::encodeRows(int numRows) {
	// ...
  this->onEncodeRows(numRows);
}
Copy the code

JPEG image encoding is implemented by SkJpegEncoder.

// txternal/skia/src/images/SkJpegEncoder.cpp
bool SkJpegEncoder::onEncodeRows(int numRows) {
	// ...
  for (int i = 0; i < numRows; i++) {
    // Perform the libjpeg-turbo encoding operation
  	jpeg_write_scanlines(fEncoderMgr->cinfo(), &jpegSrcRow, 1); }}Copy the code

Sampling algorithm

When adjusting the size of the image, it is necessary to reprocess the pixel data of the original image, which is called image sampling processing.

Currently, Nearest Neighbor and Bilinear sampling algorithms are supported by default on Android.

  • Nearest neighbor

    Each pixel in a resampled grid gets the same value as the nearest pixel in the original grid, which is the fastest processing time, but also causes the image to jagged.

  • Bilinear (Bilinear sampling)

    Each pixel in the resampled grid is the result of a weighted average of the 2×2 4 nearest pixels in the original grid.

In addition to the above two, there are several better algorithms:

  • Bicubic (Double cube sampling)

    Each pixel in the resampled grid is the result of a weighted value of 4×4 16 nearest pixel values from the original grid, with closer pixels having higher weights.

  • Lanczos

    Higher-order interpolation algorithm, which considers more surrounding pixels and retains the most image information.

  • Magic Kernel

    Fast and efficient, but it can produce amazing clear and sharp as a result, more detail: www.johncostella.com/magic/.

Spectrum

Spectrum is Facebook’s open source cross-platform image transcoding dependency library. Compared with jPEG-Turbo, which comes with the default Android operating system, Spectrum has the following advantages:

  • JPEG encoding is based on MozJPEG, which improves compression rates compared to JPEG-Turbo, but also increases compression processing time.
  • Bicubic and Magic Kernel sampling algorithms are supported.
  • The core is implemented using CPP, which can achieve consistent compression effect on both Android and iOS platforms.
  • Support for more custom configurations, including chroma sampling mode and more.

The benchmark

Using Google/Butteraugli to compare the quality difference between the original image and the compressed image, the smaller the value, the better.

Device information: Huawei P20 Pro, Android 10

A Compression mass 80

The core The compression quality Chroma sampling mode The quality differences The file size Time consuming The compression ratio
The original image – S444 – 8.7 MB – –
jpeg-turbo 80 S444 2.943352 2.5 MB 2255ms 71%
mozjpeg 80 S444 2.486266 2.8 MB 3567ms 67%
mozjpeg 80 S420 2.493475(-15%) 2.3 MB 2703ms 73% (+ 2%)

B Compression mass 75

The core The compression quality Chroma sampling mode The quality differences The file size Time consuming The compression ratio
The original image – S444 – 8.7 MB – –
jpeg-turbo 75 S444 3.075884 2.3 MB 2252ms 73%
mozjpeg 75 S444 2.698983 2.4 MB 3188ms 72%
mozjpeg 75 S420 2.670076(-13%) 2MB 2470ms 77% (+ 4%)

C Compression mass 70

The core The compression quality Chroma sampling mode The quality differences The file size Time consuming The compression ratio
The original image – S444 – 8.7 MB – –
jpeg-turbo 70 S444 2.739794 2.1 MB 2230ms 75%
mozjpeg 70 S444 2.838595 2.2 MB 3089ms 74%
mozjpeg 70 S420 2.810702 (+ 2%) 1.8 MB 2404ms 79% (+ 4%)

D Compression mass 65

The core The compression quality Chroma sampling mode The quality differences The file size Time consuming The compression ratio
The original image – S444 – 8.7 MB – –
jpeg-turbo 65 S444 3.734105 1.9 MB 2227ms 78%
mozjpeg 65 S444 3.177706 2MB 2775ms 77%
mozjpeg 65 S420 3.251182(-12%) 1.6 MB 2116ms 81% (+ 3%)

E Compression mass 60

The core The compression quality Chroma sampling mode The quality differences The file size Time consuming The compression ratio
The original image – S444 – 8.7 MB – –
jpeg-turbo 60 S444 4.526981 1.8 MB 2189ms 79%
mozjpeg 60 S444 3.486347 1.8 MB 2454ms 79%
mozjpeg 60 S420 3.479777(-23%) 1.5 MB 2035ms 82% (+ 3%)

According to the above data, the average compression rate and image quality of MozJPEG + S420 are improved by 3% and 12% compared to JPEG-Turbo + S444.

Refer to the link

www.cnblogs.com/Arvin-JIN/p…

zh.wikipedia.org/wiki/JPEG

Scc.ustc.edu.cn/zlsc/sugon/…

www.robertstocker.co.uk/jpeg/jpeg_n…

libjpeg.sourceforge.net/

libjpeg-turbo.org/

Github.com/bither/bith…

Groups.google.com/g/skia-disc…

Mp.weixin.qq.com/s/H9Tz1n4O2…

Abraia. Me/docs/image -…