Image decoding optimization in WebGL

Author – Oasis team – Tsukiki

Although WebGL doesTexture compression, there is no problem of decoding time when uploading GPU, but compressed image formats such as PNG/JPG/webP are still used in daily applications. These formats need to be converted to bitmaps for rendering in WebGL, i.e. each pixel is represented using RGB or RGBA. This process is calledImage decoding. Image decoding is a very important part of rendering. If you upload an Image object directly (texImage2D) to GPU, it usually takes a long time and blocks the main thread. For example, it will cause animation playback to stall and affect user experience. So, here we do some research and test on some WebGL image decoding schemes in browsers.The first picture is synchronous decoding, the second picture is asynchronous decoding, you can see the obvious relief of animation stuck

This article focuses on testing the image. decode method and createImageBitmap method.

Image.decode

Image.decode can decode images asynchronously without blocking main thread animation and interaction. The usage method is as follows:

const img = new Image();
img.src = '... ';
img.decode().then(function() {
  document.body.appendChild(img);
});
Copy the code

createImageBitmap

ImageBitmap is a data format specifically used for Canvas and WebGL rendering. CreateImageBitmap asynchronously returns a Promise with an ImageBitmap object. CreateImageBitmap can be used within workers, and ImageBitmap can be transferred between workers. CreateImageBitmap accepts multiple data sources, and this article focuses on testing Blob and HTMLImageElement, the two objects most commonly used in rendering engines.

// Use image as the source
createImageBitmap(image).then((imageBitmap) = >{
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, imageBitmap);
})

// Use blob as source
createImageBitmap(blob).then((imageBitmap) = >{
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, imageBitmap);
})
Copy the code

The performance test

After introducing the basic usage of the two asynchronous decoding apis, I tested 100 different 1024 * 1024 images (randomly generated by the script) in 5 ways, comparing the decoding time and texture upload time of the images. The five ways are as follows:

1. Use the createImageBitmap method with Image as the source. (sample)
2. Use createImageBitmap using Blob as source. (sample)
3. Enable the createImageBitmap method for 5 workers. (sample)
4. Decode using image.decode. (sample)
5. Upload textures directly using image. (sample)

After entering the above several tests, the results are obtained (up and down about 100ms) :

1. MacOS (2.6 GHz I7 Chrome 87 6 times less performance)

2. Android U4 (Mi 10 Pro U4 3.21.0.172)

3. Android Chrome (Mi 10 Pro Android Chrome 87)

4. IOS safari (iPhone7 iOS 14.2)

conclusion

Through the above tests, the following conclusions can be drawn:

1. Recommended for Android and Mac ChromecreateImageBitmapThe data source must be usedBlob, decoding can improve performance by about 10% :
   1. If the data source usesBlob, no decoding time; If the data source usesImageFirst, it takes a long time to create bitmap, and second, there is still decoding time in performance (it is expected that there should not be decoding time, which is a Bug in Chrome and has been mentioned to ChromiumissueChrome has officially confirmed the problem.
   2. Called in the workercreateImageBitmapMultithreading capabilities can be utilized to further improve performance by around 15%. Because worker threads are not particularly stable, whether to enable worker decoding is decided by user configuration. Users decide whether to use worker decoding according to the current CPU load, the number of required decoding and business scenarios.
2. Do not use any asynchronous decoding schemes for iOS:
   1. Does not supportcreateImageBitmap;
   2. useImage.decodeThe total time is twice that of synchronous decoding;

According to the above test results and the derived conclusion, the best image request decoding scheme adopted in WebGL is as follows:

The above solution will be applied to OASIS-Engine, welcome to discuss it in PR.