Title: Analysis of the optimization idea of Glide loading Gif

date: 2020-07-26 10:08

category: NDK

tag: Glide

Project address: GifSampleForGlide

Based on Glide 4.9.0 version analysis

preface

Glide picture framework can load GIF directly, but when doing a bank cooperation project, because there is a need to load a GIF page, but found that using Glide framework to load GIF pictures, obviously found that there is a delay.

After viewing glide loading Gif picture source that Glide in loading Gif picture frame, the rendering of the last frame and the preparation of the next frame is serial, this process, if the next frame of the preparation stage time more than the length of the Gif interval playback, will cause the playback of the card. Also,StandardGifDecoder only keeps the data from the last frame, and every time it retrieves the current frame it needs to draw a new Bitmap from the BitmapPool (note that this is a new Bitmap object), so Glide needs at least two Bitm to load the Gif ap. This causes memory consumption to be too high.

Glide is how to load Gif, and how to optimize the stuck:

Glide loading Gif principle


This article is introduced around the following keywords

  • Glide
  • StreamGifDecoder
  • ByteBufferGifDecoder
  • StandardGifDecoder
  • GifDrawable

1) Let’s start with gif-related decoders

Information about GIFs can be found in Glide’s construction.


Glide(
      @NonNull Context context,
     / *... * /) {
    / /...
 List<ImageHeaderParser> imageHeaderParsers = registry.getImageHeaderParsers();  / /..  GifDrawableBytesTranscoder gifDrawableBytesTranscoder = new GifDrawableBytesTranscoder();   / /...  registry  / /...  /* GIFs */  .append(  Registry.BUCKET_GIF,  InputStream.class,  GifDrawable.class,  new StreamGifDecoder(imageHeaderParsers, byteBufferGifDecoder, arrayPool))  .append(Registry.BUCKET_GIF, ByteBuffer.class, GifDrawable.class, byteBufferGifDecoder)  .append(GifDrawable.class, new GifDrawableEncoder())  /* GIF Frames */  // Compilation with Gradle requires the type to be specified for UnitModelLoader here.  .append(  GifDecoder.class, GifDecoder.class, UnitModelLoader.Factory.<GifDecoder>getInstance())  .append(  Registry.BUCKET_BITMAP,  GifDecoder.class,  Bitmap.class,  new GifFrameResourceDecoder(bitmapPool))  / /...  .register(GifDrawable.class, byte[].class, gifDrawableBytesTranscoder);   ImageViewTargetFactory imageViewTargetFactory = new ImageViewTargetFactory();  //.... } Copy the code

So the first step is to see that Glide decodes the InputStream of giFs by creating a StreamGifDecoder.


public class StreamGifDecoder implements ResourceDecoder<InputStream.GifDrawable> {

  @Override
  public Resource<GifDrawable> decode(@NonNull InputStream source, int width, int height,  @NonNull Options options) throws IOException {  // 1. Use a byte array to receive an InputStream  byte[] data = inputStreamToBytes(source);  if (data == null) {  return null;  }  // 2. Use the ByteBuffer wrapper to process the original data stream.  // Why use ByteBuffer?  / * * @link StandardGifDecoder#setData();  // Initialize the raw data buffer.  rawData = buffer.asReadOnlyBuffer();  rawData.position(0); rawData.order(ByteOrder.LITTLE_ENDIAN); // Align the small ends. Sort from lowest to highest* /  ByteBuffer byteBuffer = ByteBuffer.wrap(data);  return byteBufferDecoder.decode(byteBuffer, width, height, options);  } }  Copy the code

Details are as follows:

  • Receive an InputStream using the byte[] array
  • Then byte[] after passing the processing is delivered to ByteBufferGifDecoder for the next stage of processing (to improve the decoding of InputStream);

public class ByteBufferGifDecoder implements ResourceDecoder<ByteBuffer.GifDrawable> {

 / /...

 @Override  public GifDrawableResource decode(@NonNull ByteBuffer source, int width, int height,  @NonNull Options options) {  final GifHeaderParser parser = parserPool.obtain(source);  try {  return decode(source, width, height, parser, options);  } finally {  parserPool.release(parser);  }  }   @Nullable  private GifDrawableResource decode(  ByteBuffer byteBuffer, int width, int height, GifHeaderParser parser, Options options) {  long startTime = LogTime.getLogTime();  try {  // 1. Get the GIF header information  final GifHeader header = parser.parseHeader();  if (header.getNumFrames() <= 0|| header.getStatus() ! = GifDecoder.STATUS_OK) { // If we couldn't decode the GIF, we will end up with a frame count of 0.  return null;  }  //2. Determine the type of Bitmap based on whether the GIF background has transparent channel (Alpha)  Bitmap.Config config = options.get(GifOptions.DECODE_FORMAT) == DecodeFormat.PREFER_RGB_565  ? Bitmap.Config.RGB_565 : Bitmap.Config.ARGB_8888;   //3. Calculate the Bitmap sampling rate  int sampleSize = getSampleSize(header, width, height);   //4. Get Gif data from StandardGifDecoder====> by static inner class GifDecoderFactory  GifDecoder gifDecoder = gifDecoderFactory.build(provider, header, byteBuffer, sampleSize);  gifDecoder.setDefaultBitmapConfig(config);  gifDecoder.advance();   //5. Get the next frame of Gif data  Bitmap firstFrame = gifDecoder.getNextFrame();  if (firstFrame == null) {  return null;  }   Transformation<Bitmap> unitTransformation = UnitTransformation.get();  //6. Construct a GifDrawable from the Gif data frame to play the animation of the Gif frame  GifDrawable gifDrawable =  new GifDrawable(context, gifDecoder, unitTransformation, width, height, firstFrame);  //7. Wrap GifDrawable as GifDrawableResource to maintain GifDrawable recycle and stop playing animation.  return new GifDrawableResource(gifDrawable);  } finally {  if (Log.isLoggable(TAG, Log.VERBOSE)) {  Log.v(TAG, "Decoded GIF from stream in " + LogTime.getElapsedMillis(startTime));  }  }  } }   @VisibleForTesting static class GifDecoderFactory {  GifDecoder build(GifDecoder.BitmapProvider provider, GifHeader header,  ByteBuffer data, int sampleSize) {  // Get a standard Gif decoder that reads Gif frames and draws them as bitmaps for external use  return new StandardGifDecoder(provider, header, data, sampleSize);  }  } Copy the code

A quick summary:

  • First, ByteBufferDecoder extracts the Gif header information
  • Get the background color of the Gif from the header information to set the Config options for the Bitmap
  • The sampling rate is still calculated based on the header information
  • The StandardGifDecoder that gets giFs is used to build GIF frame output into bitmaps for external use
  • Build GifDrawable(for playing GIFs)
  • Build GifDrawableResource(for managing GifDrawable)

2) Second look at Gif image frame acquisition and how to inject image frame into Bitmap

To see how Gif frames are decoded into bitmaps, see StandardGifDecoder


public class StandardGifDecoder implements GifDecoder {
  private static final String TAG = StandardGifDecoder.class.getSimpleName();
  / /...

 // From the decode method of ByteBufferGifDecoder, StandardGifDecoder retrieves the next frame of Gif data for conversion to Bitmap.  @Nullable  @Override  public synchronized Bitmap getNextFrame(a) {  / /...   // Get the frame data of the current Gif frame according to the Gif header information  GifFrame currentFrame = header.frames.get(framePointer);  GifFrame previousFrame = null;  int previousIndex = framePointer - 1;  if (previousIndex >= 0) {  previousFrame = header.frames.get(previousIndex);  }   // Set the appropriate color table.  LCT == local color table; gct == global color table; So what this is telling us is local before global act = currentFrame.lct ! =null ? currentFrame.lct : header.gct;  if (act == null) {  if (Log.isLoggable(TAG, Log.DEBUG)) {  Log.d(TAG, "No valid color table found for frame #" + framePointer);  }  // No color table defined.  status = STATUS_FORMAT_ERROR;  return null;  }   // Reset the transparent pixel in the color table  // Resets the transparency of the pixels in the color table  if (currentFrame.transparency) {  // Prepare local copy of color table ("pct = act"), see #1068  System.arraycopy(act, 0, pct, 0, act.length);  // Forget about act reference from shared header object, use copied version  act = pct;  // Set transparent color if specified.  // This rimmer thinks black is transparent  act[currentFrame.transIndex] = COLOR_TRANSPARENT_BLACK;  }   // Transfer pixel data to image.  // Convert pixel data to an image  return setPixels(currentFrame, previousFrame);  }  / /...    private Bitmap setPixels(GifFrame currentFrame, GifFrame previousFrame) {  // Final location of blended pixels.  // Store the Bitmap pixel data of the previous frame  final int[] dest = mainScratch;   // clear all pixels when meet first frame and drop prev image from last loop  if (previousFrame == null) {  if(previousImage ! =null) {  // Retrieve the Bitmap of the previous frame  bitmapProvider.release(previousImage);  }  previousImage = null;  // And fill the Bitmap pixels with black  Arrays.fill(dest, COLOR_TRANSPARENT_BLACK);  }  if(previousFrame ! =null && previousFrame.dispose == DISPOSAL_PREVIOUS  && previousImage == null) {  // The last frame was discarded  Arrays.fill(dest, COLOR_TRANSPARENT_BLACK);  }   // fill in starting image contents based on last image's dispose code  //1. Inject the last frame into the dest array  if(previousFrame ! =null && previousFrame.dispose > DISPOSAL_UNSPECIFIED) {  if (previousFrame.dispose == DISPOSAL_BACKGROUND) {  // Start with a canvas filled with the background color  @ColorInt int c = COLOR_TRANSPARENT_BLACK;  if(! currentFrame.transparency) { c = header.bgColor;  if(currentFrame.lct ! =null && header.bgIndex == currentFrame.transIndex) {  c = COLOR_TRANSPARENT_BLACK;  }  } else if (framePointer == 0) {  isFirstFrameTransparent = true;  }  // The area used by the graphic must be restored to the background color.  int downsampledIH = previousFrame.ih / sampleSize;  int downsampledIY = previousFrame.iy / sampleSize;  int downsampledIW = previousFrame.iw / sampleSize;  int downsampledIX = previousFrame.ix / sampleSize;  int topLeft = downsampledIY * downsampledWidth + downsampledIX;  int bottomLeft = topLeft + downsampledIH * downsampledWidth;  for (int left = topLeft; left < bottomLeft; left += downsampledWidth) {  int right = left + downsampledIW;  for (int pointer = left; pointer < right; pointer++) {  dest[pointer] = c;  }  }  } else if(previousFrame.dispose == DISPOSAL_PREVIOUS && previousImage ! =null) {  // Start with the previous frame  // Get the data from the Bitmap of the last frame and update the data to Dest.  previousImage.getPixels(dest, 0, downsampledWidth, 0.0, downsampledWidth,  downsampledHeight);  }  }   // Decode pixels for this frame into the global pixels[] scratch.  // 2. Parse the current frame into dest  decodeBitmapData(currentFrame);   if(currentFrame.interlace || sampleSize ! =1) {  copyCopyIntoScratchRobust(currentFrame);  } else {  copyIntoScratchFast(currentFrame);  }   // Copy pixels into previous image  //3. Obtain the dest data of the current frame and store the data in the image(Bitmap) of the previous frame.  if (savePrevious && (currentFrame.dispose == DISPOSAL_UNSPECIFIED  || currentFrame.dispose == DISPOSAL_NONE)) {  if (previousImage == null) {  previousImage = getNextBitmap();  }  previousImage.setPixels(dest, 0, downsampledWidth, 0.0, downsampledWidth,  downsampledHeight);  }   // Set pixels for current image.  // 4. Obtain a new Bitmap, copy the data in Dest to the Bitmap, and provide it to GifDrawable.  Bitmap result = getNextBitmap();  result.setPixels(dest, 0, downsampledWidth, 0.0, downsampledWidth, downsampledHeight);  return result;  }  } Copy the code

After looking at the above code flow, it is not intuitive. Here is a picture for comparison and analysis:


As can be seen from the above figure:

  • Get the frame data from the Bitmap of the previous frame and populate it with the DEST array
  • It then takes the frame number data from this array and fills it into a Bitmap (converting Gif frame data to a preBitmap for the first time)
  • Parse the current frame’s data into the DEST array and store the data in preBitmap
  • Get the new Bitmap from the BitmapProvider(which provides the reuse of bitmaps) and copy the dest array parsed by the current frame into the Bitmap for external use

3)Glide to play GIF animation with GifDrawable 

public class GifDrawable extends Drawable implements GifFrameLoader.FrameCallback.    Animatable.Animatable2Compat {
    
  @Override
  public void start(a) {
 isStarted = true;  resetLoopCount();  if (isVisible) {  startRunning();  }  }   private void startRunning(a) { . if (state.frameLoader.getFrameCount() == 1) {  invalidateSelf();  } else if(! isRunning) { isRunning = true;  Subscribe to GifFrameLoader  state.frameLoader.subscribe(this);  invalidateSelf();  }  }    @Override  public void onFrameReady(a) { . // 2. Perform drawing  invalidateSelf(); . }  } Copy the code

As can be seen from the interface of GifDrawable implementation, GifDrawable is a Drawable of Animatable, so GifDrawable can support GIF animation playback. Another important class is GifFrameLoader, which is used to help GifDrawable realize GIF animation playback The scheduling.

The GifDrawable start method is the entry point for the animation to start. In this method, the GifDrawable is registered with GifFrameLoader as an observer. Once the GifFrameLoader triggers the drawing, the onFrameReady method is called, followed by the call in ValidateSelf performs this drawing.

How does GifFrameLoader perform animation scheduling


class GifFrameLoader {
  / /..
  public interface FrameCallback {
    void onFrameReady(a);
 }   / /..   void subscribe(FrameCallback frameCallback) {  if (isCleared) {  throw new IllegalStateException("Cannot subscribe to a cleared frame loader");  }   if (callbacks.contains(frameCallback)) {  throw new IllegalStateException("Cannot subscribe twice in a row");  }  // Check whether the observer queue is empty  boolean start = callbacks.isEmpty();  // Add an observer  callbacks.add(frameCallback);  // Not empty, perform GIF drawing  if (start) {  start();  }  }   private void start(a){  if(isRunning){  return;  }  isRunning =true;  isCleared=false;  loadNextFrame();  }   void unsubscribe(FrameCallback frameCallback) {  callbacks.remove(frameCallback);  if (callbacks.isEmpty()) {  stop();  }  }   private void loadNextFrame(a) {   / /..  // There is no frame data currently drawn  if(pendingTarget ! =null) {  DelayTarget temp = pendingTarget;  pendingTarget = null;  // Call onFrameReady directly to tell the observer to draw the current frame.  onFrameReady(temp);  return;  }  isLoadPending = true;  // Get the interval between frames to draw  int delay = gifDecoder.getNextDelay();  long targetTime = SystemClock.uptimeMillis() + delay;  // Place the next frame first for easy drawing.(Position)  gifDecoder.advance();  // Create a delayed message with the Handler in DelayTarget.  next = new DelayTarget(handler, gifDecoder.getCurrentFrameIndex(), targetTime);  // Glide loading process.... with().load().into(); At targetTime, the data frame is captured and drawn.  requestBuilder.apply(signatureOf(getFrameSignature())).load(gifDecoder).into(next);  }   @VisibleForTesting  void onFrameReady(DelayTarget delayTarget) {  //....  if(delayTarget.getResource() ! =null) {  recycleFirstFrame();  DelayTarget previous = current;  current = delayTarget;  // 1. Callback to observer to perform current frame drawing  for (int i = callbacks.size() - 1; i >= 0; i--) {  FrameCallback cb = callbacks.get(i);  cb.onFrameReady();  }  if(previous ! =null) {  handler.obtainMessage(FrameLoaderCallback.MSG_CLEAR, previous).sendToTarget();  }  }  //2. Proceed to load the next frame of the GIF  loadNextFrame();  }   private class FrameLoaderCallback implements Handler.Callback {  / /..   @Override  public boolean handleMessage(Message msg) {  if (msg.what == MSG_DELAY) {  GifFrameLoader.DelayTarget target = (DelayTarget) msg.obj;  onFrameReady(target);  return true;  } else if (msg.what == MSG_CLEAR) {  GifFrameLoader.DelayTarget target = (DelayTarget) msg.obj;  requestManager.clear(target);  }  return false;  }  }   @VisibleForTesting  static class DelayTarget extends SimpleTarget<Bitmap> {  / /...   @Override  public void onResourceReady(@NonNull Bitmap resource,  @Nullable Transition<? super Bitmap> transition) {  this.resource = resource;  Message msg = handler.obtainMessage(FrameLoaderCallback.MSG_DELAY, this);  // A delay message is sent through Handler to send the drawing work message for the next frame.  handler.sendMessageAtTime(msg, targetTime);  }  } } Copy the code

You can see that in the onResourceReady method, the frameloadercallback. MSG_DELAY message is posted to the main thread message queue for execution with targetTime delayed by the Handler.


class GifFrameLoader{

private class FrameLoaderCallback implements Handler.Callback {
    static final int MSG_DELAY = 1;
 static final int MSG_CLEAR = 2;   @Synthetic  FrameLoaderCallback() { }   @Override  public boolean handleMessage(Message msg) {  if (msg.what == MSG_DELAY) {  // Call back onFrameReady notification GifDrawable draw  GifFrameLoader.DelayTarget target = (DelayTarget) msg.obj;  onFrameReady(target);  return true;  } else if (msg.what == MSG_CLEAR) { . }  return false;  }  }   @VisibleForTesting  void onFrameReady(DelayTarget delayTarget){  //....  if(delayTarget.getResource() ! =null) {  recycleFirstFrame();  DelayTarget previous = current;  current = delayTarget;  // 1. Call the viewer collection (GifDrawable) to draw the current GIF frame  for (int i = callbacks.size() - 1; i >= 0; i--) {  FrameCallback cb = callbacks.get(i);  cb.onFrameReady();  }  if(previous ! =null) {  handler.obtainMessage(FrameLoaderCallback.MSG_CLEAR, previous).sendToTarget();  }  }  // 2. Proceed to load the next frame of the GIF  loadNextFrame();  } } Copy the code

The above message processing gives a clue: drawing the current frame and loading the next frame are serial, that is to say, improper time control of either link will affect the Gif loading lag problem.

Glide load Gif caton optimization

By introducing GIFLIB to decode GIFs in the native layer, memory consumption and CPU utilization can be significantly reduced and increased. Secondly, GIF animations are drawn using FrameSequenceDrawable’s double-buffering mechanism, which eliminates the need to create multiple bitmaps in the Java layer’s BitmapPool.

Take a look at FrameSequenceDrawable’s double buffering mechanism:

public class FrameSequenceDrawable extends Drawable implements Animatable.Runnable{

 //....

 public FrameSequenceDrawable(FrameSequence frameSequence,BitmapProvider bitmapProvider){
 / /...  final int width = frameSequence.getWidth();  final int height = frameSequence.getHeight();  // Draw a Bitmap of the previous frame  frontBitmap = acquireAndValidateBitmap(bitmapProvider,width,height);  // Draw a Bitmap for the next frame  backBitmap = acquireAndValidateBitmap(bitmapProvider,  width,height);   / /.. Start the decoding thread, used to process the background decoding Gif character  initializeDecodingThread();  } } Copy the code

It is easy to see from the above construction that two bitmaps are created through the BitmapProvider;

1.GIF animation drawing schedule

public class FrameSequenceDrawable extends Drawable implements Animatable.Runnable{

 @Override 
 public void start(a){
  if(! isRunning){ synchronized(mLock){  / /..  if(mState == STATE_SCHEDULED){  return;  }  //. Perform a decoding operation  scheduleDecodeLocked();  }  }  }   private void scheduleDecodeLocked(a){  mState = STATE_SCHEDULED;  sDecodingThreadHandler.post(mDecodeRunnable);  }   private final Runnable mDecodeRunnable = new Runnable(){   @Override  public void run(a){  / /...  try{  //1. Decode the next frame  invalidateTimeMs = mDecoder.getFrame(nextFrame,bitmap,lastFrame);  }catch(Exception e){  / /..  }   if (invalidateTimeMs < MIN_DELAY_MS) {  invalidateTimeMs = DEFAULT_DELAY_MS;  }  boolean schedule = false;  Bitmap bitmapToRelease = null;   / / lock  synchronized(mLock){  if(mDestroyed){  bitmapToRelease = mBackBitmap;  mBackBitmap =null;  }else if (mNextFrameToDecode >=0 && mState ==STATE_DECODING){  // The data to be decoded in the next frame is 0, indicating that the next frame is decoded. Waiting for the drawing  schedule = true;  // Interval drawing time  mNextSwap = exceptionDuringDecode ? Long.MAX_VALUE:invalidateTimeMs+mLastSwap;  mState= STATE_WAITING_TO_SWAP;  }  }   if (schedule) {  // 2. Execute the drawing schedule in mNextSwap  scheduleSelf(FrameSequenceDrawable.this,mNextSwap);   }  }   @Override  public void run(a){  boolean invalidate = false;  synchronized(mLock){  if (mNextFrameToDecode > 0 && mState == STATE_WAITING_TO_SWAP) {  invalidate =true  ;  }  }  if (invalidate) {  //3. Draw decoded data  invalidateSelf();  }  }  } } Copy the code

As you can see from the above code, the start method triggers a decoding operation, and when the decoding is complete, a drawing is performed by calling the scheduleSelf for a specified amount of time. Glide loads the Gif in much the same way.

GIF drawing and double buffering


public class FrameSequenceDrawable extends Drawable implements Animatable , Runnable{

 @Override
 public void draw(@NonNull Canvas canvas){
 synchronized(mLock){  checkDestroyLocked();  if (mState == STATE_WAITING_TO_SWAP) {  if (mNextSwap - SystemClock.uptimeMillis()<=0) {  mState = STATE_READY_TO_SWAP;  }   }  if (isRunning() && mState == STATE_READY_TO_SWAP) {  //1. Assign the next frame's Bitmap(mBackBitmap) to the previous frame's Bitmap(mFrontBitmap)  Bitmap temp = mBackBitmap;  mBackBitmap = mFrontBitmap;  mFrontBitmap = temp;   //2. After completing the above steps, notify the decoding thread to proceed with the next decoding operation  if (continueLooping) {  scheduleDecodeLocked();  }else{  scheduleSelf(mFinishedCallbackRunnable,0);  }  }  }   if (mCircleMaskEnabled) {  / /...  }else{  //3. Draw current frame  mPaint.setShader(null);  canvas.drawBitmap(mFrontBitmap,mSrcRect,getBounds(),mPaint);  }   } } Copy the code

Substitution is done in FrameSequenceDrawable’s draw method with mFrontBitmap and mBackBitmap, and the decoder thread is immediately notified to decode the next frame, thus ensuring that the next frame is fetched and the current frame is drawn approximately simultaneously.


conclusion

Through understanding and analyzing the above operation process, we can draw the following conclusions:

1. With GIFLIB+ double buffering, only two bitmaps are created and memory consumption is very stable

2, Compared to Glide’s native loading, when loading too large GIF images, more than the available size of the BitmapPool, or directly create Bitmap.

3. GIFLIB decodes GIF data directly in the native layer, which is advantageous for Glide in both efficiency and memory consumption.

Glide construction current frame data and the next frame data is serial, while FrameSequenceDrawable is the use of double buffer and decoding child threads to achieve approximate synchronization to complete the last frame and the next frame data seamless connection.

This article is formatted using MDNICE