Author: Duan Jiashun
background
At present, most applications use a large number of images, which occupy the largest proportion of application bandwidth. When we added Flutter, we found that The image control cache of Flutter was completely managed by ourselves. At the same time, Flutter did not provide disk cache (version 1.22), so it was poor in performance and experience, so we had to optimize it further.
Image cache
At present, in many CDN implementations, all resources have unique URIs, so many client implementations ignore the Caches capability of HTTP protocol, but directly use URIs as unique identifiers to judge whether image resources are unique. This greatly reduces the time and request required to confirm 304 to the server.
On the client side, there are usually at least two levels of cache, memory and disk. When we access the Flutter library, we hope that the client cache can be connected to the cache in the Flutter to reduce memory and network consumption.
Currently, there are three directions for reusing cache:
- Reuse views. The image capabilities of Flutter are provided entirely by the client, just like React Native.
- Reuse disk cache instead of memory cache, this scheme is relatively simple to implement, but will result in two images in memory.
- The memory cache is multiplexed, loaded from disk to memory by the client, and the client manages the entire cache life cycle, such as deep fusion with SDWebImage. This solution seems to be perfect for reuse, and the client has the ability to have precise control over the size of the image cache for the entire application.
So let’s take a look at the implementation of these schemes, which seem to be good schemes, and what pits we have stepped on.
The reuse view
Flutter provides a way to seamlessly merge with the client’s native view. The original motivation for Flutter was for scenes like maps and Webviews. It is impossible for Flutter to implement such a complex set of controls. So what if we use this for a client-side image bridge scheme?
First, we need to understand how PlatformView is bridged (the following is an iOS implementation). Insert a client View Layer into the Widget. Instead of simply drawing the View to the Flutter Root Layer as we might think. Since the draw call of a Flutter does not take place on the main thread, but on the raster thread, if we want to draw the client View onto the Flutter, we must first raster it into an image and then draw it. The performance overhead and latency of this is obviously unacceptable, and it is unrealistic to have to do this on every frame.
Therefore, a Flutter takes the form of breaking up the Flutter Layer. When a client View is inserted, the Flutter automatically splits itself into 2 layers:
|-----| Flutter Overlay View 2
|-----| Native View
|-----| Flutter Root View 1
Copy the codeThe client View is sandwiched between two Flutter Views like a sandwich cookie. All the subsequent sister widgets on the upper layer of the Platform View are drawn to the upper View, while the others are drawn to the lower layer. When the position of the upper layer View changes, the corresponding Overlay View needs to be re-created. In order to reduce this overhead, Flutter adopts a trick method. The Overlay View covers the screen and controls the display area by moving the mask on it.
// The overlay view wrapper masks the overlay view.
// This is required to keep the backing surface size unchanged between frames.
//
// Otherwise, changing the size of the overlay would require a new surface,
// which can be very expensive.
//
// This is the case of an animation in which the overlay size is changing in every frame.
//
// +------------------------+
// | overlay_view |
// | +--------------+ | +--------------+
// | | wrapper | | == mask => | overlay_view |
// | +--------------+ | +--------------+
// +------------------------+
Copy the codeThe ability of client views to access a Flutter has been addressed, but as you can see, when a client View is inserted, a Flutter needs to create 2 additional views for subregion drawing. When there are multiple images on a page, the overhead is obviously unacceptable, and the performance is unacceptable.
Here are the performance considerations described by Flutter officials in the Platform View.
Platform views in Flutter come with performance trade-offs.
For example, in a typical Flutter app, the Flutter UI is composed on a dedicated raster thread. This allows Flutter apps to be fast, as the main platform thread is rarely blocked.
While a platform view is rendered with Hybrid composition, the Flutter UI is composed from the platform thread, which competes with other tasks like handling OS or plugin messages, etc.
Prior to Android 10, Hybrid composition copies each Flutter frame out of the graphic memory into main memory, and then copies it back to a GPU texture. In Android 10 or above, the graphics memory is copied twice. As this copy happens per frame, the performance of the entire Flutter UI may be impacted.
Virtual display, on the other hand, makes each pixel of the native view flow through additional intermediate graphic buffers, which cost graphic memory and drawing performance.
Copy the codeReuse disk cache
Let’s all take a step back and solve the network bandwidth problem first, so a simple solution is to reuse disk caching.
Reusing disk caching is relatively simple and minimally invasive. We just need to design a channel interface to synchronize the state and address of the cache.
getCacheInfo({
String url,
double width,
double height,
double scale,
BoxFit fit})
-> {String path, bool exists}
Copy the codeSo in the use of time, we only need to customize a new set of ImageProvider, network and local providers unified.
_CompositeImageStreamCompleter({
String url,
double width,
double height
}) {
getCacheInfo({url: url, width: width, height:height})
.then((info) {
if(info ! =null&& info.path ! =null && info.path.length > 0) {
var imageProvider;
var decode = this.decode;
if (info.exists) {
final imageFile = File(info.path);
imageProvider = FileImage(imageFile, scale: this.scale);
} else {
imageProvider = NetworkImage(info.fixUrl ?? this.url,
scale: this.scale, headers: this.headers);
decode = (Uint8List bytes,
{int cacheWidth, int cacheHeight, bool allowUpscaling}) {
final cacheFile = File(info.path);
// Cache to disk
cacheFile.writeAsBytes(bytes).then((value) => { });
return this.decode(bytes,
cacheWidth: cacheWidth,
cacheHeight: cacheHeight,
allowUpscaling: allowUpscaling);
};
}
_childCompleter = imageProvider.load(imageProvider, decode);
final listener =
ImageStreamListener(_onImage, onChunk: _onChunk, onError: _onError);
_childCompleter.addListener(listener);
}
}).catchError((err, stack) {
print(err);
});
}
Copy the codeNote that Flutter is used to download images when there is no disk cache. You need to manually save them to disk to ensure disk cache consistency.
Multiplexed memory cache
Reusing the disk cache is a less risky change, but at the cost of not being able to reuse the memory cache. Not only do they have to be read separately, but multiple copies of the memory cache are kept simultaneously because the memory cache portions of the two sides are completely independent.
If we want to further optimize, we need to use the scheme of multiplexing memory cache. Currently, there are roughly several schemes of synchronous memory cache as follows:
- Memory is transferred to Flutter using channel communication
- Memory is transferred directly to Flutter using the new ffI channel feature
- Use the Texture control to reuse from the Texture level
Channel
Flutter is the official stable message communication solution with high compatibility and stability. When displaying cached images, upload the image data to Flutter via BinaryMessenger.
Since the Channel itself must be an asynchronous process, there is some overhead associated with communicating this way.
At the same time, because the Channel is processed in the main thread on the client side, it is also necessary to avoid time-consuming operations such as loading and decoding directly in the main thread.
In the process of data transfer by Channel, due to the mechanism (which is also necessary from the perspective of security), the binary data must be copied. As a result, the memory cache maintained by the Flutter and the client’s own cache are still two copies, which does not perfectly achieve the reuse effect mentioned above.
ffi
From the perspective of message communication overhead and message memory copy problems, ffI seems to be the perfect solution to all problems in Channel.
The principle and implementation process are exactly the same as Channel, and you only need to replace it with FFI Channel. Ffi has no communication process as long as Channel, no message serialization and parsing, and no thread switching, just like an HTTP request is different from a simple API call.
The important thing to note here is that the FFI interface is executed synchronously, that is, the client is executed in a flutter. UI thread. We must be aware of thread-safety issues. For Flutter, because it is executed on the UI thread, the method must return as quickly as possible and cannot perform some time-consuming operations.
But can ffI really solve these problems? After careful study, it was found that the fundamental problem of memory overcommitment could not be solved. The following is the process of FFI conversion.
When we load the client image into memory, it is passed to Flutter as a Buffer, such as this structure:
struct Buffer {
int8 *ptr;
size_t length;
}
Copy the codeUint8List = Uint8List = Uint8List = Int8Pointer = Int64
Pointer<UInt8> bufferPtr;
int length;
Uint8List buffer = bufferPtr.asTypedList(length);
Copy the codeDuring this transformation, a copy of memory occurs (the Uint8List underlying holds data using STD ::vector).
Therefore, the final result is no higher cache reuse capability than Channel.
Texture
The other option is to share PixelBuffer, which is the decoded image data. Texture can be reused in Flutter.
Ali has studied the specific implementation scheme very thoroughly, so we will not repeat it here. We will mainly analyze its performance and reuse ability.
Texture uses TextureId, which is an int value, so there is no performance overhead in terms of data volume between the two ends. The main process is:
- The client registers the texture to Flutter and returns an id as a unique identifier (i++). This happens in the Platform thread, the main thread of the client, while the actual registration into TextureRegistry is done in the Raster thread.
- This ID is passed to the TextureLayer when the flutter. UI thread processes the paint event.
- In the Raster thread, the draw call is fetched from TextureRegistry using TextureId and generated.
From the perspective of the overall process, Flutter only uses TextureId in the whole process of the intermediate flow. It does not operate memory or texture and does not have the problem of multiple caches. Therefore, this scheme perfectly solves the above two problems.
Memory optimization
Texture had the highest cache utilization from the above analysis, but the memory analysis revealed a surprising result.
The Image above shows a memory map that loads several large images using the Flutter Image control, adding a total memory consumption of 10M.
The image above shows the memory consumption of loading the same image using the Texture scheme, which is a huge difference of 37M.
At the same time, it can be seen that the original Flutter image has a relatively large wave peak at the initial stage, as does the texture, but it is relatively gentle.
This big difference starts with the rendering process of the Flutter Image control.
- Once the ImageProvider loads the image into memory, it first decodes the image, which is done in the flutter. IO thread.
- After decoding the image data, there is a very large memory consumption because the image data is stored as pixel buffer. The Flutter will be optimized during this process. The decoded data will not be 100% in size. Instead, the current widget size will be adjusted to calculate an optimal size and the Flutter will be decoded at this size. So the native Image is actually better than the client in terms of memory footprint.
- Immediately after the image is removed, Flutter reclaims the decoded memory. That is, Flutter only stores the original compressed data of the image and does not cache the Pixel buffer. Our client (SDWebImage) caches all the decoded data. This is another reason why Flutter memory performance is better than the client.
So is a strategy that beats clients in memory usage necessarily a good one?
In the rendering process, Flutter simply trades decoding time for memory space. In the actual Demo, the Image display of the Flutter Image control was significantly delayed when the list was sliding quickly, while the Texture scheme was almost undiscernible to the naked eye. So the Texture scheme is not without its advantages in terms of overall performance.
Image size
As you can see from the above, the Texture scheme has poor memory performance, so how can we further optimize it?
For many scenes, such as user profile picture, there is a fixed size, so we can take the size as a parameter and send it to CDN, and then cut it to the size we need on CDN, which will also save a lot of traffic.
But there are also many scenarios where we cannot get the control size, such as the full container size scenario. How do we automatically add Size to all images?
Paint is triggered after the Layout from the rendering process, and the size of the control must already be fully determined, so we can make a fake placeholder control here, calculate the size, and then replace it with the real image.
typedef ImageSizeResolve = void Function(Size size);
class ImageSizeProxyWidget extends SingleChildRenderObjectWidget {
const ImageSizeProxyWidget({Key key, Widget child, this.onResolve})
: super(key: key, child: child);
final ImageSizeResolve onResolve;
@override
ImageSizeProxyElement createElement() => ImageSizeProxyElement(this);
@override
ImageSizeRenderBox createRenderObject(BuildContext context) =>
ImageSizeRenderBox(onResolve);
@override
void updateRenderObject(
BuildContext context, covariant ImageSizeRenderBox renderObject) {
super.updateRenderObject(context, renderObject); renderObject.onResolve = onResolve; }}class ImageSizeProxyElement extends SingleChildRenderObjectElement {
ImageSizeProxyElement(RenderObjectWidget widget) : super(widget);
}
class ImageSizeRenderBox extends RenderProxyBox with RenderProxyBoxMixin {
ImageSizeRenderBox(ImageSizeResolve onResolve, [RenderBox child])
: onResolve = onResolve,
super(child);
ImageSizeResolve onResolve;
@override
void paint(PaintingContext context, ui.Offset offset) {
if (hasSize) {
if(onResolve ! =null) onResolve(size);
}
super.paint(context, offset); }}Copy the codeIn this way, we can force the Size parameter on all images.
After this optimization, the memory footprint is down to about 2M (since THE test images I used were all in HIGH definition, the effect is obvious).
conclusion
Many ideas and strategies of Flutter are obviously different from those of the client. From the perspective of its image capability, it can be adapted and optimized in all aspects. To achieve a perfect and usable state, it seems that constant investment and exploration are needed.
The appendix
See Alibaba.com’s Ways to Explore Flutter: Optimizing Flutter Image Performance for your implementation of Texture.
This article is published from netease Cloud Music big front end team, the article is prohibited to be reproduced in any form without authorization. Grp.music – Fe (at) Corp.Netease.com We recruit front-end, iOS and Android all year long. If you are ready to change your job and you like cloud music, join us!