The author | Wan Hongbo lake (far) new retail product | alibaba tao technology
preface
Flutter has attracted a lot of attention since its birth as a cross-platform application framework. Through self-drawing UI, it solves the multi-terminal consistency problem that RN and WEEX solutions were difficult to solve. Dart AOT and streamlined rendering pipelines offer a higher performance experience than a combination of JavaScript and WebView.
At present, there are also a lot of BU in use and exploration within the group. Understanding the working principle of the underlying engine can help us customize and optimize the engine in a more in-depth way to better innovate and support the business. On Taobao, we also explored the rendering engine of our own UI based on Flutter Engine. This paper first makes an in-depth analysis and arrangement of the underlying rendering engine of Flutter, in order to clarify the mechanism and thinking of Flutter rendering, and then shares some of our explorations based on the Flutter engine for your reference.
The analysis in this paper mainly takes Android platform as an example. The principle of I OS is roughly similar, and the relevant reference code is based on stable/ V1.12.13 + Hotfix.8.
Rendering engine analysis
Rendering line
The UI generation and rendering of the whole Flutter are divided into the following steps:
1-6 of them in after receiving the vsync signal, the UI thread of execution, mainly involving in the Dart framework Widget/Element/RenderObject three tree generation as well as the bearing drawing instruction LayerTree created, 7-8 is executed in GPU thread, mainly involving the combination of grating into the upper screen.
1-4 is not directly related to rendering. It is mainly about managing UI component life cycle, page structure, Flex layout and other related implementation. This article will not give in-depth analysis. 5-8 refers to the process related to rendering, of which 5-6 are executed in THE UI thread. The product is the Layer tree containing the rendering instruction, which is generated in the Dart Layer. It can be considered as the first half of the whole rendering process and belongs to the producer role. 7-8 upload the Layer Tree generated by dart Layer to the C++ code of Flutter engine through window, realize raster and synthesize output through flow module. Think of it as the second half of the rendering process, the consumer role. Below is a sequence diagram rendering a frame of Flutter UI on Android platform:
Specific runtime steps:
- When the Flutter engine starts, register with the Choreographer instance of the system to receive Vsync callbacks.
- After sending the Vsync signal platform, step on the registered callback is invoked, after a series of calls, perform to VsyncWaiter: : fireCallback.
- VsyncWaiter: : fireCallback BeginFrame will actually perform Animator class member function.
- The BeginFrame goes through a series of calls to the Window BeginFrame. The Window instance is an important bridge between the underlying Engine and the Dart framework. Basically, all platform-related operations are connected by the Window instance. Including events, rendering, accessibility, etc.
- The Window BeginFrame is called to the Dart Framework RenderBinding class, which has a drawFrame method that drives the REarrangement and drawing of dirty nodes on the UI. If an image is displayed, The IO thread and the worker thread will be thrown to perform image loading and decoding. After decoding, the IO thread will be thrown again to generate image texture. Since the IO thread and GPU thread belong to share GL context, So the image texture generated in the IO thread can be directly processed and displayed by the GPU thread.
- Dart layer drawing instructions and related render attribute configurations are stored in the LayerTree. The LayerTree is submitted to the GPU thread through the Animator::RenderFrame. After the GPU thread holds the LayerTree, Rasterize and screen up (we’ll talk more about LayerTree later). The Animator::RequestFrame will then request the next Vsync signal from the system, which will continue from step 1, driving the UI updates.
The overall operation process of the underlying engine of Flutter is analyzed. The concepts and details involved in the above rendering pipeline will be analyzed in relative detail below. You can read them selectively according to your own situation.
Threading model
To understand the rendering pipeline of Flutter, one must first understand the thread model of Flutter. From the rendering engine’s perspective, the four threads of Flutter are responsible for the following:
- Platform thread: Responsible for providing Native Windows as targets for GPU rendering. It receives the platform’s VSync signal and sends it to the UI thread to drive the rendering pipeline.
- UI thread: responsible for UI component management, maintenance of 3 trees, Dart VM management, and UI rendering instruction generation. It is also responsible for submitting the LayerTree bearing the rendering instructions to the GPU thread for rasterization.
- GPU thread: Raster through flow module, call the underlying rendering API (OpengL/Vulkan/Meta), synthesize and output to the screen.
- IO thread: Several worker threads will request image resources and complete image decoding, and then generate texture in IO thread and upload GPU. Since the EGL Context is shared with GPU thread, GPU thread can directly use the texture uploaded by IO thread. Through parallelization, The concepts that will be introduced later on will run through all four threads. For more information on the threading model, see the following two articles:
Insight into the Flutter Engine Threading Model
The Engine Architecture
VSync
When the Flutter engine starts, it registers a callback function to receive Vsync with the Choreographer instance of the system. After Vsync is issued by the GPU hardware, the system will trigger the callback function and drive the UI thread for layout and drawing.
@ shell/platform/android/io/flutter/view/VsyncWaiter.java
private final FlutterJNI.AsyncWaitForVsyncDelegate asyncWaitForVsyncDelegate = new FlutterJNI.AsyncWaitForVsyncDelegate() {
@Override
public void asyncWaitForVsync(long cookie) {
Choreographer.getInstance().postFrameCallback(new Choreographer.FrameCallback() {
@Override
public void doFrame(long frameTimeNanos) {
floatfps = windowManager.getDefaultDisplay().getRefreshRate(); Long refreshPeriodNanos = (long) (1000000000.0 / FPS); FlutterJNI.nativeOnVsync(frameTimeNanos, frameTimeNanos + refreshPeriodNanos, cookie); }}); }}Copy the codeThe following is the call stack when Vsync is triggered:
On Android, the Java layer receives the system’s Vsync callback and sends it to the Flutter Engine via JNI. The Dart layer is then routed back to the Dart layer via Animator,Engine, Window and other objects to drive the DART layer to perform drawFrame operations. In the Dart framework RenderingBinding: : triggers in drawFrame function of all dirty node layout/paint/compositor related operations, then generate LayerTree, This is then rasterized and synthesized by the Flutter engine.
//@rendering/binding.dart
void drawFrame() { assert(renderView ! = null); pipelineOwner.flushLayout(); pipelineOwner.flushCompositingBits(); pipelineOwner.flushPaint(); renderView.compositeFrame(); // this sends the bits to the GPU pipelineOwner.flushSemantics(); // this also sends the semantics to the OS. }Copy the codeThe layer
After the dirty nodes are typeset by drawFrame in the Dart layer, the nodes that need to be redrawn are drawn. The widget in a Flutter is an abstract description of a UI Element. When drawing, the inflate becomes an Element and the corresponding RenderObject is generated to drive rendering. Generally speaking, all renderobjects of a page belong to one layer, the concept of Flutter itself does not have a layer. The layer mentioned here can be roughly interpreted as a buffer. All renderobjects belonging to this layer should be drawn in the corresponding buffer of this layer.
If the RepaintBoundary property of the RenderObject is true, an additional layer will be generated and all its child nodes will be drawn on this new layer. Finally, all layers will be composited by GPU and displayed on the screen.
The concept of Layer is used to represent all renderobjects on a Layer of Flutter. There is an N: 1 relationship between Layer and Layer. The root node RenderView creates the root Layer, which is typically a Transform Layer and contains multiple sub-layers, each of which contains renderObjects. When each RenderObject is drawn, The relevant drawing instructions and drawing parameters are generated and stored in the corresponding Layer.
Layer is actually used to organize and store rendering-related instructions and parameters. For example, the Transform Layer is used to store the matrix of Layer transformations, and the ClipRectLayer contains the size of the Layer’s clipping field. The PlatformViewLayer contains the texture ID of the rendering component of the same layer, and the PictureLayer contains the SkPicture(SkPicture records the instructions for SkCanvas drawing, which will be used to raster the GPU thread during rasterization).
Rendering instructions
When the first frame is rendered, all the child nodes are drawn one by one, starting with the root node RenderView.
Override void paint(PaintingContext context, Offset Offset) {override void paint(PaintingContext context, Offset) {if(child ! = null) context.paintChild(child, offset); }Copy the codeWe can look specifically at how a node is drawn:
- To create a Canvas. A PictureLayer is created inside the Canvas acquired by PaintContex, and an instance of Skia’s SkPictureRecorder is created by calling ui.PictrureRecorder to the C++ layer. The SkCanvas is then created using the SkPictureRecorder and returned to the Dart layer for use.
//@rendering/object.dart
@override
Canvas get canvas {
if (_canvas == null)
_startRecording();
return _canvas;
}
void _startRecording() { assert(! _isRecording); _currentLayer = PictureLayer(estimatedBounds); _recorder = ui.PictureRecorder(); _canvas = Canvas(_recorder); _containerLayer.append(_currentLayer); }Copy the code-
Perform the concrete drawing through the Canvas. The Dart layer takes the object bound to the underlying SkCanvas and uses the Canvas to perform specific drawing operations. These drawing commands are recorded by the underlying SkPictureRecorder.
-
Finish drawing and prepare to screen. When the drawing is complete, stopRecordingIfNeeded will be called to Canvas. It will finally call endRecording of C++ SkPictureRecorder to generate a Picture object. Stored in PictureLayer.
//@rendering/object.dart
void stopRecordingIfNeeded() {
if(! _isRecording)return;
_currentLayer.picture = _recorder.endRecording();
_currentLayer = null;
_recorder = null;
_canvas = null;
}
Copy the codeThis Picture object corresponds to Skia’s SkPicture object, which stores all the drawing instructions. If you are interested, please check the official instructions of SkPicture.
All layers are drawn to form a LayerTree, At renderView.com positeFrame () by the Dart SceneBuilder Layer mapped to Flutter the flow of the engine: : Layer, it will also generate a c + + Flow ::LayerTree, stored in the Scene object, is submitted to the Flutter Engine via the Window’s Render interface.
//@rendering/view.dart
void compositeFrame() {... final ui.SceneBuilder builder = ui.SceneBuilder(); final ui.Scene scene = layer.buildScene(builder); _window.render(scene); scene.dispose(); }Copy the codeAfter all drawing operations have been completed, a flow::LayerTree should be created in the Flutter engine, which should look like this:
The flow::LayerTree, which contains all the drawing information and drawing instructions, will be called to Animator::Render through the window instance, and finally submitted to the GPU thread in Shell::OnAnimatorDraw for rasterization. The code can be referred to:
@shell/common/animator.cc/Animator::Render
@shell/common/shell.cc/Shell::OnAnimatorDraw
Flow is a SKIA based synthesizer that generates pixel data based on render instructions. Flutter operates Skia based on flow modules, which are rasterized and synthesized.
Image texture
When we talked about the thread model, we mentioned that the IO thread is responsible for loading and decoding images and uploading the decoded data to the GPU to generate a texture that will be used later in the rasterization process. Let’s look at this part.
When loading images, the UI thread will call InstantiateImageCodec* to the C++ layer to initialize the image decoder library. After decoding bitmap data from skia’s own decoder library, Can call SkImage: : MakeCrossContextFromPixmap to generate in multiple threads share SkImage, use it to generate a GPU texture in IO thread.
//@flutter/lib/ui/painting/codec.cc sk_sp<SkImage> MultiFrameCodec::GetNextFrameImage( fml::WeakPtr<GrContext> resourceContext) { ... // If resourceContext is not empty, a SkImage is created, and the SkImage is in resouceContext,if (resourceContext) {
SkPixmap pixmap(bitmap.info(), bitmap.pixelRef()->pixels(),
bitmap.pixelRef()->rowBytes());
// This indicates that we do not want a "linear blending" decode.
sk_sp<SkColorSpace> dstColorSpace = nullptr;
return SkImage::MakeCrossContextFromPixmap(resourceContext.get(), pixmap,
false, dstColorSpace.get());
} else{ // Defer decoding until time of draw later on the GPU thread. Can happen // when GL operations are currently forbidden such asin the background
// on iOS.
returnSkImage::MakeFromBitmap(bitmap); }}Copy the codeAs we know, The OpenGL environment is thread-safe, and image textures generated in one thread cannot be used directly in another thread. However, due to the time-consuming operation of uploading texture, all operations are placed on the GPU thread, which will reduce the rendering performance. Multithreaded texture upload is currently supported by OpenGL via the share Context, so the IO thread is responsible for texture upload in Flutter and the GPU thread is responsible for texture use.
The basic operation is to create an EGLContextA on the GPU thread, and then pass the EGLContextA to the IO thread. The IO thread creates the EGLContextB using the EGLCreateContext. Use EGLContextA as a shareContext parameter so that EGLContextA and EGLContextB can share texture data.
Specific related codes are not listed, you can refer to:
@shell/platform/android/platformviewandroid.cc/CreateResourceContext
@shell/platform/android/androidsurfacegl.cc/ResourceContextMakeCurrent
@shell/platform/android/androidsurfacegl.cc/AndroidSurfaceGL
@shell/platform/android/androidsurfacegl.cc/SetNativeWindow
For the process of loading images, see this article: TODO
Rasterization and synthesis
The process of converting drawing instructions into pixel data is called rasterization, and the related superposition of data after rasterization of each layer and the processing related to special effects become synthesis, which is the main work in the second half of rendering.
As mentioned above, after the LayerTree is generated, it will be submitted to the GPU thread for rasterization through the Render interface of Window. The general process is as follows:
Steps 1-4 are performed in the UI thread. LayerTree is submitted to the rendering queue of Pipeline object through the Animator class. After that, the Pipeline object is submitted to the GPU thread for rasterization through Shell. The code in the animator. Cc&pipeline. H
In steps 5-6, perform specific rasterization operations on the GPU thread. This part is mainly divided into two parts, one is the management of Surface. One is how to draw the render instructions in the Layer Tree into the Surface created earlier.
The following figure shows the Surface in Flutter. Different types of Surface correspond to different underlying rendering apis.
Take GPUSurfaceGL as an example. In Flutter, GPUSurfaceGL is a management and encapsulation of Skia GrContext, which Skia uses to manage GPU drawing. And that’s how you end up using the OpenGL API to do all the up-screen stuff. At engine initialization, when FlutterViewAndroid is created, GPUSurfaceGL is created, and Skia’s GrContext is created synchronously in its constructor.
Rasterizer Rasterizer: mainly in function: DrawToSurface implemented:
//@shell/rasterizer.cc RasterStatus Rasterizer::DrawToSurface(flutter::LayerTree& layer_tree) { FML_DCHECK(surface_); .if(Compositor_frame) {//1. RasterStatus = Compositor_frame ->Raster(layer_tree, layer_tree, layer_tree, layer_tree)false);
if (raster_status == RasterStatus::kFailed) {
returnraster_status; } //2. Submit();if(external_view_embedder ! = nullptr) { external_view_embedder->SubmitFrame(surface_->GetContext()); } / / 3. The screen FireNextFrameCallbackIfPresent ();if (surface_->GetContext()) {
surface_->GetContext()->performDeferredCleanup(kSkiaCleanupExpiration);
}
return raster_status;
}
return RasterStatus::kFailed;
}
Copy the codeAfter rasterization is complete, execute frame->Submit() for composition. This calls the PresentSurface below to transfer the offScreenSurface content to onscreencanvas and finally to the screen via GLContextPresent().
//@shell/GPU/gpu_surface_gl.cc
bool GPUSurfaceGL::PresentSurface(SkCanvas* canvas) {
...
if(offscreen_surface_ ! = nullptr) { SkPaint paint; SkCanvas* onscreen_canvas = onscreen_surface_->getCanvas(); onscreen_canvas->clear(SK_ColorTRANSPARENT); Onscreen_canvas ->drawImage(offscreen_surface_->makeImageSnapshot(), 0, 0, &paint); } {//2. Flush onscreen_surface_->getCanvas()->flush(); } / / 3 on the screenif(! delegate_->GLContextPresent()) {return false; }...return true;
}
Copy the codeThe GLContextPresent interface code is shown below, and is actually called to the eglSwapBuffers interface of EGL to display the contents of the graph buffer.
//@shell/platform/android/android_surface_gl.cc
bool AndroidSurfaceGL::GLContextPresent() {
FML_DCHECK(onscreen_context_ && onscreen_context_->IsValid());
return onscreen_context_->SwapBuffers();
}
Copy the codeThe onscreen_context in the code snippet above is obtained by setNativeWindow when the Flutter engine is initialized. The main method is to pass the ANativeWindow pointer corresponding to an Android SurfaceView component to EGL. According to this window, EGL calls eglCreateWindowSurface to establish association with the display system, and then displays the rendered content on the screen through this window.
The code can be referenced:
@shell/platform/android/androidsurfacegl.cc/AndroidSurfaceGL::SetNativeWindow
By summarizing the last half of the rendering process above, it can be seen that the rendering instructions in LayerTree are rasterized and drawn to the corresponding Surface of SkSurface. Surface is an offscreensurface created by AndroidSurfaceGL. Offscreen_surface was then transferred to onScreen_surface via PresentSurface operation, and eglSwapSurfaces were called to finish the rendering of a frame.
explore
Based on our in-depth understanding of the rendering mechanism of the Flutter engine, we also made some relevant explorations based on our business needs. Here we briefly share them.
Applets rendering engine
Based on the Flutter engine, we removed the native dart engine, introduced js engine, and rewrote the core logic of rendering, painting, and widgets in the Flutter Framework in C++. The basic components will continue to be packaged upward to realize cssom and C++ version of the responsive framework, provide unified JS Binding API externally, and then connect to the DSL of small program for use by small program business side. For small programs with high performance requirements, we can choose to use this link for rendering. Offline, we ran the UI rendering of Starbucks small program and had a good performance experience.
Applets interactive rendering engine
Limited by the structure of small program worker/ Render, frequent drawing operations in interactive business need to be serialized/deserialized and send messages from worker to Render to execute the render command. Based on the Flutter Engine, we provide a set of independent 2D rendering engine, which introduces canvas rendering pipeline and provides standard Canvas API for businesses to use directly in worker threads to shorten rendering links and improve performance. At present, it has supported the online operation of relevant interactive business, with good performance and stability.
Summary and Reflection
This article focuses on the rendering pipeline of flutter Engine and its related concepts and briefly shares some of our explorations. Familiarity and understanding of the rendering engine turned out to help us quickly build a differentiated and efficient rendering link on both Android and IOS. This provides a more customizable and optimized solution to the current dual-end, predominantly Web based cross-platform rendering.
Pay attention to “Tao Technology” wechat public number, a technical community with content and temperature.
