On the afternoon of July 17th, In the front end special tour salon Beijing station, Lu Xuhui, cross-platform development engineer of Sound Net Agora, shared the theme of Flutter2 Rendering Principle and How to Achieve Video Rendering. This article is a summary of the speech.
This sharing mainly includes three parts:
- Flutter2 overview.
- Practice with the Flutter2 video rendering plugin.
- Flutter2 Rendering principle (source code).
preface
Flutter1 does not have a large domestic market share. Many developers may know that Flutter’s upper language is based on Google’s Dart (a failed attempt to replace JavaScript). The Dart language is also one of the areas where Flutter is not readily accepted by many developers. Many domestic companies may still choose ReactNative or stick to native development, but with the release of Flutter2 (full platform support) and Ali’s Beihai framework (a cross-platform framework that uses JavaScript based on the rendering capabilities of Flutter Engine), I believe Flutter2 has a future. Considering that many readers may be front-end developers, in part 3 I will start from the perspective of the Web, and you will see a lot of familiar and unfamiliar content. It doesn’t matter whether you are a developer of Flutter or whether you know about Flutter, what matters is the design idea of Flutter. I hope it will help you.
Flutter2 overview
Flutter2 is the latest version of Flutter released by Google in March 2021. It supports null-safety based on Dart1.12. The compiler will ask you to validate data that may be null to avoid some null pointer problems during development. More importantly, there is stable support for the Web side, and support for the desktop side has also been incorporated.
Let’s take a look at the overall architecture of Flutter2:
The Web part of Flutter2 includes Framework layer and Browser layer, wherein Framework layer covers rendering, drawing, gesture processing, etc., while Browser layer covers CSS, HTML, Canvas, WebGL, etc. (after all, it runs on the Browser). The final WebAssembly was designed to use C and C++ to schedule the Skia rendering engine, which we’ll cover in more detail in Part 3.
In addition to the common Framework layer, the Native part also includes Engine layer and Embedder layer. Engine layer mainly includes initialization of Dart VIRTUAL machine and Isolate, layer composition, GPU rendering, platform channel, text layout, etc. The Embedder layer is mainly used for feature adaptation of different platforms.
At first glance, the difference between Web and Native is quite large, but there is also an Engine layer on the Web based on Dart development called web_UI, which deals with Composition and Rendering on the Web.
Here’s a quick look at the platform differences of Flutter2, as shown in the figure above. Currently, Flutter2 supports 6 major platforms, namely Web, Android, iOS, Windows, macOS and Linux. Compared to other cross-platform frameworks such as ReactNative and Electron (representing mobile and desktop respectively), Flutter2 has richer platform support, although ReactNative also has desktop support from Microsoft, and Expo has support for the Web. But it’s not uniform enough.
For some build tools or package management tools, Flutter2 uses a standard way of comparing platforms, such as Web based JavaScript, which helps Dart compile to JavaScript with Dart2JS; Android is still based on Gradle system; In iOS and macOS, Flutter is introduced into the project based on CocoaPods. In Windows and Linux, it is mainly based on CMake.
Some of Flutter’s features, such as PlatformView, provide the ability to bridge native controls, such as displaying an Element on the Web or displaying custom views on Android or iOS. PlatformView is not currently supported on the desktop. That’s not to say it’s not technically possible, but it’s not being developed yet. ExternalTexture is an ExternalTexture that allows users to render their own graphics data. Dart :: FFI gives Flutter the ability to call C and C++ directly, both of which are supported in addition to the Web.
Implementation of Flutter2 video rendering plug-in
1. Implementation process of rendering video plug-in
Next, I will share some of sonnet’s practices in video rendering plugins, mainly for the Web and desktop.
As described earlier in platform differences, Web does not support ExternalTexture and Desktop does not support PlatformView. So on the Web, we use PlatformView to achieve video rendering. The basic process is to use UI. platformViewRegistry to register PlatformView and return DivElement. After DivElement is created, you need to use Package :js to call Dart and JavaScript to each other.
Soundnet has a dedicated Web AUDIO and video SDK, so we didn’t do too much in the Dart layer. Instead, we wrapped the JS layer, and this wrapper library scheduled the SDK to operate on WebRTC to create VideoElement. Finally append to the DivElement you created earlier to render the video.
Let’s take a look at the desktop solution. Since it doesn’t support PlatformView, we can only use the ExternalTexture solution for custom video rendering. Call the custom createTextureRender function in the Native layer with MethodChannel, which dispatches FlutterTextureRegistry to create FlutterTexture. Throw textureId back into the Dart layer and bind the Texture Widget. Native SDK video data will be converted to image format in the AgoraRtcWrapper layer, Then we can inform FlutterTexture FlutterTextureRegistry MarkTextureFrameAvailable function of image data from the callback.
2. Some pits encountered in Flutter2 development
We also encounter some problems during plug-in development. Here is a brief share for you:
On the desktop side, macOS is an OC header and Windows is a C++ header. Linux, on the other hand, is the C header file, and this part is not completely uniform, even some oF the apis are different, so there is a lot of trouble in the desktop development process, after all, it is not completely stable.
For example, as shown in the figure above, the first three are problems encountered on the Web.
1. UI. PlatformViewRegistry error on the Web, because it is not in the Framework of the UI. The dart defined, but the definition in web_ui/UI. The dart, but it does not affect the operation, so you can choose to use ignore comments to ignore it.
Dart ::js: dart::js: dart::js: dart::js: dart::js: dart::js: dart::js: dart::js: Dart ::js: Dart ::js: Dart ::js: Dart ::js: Dart ::js However, errors are reported in Profile and Release mode.
Dart :: IO is used to make platform-specific calls, such as platform determination, which is not available on the Web. We can use if(Dart.library.html) to point to a custom DART file while importing and define an empty implementation for the API, or we can use kIsWeb to not execute the API on the Web.
4. On Windows, is to use EncodableValue for Dart and c + + communication (based on the c + + 17 STD: : the variant, can understand as the TypeScript type1 | type2 | type3). When processing int32 and int64, the Framework determines whether the maximum int32 value is exceeded. If the maximum int32 value is exceeded, the Framework marks it as INT64. Developers using the SDK may know that our user ID type is uint32. Uint32 The value range may be greater than int32 and smaller than INT64. If STD :: GET is used only, an error may be reported regardless of whether int32_t or INT64_t is specified. Fortunately, it provides the LongValue function. Internal judgment is made and returns are uniformly int64.
The following is the focus of this topic: Flutter2 rendering principle. Many principles of Flutter engine are common, but they are implemented with Dart on the Web and C and C++ in Native.
The rendering principle of Flutter2
1, Flutter Framework
Before we start, let’s briefly review that the Flutter Framework is divided into a Framework part and an Engine part. The rendering process of Flutter is also done by the two parts in coordination. However, unlike other frameworks, Flutter is handled by the upper layer and handed over directly to the lower layer. The Flutter Engine provides several Builders for the Framework to use, so many processes are scheduled back and forth between these two parts.
The UserInput process processes UserInput, and the Animation process processes Animation. However, these two parts are not the focus of today’s discussion. Build is mainly used to make widgets generate renderobjects that are recognized by the Flutter framework. Layout is mainly used to determine the location and size of components, etc. Paint is mainly used to transform the rendering object into Layer, which is combined by Composition, and finally Rasterize for GPU rendering.
Flutter processes the UI based on a Tree structure. In the figure below, we can see three Tree structures: Widget Tree, Element Tree and Render Tree.
We start with the Widget and create a Container that contains Row (the Flex layout Container), which in turn contains Image and Text. The Container contains the ColoredBox, which can be used as a background or border. Image contains RawImage and Text contains RichText. Only ColoredBox, Row, RawImage and RichTexth are converted to RenderObjectElement. They will eventually generate the corresponding RerderObject, respectively.
The Attach function directs the rendering process to PipelineOwner. The RenderObject function is used to determine whether Layout needs to be processed or synthesized. And whether you need to draw.
PipelineOwner is used to manage the rendering process. First, a frame callback is registered when a Flutter is initialized. The frame of the Flutter is managed by itself. The three functions flushLayout, flushCompositingBits, and flushPaint are triggered in the callback, which corresponds to the three mark functions mentioned earlier for RenderObject.
PipelineOwner has three arrays in which the renderobjects that were previously marked are stored and then flushed quickly through. After PipelineOwner processing, it calls RenderView’s compositeFrame function, which is described below.
FlushPaint calls the paint function of RenderObject. This is an abstract function that is not implemented by itself, but is implemented by subclasses that inherit from it.
As you can see, the first argument to paint is PaintingContext. Let’s look at some of its APIS. They all return Layer, including pushClipRect, which returns a different subclass of Layer. So one of the responsibilities of the paint function is to convert the RenderObject into a Layer and add it to its member’s ContainerLayer. By the way, LayerHandle is a reference count that handles auto-release.
The paint function is also responsible for storing Canvas drawing instructions in PictureRecorder for renderObjects that need to be drawn.
Canvas is mainly used to draw objects that need to be drawn, such as RichText and RawImage mentioned above. Besides, it can also perform operations such as transform and clipPath.
In the Canvas factory construction here, useCanvasKit will be judged and different Canvas will be constructed. Why there is such logic, here we first press no table, which will be introduced later. Let’s go down the Render Pipeline first.
After the PipeLineOwner process mentioned earlier completes, the compositeFrame function of RenderView is called for Layer composition. In the compositeFrame function, we can see several very important classes, Scene and SceneBuilder. Scene is the product of Layer compositing and is built by SceneBuiler.
The _window is SingletonFlutterWindow, which is a singleton RenderView that will be described in more detail later. Let’s first look at the Build Scene process.
BuildScene is the function that calls ContainerLayer buildScene (see the right part of the figure above), It then calls Layer’s addToScene function, which, like RenderObject’s paint function, is an abstract function that needs to be implemented by subclasses of Layer. For example, ContainerLayer’s addToScene function is going to go through the Child Tree and call addToScene for each of the Child layers.
So what addToScene does, it actually calls pushXXX which is provided by SceneBuilder, and they return Layer, just EngineLayer, Layer is an abstraction of layers in the Framework, The EngineLayer is an abstraction of the layers in Engine, and then you combine those EngineLayer layers into the Scene in Engine.
2, Flutter Engine
Now that we’ve covered the Framework layer, let’s look at the Engine layer.
To recap, our Widget goes through the following conversion process: Widget->RenderObject->Layer->EngineLayer->Scene.
Here we see SingletonFlutterWindow, whose render function calls the EnginePlatformDispatcher’s render function. Here we see the familiar useCanvasKit, So, what does this useCanvasKit mean? Let’s move on.
This time we must introduce the concept of the Web Renderer. There are two rendering modes in Flutter Web: One is the RENDERING mode based on HTML tags, which maps all the Flutter widgets into different tags. Those that cannot be represented by tags alone will be drawn on Canvas, which is similar to ReactNative.
The other is the CanvasKit-based rendering mode, which downloads a 2MB WASM file to call the Skia rendering engine, through which widgets are rendered.
We can specify a rendering mode for flutter build or run using a command line parameter. It is worth mentioning that the default rendering mode is Auto, CanvasKit on desktop browsers and HTML on mobile WebViews.
First, let’s take a look at the HTML rendering mode of the Flutter SDK. Take the API Example of the Flutter SDK as an Example. As you can see from the Elements Tree, there are many levels of tags on the Flutter SDK. This means that text rendering in this mode is using Canvas, so why draw text using Canvas instead of using the browser’s default text rendering capability? That’s because to eliminate differences in platform rendering performance, especially in the way text is wrapped, Flutter has a text typography engine built into it and renders based on that engine. If the focus Flutter is acquired, it will add a label and then receive the input Text message. When the focus is lost, it will hide it. This is a very clever scheme.
Let’s look at some of the details in HTML rendering mode. SurfaceCanvas will be built while in HTML rendering mode. As you can see from the image on the right, the List is a collection of drawing instructions.
For SceneBuilder, the subclass SurfaceSceneBuilder, see PersistedSurface on the right.
It is a subclass of EngineLayer and has a rootElement property and a visitChildren function, which is also an abstract function. PersistedLeafSurface is a plugin with no child, so its visitChildren implementation is empty. It is derived from PersistedPicture and PersistedPlatformView. They correspond to image text (which we mentioned is drawn using Canvas) and platform View. EngineLayer PersistedContainerSurface is a container, it also has a huge number of subclasses, such as PersistedClipPath, PersistedTransform, etc., These EngineLayer correspond to the various custom tags in the Elements Tree from the previous API Example.
When SurfaceSceneBuilder build is executed, the webOnlyRootElement generated for the SurfaceScene already contains our entire Html Element.
Finally, we can see that SurfaceScene will call DomRenderer’s renderScene function to add these elements to the _sceneHostElement.
This is the end of HTML rendering mode.
Let’s take a look at the rendering mode of CanvasKit. We can see from the Elements Tree that the hierarchy is very simple. All the rendering is done in a canvas. You can do style isolation.
Again, we’ll start with Canvas, which is CanvasKitCanvas, and the drawing instructions are stored in the _Commands property of CkPictureSnapshot.
For SceneBuilder, the subclass of CanvasKit is LayerSceneBuilder. This Layer is similar to PersistedSurface for HTML and is derived from EngineLayer. And there’s a ContainerLayer that contains all the children, and there’s a PictureLayer and a PlatformViewLayer. The difference, however, is that it has a paint function, which in this case is the function that actually does the drawing on the GPU.
The LayerScene generated by the Build function of LayerSceneBuilder contains a root node called LayerTree, corresponding to webOnlyRootElement in HTML rendering mode.
Since the paint function is the real draw, let’s take a look at when it is called.
When I mentioned How To Render Scene earlier, LayerScene is drawn by calling rasterizer’s draw function. AcquireFrame acquireFrame obtains frameSize from LayerTree to build SurfaceFrame, and builds SkSurface inside it. Bind WebGLContext and a series of scheduling operations on Skia.
Context. AcquireFrame The generated Frame is just a simple aggregate class, don’t worry about it, then call the Frame raster function to raster it. The final addToScene adds the CANVAS HTML tag from baseSurface to skiaSceneHost.
The rasterization phase consists of preroll and paint, calculating the draw boundaries separately, and iterating through the LayerTree and calling all Layer paint functions, PaintContext differs from the Framework’s PaintingContext in that it holds all the Canvas so that different layers can paint it.
Now that we’re almost done with the process in CanvasKit rendering mode, let’s take a final look at how it will be displayed in HTML. In fact, DomRenderer was eventually used in CanvasKit rendering mode. In the initialization process of Flutter, we can see that The first half of the initializeCanvasKit function is the wASM resource and the corresponding JavaScript file that we introduced to Skia; The second half creates a skiaSceneHost root node, the Element referenced earlier in basesurface.addToScene.
The whole rendering principle is covered here, of course, there are many details in the whole rendering, such as SurfaceFactory in addition to baseSurface and backupSurface can cache the drawing, etc., each of these points can be discussed as a separate topic. Finally, post a summary of the flow chart, you can review the whole process with the previous.
At the end of this post, we have attached a GitHub link to the Flutter RTC SDK. We have adapted Flutter2 to the Dev/Flutter branch. Screen sharing is also supported on the Web and desktop. You can experience it by yourself. If you have any questions or suggestions, you are welcome to give feedback. If the experience is good, you are also welcome to point the Star on our warehouse.
