[TOC]

preface

  • We were exploring earlierIOS animation and rendering principlesWhen, have learned:
    • Computer graphics rendering principle;
    • Mobile terminal screen imaging and Caton principle;
    • IOS rendering framework and iOS layer rendering principle;
  • There are several ways that we, as programmers, can learn to improve our level of technical competence. For example,Chain learning,Loop learning,Comparative learning methodAnd so on.
    • Obviously, we are using chain learning to explore the principles of iOS animation and rendering.
    • Next, in order to broaden our knowledge and make a relatively systematic understanding of the same subject, we will adoptLoop learning, to explore several related topics:
      • IOS -OffScreenRendering off-screen rendering principle
      • Causes and solutions of iOS CPU and GPU resource consumption
  • All of the above areiOS NativeFrom a development perspective, from another dimension,Of course we should pay attention to WebApps,Cross-platform developmentYou guessed it, I’m using hereComparative learning method, related topics:
    • Rendering principles for Web and Class RN large front ends
    • Flutter rendering principle.This article is designed to explain the page rendering principle of Flutter!!!!!!!!!

Flutter:

  • Flutter is a cross-platform mobile UI framework developed using the Dart language, enabling high-performance, high-fidelity Android and IOS development with its own rendering engine. As a cross-platform application framework, after its birth, it has been highly concerned.
  • Through self-drawing UI, it solves the multi-terminal consistency problem that is difficult to be solved by previous RN and WEEX technology solutions.
  • Flutter is developed in Dart language and compiled by AOT (iOS AOT/ Android JIT)This article)
    • Dart,AOT 和Streamlined rendering pipeline, compared with the combination of JavaScript and WebView, has a higher performance experience.
  • Understanding the working principle of the underlying engine can help us customize and optimize the engine in a more in-depth way to better innovate and support the business.
  • This paper first makes an in-depth analysis and arrangement of the underlying rendering engine of Flutter, so as to clarify the mechanism and thinking of Flutter rendering, and provide knowledge reserve for further study of Flutter development and understanding of the underlying rendering principle.

An overview of the

  • The Flutter interface is made up of widgets. All widgets make up the Widget Tree. When the page is updated, the Widget Tree is updated, then the Element Tree is updated, and finally the RenderObject Tree is updated
  • Next render flow:
    • In the Framework layer:
      • Flutter renderingIn the Framework layerThere will beBuild,Wiget Tree,Element Tree,RenderObject Tree,Layout,Paint,Composited LayerWait several stages
    • In the GPU:
      • Combine the layers to generate a texture
      • Use OpenGL’s interface to submit rendering content to the GPU for rasterization and composition, in the C++ layer of Flutter.The Skia library is used
  • Including submitted to the GPU process, synthesis calculation, display screen processAnd NativeIt’s basically the same thing, soThe performance is similar.

1. Structure of Flutter (Rendering Framework)

From the attached figure, we can see that the Framework of Flutter is divided into Framework and Engine:

  • Applications are developed based on the Framework layer:Flutter rewrites the UI framework.Re-implement everything from UI controls to rendering itself. The Framework is responsible for renderingBuild.Layout.Paint.Generate LayerSuch as link.
  • Do not rely on native controls on iOS and Android platforms,
  • The Engine layer is the rendering Engine for the C++ implementation:FlutterSkia graphics library relies on Engine (C++) layer and system graphics rendering interface, responsible for the Framework generationLayer combination.To generate the textureAnd then throughThe Open GL interface submits render data to the GPU.

The rendering process of Flutter

The UI generation and rendering of the whole Flutter are divided into the following steps:

1-6 of them in after receiving the vsync signal, the UI thread of execution, mainly involving in the Dart framework Widget/Element/RenderObject three tree generation and load drawing instruction LayerTree create; 7-8 is executed in GPU thread, mainly involving the combination of grating into the upper screen;

1-4 is not directly related to rendering. It mainly manages UI component life cycle, page structure, Flex layout and other related implementation. This paper will not make in-depth analysis. 5-8 refers to the process related to rendering, of which 5-6 are executed in THE UI thread. The product is the Layer tree containing the rendering instruction, and the Layer is generated in the Dart Layer, which can be considered as the first half of the whole rendering process and belongs to the producer role. 7-8 upload the Layer Tree generated by dart Layer to the C++ code of Flutter engine through window, realize raster and synthesize output through flow module. Think of it as the second half of the rendering process, the consumer role

We can further divide the rendering process into five stages:

  1. The Framework notifies Engine when a UI update is needed

    • Simply put, the interface of Flutter consists of widgets.

    • All widgets make up the Widget Tree.

    • Widget Tree is updated when the interface is updated,

    • Update the Element Tree, and finally update the RenderObjectTree.

    • The logic for updating widgets is as follows:

  2. Engine waits until the next Vsync signal arrives and notifies the Framework

  3. The Framework does animations, build, layout, Compositing, paint, and finally generates a layer to submit to the Engine

  4. The Engine combines the layers, generates textures, and finally submits data to the GPU through the Open Gl interface

    • A Flutter rendering contains Build, Widget Tree, Element Tree, RenderObject Tree, Layout, Paint, Composited Layer, etc

    • In the C++ Layer of Flutter, Skia library is used to combine layers to generate textures, and OpenGL interface is used to submit rendering content to GPU for rasterization and composition.

    • After submitting to the GPU process, the process of compositing and displaying the screen is basically similar to iOS native rendering, so the performance is similar.

  5. The GPU is processed and displayed on the display. The whole process is shown as follows:

As you can see from the flow chart, only when there is a UI update needs to be re-rendered, of course the application is started by default to render.

3. Render trigger

Let’s take a look at how rendering is triggered when a UI needs to be updated, from the application to the Framework to the Engine. When a Flutter application needs to update its UI, the setState method needs to be called. Then the UI can be updated. Let’s examine what this method does.

Void _drawFrame() {//Engine invoke Framework entry _invoke(window.ondrawframe, window._ondrawframezone); } // Set onDrawFrame to _handleDrawFrame void initInstances() {super.initInstances(); _instance = this; ui.window.onBeginFrame = _handleBeginFrame; ui.window.onDrawFrame = _handleDrawFrame; SystemChannels.lifecycle.setMessageHandler(_handleLifecycleMessage); } void _handleDrawFrame() { if (_ignoreNextEngineDrawFrame) { _ignoreNextEngineDrawFrame = false; return; } handleDrawFrame(); } void handleDrawFrame() { _schedulerPhase = SchedulerPhase.persistentCallbacks; For (FrameCallback callback in _persistentCallbacks) _invokeFrameCallback(callback, _currentFrameTimeStamp); } } void initInstances() { .... addPersistentFrameCallback(_handlePersistentFrameCallback); } void _handlePersistentFrameCallback(Duration timeStamp) { drawFrame(); } void drawFrame() { ... if (renderViewElement ! = null) buildOwner.buildScope(renderViewElement); // rebuild widget super.drawFrame(); buildOwner.finalizeTree(); } void drawFrame() {// This method completes Layout, CompositingBits,Paint, generates Layer and submits work to Engine assert(renderView! = null); pipelineOwner.flushLayout(); pipelineOwner.flushCompositingBits(); pipelineOwner.flushPaint(); renderView.compositeFrame(); / / generated Layer and submitted to Engine pipelineOwner. FlushSemantics (); }Copy the code

From the Engine callback, the Framework will build, Layout, Paint, generate Layer, etc. Next, let’s analyze how these links are realized.

4. Details of rendering process

4.1 Build

Stateless widgets build their UI using the Build method of statelessWidgets, while stateful widgets build their UI using the build method of State. Let’s take a closer look at the flow from the setState call to the custom State build (for now, just look at the stateful widget rendering process).

// This is the official demo class _MyHomePageState extends State<MyHomePage> {int _counter = 0; void _incrementCounter() { setState(() { _counter++; }); @override Widget Build (BuildContext context) {return new Scaffold(...) ; Build void buildScope(Element Context, [VoidCallback callback]) {// Build void buildScope(Element context, [VoidCallback callback]) {... _dirtyElements.sort(Element._sort); _dirtyElementsNeedsResorting = false; int dirtyCount = _dirtyElements.length; int index = 0; while (index < dirtyCount) { ... _dirtyElements[index].rebuild(); index += 1; } for (Element element in _dirtyElements) { assert(element._inDirtyList); element._inDirtyList = false; } _dirtyElements.clear(); } void rebuild() { ... if (! _active || ! _dirty) return; performRebuild(); } void performRebuild() { ... built = build(); . } Widget build() => widget.build(this);Copy the code

As you can see above, buildScope iterates over _dirtyElements, calling rebuild for each element in the array, and ultimately calling the build method of the corresponding widget. When setState is set, the corresponding element is added to the _dirtyElements array, and element identifies the dirty state.

4.2 Layout

The wiget that supports Layouts, including Container, Padding, Align, and so on, is powerful and easy to implement with Flutter. In the rendering process, the widget build will be followed by layout. The layout entry is flushLayout.

void flushLayout() { ... while (_nodesNeedingLayout.isNotEmpty) { final List<RenderObject> dirtyNodes = _nodesNeedingLayout; _nodesNeedingLayout = <RenderObject>[]; for (RenderObject node in dirtyNodes.. Sort ((RenderObject a, RenderObject b) => a.depth-b.depth)) {nodesNeedingLayout, RenderObject's markNeedsLayout method adds itself to _nodesNeedingLayout if (node._Needslayout && Node.owner == This)// Layout node._LayoutwithOutreSize () for RenderObject that needs layout; }}... } void _layoutWithoutResize() { ... performLayout(); // This method calculates the implementation of a Layout. Different Layout widgets have different implementations markNeedsSemanticsUpdate(); . _needsLayout = false; markNeedsPaint(); } // This is the layout of the RenderView. It is easy to read the size of the configuration and then call the Layout of the Child. The layout of other widgets is very complicated. Void performLayout() {assert(_rootTransform! = null); _size = configuration.size; assert(_size.isFinite); if (child ! = null) child.layout(new BoxConstraints.tight(_size)); // The method parent calls the entry of the child's layout, and the parent passes restrictions to the child, Layout void layout(Constraints Constraints, {bool parentUsesSize: false}) {... RenderObject relayoutBoundary; if (! parentUsesSize || sizedByParent || constraints.isTight || parent is! RenderObject) { relayoutBoundary = this; } else { final RenderObject parent = this.parent; relayoutBoundary = parent._relayoutBoundary; } if (! _needsLayout && constraints == _constraints && relayoutBoundary == _relayoutBoundary) { return; } _constraints = constraints; _relayoutBoundary = relayoutBoundary; if (sizedByParent) { performResize(); } RenderObject debugPreviousActiveLayout; performLayout(); // The implementation of the actual calculation layout markNeedsSemanticsUpdate(); _needsLayout = false; markNeedsPaint(); } void performResize() { ... size = constraints.biggest; switch (axis) { case Axis.vertical: offset.applyViewportDimension(size.height); break; case Axis.horizontal: offset.applyViewportDimension(size.width); break; } // This is marked as layout dirty and added to the PipelineOwner void markNeedsLayout() {if (_relayoutBoundary! = this) { markParentNeedsLayout(); } else { _needsLayout = true; if (owner ! = null) { return true; } ()); owner._nodesNeedingLayout.add(this); owner.requestVisualUpdate(); }}}Copy the code

When RenderOjbect needs to relayout, it adds itself to the rendering pipe, and then triggers the layout link. It first iterated through the rendering pipe to find the RenderObject to render. Then call performLayout to calculate the layout. Different objects implement different performLayout methods and calculate the layout in different ways. Then call the Child layout entry and pass the parent restrictions to the child. Child calls its own performLayout.

4.3 Paint

When you need to draw custom images, you can implement the Paint method by inheriting the CustomPainter method, and then use the interface provided by Flutter inside the Paint method to create complex images. Let’s look at how the paint process is implemented.

// This is the official paint demo class Sky extends CustomPainter {@override void paint(Canvas Canvas, Size size) { var rect = Offset.zero & size; Var gradient = new RadialGradient(Center: const Alignment(0.7, -0.6), radius: 0.2, colors: [const Color(0xFFFFFF00), const Color(0xFF0099FF)], stops: [0.4, 1.0],); canvas.drawRect( rect, new Paint().. shader = gradient.createShader(rect), ); } @override bool shouldRepaint(Sky oldDelegate) => false; } // This is the entry for paint in the render pipe void flushPaint() {final List<RenderObject> dirtyNodes = _nodesNeedingPaint; _nodesNeedingPaint = <RenderObject>[]; // Sort the dirty nodes in reverse order (deepest first). for (RenderObject node in dirtyNodes.. Sort ((RenderObject a, RenderObject b) => b.depth-a.depth)) {assert(node._layer! = null); if (node._needsPaint && node.owner == this) { if (node._layer.attached) { PaintingContext.repaintCompositedChild(node); } else { node._skippedPaintingOnLayer(); } } } } static void repaintCompositedChild(RenderObject child, { bool debugAlsoPaintedParent: false }) { ... if (child._layer == null) { child._layer = new OffsetLayer(); } else { child._layer.removeAllChildren(); } final PaintingContext childContext = new PaintingContext._(child._layer, child.paintBounds); // Create the layer painting Context child._paintwithContext (childContext, offset.zero); childContext._stopRecordingIfNeeded(); } void _paintWithContext(PaintingContext context, Offset offset) { ... paint(context, offset); . } void paint(PaintingContext context, Offset offset) { if (_painter ! = null) {// Continue to call the custom Paint method only if you own CustomPainter, Pass canvas _paintWithPainter(context.canvas, offset, _painter); _setRasterCacheHints(context); } super.paint(context, offset); Call the parent paint method if (_foregroundPainter! = null) { _paintWithPainter(context.canvas, offset, _foregroundPainter); _setRasterCacheHints(context); Void paint(PaintingContext context, Offset Offset) {if (child! = null){ context.paintChild(child, offset); } void _paintWithPainter(Canvas canvas, Offset offset, CustomPainter painter) { int debugPreviousCanvasSaveCount; canvas.save(); if (offset ! = Offset.zero) canvas.translate(offset.dx, offset.dy); painter.paint(canvas, size); Layer ->PaintingContext->Canvas; paint is painted on Canvas... canvas.restore(); }Copy the code

To summarize, the render pipe first finds the RenderObject that needs to be repainted. Then, if CustomPainter is implemented, call the CustomPainter paint method and then call child paint.

4.4 Composited Layer

Timeline.startSync('Compositing', arguments: timelineWhitelistArguments); try { final ui.SceneBuilder builder = new ui.SceneBuilder(); layer.addToScene(builder, Offset.zero); final ui.Scene scene = builder.build(); ui.window.render(scene); scene.dispose(); assert(() { if (debugRepaintRainbowEnabled || debugRepaintTextRainbowEnabled) debugCurrentRepaintColor = DebugCurrentRepaintColor. WithHue (debugCurrentRepaintColor. Hue + 2.0); return true; } ()); } finally { Timeline.finishSync(); } } void addToScene(ui.SceneBuilder builder, Offset layerOffset) { addChildrenToScene(builder, offset + layerOffset); } void addChildrenToScene(ui.SceneBuilder builder, Offset childOffset) { Layer child = firstChild; while (child ! = null) { child.addToScene(builder, childOffset); child = child.nextSibling; }}Copy the code

The Composited Layer combines all the layers into a Scene and submits the Scene to Engine using the uI.window.render method. At this point, the Framework is almost done submitting data to Engine. Engine calculates the final display effect of all layers according to size, level and transparency, and renders it to the screen through Openg Gl interface.

Five, the summary

This article looks at Flutter from a rendering perspective to help us understand the reliability of Flutter, a cross-platform UI SDK. The whole chapter describes the whole structure of Flutter, Flutter rendering pipeline and so on. We also draw a conclusion that the rendering performance of Flutter pages is ≈ Native

The main points not discussed in this article are as follows:
  • How to learn Flutter?
  • Dart Language Learning
  • Learn the practical techniques of Flutter
  • Flutter underlying technology, source code reading
  • Flutter custom plugin developed and open source
  • .

Six, reference

  • 01- “Flutter’s Rendering Pipeline”
  • 02- How is Flutter actually rendered?
  • 03 – super explanation analysis tao is technology 】 【 Flutter rendering engine | business want to innovation, do not understand the underlying principle how line?

Recommended reading

  • 01-[Free Fish Technology] Release the last mile of the Flutter

  • 02- Make Flutter run on The Hongmeng system

Related reading (14 articles in total)

  • On principle of 01 – computer | computer graphics rendering this article
  • 02 – | mobile terminal screen computer imaging and caton

IOS related Topics

  • 01 – iOS | the underlying principle of iOS rendering framework and principle of iOS layer rendering
  • 02 – iOS underlying principle | iOS animation rendering principle
  • 03 – iOS underlying principle | iOS OffScreen Rendering off-screen Rendering principle
  • 04 – iOS underlying principle | caton caused by CPU and GPU resources consumption reasons and solutions

WebApp related topics

  • 01- Rendering principles for Web and Class RN large front ends

Topics related to cross-platform development solutions

  • 01-Flutter page rendering principle

Stage summary: Performance comparison of Native, WebApp and cross-platform development schemes

  • 01- Performance comparison of Native, WebApp and cross-platform development schemes

Android and HarmonyOS page rendering features

  • 01-Android page rendering principle
  • 02-HarmonyOS page rendering principles (For the output)

Small program page rendering thematic

  • 01- Applets framework rendering principle

conclusion

  • 01- Big front-end technology collocation and selection of the project scheme (For the output)
  • 02- Thoughts on stack accumulation for big front-end engineers (For the output)