Analysis of the Flutter framework (VII) — Drawing

Analysis of Flutter Framework

Analysis of the Flutter Framework (I) — Overview and Window

Analysis of the Flutter Framework (II) — Initialization

Analysis of the Flutter Framework (iii) — Widget, Element and RenderObject

Analysis of The Flutter Framework (IV) — The Operation of the Flutter Framework

Analysis of the Flutter Framework (5) — Animation

Analysis of the Flutter Framework (VI) — Layout

Analysis of the Flutter Framework (VII) — Drawing

preface

This article introduces the final stage of the rendering pipeline, Paint, with the Source of Flutter. The content covered in this article may be a little bit further from the framework that you need to know to develop a Flutter app than the previous chapters. The thing that you might want to notice at this point is the RepaintBoundary Widget, whose corresponding RenderObject is RenderRepaintBoundary. The purpose of this Widget will be more clearly understood after you walk through the drawing phase of the rendering pipeline.

An overview of the

As we all know, the Render Tree in the Flutter framework is responsible for layout and rendering. When rendering, Flutter traverses the RenderObject subtrees that need to be redrawn one by one. The page of the Flutter app we see on the screen is actually composed of different layers. These layers are organized in the form of a tree, another important tree we see in Flutter: the Layer tree.

paint()

Layer

The layers in a Flutter are represented by the Layer class.

abstract class Layer extends AbstractNode with DiagnosticableTreeMixin {
    
  @override
  ContainerLayer get parent => super.parent;
    
  Layer get nextSibling => _nextSibling;
  Layer _nextSibling;

  Layer get previousSibling => _previousSibling;
  Layer _previousSibling;
}
Copy the code

Class Layer is an abstract class that, like RenderObject, inherits from AbstractNode. That means it’s also a tree. The parent property represents the parent node of type ContainerLayer. This class inherits from Layer. Only layers of the ContainerLayer type and its subclasses can have children. All other Layer subclasses are leaf layers. NextSibling and previousSibling represent the siblings before and after the same layer, i.e. the layer children are stored in a bidirectional linked list.

class ContainerLayer extends Layer {
  Layer _firstChild;
  Layer _lastChild;
  
  void append(Layer child) {
    adoptChild(child);
    child._previousSibling = lastChild;
    if(lastChild ! =null) lastChild._nextSibling = child; _lastChild = child; _firstChild ?? = child; }void _removeChild(Layer child) {
    if (child._previousSibling == null) {
      _firstChild = child._nextSibling;
    } else {
      child._previousSibling._nextSibling = child.nextSibling;
    }
    if (child._nextSibling == null) {
      _lastChild = child.previousSibling;
    } else {
      child.nextSibling._previousSibling = child.previousSibling;
    }
    child._previousSibling = null;
    child._nextSibling = null;
    dropChild(child);
  }

  void removeAllChildren() {
    Layer child = firstChild;
    while(child ! =null) {
      final Layer next = child.nextSibling;
      child._previousSibling = null;
      child._nextSibling = null;
      dropChild(child);
      child = next;
    }
    _firstChild = null;
    _lastChild = null; }}Copy the code

ContainerLayer adds header and tail child attributes and provides methods for adding and removing child nodes.

ContainerLayer subclass OffsetLayer, ClipRectLayer and so on.

Leaf type layer have TextureLayer, PlatformViewLayer, PerformanceOverlayLayer, PictureLayer, etc., the framework of most RenderObject draw target layer are PictureLayer.

class PictureLayer extends Layer {

    final Rect canvasBounds;
    
    ui.Picture _picture;
}
Copy the code

The canvasBounds property represents the boundary of the layer canvas, but this property is advisory. The picture property comes from the DART: UI library.

Analysis of the

Back to our familiar drawFrame () function, pipelineOwner. FlushLayout () call is finished rendering pipeline has entered the phase of drawing (paint).

void drawFrame() {
  pipelineOwner.flushLayout();
  pipelineOwner.flushCompositingBits();
  pipelineOwner.flushPaint();
  renderView.compositeFrame(); // this sends the bits to the GPU
  pipelineOwner.flushSemantics(); // this also sends the semantics to the OS.
}

Copy the code

Rendering stage, the first call is pipelineOwner flushCompositingBits ().

pipelineOwner.flushCompositingBits()

This call is used to update the _needsCompositing flag bit of the Render Tree RenderObject.

Before we introduce this call, let’s look at some of the flag bits of the RenderObject.

Bool _needsCompositing: indicates that it or a child node has a compositing layer. If the current node needs to be synthesized, then all ancestor nodes need to be synthesized as well.

Bool _needsCompositingBitsUpdate: sign whether the current node needs to be updated _needsCompositing. This flag bits from the bottom of the markNeedsCompositingBitsUpdate set () function.

bool get isRepaintBoundary => false; : indicates whether the current node is redrawn separately from the parent node. When this flag bit is true, the parent does not necessarily need to be redrawn when the child is redrawn, and similarly, the parent does not need to be redrawn when the parent is redrawn. RenderObject with this flag bit true has the root node of the Render tree, RenderView, and familiar RenderRepaintBoundary, TextureBox, etc.

bool get alwaysNeedsCompositing => false; : indicates whether the current node always needs to be synthesized. The flag bit true means that the current node is always drawn with a new Composited layer. Examples include TextureBox and the familiar RenderPerformanceOverlay that shows runtime performance.

During the construction phase of the render pipeline, there are situations in which nodes in the Render Tree need to be updated _needsCompositing, such as adding and deleting nodes in the Render Tree. This tag work by function markNeedsCompositingBitsUpdate () finish.

void markNeedsCompositingBitsUpdate() {
    if (_needsCompositingBitsUpdate)
      return;
    _needsCompositingBitsUpdate = true;
    if (parent is RenderObject) {
      final RenderObject parent = this.parent;
      if (parent._needsCompositingBitsUpdate)
        return;
      if(! isRepaintBoundary && ! parent.isRepaintBoundary) { parent.markNeedsCompositingBitsUpdate();return; }}if(owner ! =null)
      owner._nodesNeedingCompositingBitsUpdate.add(this);
  }
Copy the code

This will call up from the current node, all the parent node _needsCompositingBitsUpdate setting flags are true. Until its own or its parent’s isRepaintBoundary is true. Finally will join themselves to PipelineOwner _nodesNeedingCompositingBitsUpdate list. While the function calls pipelineOwner. FlushCompositingBits () is used to deal with this list.

FlushCompositingBits ()

void flushCompositingBits() {
    
    _nodesNeedingCompositingBitsUpdate.sort((RenderObject a, RenderObject b) => a.depth - b.depth);
    for (RenderObject node in _nodesNeedingCompositingBitsUpdate) {
      if (node._needsCompositingBitsUpdate && node.owner == this)
        node._updateCompositingBits();
    }
    _nodesNeedingCompositingBitsUpdate.clear();
  }
Copy the code

First turn on the list of _nodesNeedingCompositingBitsUpdate ordered according to the depth of the node in the tree. Then iterate over node._updatecompositingbits ()

  void _updateCompositingBits() {
    if(! _needsCompositingBitsUpdate)return;
    final bool oldNeedsCompositing = _needsCompositing;
    _needsCompositing = false;
    visitChildren((RenderObject child) {
      child._updateCompositingBits();
      if (child.needsCompositing)
        _needsCompositing = true;
    });
    if (isRepaintBoundary || alwaysNeedsCompositing)
      _needsCompositing = true;
    if(oldNeedsCompositing ! = _needsCompositing) markNeedsPaint(); _needsCompositingBitsUpdate =false;
  }
Copy the code

What we’re doing here is looking down from the current node and setting _needsCompositing to true if one of the child nodes isRepaintBoundary is true or alwaysNeedsCompositing is true. If the flag bit of the child node is true, the flag bit of the parent node is also set to true. If _needsCompositing changes, markNeedsPaint() is called to inform the rendering pipeline that the RenderObject needs to be redrawn. Why redraw it? The reason is that the layer on which the RenderObject is located may have changed.

pipelineOwner.flushPaint()

The function flushPaint() handles the objects that were previously added to the list _nodesNeedingPaint. MarkNeedsPaint () is called when a RenderObject needs to be redrawn

void markNeedsPaint() {
    if (_needsPaint)
      return;
    _needsPaint = true;
    if (isRepaintBoundary) {
      if(owner ! =null) {
        owner._nodesNeedingPaint.add(this); owner.requestVisualUpdate(); }}else if (parent is RenderObject) {
      final RenderObject parent = this.parent;
      parent.markNeedsPaint();
    } else {
      if(owner ! =null) owner.requestVisualUpdate(); }}Copy the code

The first thing the function markNeedsPaint() does is set its flag bit _needsPaint to true. It then looks up to the nearest ancestor node whose isRepaintBoundary is true. Until such a node is found, it is not added to the _nodesNeedingPaint list. That is, not every RenderObject that needs to be redrawn is added to the list. It’s going to go up until it finds the nearest one where isRepaintBoundary is true, so in other words, there’s only one node in the list where isRepaintBoundary is true. That is to say, the starting point of redrawing is “redrawing boundaries”.

  void flushPaint() {
    try {
      final List<RenderObject> dirtyNodes = _nodesNeedingPaint;
      _nodesNeedingPaint = <RenderObject>[];
      // Sort the dirty nodes in reverse order (deepest first).
      for (RenderObject node indirtyNodes.. sort((RenderObject a, RenderObject b) => b.depth - a.depth)) {if (node._needsPaint && node.owner == this) {
          if (node._layer.attached) {
            PaintingContext.repaintCompositedChild(node);
          } else{ node._skippedPaintingOnLayer(); }}}}finally{... }}Copy the code

FlushLayout: flushLayout() {flushLayout();} flushLayout() {flushLayout() {flushLayout();}} flushLayout() {flushLayout();}} In the body of the loop, it determines whether the _layer property of the current node is attached. If _layer attached to true words call PaintingContext. RepaintCompositedChild (node); Otherwise, call Node._skippedPaintingonLayer () to set _needsPaint to true for both itself and the nodes between the upper draw boundary. This will be drawn directly the next time _layer.attached is true.

Can be seen from the above code is, the equivalent of border redrawn the Flutter drawing did block processing, redraw the beginning from the upper redrawing borders, by the end of the lower redrawing borders between the RenderObject need to be redrawn, and boundaries may do not need to be redrawn, this is also a performance considerations, avoid unnecessary to draw. Therefore, how to arrange the RepaintBoundary reasonably is a direction we need to consider when optimizing the performance of Flutter APP.

The _layer property here is the layer we talked about earlier, and this property only has a value for the boundary RenderObject. Normally, the RenderObject property is null.

  static void _repaintCompositedChild(
    RenderObject child, {
    bool debugAlsoPaintedParent = false,
    PaintingContext childContext,
  }) {
    if (child._layer == null) {
      child._layer = OffsetLayer();
    } else{ child._layer.removeAllChildren(); } childContext ?? = PaintingContext(child._layer, child.paintBounds); child._paintWithContext(childContext, Offset.zero); childContext.stopRecordingIfNeeded(); }Copy the code

The _repaintCompositedChild() function checks the layer properties of the RenderObject and creates a new OffsetLayer instance if it is null. Clear the child if the layer already exists.

If you don’t have a PaintingContext you’re going to create a new one and let it start drawing. Let’s start with the PaintingContext class:

class PaintingContext extends ClipContext {
  @protected
  PaintingContext(this._containerLayer, this.estimatedBounds)

  final ContainerLayer _containerLayer;
  
  final Rect estimatedBounds;
  
  PictureLayer _currentLayer;
  ui.PictureRecorder _recorder;
  Canvas _canvas;

  @override
  Canvas get canvas {
    if (_canvas == null)
      _startRecording();
    return _canvas;
  }

  void _startRecording() {
    _currentLayer = PictureLayer(estimatedBounds);
    _recorder = ui.PictureRecorder();
    _canvas = Canvas(_recorder);
    _containerLayer.append(_currentLayer);
  }
  
   void stopRecordingIfNeeded() {
    if(! _isRecording)return;
    _currentLayer.picture = _recorder.endRecording();
    _currentLayer = null;
    _recorder = null;
    _canvas = null;
  }
Copy the code

The class PaintingContext literally means draw context, and the attribute _containerLayer is the container layer, which comes from the construction input. So the PaintingContext is associated with the container layer. Then there is the _currentLayer property of PictureLayer, the _Recorder property of UI. PictureRecorder and the familiar Canvas property _canvas. The _startRecording() function instantiates these properties. _recorder is used to record drawing commands, and _canvas is bound to a recorder. Finally, _currentLayer is added to _containerLayer as a child node. If there is a start then there is an end. StopRecordingIfNeeded () is used to end the currently drawn recording. The finished Picture is assigned to the current Picturelayer.picture.

Once you have PaintingContext, you can call renderObject._paintwithContext () to start drawing. This function will call renderObject.paint (context, offset) directly. We know that the function paint() is implemented by RenderObject subclasses themselves. From the previous source analysis we know that the starting point is “draw the boundary”. Here we take a familiar “draw boundary”, RenderRepaintBoundary, as an example to go through the drawing process. Its drawing function is implemented in RenderProxyBoxMixin class:

  @override
  void paint(PaintingContext context, Offset offset) {
    if(child ! =null)
      context.paintChild(child, offset);
  }
Copy the code

This call goes back to the paintChild() method of the PaintingContext:

  void paintChild(RenderObject child, Offset offset) {
    if (child.isRepaintBoundary) {
      stopRecordingIfNeeded();
      _compositeChild(child, offset);
    } else {
      child._paintWithContext(this, offset); }}Copy the code

This checks to see if the child has drawn a boundary, if not, it’s just plain drawn, and then calls down _paintWithContext() to continue drawing on the current PictureLayer. If so, stop the current drawing. Then call _compositeChild(child, offset);

  void _compositeChild(RenderObject child, Offset offset) {
    if (child._needsPaint) {
      repaintCompositedChild(child, debugAlsoPaintedParent: true);
    }
    child._layer.offset = offset;
    appendLayer(child._layer);
  }
Copy the code

If the subdraw boundary is marked as needing to be repainted, then repaintCompositedChild() is called to regenerate the layer and repaint it. If the subdraw boundary is not marked as needing to be redrawn, the regeneration layer and redrawing are skipped. All you need to do is add the sublayer to the current container layer.

What if the child node is a normal RenderObject? Here is an example of drawing a Flutter app error control:

void paint(PaintingContext context, Offset offset) {
    try {
      context.canvas.drawRect(offset & size, Paint() .. color = backgroundColor);
      double width;
      if(_paragraph ! =null) {
        // See the comment in the RenderErrorBox constructor. This is not the
        // code you want to be copying and pasting. :-)
        if (parent is RenderBox) {
          final RenderBox parentBox = parent;
          width = parentBox.size.width;
        } else{ width = size.width; } _paragraph.layout(ui.ParagraphConstraints(width: width)); context.canvas.drawParagraph(_paragraph, offset); }}catch (e) {
      // Intentionally left empty.}}Copy the code

This is going to look like a normal drawing, we’re going to use the canvas from the PaintingContext to draw the rectangle, draw the text and so on. As you can see from the previous analysis, all the drawing here is done on a PictureLayer layer.

So far pipelineOwner. FlushPaint (); This function call is finished, through analysis we can know that the drawing work is actually mainly done in this function. Let’s look at the last important function call in the drawing process:

renderView.compositeFrame()

The renderView is the root of the Render Tree we talked about earlier. This function call basically sends the entire Layer Tree generated scene to the engine for display.

void compositeFrame() {
    try {
      final ui.SceneBuilder builder = ui.SceneBuilder();
      final ui.Scene scene = layer.buildScene(builder);
      if (automaticSystemUiAdjustment)
        _updateSystemChrome();
      _window.render(scene);
      scene.dispose();
    } finally{ Timeline.finishSync(); }}Copy the code

Ui.scenebuilder () finally calls the Native method SceneBuilder_constructor. This means that the UI.SceneBuilder instance is created by Engine. The next step is to call the layer.buildScene(Builder) method, which returns a UI.scene instance. The caller to the method compositeFrame() is renderView. So this layer right here is a property from the renderView, and we said that only the draw boundary node has a layer. RenderView, the root node of the Render Tree, is also a draw boundary. So where does this layer come from? As we explained in the article “Flutter Framework Analysis ii — Initialization”, renderView will schedule the first frame of creation during frame initialization:

void scheduleInitialFrame() {
    scheduleInitialLayout();
    scheduleInitialPaint(_updateMatricesAndCreateNewRootLayer());
    owner.requestVisualUpdate();
  }
  
  Layer _updateMatricesAndCreateNewRootLayer() {
    _rootTransform = configuration.toMatrix();
    final ContainerLayer rootLayer = TransformLayer(transform: _rootTransform);
    rootLayer.attach(this);
    return rootLayer;
  }
  
  void scheduleInitialPaint(ContainerLayer rootLayer) {
    _layer = rootLayer;
    owner._nodesNeedingPaint.add(this);
  }

Copy the code

In the method _updateMatricesAndCreateNewRootLayer (), instantiate a TransformLayer we see here. TransformLayer inherits from OffsetLayer. Construct by passing a transform parameter of type Matrix4. This Matrix4 is actually the same thing as the Matrix we see in Android. It stands for matrix transformation. The transform here comes from ViewConfiguration, which converts the ratio of device pixels into a matrix. Finally this layer is attached to the renderView. So this TransformLayer here is actually the root of the Layer tree.

Back to our drawing process. layer.buildScene(builder); This method is in its parent class, OffsetLayer. From this call we operate on the layer and convert the Layer tree into a scene:

ui.Scene buildScene(ui.SceneBuilder builder) {
    List<PictureLayer> temporaryLayers;
    updateSubtreeNeedsAddToScene();
    addToScene(builder);
    final ui.Scene scene = builder.build();
    return scene;
  }
Copy the code

A function call updateSubtreeNeedsAddToScene (); The layer Tree is traversed to set the _subtreeNeedsAddToScene flag bit. If any sublayers are added or removed, the sublayer and its ancestors will be set to the _subtreeNeedsAddToScene flag bit. Then addToScene(Builder) is called;

  @override
   @override
  ui.EngineLayer addToScene(ui.SceneBuilder builder, [ Offset layerOffset = Offset.zero ]) {
    _lastEffectiveTransform = transform;
    final Offset totalOffset = offset + layerOffset;
    if(totalOffset ! = Offset.zero) { _lastEffectiveTransform = Matrix4.translationValues(totalOffset.dx, totalOffset.dy,0.0)
        ..multiply(_lastEffectiveTransform);
    }
    builder.pushTransform(_lastEffectiveTransform.storage);
    addChildrenToScene(builder);
    builder.pop();
    return null; // this does not return an engine layer yet.
  }
Copy the code

Builder. pushTransform calls the Engine layer. That tells Engine I’m going to add a transform layer here. Then call ddChildrenToScene(Builder) to add the child layer to the scene, and then push the change layer off the stack.

void addChildrenToScene(ui.SceneBuilder builder, [ Offset childOffset = Offset.zero ]) {
    Layer child = firstChild;
    while(child ! =null) {
      if (childOffset == Offset.zero) {
        child._addToSceneWithRetainedRendering(builder);
      } else{ child.addToScene(builder, childOffset); } child = child.nextSibling; }}Copy the code

This is the call to iterate over adding sublayers. Basically, call addToScene() layer by layer. This method is implemented differently for different layers, but for container layers, it does three things: 1. Add the effects of your own layer and push it into the stack. 2. Add sub-layers. 3.

After all the layers are processed. Back to renderView.com positeFrame (), finally can finish processing the scene visible through _window. Render (scene); The call is sent to engine to display.

The paint phase of the rendering pipeline is now complete.

And so on, a little bit of what seems to lack, the analysis of the drawing process. We see a major call pipelineOwner flushCompositingBits () is in the update tree node _needsCompositing flag bit in the render. But we have covered the process here, it seems that we do not see where this flag bit is used. This flag bit must be used somewhere, otherwise why bother updating it? Go back to the code……

This flag bit is used by some renderObjects in their paint() function, and is used by calls to the PaintingContext function:

  void pushClipRect(bool needsCompositing, Offset offset, Rect clipRect, PaintingContextCallback painter, { Clip clipBehavior = Clip.hardEdge }) {
    final Rect offsetClipRect = clipRect.shift(offset);
    if (needsCompositing) {
      pushLayer(ClipRectLayer(clipRect: offsetClipRect, clipBehavior: clipBehavior), painter, offset, childPaintBounds: offsetClipRect);
    } else {
      clipRectAndPaint(offsetClipRect, clipBehavior, offsetClipRect, () => painter(this, offset)); }}void pushClipRRect(bool needsCompositing, Offset offset, Rect bounds, RRect clipRRect, PaintingContextCallback painter, { Clip clipBehavior = Clip.antiAlias }) {
    final Rect offsetBounds = bounds.shift(offset);
    final RRect offsetClipRRect = clipRRect.shift(offset);
    if (needsCompositing) {
      pushLayer(ClipRRectLayer(clipRRect: offsetClipRRect, clipBehavior: clipBehavior), painter, offset, childPaintBounds: offsetBounds);
    } else {
      clipRRectAndPaint(offsetClipRRect, clipBehavior, offsetBounds, () => painter(this, offset)); }}void pushClipPath(bool needsCompositing, Offset offset, Rect bounds, Path clipPath, PaintingContextCallback painter, { Clip clipBehavior = Clip.antiAlias }) {
    final Rect offsetBounds = bounds.shift(offset);
    final Path offsetClipPath = clipPath.shift(offset);
    if (needsCompositing) {
      pushLayer(ClipPathLayer(clipPath: offsetClipPath, clipBehavior: clipBehavior), painter, offset, childPaintBounds: offsetBounds);
    } else {
      clipPathAndPaint(offsetClipPath, clipBehavior, offsetBounds, () => painter(this, offset)); }}void pushTransform(bool needsCompositing, Offset offset, Matrix4 transform, PaintingContextCallback painter) {
    final Matrix4 effectiveTransform = Matrix4.translationValues(offset.dx, offset.dy, 0.0).. multiply(transform).. translate(-offset.dx, -offset.dy);if (needsCompositing) {
      pushLayer(
        TransformLayer(transform: effectiveTransform),
        painter,
        offset,
        childPaintBounds: MatrixUtils.inverseTransformRect(effectiveTransform, estimatedBounds),
      );
    } else{ canvas .. save() .. transform(effectiveTransform.storage); painter(this, offset);
      canvas
        ..restore();
    }
  }
Copy the code

NeedsCompositing is an input to these functions. As you can see from the code, it controls how these particular draw operations are implemented. If needsCompositing is true, pushLayer is called with the various layers we have seen before

  void pushLayer(ContainerLayer childLayer, PaintingContextCallback painter, Offset offset, { Rect childPaintBounds }) {
    stopRecordingIfNeeded();
    appendLayer(childLayer);
    final PaintingContext childContext = createChildContext(childLayer, childPaintBounds ?? estimatedBounds);
    painter(childContext, offset);
    childContext.stopRecordingIfNeeded();
  }

  @protected
  PaintingContext createChildContext(ContainerLayer childLayer, Rect bounds) {
    return PaintingContext(childLayer, bounds);
  }
Copy the code

The process is basically the same as what we saw before when we redrew a new layer.

If needsCompositing is false then we are going through various canvas transformations. If you are interested, you can take a look at the source code. I will not go into details here.

conclusion

This concludes the analysis of the Paint phase of the Flutter frame rendering pipeline. The drawing process is not as straightforward as the previous build and layout processes, which simply traverse the Element Tree or Render Tree. Another tree, layer Tree, will appear in the render phase. The whole drawing process is a process of converting render Tree into appropriate Layer tree and finally regenerating scene.

Finally, in addition to understanding the rendering process, I recommend you take a look at this video from a Google engineer: An in-depth look at the high-performance graphics rendering of Flutter. After watching this video, you will have a better understanding of the rendering of the Flutter framework and some of the performance issues you may encounter.