Original: www.didierboelens.com/2019/09/flu…

How does Flutter work?

Widgets, Elements, BuildContext, RenderObject, Bindings…

Reading Difficulty: Elementary

introduce

When I first got into Flutter last year, it was hard to find material on the Internet about Flutter. Although there were many articles about Flutter, there was almost nothing about the internal implementation of Flutter.

What exactly is Widgets, Elements, BuildContext? Why is Flutter execution so efficient? When executing, why do the results sometimes not meet the expectations? What is a tree, often mentioned?

When you develop a Flutter app, you are mostly dealing with Widgets. But have you ever wondered how these features work? How does the system know when and which parts need to be refreshed?

Part ONE: Background

The first part of the article will introduce some important concepts that will help you understand the whole article.

Start with the device

First, let’s start with the basics of the device.

When an app is running on your device, you can only see the specific content of the app through the screen.

More precisely, what you see is a series of pixels that make up a 2D image. When you touch the screen with your finger, the device simply recognizes where your finger is on the screen.

In most cases, app pages are updated because of four interaction scenarios:

  • Interaction with the device screen (e.g., finger tapping on the screen)
  • Interaction with the network (e.g., requesting data from a server)
  • Time-dependent interactions (e.g. animations)
  • Interaction with other external sensors

The image is rendered to the screen by the hardware (the monitor), which refreshes the display at a certain rate (usually 60 times per second). This refresh rate is also known as the ** “refresh rate,” which, in notation, is Hertz (Hertz).

The display obtains data from the Graphics Processing Unit (GPU) and displays the data on the screen. A GPU is a professional integrated circuit that does a lot of optimization to turn data (polygons and textures) into images. The number of images a GPU can generate * per second (frame buffering) * and send to the display is called the frame rate, and usually we use FPS as the unit of frame rate (e.g 60 frames per second, called 60 FPS).

You may ask me, why is this article starting with the concept of 2D graphics, Gpus, displays? What do these have to do with Flutter widgets?

Simply put, one of the main purposes of the Flutter app is to generate 2D images and make the 2D images interactive. I think it will be easier to understand when learning the actual workflow of Flutter if we can see it on a visual level.

And almost everything in Flutter is implemented based on a screen refresh.

Interface between code and physical devices

Everyone who is interested in a Flutter has seen this diagram describing the structure of a Flutter.

When we write the Flutter application with Dart, we are in the hierarchy of the Flutter framework (in green).

The Flutter framework interacts with the Flutter Engine (shown in blue) via the Window. The Window abstraction layer exposes a set of apis that allow the Flutter framework to interact indirectly with devices.

It is also through this abstraction layer that the Flutter Engine notifies the Flutter framework when:

  • When some events occur on the device, such as direction changes, Settings changes, memory problems, app running status changes, etc.
  • When something happens on the screen, like a gesture
  • When a platform channel sends some data
  • Most importantly, when the Flutter Engine is ready to render a new frame

The Flutter framework is driven by the RENDERING module in the Flutter Engine framework

It’s hard to believe, but it’s true.

The code in the Flutter framework is only executed when driven by the rendering module in the Flutter Engine framework. This includes the following four situations:

  • Gesture(Touch screen related events)
  • PlatformMessages (a message sent by a device, such as GPS)
  • DeviceMessages (device status changes such as device direction, APP entering background, memory warnings, device Settings, etc.)
  • FutureorThe HTTP request

The Flutter Engine framework will not execute any UI-related code if the RENDERING module in the Flutter Engine framework does not initiate a request.

(In fact, the Flutter framework itself can change the UI without a Flutter Engine driving it, but this is not recommended)

You might ask, how does the interface refresh when a gesture causes a UI change, or when a timer is used to perform a task (such as animation) that causes a UI change?

If you want the UI to change, or if you want to add a timer to execute some code, you need to tell the Flutter Engine that something needs to be refreshed.

Generally speaking, on the next refresh, the Flutter Engine will ask the Flutter framework to execute some code that will eventually generate a new scene for the Flutter Engine to render.

Therefore, the biggest question is how does the Flutter Engine get a new scene to provide to the rendering module?

To understand the inner workings of Flutter, take a look at the following GIFs.

A quick note (more on that later) :

  • Some external event (gesture, HTTP request, etc.) orfutureWill perform the task of changing the UI. In this case, a message (Schedule Frame) is sent to the Flutter Engine to inform it that the Flutter Engine needs to refresh.
  • inFlutter EngineBefore it’s ready to render, it emits oneBegin Framerequest
  • thisBegin FrameThe request will beFlutter frameworkIntercept, then executeTickersRelated tasks (e.g. animation)
  • These tasks may initiate a request to re-render (for example, an animation is not yet complete and it needs to be in the next stageBegin FrameTo continue executing the animation code)
  • And then,Flutter EngineinitiateDraw Framerequest
  • Draw FrameWill beFlutter frameworkInterception,Flutter frameworkAll tasks related to structure and size are found to update the layout and executed.
  • By the time all these missions are completed,Flutter frameworkThe task of updating the layout related to drawing continues
  • If there’s an update to the UI,Flutter frameworkNew ones will be sent to be renderedScene 给 Flutter Engine.Flutter EngineThe screen will be updated
  • And then,Flutter frameworkAll post-rendering tasks are performed, as well as some non-rendering subtasks.
  • And then this guy keeps repeating the process

RenderView and RenderObject

Before diving into the entire workflow, we need to understand the concept of Rendering Tree.

As mentioned earlier, all data will eventually be displayed as pixels on the screen. The Flutter framework transforms the Widgets we use to build our app into visually relevant content that will eventually be rendered onto the screen.

The corresponding class for the visually relevant content that will be rendered on screen is called RenderObject, which is used for:

  • Define areas in terms of size, position, geometry (collectively referred to as rendered content)
  • Define areas on the screen that recognize gestures

This set of renderObjects forms a tree called the Render tree; At the very top of the tree (=root), we’ll find the RenderView.

The RenderView represents the entire Render plane of the Render tree, and is itself a special RenderObject.

We can use the following picture to describe:

We’ll cover the relationship between Widgets and RenderObjects in a later article.

Now, let’s dig a little deeper.

First, the important thing — Bindings initialization

When you start a Flutter application, the main() method is called, and eventually the RunApp(Widget app) method is called.

When the runApp() method is executed, a Flutter initializes the interface between the Flutter framework and the Flutter Engine. This interface is called Bindings.

Introduce Bindings –

Bindings are the glue codes between the Flutter Engine and the Flutter framework. Between these two Flutter modules (The Flutter Engine and the Flutter framework), data can only be exchanged through Bindings. (One exception is RenderView, which we’ll talk about later.)

Each Binding handles a particular set of tasks, actions, and events, and is regrouped by activity domain.

As of this writing, the Flutter framework consists of eight bindings.

We will discuss the following four in this article:

  • SchedulerBinding
  • GestureBinding
  • RendererBinding
  • WidgetsBinding

For ease of understanding, we also list the remaining four (which we won’t cover in detail in this article) :

  • ServicesBinding: Responsible for handling fromplatformChannel data
  • PaintingBinding: Handles the image cache
  • SemanticsBinding: Reserve all semantics-related content for later implementation
  • TestWidgetsFlutterBinding: for the use of widgets test library

I’ll also mention WidgetsFlutterBinding, but it’s not really a binding, it’s more like a “Binding initializer.”

The diagram below shows the relationship between Bindings and Flutter Engine that I will describe later,

Let’s take a look at these “major” bindings.

SchedulerBinding

This binding has two main responsibilities:

  • First, tellFlutter Engine: “Hey! Next time you are not busy, remember to wake me up so I can work for a while and tell you what you need to render or if you need to wake me up later ”
  • Second, listen for and respond to the * wake up event * mentioned in the previous article (see below)

When does a SchedulerBinding request a wake event?

  • When the Ticker performs the UI change task,

    For example, suppose you start an animation that is driven by a Ticker, and the Ticker executes a callback every once in a while. To perform this callback, we need to tell the Flutter Engine to wake us up (by calling Begin Frame) on the next refresh. This will execute the Ticker callback to accomplish this task. After performing this task, if the Ticker needs to continue performing the animation task, it will call SchedulerBinding to schedule the next task for the next frame.

  • When the layout changes.

    For example, when you’re dealing with an event that changes the UI (e.g., changing colors, scrolling, adding/removing content from the screen), we need to take the necessary steps to finally render the content to the screen. In this case, when these changes occur, the Flutter framework will invoke the SchedulerBinding to make the Flutter Engine perform the task of refreshing the next frame. (See how it works in a moment)

GestureBinding

This Binding listens for gesture events in the Flutter Engine.

In particular, it is responsible for receiving gesture-related data, determining which area of the screen receives this data, and notifying the appropriate area.

RendererBinding

This binding is the glue code between the Flutter Engine and the Render Tree. It has two different responsibilities:

  • First, the user listening to Engine emit changes to device Settings that cause the UI orsemanticsChanging event
  • Second, provide the Engine with the data rendered to the screen

This Binding is responsible for driving PipelineOwner and initializing the RenderView in order to render data to the screen.

The PipelineOwner is a coordinator that knows which RenderObjects need to be rearranged and schedules them to complete those tasks.

WidgetsBinding

This Binding listens for events in device Settings that affect locales and semantic events.

sidenote

My guess is that future events related to SemanticsBinding will be migrated to SemanticsBinding, but at the time of this writing, the code has not been migrated.

In addition, WidgetsBinding is also the glue code between the Widget and the Flutter Engine. It has two main responsibilities:

  • The first is responsible for building the Widget structure
  • The second is to trigger the rendering process

BuildOwner builds the Widgets structure.

BuildOwner records which widgets need to be rebuilt and handles other tasks that cause structural changes in widgets.

Part two: From Widget to pixel

Now that we’ve covered the basics of the internals, it’s time to talk about widgets.

All documentation for Flutter will tell you that everything is a Widget.

That’s true, of course, but for the sake of accuracy, let me rephrase it:

From a developer’s perspective, the layout and interaction aspects of the user interface are all done through Widgets.

Why is that? Because a Widget can do more than just define dimensions, content, layout, and interaction on the screen. So what exactly is a Widget?

Immutable configuration

When you read the source code for Flutter, you will notice the definition of the Widget class:

@immutable
abstract class Widget extends DiagnosticableTree {
  const Widget({ this.key });

  finalKey key; . }Copy the code

What does that mean?

The annotation “@immutable” is important because it tells us that variables in any Widget class must be final, in other words: “assigned only once, at initialization.” Once initialized, the Widget can no longer change its internal variables.

Because a Widget is immutable, it is more like an immutable configuration fileCopy the code

The hierarchy of Widgets

When you develop Flutter, you write the interface with widgets that look like this:

Widget build(BuildContext context){
    return SafeArea(
        child: Scaffold(
            appBar: AppBar(
                title: Text('My title'),
            ),
            body: Container(
                child: Center(
                    child: Text('Centered Text'(), ((), ((), ((), ((), (() }Copy the code

This example uses seven widgets, which together form a hierarchy.

If simply enumerated, the hierarchy would look like this:

As you can see, this looks like a tree, and SafeArea is the root of the tree.

Trees and forests

As you surely know, a Widget can contain multiple widgets by itself. For example, in the code above, I could reconstitute it like this:

Widget build(BuildContext context){
    return MyOwnWidget();
}
Copy the code

The “MyOwnWidget” here includes SafeAre, Scaffold, and so on, but the most important thing to note in this example is that:

A Widget can be a leaf node, a node, a tree itself, or even a forest of trees…

—

The concept of Element

Why do we mention Element here?

As we will see later, in order to be able to render the pixels of the corresponding image on the device, Flutter needs to know all the details of the various parts that make up the screen, so it inflate all the widgets in order to know all the data.

For example, imagine Russian nesting dolls: at first, you just see one nesting doll, which in fact contains another nesting doll, which in turn contains another nesting doll, and so on…

The process of Flutter any widget is similar to the process by which we take out all the dolls in a Russian nesting doll.

The following figure shows the hierarchy behind the Widget inflate (part of it) in the previous code. The yellow highlights the widgets mentioned in the previous code so that you can recognize them from the Widget tree.

Important statement

We say “Widget tree” only for ease of understanding, because developers develop with widgets, but the Widget tree does not exist in Flutter!

What we’re really describing is: “Element tree.”

Now it’s time to introduce Element…

Each widget corresponds to an element. The elements are related to each other to form a tree. Thus, an element is a reference in the tree.

First, let’s imagine that an element is a node with a parent and child (perhaps none). These nodes are joined together by the parent relationship, and we have a tree structure.

As shown above, Element holds a Widget and possibly a RenderObject.

To be more precise… The Widget that Element holds creates an Element instance of its own type!

Let’s review…

  • There are no Widget trees, only Element trees
  • Element is created by Widget
  • Element holds widgets that create its own
  • Element byparentRelationships are linked together
  • Element may have a child or children
  • Element may also holdRenderObject

Does Element define how interfaces relate to each other

To better understand the concept of Element, let’s take a look at the following figure:

As you can see, the Element tree is the actual link between the Widget and the RenderObject.

But why use widgets to create an Element?

There are three main types of widgets

In Flutter, widgets are divided into 3 categories, which I call (just my own way of categorizing them) :

  • The proxy class

    The main purpose of this type of Widget is to hold information that can be accessed by its child. Typical examples are inheritedWidgets and LayoutId.

  • Apply colours to a drawing class

    Such widgets contribute directly or indirectly to layout in three ways:

    • size
    • location
    • Layout, rendering

    For example Row, Column, Stack, Padding, align, Opacity, RawImage…

  • Component classes

    These widgets do not directly provide final information, such as size, location, or appearance, but rather provide data (or clues) that lead to the final information. Such widgets are commonly referred to as components.

    For example: RaisedButton, Scaffold, Text, GestureDetector, Container…

This PDF lists most of the widgets by type.

Why is categorization so important? Because each type of Widget has an Element of its own type…

Element type

Here are the different types of elements:

As shown in the figure, elements fall into two main categories:

  • ComponentElement

    Such elements do not directly participate in UI rendering.

  • RenderObjectElement

    Elements of this class are directly involved in UI rendering.

So far, many concepts have emerged. Why are we introducing them? How do they relate?

How do Widgets and Element work together?

In Flutter, the entire mechanism depends on the Invalidate element or renderObject.

There are two ways to invalidate an element:

  • By calling thesetStateCan invalidateStatefulElement(Notice I didn’t mention it hereSatefulWidget ）
  • proxyElement(e.g.InheritedWidget) bynoticeTo invalidate all dependencies on itelement 。

The invalidate element is used to mark it as dirty.

If the structure of an element does not change, invalidate the renderObject. For example:

  • UI changes in size, position, and shape
  • Background color, font changes, etc. need to be redrawn

As a result, the corresponding renderObject is rebuilt or repainted.

A SchedulerBinding is called to cause the Flutter Engine to refresh the UI when the next frame is refreshed. This is where the rendering process begins.

onDrawFrame()

At the beginning of this article, we mentioned that the SchedulerBinding has two main responsibilities, one of which is to process rebuild requests from the Flutter Engine. Now let’s go through the process in detail…

The sequence diagram below shows what happens when the Flutter Engine executes a SchedulerBinding onDrawFrame() method:

Step 1: Elements

The WidgetBinding will be invoked, so let’s first look at the process associated with Element.

We know that BuildOwner handles the Element tree. In fact, WidgetBinding calls BuildOwner’s buildScope method.

This method iterates through the invalidated Elements (element in the dirty state) and calls the rebuild() method on Element.

Rebuild () actually does the following things:

  1. rebuild()Will performelementHolds the widgetbuild()Method, this is alsorebuild()The most time-consuming operation in. thisbuild()A new widget is returned.
  2. If element does not have a child, the new widget is inflate (see below), otherwise
  3. Compare the new widget to the widget held by Element’s Child,
    • If it is interchangeable (the widget type and key are the same), the old widget is replaced with the new widget, with the child element unchanged.
    • If it is not interchangeable, the child Element is thrown away and the new widget is inflate.
  4. With the widget inflate, a new element is generated, and this element is mounted to the current element as a child. (To mount oneself as a child to an Element tree)

The whole process is shown below.

An explanation of Widget Constructors

With widget constructors, the widget is asked to create an Element whose type is related to the type of the widget that created it.

As a result,

  • InheritedWidgetWill be generatedInheritedElement
  • StatefulWidgetWill be generatedStatefulElement
  • StatelessWidgetWill be generatedStatelessElement
  • InheritedModelWill be generatedInheritedModelElement
  • InheritedNotifierWill be generatedInheritedNotifierElement
  • LeafRenderObjectWidgetWill be generatedLeafRenderObjectElement
  • SingleChildRenderObjectWidgetWill be generatedSingleChildRenderObjectElement
  • MultiChildRenderObjectWidgetWill be generatedMultiChildRenderObjectElement
  • ParentDataWidgetWill be generatedParentDataElement

Different elements have different behaviors, such as

  • StatefulElementIs called at initialization timewidget.createState()To create aStatethestateWill beelemntHold.
  • RenderObjectElementWhen mounted, one is createdRenderObjecttherenderObjectWill be added torender tree, will beelementHold.

Step 2: renderObjects

After all the dirty elements have been processed and the Element Tree has been built, the render process begins.

RendererBinding is responsible for rendering tree, so after WidgetsBinding completes the build task, It calls the drawFrame method of RendererBinding to do the render work.

The sequence diagram for executing drawFrame() in RendererBinding looks like this:

During the whole process, the following process is followed:

  • eachdirtyThe state of therenderObjectExecutes their Layout method, which recalculates the size and shape;
  • eachneeds paintThe state of therenderObjectWill userenderObject 的 layerRedraw;
  • Final generatedsceneWill be sent toFlutter Engine ，Flutter EngineThen willsceneRender to screen;
  • SemanticWill also be updated and handed overFlutter EngineRendering;

After the entire process, the screen is refreshed.

—

Part three: Gesture processing

GestureBinding is responsible for handling gesture-related logic.

Flutter Engine will through the window. The onPointerDataPacket send information about gestures, and GestureBinding can intercept the message, and then perform the following process:

  1. conversionFlutter EngineCoordinates to match the resolution of the current device,
  2. According to the transformed coordinates, fromrenderViewTo deriveallIn this coordinate regionRenderObject ，
  3. Traverse theserenderObjectAnd send gesture events to them in turn,
  4. renderObjectOnce you receive the event, you handle it yourself.

This shows how important renderObject is.

Part four: Animation

The last part of this article focuses on the following concepts of animation, in particular the Ticker concept.

When you initialize an animation, you usually use the AnimationController class, or the associated Widget or Conponent.

In Flutter, everything related to animation is about Ticker.

The Ticker actually does only one thing when the Ticker is active: “It tells the ScheduleBinding to register a callback and tells the Flutter Engine to wake itself up at the next frame”.

When the Flutter Engine is ready, it wakes up the SchedulerBinding via onBeginFrame.

The SchedulerBinding intercepts the request and then iterates through and invokes all of the Ticker’s callbacks.

Each ticker tick is intercepted by the corresponding controller and the corresponding code is executed. If the animation is complete, the Tikcer’s state is changed to “Disabled”; otherwise, the Ticker calls SchedulerBinding again to schedule another callback.

The overview

We now know the inner workings of Flutter. Here is the process:

BuildContext

If you look back at images showing different types of elements, you’ll probably notice the definition of base Element:

abstract class Element extends DiagnosticableTree implements BuildContext {... }Copy the code

Here we see the famous BuildContext.

So what exactly is BuildContext?

BuildContext is an interface that defines getters and methods. Element implements this interface.

BuildContext is used primarily in the build() methods of statelessWidgets and StatefulWidgets, as well as in the State of statefulWidgets.

BuildContext is Element itself in the following cases:

  • WidgetRebuild (context in the Build or Builder method)
  • StatefulWidgetIn theStateRefer to thecontext

That said, most developers work with Element every day without knowing it.

How do I use BuildContext?

BuildContext corresponds to the element of the widget and represents the location of the widget in the tree, so BuildContext is often used in the following scenarios:

  • To obtainwidgetThe correspondingRenderObject
  • To obtainRenderObjectThe size of the
  • Walk through the Element tree. In fact, in generalofThe implementation of the Theme. Of (context) method, for example, depends onBuildContext 的

An inappropriate example

Now we know that BuildContext is actually element, and to show you another way to use it, here’s an unfortunate example…

In the next example, let’s update the StatelessWidget to look like a StatefulWidget without the setState() method, which we do by BuildContext.

warning

Never code like this!

Its sole purpose is to let us know that the StatelessWidget can also update itself.

If you need to change your status, use the StatefulWidget

void main(){
    runApp(MaterialApp(home: TestPage(),));
}

class TestPage extends StatelessWidget {
    // final because a Widget is immutable (remember?)
    final bag = {"first": true};

    @override
    Widget build(BuildContext context){
        return Scaffold(
            appBar: AppBar(title: Text('Stateless ?? ')),
            body: Container(
                child: Center(
                    child: GestureDetector(
                        child: Container(
                            width: 50.0,
                            height: 50.0,
                            color: bag["first"]? Colors.red : Colors.blue, ), onTap: (){ bag["first"] = !bag["first"];
                            //
                            // This is the trick
                            //
                            (context as Element).markNeedsBuild(); }),),),); }}Copy the code

In fact, when you call setState() in StatefulWidget, you’re actually doing _element.markNeedsbuild () as well.

conclusion

Ok, finally finished another article.

I think the architecture of Flutter is interesting. It is designed to be efficient, scalable, and open to future expansion.

Also, concepts like Widget, Element, BuildContext, and RenderObject are not always easy to understand.

Hope you found this article useful.

Stay tuned for new articles, and in the meantime, have fun coding.