Flutter UI system

1. Screens and images

Before introducing the UI system of Flutter, let’s look at how the display displays graphics, which will help to understand the design of the UI system.

1.1 Basic Unit – pixel

The visualization of everything in nature is relying on the drawing, but it is the display in the drawing the basic unit of a physical display units, each unit we can call it a physical pixels, each pixel and can emit a variety of color, display imaging principle is displayed on the different physical pixel different color, Finally, the complete image is formed.

(figure from www.bilibili.com/video/BV1dA… 22s screenshot in the video)

• RGB arrangement: Each pixel is composed of three closely adjacent RGB sub-pixels arranged in a neat arrangement, which is common on LCD screens.
• Delta arrangement: The number of sub-pixels in each of the three colors is reduced by one-third compared to the standard RGB arrangement. Each pixel is arranged r-G, G-B or B-R, and the six pixels share a surrounding sub-pixel. The actual pixel density is only about 70% of the standard RGB arrangement.
• Diamond Pentile arrangement: Each row of the Pentile arrangement has three fewer sub-pixels, and the total number of sub-pixels is reduced by a third.
• Boe Arrangement: a variant of Delta, designed by BOE. Divide a green pixel into two sub-pixels. The arrangement of pixels looks like red, green and blue figures with small eyes and thick lips, just like the expression of Zhou Dongyu, also known as Zhou Dongyu arrangement.

(figure from network: wantubizhi.com/image.aspx?…).

The layout scheme described above, RGB, is often seen in LCD screens, while the others are created for OLED screens, because OLED is self-luminous and belongs to organic electroluminescence, while organic matter is prone to failure and self-decomposition, and the failed organic matter can no longer emit light. So, in order to extend the life of organic matter, a variety of arrangements have been created to increase the life of OLED screens.

1.2 Screen Resolution

A pixel point can display 16 million colors, consisting of RGB three primary colors of the screen, each base color (R, G, B) depth expanded to 8 bit(bit), that is, 2 to the power of 24 16 million colors, color depth, can be displayed more rich and beautiful. So how big are pixels? Depending on the resolution of the monitor, different resolutions of the same area of the display, its pixel size is different.

(figure comes from www.dazhuanlan.com/2019/10/12/.)

1.3 DPI

DPI, also known as PPI (pixels per Inch), is calculated by dividing the number of pixels a screen has in landscape or portrait by its width or height in inches. A higher DPI means that each pixel has to be smaller in order to fit into the available space, which means the screen is sharper and the level of detail the screen can be drawn at is higher.

(figure comes from www.dazhuanlan.com/2019/10/12/.)

(figure comes from www.dazhuanlan.com/2019/10/12/.)

1.4 Display Principle

To update the display, the screen is refreshed (taking data from the GPU) at a fixed rate, such as 60Hz for a phone screen. When one frame is drawn and ready to be drawn to the next, the display emits a vertical synchronization signal (such as VSync), which a screen at 60Hz emits 60 times a second. This signal is used to synchronize the CPU, GPU, and display. Generally speaking, in a computer system, the CPU, GPU, and display work together in a specific way: the CPU submits the calculated display content to the GPU, the GPU renders it and puts it into the frame buffer, and then the video controller takes the frame data from the frame buffer and passes it to the display according to the synchronization signal.

(figure comes from www.jianshu.com/p/7b7975158.)

Due to the VSync mechanism, if the CPU or GPU does not complete the submission within a VSync period, the frame is discarded until the next opportunity to display, while the display remains unchanged. That’s why the interface is stuck.

CPU and GPU have different tasks. CPU is mainly used for basic mathematics and logic calculation, while GPU mainly performs complex mathematics related to graphics processing, such as matrix change and geometric calculation. The main function of GPU is to determine the color value of each pixel finally transmitted to the display.

1.5 the UI system

Due to the final graphic calculation and drawing is done by the corresponding hardware, and direct operation of hardware instructions there are usually operating system block, developers usually don’t directly in the face of hardware, hardware encoding development efficiency is low, usually blocked access to the underlying hardware operating system will provide some packaging after operation of API for the application of the operating system calls.

For application developers, directly calling the APIS provided by the operating system is complex and inefficient, because the apis provided by the operating system are often relatively basic, and direct calls require a lot of details about the API. So every programming language that develops GUI programs encapsulates a layer on top of the operating system, encapsulates the operating system’s native API in a programming framework and model, and then defines a simple development rule for developing GUI applications. And this layer of abstraction is the “UI” system. For example, Android SDK encapsulates the API of Android operating system and provides a UI system of “UI description file XML+Java operation DOM”, while iOS UIKit abstracts View in the same way. They abstracted the OPERATING system API into a basic object (such as the Canvas for 2D graphics) and then defined a set of rules to describe the UI, such as the UI tree structure, the single thread principle for UI operations, and so on.

That’s what Flutter does. It provides a set of Dart apis and then underneath it implements a set of code across multiple ends using skia, a cross-platform drawing library that internally calls the OPERATING system API.

Ii. Flutter UI system

Flutter comes with a rendering engine that provides a complete solution from the underlying rendering logic to the upper development language: view rendering is completely closed loop within its framework and does not depend on any components provided by the underlying operating system, ensuring a high degree of consistency of view rendering on Android and iOS.

So how did Flutter achieve this solution? Let’s take a look at the framework design of the Flutter.

2.1 Framework design of Flutter

(Photo from flutter.cn/docs/resour…)

The Flutter was divided into three layers, Embedder (operating system adaptation layer), Engine (rendering Engine and Dart VM layer), and Framework (UI SDK layer). As you can see from the diagram above, the components of each layer of the Flutter framework are defined with clear boundaries, and the capabilities they provide up and the ability to rely on them down.

2.2 Embedder – An operating system adaptation layer

Embedder is an operating system adaptation layer that implements platform-related features such as render Surface Settings, thread Settings, and platform plug-ins. As we can see from this, the Flutter platform has few relevant features, which makes it relatively inexpensive to maintain cross-end consistency at the framework level.

(Photo from flutter.cn/docs/resour…)

2.3 Engine – Rendering Engine and Dart VM layer

The Engine layer consists of Skia, Dart, and Text, and implements the rendering Engine, Text layout, event handling, and Dart runtime. Skia and Text provide upper-level interfaces with the ability to call lower-level rendering and typography, and Dart provides Flutter with the ability to call Dart and the rendering engine at runtime. The Engine layer combines them and renders views from the data they generate.

(Photo from flutter.cn/docs/resour…)

2.4 Framework – UI SDK layer

The Framework layer is a UI SDK implemented with Dart, including animation, drawing, and gesture recognition. In order to provide a more intuitive and convenient interface for drawing fixed style graphics such as controls, Flutter also builds upon these basic capabilities to encapsulate a set of UI component libraries based on the Material and Cupertino visual design styles. We were able to use these component libraries directly when developing Flutter.

(Photo from flutter.cn/docs/resour…)

2.5 Drawing Process

After looking at the framework design of the Flutter, let’s look at how it works. A schematic diagram can be found on the Flutter website.

(Photo from flutter.cn/docs/resour…)

According to the schematic disassembly, the GPU synchronizes the UI thread with the VSync signal. The UI thread uses Dart to build the abstract view structure, and then submits the data required by the Skia engine to render to the GPU, which is provided to the GPU through OpenGL or Vulkan. Layer composition is performed on the GPU thread.

We can also organize the process according to the VSync signal.

(figure comes from www.debugger.wiki/article/htm.)

The Flutter calculates and synthesizes the view data between the VSync signals of the two hardware clocks, which Skia gives to the GPU to render: Dart is used by the UI thread to build the view structure data, layer composition is performed on the GPU thread, and then the Skia engine processes the data into GPU data, which is finally provided to the GPU rendering through OpenGL.

The entire rendering process of the Flutter is based on the Dart API and calls the OPERATING system API, so the client provides a single canvas for a highly consistent rendering experience from business logic to functional presentation with near-native performance.

The foundation of the Flutter UI system

After understanding this part of the interaction between the Flutter UI system and the operating system, take a look at how the Flutter uses the Dart API to manipulate the canvas.

3.1 Page Structure

Usually when we draw on a canvas, we have to have a general outline of the painting called a line draft, dividing the canvas into areas, and the Flutter was the same.

In the structure of the Flutter sample page above, you can see that the page tree structure is formed by individual controls. In the Flutter architecture, however, the real task of Rendering components on the screen is not in the Widget layer, but in the Rendering layer. The render layer has two important trees, the Element tree and the RenderingObject tree.

3.2 the three trees

(figure comes from the book. Flutterchina. Club/chapter14 / e…).

The widgets on a Flutter page are organized as a tree, a control tree. Flutter creates different types of render objects with each control in the control tree, forming the render object tree. But what the Flutter really represents is the Element that is displayed on the screen, and the Widget just describes the configuration data of the Element. So the UI tree is actually made up of individual Element nodes.

The final Layout and rendering of components are completed by RenderObject. The general process from creation to rendering is as follows: The Element is generated according to the Widget, and then the corresponding RenderObject is created and associated with the Element. RenderObject property. Finally, the layout and drawing are completed by the RenderObject.

3.3 summarize

Flutter creates different types of render objects with each control in the control tree, forming the render object tree. Then RenderObject will calculate the final display effect of all layers according to the rules of size, level, transparency and so on, and classify and merge the same layers to simplify the rendering tree and improve the rendering efficiency. After the synthesis, Flutter will transfer the geometric layer data to the Skia engine for processing into two-dimensional image data, which will be rendered by THE GPU to complete the display of the interface.

References:

Flutter website

Flutter of actual combat

Screen arrangement

Pixels, points and resolutions

Screen display image principle

Author: Lin Kaifeng