I. Evolution of mobile cross-platform technology

1. The introduction

Mobile Internet has been developing for more than ten years. With the continuous popularity of Android, iOS and other smart phones, mobile terminal has gradually replaced PC terminal and become a battle field for military experts. As the saying goes, “whoever gains the mobile terminal gains the world”, mobile terminal has become the largest traffic distribution portal in the Internet field, and a large number of Internet companies are rising under this general trend.

2. Why cross-platform technology

With the rapid development of mobile Internet, the competition between companies is becoming more and more fierce. How to quickly implement good ideas and quickly try and error has become a concern. Improve r&d efficiency, shorten r&d cycle, ensure rapid trial-and-error of products and rapid iteration of new features, so that new products and new features can reach Android, iOS and other multi-end users at the fastest speed.

As we all know, Android apps are written in Java or Kotlin, iOS apps in Objective-C or Swift, and Web apps in HTML /CSS/JavaScript. When you need to develop multi-end applications, each end needs independent research and development, testing, up to the launch, and subsequent maintenance work, the workload increases exponentially, is bound to extend the development cycle.

In order to solve the problem of multi-stage independent development, cross-platform technology emerges at the right moment. Major Internet companies have invested a lot of manpower to solve this problem, so various cross-platform technology frameworks emerge. Facing the endless emergence of cross-platform technology solutions in the mobile field, how to choose the technology?

3. Selection of mobile terminal technology

As a cross-terminal technology solution for mobile terminals, we pay attention to the following four aspects: R&D efficiency, dynamic, multi-terminal consistency, and performance experience.

  1. Research and development efficiency: maximize code reuse, reduce adaptation workload of multi-end differences, reduce development costs, focus on business development, and achieve the ultimate goal of “write once, Run everywhere”. Efficiency improvement runs through the life cycle of the whole business. Even after the launch of the business, the subsequent maintenance cost can be continuously reduced and the iteration speed of new features can be accelerated, which is a continuous efficiency benefit. Of course, it has to be said that any new technology has some costs in the initial learning phase of development, but the benefits of getting started are long-term.
  2. Dynamic: Breaking through the update frequency of channels and quickly iterating new functions, which is not only the appeal of cross-platform technology, but also the essential trump card for Native technology, which is also an important assessment point for cross-end technology evaluation.
  3. Multi-terminal consistency: The overall style of multi-terminal UI design of a good product is often unified. Therefore, after the business side adopts the original independent development, it needs to spend a lot of extra time to modify the UI to ensure multi-terminal consistency. It can be seen that the efficiency lag brought by the independent development mode of each end is not only the workload of developing a code for Android and iOS, but also the work of consistent alignment of the UI of both ends.
  4. Performance experience: In general, cross-end solutions have all of these advantages, but are less fluid in performance than native solutions. It is reasonable to sacrifice part of experience in exchange for improved efficiency. If cross-platform technology solution achieves these four points at the same time, then the original technology may have retired from the stage of history and has been dominated by cross-platform technology. Therefore, the performance of cross-platform technology often becomes the core indicator.

4. Cross-platform technology division

In the constant pursuit of R&D efficiency and experience, cross-platform technology frameworks on mobile terminals emerge in an endless stream. However, there are many martial arts in the world, and all changes are consistent with one another. From their core essence, they can be roughly divided into the following three categories:

  1. Web technologies: Technologies that rely on WebView and have limited functionality and poor performance experience, such as PhoneGap, Cordova, and applets.
  2. Native rendering: Using JavaScript as the programming language, the UI is rendered by converting the middle layer to Native controls such as React Native and Weex.
  3. Self-rendering: A self-rendering framework that can be implemented by calling skia and other methods without relying on native controls such as Flutter and Unity.

5. Cross-platform evolution

Cross-platform technology has always been the dream pursued by every developer with pursuit, but also the nightmare of the old guard. The cross-platform multi-terminal integration scheme is bound to overturn the existing original independent development mode of each terminal. The following lists the evolution stages of several key technical schemes among many cross-platform technologies.

As can be seen from the figure above, the technological evolution process can be roughly divided into the following three stages: In the first stage, the Hybrid development technology, which uses WebView technology to draw the interface, exposes some of the system capabilities to JS invocation through JS Bridge. Its disadvantages are poor performance, limited functions and poor scalability, and it is not suitable for complex interactive scenarios, such as Cordova. In the second stage, in view of WebView interface performance and other problems, rendering returned to the Native rendering, and Native controls were only called through JS. Compared with WebView technology, the performance experience was better. This is the design idea of most cross-platform frameworks at present, such as React Native and Weex. In addition, the recent small program is also more fire, the fusion of the first and second stage, still use WebView as a rendering container, by limiting the subset of the Web technology stack, standardized component use, and gradually introduce native controls on behalf of WebView rendering, in order to improve performance. In the third stage, although native controls were used through the bridge technology to solve the problem of limited functions and improve performance experience, there was still a large gap compared with the native experience, and it was very labor-consuming to deal with the platform differences. There is a lot of interest in Flutter because of its own rendering engine solution, which minimizes the differences between platforms while matching the native high performance experience.

With so many cross-platform solutions out there, why is Flutter the most popular cross-platform technology and what are its advantages?

RN and Weex both use JavaScript as a programming language. As a front-end development language, JavaScript shines brilliantly in cross-platform development. Web technology can be used to develop not only websites, but also mobile web applications and mobile applications. There seems to be a tendency to unify the three worlds (Android, iOS, Web), and this is often referred to as the “big front end” era. These technical solutions are not very smooth, platform consistency is poor, so far has not been able to replace a large area of native development.

Written in the Dart language, the development experience of Flutter is closer to that of the client. Based on the feedback, the process of Flutter development environment is not very friendly for front-end development. Flutter is also positioned as a multi-end integration process, but with the client as the first step, it will smooth out the Android and iOS development experience and gradually move into the Web side. As we can see from the Roadmap of Flutter planning, Flutter for Web is still in the early stage. The Flutter client has already launched a number of applications.

Previously, the term “big front-end” has been used to describe Flutter technology. In my opinion, it is more appropriate to refer to Flutter as “big mobile”. The UI framework of Flutter supports client applications first (Android/iOS) and then Web applications. In the era of mobile Internet, many companies have formulated the “mobile first” strategy, and even only develop mobile terminal without Web terminal. The era of the mobile Internet has created the “big mobile”, and Flutter, as a high-performance cross-platform technology solution capable of matching native technologies, may take over the world.

In the field of cross-platform technology, as long as the challenges remain, the technology will not stop. With the continuous evolution and innovation of technology, it will eventually become better.

6. Technological advantages of Flutter

Flutter is a completely cross-platform solution. Instead of using webView or JS to bridge native controls, Flutter implements its own UI framework and renders to the screen via Skia at the bottom of the engine. For services provided by mobile devices themselves that need to be used outside the UI, such as camera, positioning, screen touch, etc., Platform Channels are used to communicate with the native system.

As for the advantages of Flutter, go back to the previous four factors of mobile terminal technology selection: r&d efficiency, dynamics, multi-terminal consistency and performance experience, which correspond to the following set of words respectively.

  1. High efficiency: Writing code in DART takes a while to get started, but it’s efficient once you’re good at it. One set of code for multiple platforms (Android, iOS, Web), and the efficient Hot Reload can quickly assist debugging;
  2. Dynamic: In March 2017, Apple sent a warning email banning hot updates to iOS apps like JSPatch, making iOS dynamic a topic that should not be discussed publicly. Similarly, the Flutter engine made some attempts at dynamization in one official version, but was later removed due to risk considerations. Of course, this has not prevented anyone from exploring the technology.
  3. High consistency: To achieve pixel-level UI control, the Flutter rendering engine relies on the cross-platform Skia graphics library. It only relies on the system’s graphics-related interfaces, such as Vulkan on Android and Metal on iOS, which are called by the Skia wrapper. The experience consistency of different platforms can be guaranteed to the greatest extent, as shown in the figure below.

  1. High performance: The rendering performance is superior to existing cross-platform frameworks and comparable to native cross-platform technology solutions. Dart code is more efficient than JS code, compiled into platform native code through AOT, and rendered using self-rendering SKia solution, which requires neither JS Bridge nor Art VIRTUAL machine. Let’s look at the performance of Flutter from the rendering principle.

Illustration:

  • Android native Framework, by calling the Java Framework layer, and then calling skia to render interface;
  • Other cross-platform solutions (such as RN) convert JS-written APP into corresponding Native rendering logic through JSBridge intermediate layer, which increases more logic than Native code and has inferior performance to Native framework.
  • The Application of Flutter Framework calls Dart Framework layer and then directly calls SKia to render the interface without going through the process of Native Framework. It can be seen that the rendering performance of Flutter Framework is not weaker than that of Native technology, which is a cross-platform technology with a high performance ceiling.

Of course, it has to be said that the current Flutter is indeed not perfect, and there are some shortcomings, such as incomplete ecology and package size. However, the upper limit of Flutter is relatively high, and there is a large space for imagination. I believe that with more developers involved and more polishing, Flutter will do better in the future.

7. Recent development of the industry

Flutter was officially announced at Google I/O in May 2017 and Flutter1.0 was released in December 2018, triggering a large number of developers and enterprises around the world to start researching Flutter. Flutter was voted one of the most popular frameworks among developers in StackOverflow’s 2019 Global Developer Docs survey, ahead of TensorFlow and Node.js.

Up to now, more and more companies around the world have applied Flutter technology in well-known apps and implemented Flutter. In particular, well-known Domestic Internet companies have invested heavily in Flutter and their communities are very active.

8. Flutter future trends

Currently available on both Android and iOS mobile devices, Flutter’s vision is to be a multi-terminal UI framework that supports not only mobile devices, but also the Web, desktop, and even embedded devices. A framework for developing Web applications using Flutter was unveiled at Google I/O developer Conference 2019. In September of that year, Flutter 1.9 was released and The Flutter Web was incorporated into the Main Flutter repository.

The architecture diagram shows that Flutter uses the same Dart Framework layer to unify the Flutter C++ engine and Web engine. Flutter can run on Android, iOS, and Browser. The Code for the Flutter engine shows that the Flutter also supports the Fuchsia operating system.

Fuchsia is a new operating system being developed internally by Google that uses Flutter as its default UI framework. This means that Flutter naturally supports Fuchsia, which gives Flutter an advantage over many other cross-platform solutions.

In terms of Fuchsia technology architecture, The core layer of Zircon, LK, is designed for small and medium sized systems in embedded applications. The code is simple and suitable for embedded devices and high performance devices, such as IOT and mobile wearable devices, where there is no standardized monopolist. In addition, there are modules of voice interaction, cloud and intelligence in the framework layer. Therefore, the author speculates that Fuchsia will take the lead in the application of intelligent embedded devices such as voice control in the future.

The two most promising technologies in the future are 5G and IoT. The demand for 5G is largely due to the “IoT era” of mobile Internet development. At this stage of development, the number of internet-connected devices in the world could reach 50 billion. With the arrival of 5G+IOT era, the size of the Flutter package is no longer an issue. Maybe the Flutter technology has a longer life than the client, or maybe Fuchsia is riding the Internet of Things. Your stack of Flutter technologies can move seamlessly, a chance to overtake on a curve.

This concludes the cross-platform technology evolution and the advantages of Flutter. Seeing this, I believe you may be interested in the technology of Flutter. In order to give you a quick and not boring insight into the inner workings of Flutter, Gityuan has used a series of diagrams to help you quickly understand Flutter from its overall architecture.

Ii. The Structure of the Flutter engine

1. Flutter technical architecture

The overall technical architecture of Flutter is divided into four layers, including Dart APP, Dart Framework, C++ Engine and Platform from top to bottom.

The core of the Flutter architecture is the Framework and Engine:

  • Flutter Framework layer: Dart encapsulates the core functions of the whole Flutter architecture, including widgets, animations, drawing, gestures, and other functions. There are Material (Android style UI) and Cupertino (iOS style UI) UI interfaces for building widgets and implementing UI layout.
  • The Flutter Engine layer is a lightweight runtime environment written in C++ for high-quality mobile applications that implements the core libraries of Flutter, including Dart virtual machine, animation and graphics, text rendering, communication channels, event notification, plug-in architecture, and more. The engine is rendered using Skia, a 2D graphics rendering library, and the virtual machine uses Dart VM, an object-oriented language, which is hosted into the embed layer of Flutter. The shell implements platform-specific code, such as interacting with the on-screen keyboard IME and system application life cycle events. Different platforms have different shells, such as Android and iOS shells.

2. Flutter compilation products

After looking at the internal architecture of Flutter, you may be wondering how Flutter can be developed without Android/iOS native language technology. How does Dart code make Flutter recognizable to different systems?

The Dart Code in the figure contains the business Code written by the developer. Engine Code is the Engine Code. If there is no custom Engine, there is no need to recompile the Engine Code.

A copy of Dart code that compiles and generates a two-ended product that implements cross-platform capabilities. After processing by compilation tools, two-end products can be generated. The compilation product in release mode is shown in the figure. The Android product is app.apk composed of VM, ISOLATE command and data segments respectively and flutter. The iOS product is Runner. App made up of App. Framework and Flutter. Framework.

This process involves frontend_server, gen_Snapshot, xcrun, and ninja compilation tools. The Frontend_server front-end compiler performs lexical analysis, syntax analysis, and related global transformations to convert dart code into AST(abstract syntax tree) and generate the DART kernel in app.dill format. Gen_snapshot generates optimized FlowGraph objects according to intermediate code after CHA, inline and a series of execution flow optimization, and then converts them into binary instructions of specific system architecture (ARM/ARM64, etc.).

3. Start the Flutter engine

Now that you know about the compilation of Flutter, you may wonder how the engine of Flutter starts and how it connects with Native.

If you are familiar with Android, you should be familiar with the APP startup process and execute the onCreate() method of Application and Activity. The onCreate() method of FlutterApplication and FlutterActivity is the hub that connects Native to Flutter.

  • The onCreate process of FlutterApplication. Java mainly completes the initial configuration, loads the engine libflutter.
  • The onCreate process of FlutterActivity. Java initializes objects such as Engine, Dart VIRTUAL machine, Isolate, and taskRunner through FlutterJNI AttachJNI(). Dart calls main() and executes runApp(Widget app) to process the entire Dart business code.

There are four Taskrunners and a vm created when the Flutter engine starts. See how they work.

How TaskRunner works

During the startup of the Flutter engine, UI, GPU, and IO threads will be created. MessageLoop objects will be created for these threads in sequence and they will be in epoll_wait state after startup. The message mechanism of Flutter has many similarities with the Android native message mechanism, including messages (or tasks), message queues (or task queues) and Looper. One difference is that Android has a Handler class for sending messages and performing callback methods, while the TaskRunner class has a similar function in Flutter.

The figure above is a task processing process extracted from the source code. It is easier to understand the timing problems of some complex processes than the official flow chart. The reasons will be explained later. The Task queue processing mechanism of Flutter is similar to the message queue processing mechanism of Android, but there are two types of Flutter: Task and MicroTask. The events of engine and Dart vm and Future belong to tasks. ScheduleMicrotask () is the Microtask generated by the Dart layer.

Every time the Flutter engine consumes a task, it calls the FlushTasks() method, iterates the entire delay task queue delayed_tasks_, adds the expired task to the task queue, and then starts processing the task.

  • Step 1: Check the task. If the task queue is not empty, execute a task first.
  • Step 2: Check the microTask. If the microTask is not empty, execute it. Repeat Step 2 until the microTask queue is empty, then return to Step 1.

Process all microtasks first, and then process tasks. Since the scheduleMicrotask() method is called within a Task itself, executing the current Task means executing the Microtask immediately.

After understanding its working mechanism, let’s take a look at the specific work content of these four Task Runners.

  • Platform Task Runner: the main thread running on Android or iOS. Although blocking this thread does not affect the Flutter rendering pipeline, the Platform thread advises not to perform time-consuming operations; Otherwise, watchdog may be triggered to end the application. For example, Both Android and iOS use platform threads to deliver user input events. Once the platform thread is blocked, gesture events will be lost.
  • UI Task Runner: runs on the UI thread, such as 1. UI, which is used by the engine to execute all Dart code in the root ISOLATE, perform rendering and processing Vsync signals, and convert widgets to generate Layer Tree. In addition to rendering, it also handles Native Plugins messages, Timers, Microtasks, etc.
  • GPU Task Runner: runs on GPU threads, such as 1. GPU, which is used to convert Layer Tree into specific GPU instructions, execute GPU-related SKia calls, and convert the drawing modes of corresponding platforms, such as OpenGL, Vulkan, and Metal. The drawing of each frame needs the cooperation of UI Runner and GPU Runner, and any link delay may lead to frame drop.
  • IO Task Runner: runs on the I/O thread, such as 1. IO. The first three Task Runners are not allowed to perform time-consuming operations. The Runner is used to read the image from the disk, decompress it into a format that can be recognized by the GPU, and then upload the image to the GPU thread. To access GPU, IO Runner and GPU Runner Context are in the same ShareGroup. For example, UI. Image makes IO Runner load images asynchronously through asynchronous calls. This thread cannot perform other time-consuming operations, otherwise the performance of image loading may be affected.

5. The Dart VM works

The Dart VM and Root Isolate are created when the Flutter engine is started. DartVM also has its own Isolate, which is completely managed by the VMS themselves and cannot be directly accessed by the Flutter engine. Dart ui-related operations are performed by Root Isolate using Dart C++ or by sending message notifications to UIRunner, which can interact with Flutter engine modules.

The definition of Isolate is literally “isolation”, and each Isolate is logically isolated. The code in the Isolate is also executed sequentially, and since Dart does not have concurrent shared memory, there is no possibility of contention, so locks are not required and there is no risk of deadlock. Concurrency for the Dart program relies on multiple ISOLates.

Illustration:

  • The ISOLATE heap is the GC-managed memory store that runs all the objects assigned by the code in the ISOLATE;
  • Vm ISOLATE is a pseudo-ISOLATE that contains immutable objects such as NULL, true, and false.
  • The ISOLATE heap can refer to objects in the VM ISOLATE heap, but the VM ISOLATE cannot refer to the ISOLATE heap.
  • Isolate Does not refer to each other.
  • Each ISOLATE has a Mutator thread that executes the DART code and a helper thread that handles tasks within the VIRTUAL machine (such as GC, JIT, etc.). You can see that the ISOLATE has memory heaps and control threads. Virtual machines have many ISOLates, but they do not share memory with each other and cannot be accessed directly. They can only communicate with each other through the dart Port. The ISOLATE has a mutator control thread and other helper threads such as background JIT compilation threads, GC cleanup/concurrent markup threads.

6. Widget architecture Overview

Dart service is performed after the Flutter engine is started through the runApp(Widget App) method. What is the Widget?

A Widget is the cornerstone of all Flutter applications. A Widget can be a button, a font or color, a layout property, etc. The UI world of Flutter is an “everything Widget”. Common subcategories of widgets are StatelessWidget(stateless) and StatefulWidget(stateful).

  • StatelessWidget: No internal state is saved and the UI will not change after it is created.
  • StatefulWidget: There is internal save state. When the state changes, calling setState() triggers an update to the UI of the StatefulWidget. For custom subclasses that inherit from StatefulWidget, the createState() method must be overridden.

The three trees

Illustration:

  • Widgets describe the configuration required for an Element, create it, and determine whether it needs to be updated. The Flutter Framework uses a difference algorithm to determine whether the Element’s State changes by comparing changes in the Widget tree. If nothing has changed since rebuilding the Widget tree, Element does not trigger a redraw, so rebuilding the Widget tree does not necessarily trigger rebuilding the Element tree.
  • Element represents an instance of a Widget at a specific location in the Widget configuration tree, holds both the Widget and RenderObject, and is responsible for managing Widget configuration and RenderObject rendering. The Element state is managed by the Flutter Framework and the developer only needs to change the Widget.
  • RenderObject represents an object in the rendering tree that does the real rendering work, such as measuring size, position, and drawing.

As you can see, the developer updates the Element through the Widget configuration, the Framework updates the Element through the Widget configuration, and finally schedules the RenderObject Tree to arrange and draw the layout.

7. Rendering principle

Dart’s UI is implemented using widgets that are converted to RenderObjects. How does the interface render?

In the rendering process, the UI thread completes the layout and drawing operation and generates Layer Tree. The GPU thread performs synthesis and rasterization, and then hands it to GPU for processing. There are several key steps:

  • Animate: _transientCallbacks, performing animation callbacks;
  • Build: Build constructs are executed for dirty elements, not without dirty elements, corresponding to buildScope()
  • Layout: Calculates the size and position of the render object, corresponding to flushLayout(), which may be nested and then called with the build operation;
  • Compositing bits: Updates any render object with dirty Compositing bits, corresponding to flushCompositingBits();
  • Paint: Records the draw command to the Layer, corresponding to flushPaint();
  • Compositing: Sends Compositing bits to the GPU, corresponding to compositeFrame();

The GPU thread draws a frame of data to the GPU hardware through SKIA, and the GPU saves the frame information into the FrameBuffer. Then the video controller will transfer the frame data from the FrameBuffer to the display according to the VSync signal, so as to display the final picture.

8. Platform Channels

The Flutter framework provides support for UI controls. What about apps that rely on Native platforms other than UI, such as calling Camera functions? To this end, Flutter enables Dart code to interact with Native by providing Platform Channel functionality.

Platform Channel is used for message transfer between Flutter and Native. Messages and responses are executed asynchronously during the whole process without blocking the user interface. The Flutter engine framework has been bridged so that developers can write custom Android/iOS code in the Native layer and call it directly from Dart code. This is a form of the Flutter Plugin.

Read the source code of Flutter

Gityuan used to work at the bottom of the Android operating system. This year, HE was introduced to Flutter, a new cross-platform technology framework. If you take a closer look, Flutter is a small system that covers a wide range of technologies:

  • How does compilation technology convert DART code into AST(Abstract syntax tree), how does assembly translate into machine code, and what are the packaging products?
  • How does the Flutter engine start, connect to the Native system, identify the product and load it into memory?
  • How does TaskRunner distribute tasks once the engine starts, and how does it relate to the native system messaging mechanism?
  • How does Dart manage memory and how does IT relate to ISOLATE?
  • How do Widget controls written by developers render to the screen?
  • How does Flutter support services provided by mobile devices via plugin?

These questions have been briefly answered before. This depth of technology is not necessary to develop a business with Flutter only, but knowing how to do it will give you a lot of flexibility.

This article is synchronized to my official account “Cross-platform Technology Evolution and the Future of Flutter”. In addition, I have created a Flutter wechat communication group for everyone. If you want to join, you can add Gityuan wechat friends and comment on Flutter.

Start the article

  • In-depth understanding of Flutter engine startup
  • In-depth understanding of Dart VIRTUAL machine startup
  • In-depth understanding of Flutter application startup

Communication article

  • Gain insight into the Flutter messaging mechanism
  • In-depth understanding of the Platform Channel mechanism of Flutter
  • Gain insight into the Flutter asynchronous Future mechanism
  • Gain a deep understanding of the PROCESS of Flutter creation

Apply colours to a drawing paper

  • Flutter rendering mechanism – UI thread
  • Flutter rendering mechanism — GPU thread
  • In-depth understanding of the setState update mechanism
  • In-depth understanding of Flutter animation principles

In the future, the author will continue to study and sort out the internal mechanism of Flutter.

Fourth, concluding remarks

Now all say the Internet winter, in fact, as long as their technical ability is strong enough, we are not afraid! I have compiled a set of Android advanced learning video, a full set of Android interview secrets, and a full set of Android knowledge points PDF for Android developers. If there are friends who need to obtain information documents,You can get it for free by clicking on my GitHub!