In the case of increasingly complex terminal business requirements and frequent version iterations, we urgently need excellent multi-terminal unified cross-platform development schemes to improve r&d efficiency. At present, there are terminal technology solutions like RN and Weex that bridge to Native through JavaScript. However, JavaScriptCore has its own performance bottlenecks and bridge layer consumption. The complexity of the idle fish product interface and the high performance baseline we developed made it impossible for us to choose these options.

Currently we are actively trying and exploring Flutter’s business practices in pursuit of a higher performance cross-platform terminal solution. As a cross-platform technology, why does Flutter have performance features that other solutions do not?

  • Flutter compiles Dart directly to local machine code in Rlease mode, avoiding the performance cost of code interpretation.

  • Dart itself is optimized at the memory level for high frequency loop refreshes, such as 60 frames per second on the screen, making on-screen drawing a natural fit for Dart runtime.

  • Flutter implements its own graphics rendering to avoid Native bridging.

Flutter architecture:

In order to better apply and practice, we need to go deep into the engine to understand its implementation principle and structure. Thread has always been a troublesome topic in development, and we have experienced many difficulties in practice. This article discusses the thread mode of the Flutter engine.

For those who are new to Flutter, more information about Flutter can be found on its website, Flutter IO

Flutter thread model

The Flutter Engine does not create its own administrative threads. The creation and management of the Flutter Engine threads is administered by Embedder. Embeder refers to middle-tier code that ports the engine to the platform.

The Flutter Engine requires Embeder to provide four Task Runners. Although the Flutter Engine doesn’t care which thread a Runner runs on, it does need the thread configuration to remain stable throughout its lifetime. That is, a Runner should always run on the same thread. The four main Task runners include:

Platform Task Runner

The main Task Runner of the Flutter Engine and the thread running Platform Task Runner can be understood as the main thread. Similar to the Android Main Thread or iOS Main Thread. However, it is important to note that there is a difference between the Platform Task Runner and the main thread like iOS.

To the Flutter Engine, the thread of Platform Runner is not substantially different from any other thread, but we give it an artificial meaning to make it easier to understand and distinguish. We can actually start multiple instances of Engine at the same time, with each Engine corresponding to a Platform Runner, and each Runner running in its own thread. This is also how the Content Handler in Fuchsia (the operating system Google is developing) works. Generally, when a Flutter application is started, an Engine instance is created. When the Engine is created, a thread is created for Platform Runner to use.

All interactions with the Flutter Engine (interface calls) must occur on the Platform Thread. Attempting to invoke the Flutter Engine from another Thread will cause unexpected exceptions. This is similar to how iOS UI operations must be performed on the main thread. Note that there are many modules in the Flutter Engine that are not thread-safe. Once the engine is up and running properly, all engine API calls are made in the Platform Thread.

Platform Runner’s Thread doesn’t just handle interactions with the Engine; it also handles messages from the Platform. This is convenient because almost all engine calls are safe only on the Platform Thread, and Native Plugins do not need to do additional Thread operations to ensure that the operation can be performed on the Platform Thread. If the Plugin itself starts an additional Thread, it is responsible for sending the result back to the Platform Thread so that Dart can safely process it. The rule is simple: All calls to the Flutter Engine interface must be made on the Platform Thread.

Note that blocking the Platform Thread does not directly cause the Flutter application to stall (unlike the iOS android main Thread). However, platforms have enforcement restrictions on Platform threads. Therefore, it is recommended that complex computational logic operations not be placed on the Platform Thread but on other threads (not including the four threads we are discussing now). Other threads forward the results back to the Platform Thread. If a Platform Thread is stuck for a long time, it may be killed by the Watchdot system.

UI Task Runner Thread (Dart Runner)

The UI Task Runner is used by the Flutter Engine to execute the Dart root ISOLATE code (the ISOLATE will be discussed later, but first understood simply as threads inside the Dart VM). The Root isolate is special because it binds many of the functions that Flutter requires. Root ISOLATE Main code of running applications. The engine was started with the necessary bindings to enable it to schedule the submission of render frames. For each frame, the Engine does the following: – Root ISOLATE notifies the Flutter Engine that a frame needs to be rendered. – The Flutter Engine notification platform needs to be notified at the next vsync. – The platform waits for the next vsync – Layouts the created objects and Widgets and generates a Layer Tree, which is immediately submitted to the Flutter Engine. No rasterization is done in the current phase; this step simply generates a description of what needs to be drawn. – Creates or updates a Tree that contains semantic information for displaying Widgets on the screen. This thing is mainly used to configure and render platform-specific auxiliary Accessibility elements.

In addition to rendering logic, Root Isolate also handles message responses from Native Plugins, Timers, Microtasks, and asynchronous IO. We see that the Root Isolate is responsible for creating and managing the Layer Tree that ultimately decides what to draw on the screen. Therefore, the overload of this thread will directly cause frame lag. If there are some heavy computations that cannot be avoided, it is recommended that the computations be performed on an independent Isolate, for example, using the compute keyword or on a non-root Isolate to avoid UI lag. However, it is important to note that the non-root Isolate lacks some of the functional bindings required by the Flutter Engine. You cannot interact directly with the Flutter Engine from this Isolate. So use isolated Isolate only when you need a lot of computing.

GPU Task Runner

GPU Task Runner is used to perform calls related to the device GPU. The Layer Tree information created by UI Task Runner is platform-independent, that is to say, the Layer Tree provides the information needed for drawing. The specific method of drawing depends on the specific platform and method, which can be OpenGL, Vulkan, software drawing or other Skia configuration drawing implementation. Modules in GPU Task Runner are responsible for translating the information provided by Layer Tree into actual GPU instructions. GPU Task Runner is also responsible for configuration and management of GPU resources required for each frame drawing, including platform Framebuffer creation, Surface lifecycle management, and ensuring that Texture and Buffers are available during drawing.

Based on the processing time of Layer Tree and the time of GPU frame display to the screen, GPU Task Runner may delay the scheduling of the next frame in UI Task Runner. Generally speaking UI Runner and GPU Runner run on different threads. It is possible that UI Runner is delivering the last frame to the GPU while the GPU Runner is already preparing the next frame. This delayed scheduling mechanism ensures that UI Runner does not assign too many tasks to GPU Runner.

As mentioned earlier, GPU Runner can cause frame scheduling delay of UI Runner, and overload of GPU Runner can cause lag of Flutter application. In general, users don’t have the opportunity to submit tasks directly to GPU Runner because neither the platform nor the Dart code can run into GPU Runner. But Embeder can still submit tasks to GPU Runner. Therefore, it is recommended to create a dedicated GPU Runner thread for each Engine instance.

IO Task Runner

The runners discussed earlier have strong restrictions on the types of tasks they can perform. Overload of Platform Runner may cause system WatchDog to forcibly kill, while overload of UI and GPU Runner may cause lag of Flutter applications. However, GPU threads have some necessary operations that are time-consuming, such as IO, and these operations are exactly what IO Runner needs to handle.

IO Runner’s main function is to read compressed image formats from image storage (such as disk) and process the image data in preparation for GPU Runner’s rendering. To prepare Texture, IO Runner first reads compressed image binary data (such as PNG, JPEG), unzips it into a format that the GPU can process, and uplots the data to the GPU. These complex operations can cause the Flutter application UI to stall if run on the GPU thread. However, only GPU Runner can access GPU, so the IO Runner module configates a special Context when the engine starts, which is in the same ShareGroup as the Context used by GPU Runner. In fact, image data can be read and decompressed in a thread pool, but the Context can only be accessed safely in a particular thread. That’s why you need a dedicated Runner to handle IO tasks. The Flutter Framework tells IO Runner to perform the Image asynchronous operation just mentioned when an async call occurs to acquire resources such as UI. Image. This allows GPU Runner to use image data prepared by IO Runner without having to do anything extra.

User operations, neither Dart Code nor Native Plugins, have direct access to IO Runner. Although Embeder can schedule some moderately complex tasks to IO Runner, this does not directly cause Flutter applications to lag, but it can lead to delays in loading images and other resources that indirectly affect performance. It is recommended to create a dedicated thread for IO Runner.

Each platform currently implements Runner threads by default

As mentioned earlier, Engine Runner threads can be configured as required, and each platform currently has its own implementation strategy.

IOS and Android

Each Engine instance on the Mobile platform starts with a new thread for the UI, GPU, and IO Runner. All Engine instances share the same Platform Runner and thread.

Fuchsia

Each Engine instance creates its own new thread for UI, GPU, IO, and Platform Runner.

Feasible solution of customizing configuration threads

We noticed that Platform Runner and Thread were shared on the Mobile Platform. The engine source code is as follows:

Shell::Shell(fxl::CommandLine command_line) : command_line_(std::move(command_line)) { FXL_DCHECK(! g_shell); gpu_thread_.reset(new fml::Thread("gpu_thread")); ui_thread_.reset(new fml::Thread("ui_thread")); io_thread_.reset(new fml::Thread("io_thread")); // Since we are not using fml::Thread, we need to initialize the message loop // manually. fml::MessageLoop::EnsureInitializedForCurrentThread(); blink::Threads threads(fml::MessageLoop::GetCurrent().GetTaskRunner(), gpu_thread_->GetTaskRunner(), ui_thread_->GetTaskRunner(), io_thread_->GetTaskRunner()); blink::Threads::Set(threads); blink::Threads::Gpu()->PostTask([this]() { InitGpuThread(); }); blink::Threads::UI()->PostTask([this]() { InitUIThread(); }); blink::SetRegisterNativeServiceProtocolExtensionHook( PlatformViewServiceProtocol::RegisterHook); }Copy the code

Here we can change it to have each instance of the engine initialize its own thread:


    gpu_thread_.reset(new fml::Thread("gpu_thread"));
    ui_thread_.reset(new fml::Thread("ui_thread"));
    io_thread_.reset(new fml::Thread("io_thread"));

    platform_thread_.reset(new fml::Thread("platform_thread"));

    blink::Threads threads(platform_thread_->GetTaskRunner(),
                            gpu_thread_->GetTaskRunner(),
                            ui_thread_->GetTaskRunner(),
                            io_thread_->GetTaskRunner());Copy the code

In theory you can configure any thread for use, but it’s best to follow best practices.

Code reading

Download the Flutter Engine for iOS Android.

flutter/common/threads.cc
flutter/shell/common/shell.ccCopy the code

The Dart isolate mechanism

An isolated Dart execution context. This is the document’s definition of ISOLATE.

Isolate definition

Isolate is Dart’s implementation of actor concurrency. The Dart program in action consists of one or more actors, which are known as the ISOLATE within the Dart concept. Isolate is a running entity with its own memory and single thread control. The isolate itself means “isolated” because internals between isolates are logically isolated. The code in the ISOLATE is executed sequentially, and concurrency in any Dart program is the result of running multiple ISOLates. Because Dart does not have shared memory concurrency, there is no possibility of contention so locks are not required and deadlocks are not a concern.

Communication between the ISOLATE

Since there is no shared memory between the ISOLates, the only way they can communicate with each other is through Port, and messaging in Dart is always asynchronous.

The difference between ISOLATE and normal threads

We can see that the isolate is similar to Thread, but in fact the two are essentially different. The main difference is that the ISOLATE does not have shared memory between threads within the operating system.

Overview of ISOLATE Implementation

You can read the ISOLate. cc file in the Dart source code to see the implementation of THE ISOLATE. We can see that there are several main steps in the creation of the ISOLATE:

  • Initialize the ISOLATE data structure

  • Initialize Heap memory

  • Enter the newly created ISOLATE and run the ISOLATE on a one-to-one thread

  • Configure the Port

  • Configuring Message Handler

  • Configure the Debugger, if necessary

  • Register isolate with the Global Monitor

Let’s look at the main code that the ISOLATE started running

Thread* Isolate::ScheduleThread(bool is_mutator, bool bypass_safepoint) { // Schedule the thread into the isolate by associating // a 'Thread' structure with it (this is  done while we are holding // the thread registry lock). Thread* thread = NULL; OSThread* os_thread = OSThread::Current(); if (os_thread ! = NULL) { MonitorLocker ml(threads_lock(), false); // Check to make sure we don't already have a mutator thread. if (is_mutator && scheduled_mutator_thread_ ! = NULL) { return NULL; } while (! bypass_safepoint && safepoint_handler()->SafepointInProgress()) { ml.Wait(); } // Now get a free Thread structure. thread = thread_registry()->GetFreeThreadLocked(this, is_mutator); ASSERT(thread ! = NULL); // Set up other values and set the TLS value. thread->isolate_ = this; ASSERT(heap() ! = NULL); thread->heap_ = heap(); thread->set_os_thread(os_thread); ASSERT(thread->execution_state() == Thread::kThreadInNative); thread->set_execution_state(Thread::kThreadInVM); thread->set_safepoint_state(0); thread->set_vm_tag(VMTag::kVMTagId); ASSERT(thread->no_safepoint_scope_depth() == 0); os_thread->set_thread(thread); if (is_mutator) { scheduled_mutator_thread_ = thread; if (this ! = Dart::vm_isolate()) { scheduled_mutator_thread_->set_top(heap()->new_space()->top()); scheduled_mutator_thread_->set_end(heap()->new_space()->end()); } } Thread::SetCurrent(thread); os_thread->EnableThreadInterrupts(); thread->ResetHighWatermark(); } return thread; }Copy the code

As you can see, Dart abstracts both ISOLATE and Threads, and actually uses osThreads provided by the operating system underneath.

Flutter Engine Runners and Dart Isolate

Some friends may ask that since Flutter Engine has its own Runner, why Dart’s Isolate? What is the relationship between them?

Runner is an abstract concept. We can submit tasks to Runner, and the tasks will be executed by Runner in its thread, which is similar to the execution queue of iOS GCD. In fact, there is a loop in the implementation of iOS Runner. This loop is CFRunloop, and the specific implementation of Runner on iOS platform is CFRunloop. The submitted task is placed in the CFRunloop for execution.

The Dart Isolate is managed by the Dart VM and cannot be directly accessed by the Flutter Engine. The Root Isolate uses Dart’s C++ call capability to submit UI rendering tasks to UI Runner so that they can interact with Flutter Engine modules. The tasks associated with Flutter UI are also submitted to UI Runner to notify the Isolate of events. UI Runner also handles tasks from the App Native Plugin.

Therefore, Dart ISOLATE and Flutter Runner are independent of each other. They cooperate with each other through task scheduling mechanism.

Tread pit blood and tears history

Understanding the principles of the Flutter Engine and the asynchronous implementation of the Dart VIRTUAL machine allows us to develop flexibly and efficiently without pit stops. In the application process of the project, we stepped on many pits and kept learning in the process of pit mining and pit filling. I’ll briefly mention one specific case where we needed to register the Native image data to Engine to generate the Texture rendering, and we needed to remove the resource after it was used, and the seemingly clear logic caused a wild pointer problem. Later, troubleshooting was done on a child thread while removal was done on the Platform thread, and the problem was solved once the thread structure was understood.

conclusion

This paper mainly discusses the thread configuration management at the Flutter engine level and the mechanism of Dart isolate. With a deeper understanding of the Flutter thread mechanism, we are much more comfortable in the development process. In understanding the design of Flutter, we were inspired to design thread structures similar to those in applications.

Extend the discussion

At present, the team integrates Flutter into the existing App and starts Flutter in a singleton way, which avoids the time-consuming problem caused by the cold start of Flutter during the page switching between Native and Flutter. However, the logic of single-engine startup becomes complicated when multiple Flutter pages are supported. Ideally, we want each page to be a relatively independent component, which may require some engine level tweaking. One possible scenario is that one component corresponds to one engine instance, and multi-instance engines face some challenges with threading patterns. This includes issues of thread reuse, interthread communication, thread performance tuning, and inter-engine collaboration.

We will further explore and try in this aspect, please continue to pay attention to the official account, and the Xianyu team is looking forward to your joining!

Scan code to pay attention to [Xianyu Technology] public number

Forward-looking technology is at hand

The resources

  • Flutter development document (https://github.com/flutter/flutter)

  • Flutter engine document (https://github.com/flutter/engine)

  • (https://flutter.io/) Flutter IO

  • (https://www.dartlang.org/) Dart language development documentation

  • (https://book.douban.com/subject/27074797/) “Dart programming language”