This article introduces bytedance business practice on Flutter lightweight engine, introduces various problems encountered during this process and the solutions finally used.

Bytedance Terminal Technology — Hou Huayong

The background,

Flutter had very limited support for view-level development prior to version 2.0 in hybrid engineering development. There were two ways to display card views with Flutter. One was a single-engine model, by sharing the same container in different Native interfaces. It is FlutterViewController on iOS and FlutterActivity on Android. When removing and adding containers, a presentation relationship is maintained on the Flutter side and Native side to ensure the display and recovery of the page. Another option is to create a separate engine for new page presentation. The advantage of scheme 1 is that it will not produce extra memory consumption, but the disadvantage is that it will increase maintenance cost, and it is easy to cause black screen, white screen, no page response and other difficult and complicated diseases in different use scenarios. Scheme 2 has the advantages of simplicity, but has the disadvantages of large memory increments and slow startup. The official version 2.0 of the lightweight Engine opens up a new direction for Flutter use at the View level. The lightweight Engine combines the advantages of the above two solutions, spawn a new Engine when new pages are created. The new Engine shares threads and some C++ resources with the original Engine for the purpose of adding a new Engine but not much memory.

2. Implementation of the scheme

EngineGroup

Lightweight Engine introduced the concept of FlutterEngineGroup, which enables the Engine objects in the same Engine group to share resources to the maximum extent. It’s also easy to use. Create a EngineGroup and spawn new lightweight engines using makeEngine, which can then be used to render pages.

Memory footprint

One of the biggest improvements to the lightweight engine for developers is memory increments, and we did some research and testing on specific memory increments using the lightweight engine.

Official data: double side new Engine is only 180 KB. Flutter. Dev/docs/develo…

New one for AndroidFlutterCard, memory delta~~0.8 M; IOS added a new FlutterVC memory increment~~3.8 4.8 M

FlutterView number	2	3	4
Android memory value	68.8	69.6	70.5
IOS memory value	39.3	42.7	47.6

There are some differences from the official statistics. The official statistics only counted the memory increment when the engine was created, and did not count the extra memory consumption. The reason why the memory increment on iOS differed greatly from the official statistics is that the official statistics did not count the iOSurface created when the Flutter page was displayed. In a single engine, this portion of memory is not allocated for newly created pages and can be reclaimed when the page is not visible to reduce memory consumption (note that this portion of memory consumption depends on device resolution).

startup

Android: Speed increased ~2.63 times

IOS: Speed increased by ~9 times

Add cards in FlutterFragment form, statistics for Engine start created into onFlutterUiDisplayed

	Create in common mode	EngineGroup create
Android takes	140~150ms	50~60ms
IOS time-consuming	280~300ms	30~40ms

Since the lightweight engine is created from the “spawn” EngineGroup, most of the shared content has already been created, so the creation time of the lightweight engine is very short, and the startup speed is correspondingly improved. However, it should be noted that the creation of the first Engine in the EngineGroup is no different from the creation of the previous Engine. We call it the main Engine for now, and other lightweight engines are created from the main Engine. The creation of the main Engine can be preloaded in advance.

Third, business landing

Prior to 2.0, many businesses were interested in the Flutter lightweight engine. Since the launch of this feature, the Flutter Infra team has collaborated with other businesses to build and implement the Flutter lightweight engine in some businesses. Here, we would like to express our special thanks to the team of Dali Parents, Xiao He and Xing Fuli for their support of Flutter multi-engine solution. Typical business scenarios are shown below.

In the parent app of Great effort, we carried out two phases of landing practice. In the first phase, we transformed the pop-up window of photo prompt into a lightweight engine. After the launch of the first phase, we made further use of it and switched the Tab page to a lightweight engine.

The first frame time after using the lightweight engine is as follows:

With the lightweight engine, the first frame 50 quits time was reduced from 96ms to 78ms, which was expected in the context of single-engine preloading and the lightweight engine having to create a new engine every time the page was opened.

Flutter lightweight engine did encounter some problems in the process of business landing, but the landing effect and data feedback were finally recognized by the business side. Due to the lack of supporting facilities, there are still many things to be done when the real business needs to be implemented, such as plug-in registration management, engine destruction strategy, global configuration of Flutter entry, and variable synchronization management between engines. In the process of practice, the business gave us a lot of feedback and solved a lot of problems in the process of construction. Thanks again to the business side of our cooperation. We also addressed some missing feature points in an effort to create a more complete solution for the Flutter lightweight engine.

Fourth, function optimization

Set page entry parameters

In a single engine model, Flutter uses the main function as the engine entry. However, the entry function of the corresponding engine needs to be specified in the process of creating lightweight engines. On the side of Flutter, specific methods need to be specified using @pragma(‘ VM: Entry-point ‘). This method is executed after the engine starts. Compared with the main function of other languages, the main function of Flutter lacks input parameters. It is necessary for businesses using lightweight engines to pass some parameters to Flutter from the Native side for subsequent logical processing.

void main() {
  runApp(HomePage());
}

@pragma('vm:entry-point')
void home() {
  runApp(HomePage());
}
Copy the code

We changed the engine layer Settings to add entryPointsArgsJson so that the entry parameters we set can be retrieved from the Settings on the Flutter side. The use of the Flutter side is changed to the following.

@pragma('vm:entry-point')
void home(List<String> args) {
  runApp(HomePage(extras: args));
}
Copy the code

Adding parameters to the main function also has the advantage of eliminating some of the duplicate code, because the code executed in different lightweight engines is isolated from each other, and we usually have some initialization code or some global initialization Settings before the page is built. If we open up multiple EntryPoints, the duplicate code must be written again in each EntryFunction. This way, instead of defining multiple @pragma(‘ VM :entry-point’), we can define a single EntryPoint and determine the specific execution path by specifying the parameter value in the EntryFunction argument. On the Native side, you only need to know the unique definition of EntryPoint, avoiding the hard coding caused by specifying the entry function name.

Multi-engine data communication

The rationale for the lightweight engine is to take advantage of Dart’s IsolateGroup, which provides a significant increase in memory and startup speed compared to what was previously possible without the IsolateGroup. However, although multiple engines are in the same IsolateGroup and use the same Heap, the essence of the Isolate does not change, that is, data between isolates is not shared.

int count = 10;

@pragma('vm:entry-point')
void topMain(){
  count++;
  print("topMain:${count}");
}

@pragma('vm:entry-point')
void centerMain(){
  count++;
  print("centerMain:${count}");
}
Copy the code

In the example above, topMain and centerMain are two different lightweight engine portals, corresponding to two IsolateGroup isolategroups. There is a global variable count, which is +1 in both portals and printed with 11 in both. Data is not shared.

In a real-world scenario where we would have many lightweight engines sharing data, such as user login information or, for example, count above, we would prefer topMain’s changes to be synchronized to centerMain.

Since data cannot be directly shared between isolates, it is an intuitive idea to put specific data on the Native side of Flutter and then interact with the Native through channels. The official idea is to use Pigeon to generate code to provide multiple simultaneous access capabilities, but the official solution has not yet been developed due to various reasons.

We also carried out some exploration on realizing data communication through Channel, and found some problems in this process:

Multi-terminal (Android, iOS, etc.) need to write corresponding Native implementation, high development cost, personnel structure requirements;
MethodChannel serializes the data into a string, and the receiver deserializes it.

In order to solve the above problems, we designed the following scheme:

Dart isolates cannot share data with each other, but they can communicate with each other through Port. Data synchronization between multiple ISOLates can be realized through this mechanism.

Convergence of shared data into a ServiceIsolate so that shared data is still in the Dart layer and there is no need to worry about multiple aspects.
When other isolates update data, ServiceIsolate sends an update message to ServiceIsolate. In this case, ServiceIsolate broadcasts the update message to other ISOLates.
When the Isolate needs to obtain the latest data, it sends a request message to ServiceIsolate. After receiving the message, ServiceIsolate sends the data back.
FFI is used to bind ports between isolates. Dart objects can be directly transferred between different isolates without serialization, with better performance and easy use.

Data update broadcast

Listen for each individual piece of data that needs to be shared, and take action when changes occur.

Call when it is necessary to update the data channel. PostUpdateMessage (content), and other places you just need to listen the news.

Call channel.postNotification(content) directly when there is a need to broadcast, so that broadcast messages can be sent between multiple engines without affecting the built-in synchronization data. The content can be customized.

Monitoring data updates

When you use BroadcastChannel channel = BroadcastChannel(channelName), you can add a channel to the BroadcastChannel and receive content updates and message notifications

BroadcastChannel channel = BroadcastChannel('countService');
channel.onDataUpdated = (dynamic content) {
  setState(() {
    int counter = content as int;
    _counter = counter;
  });
  print('this will be invoked after data has been changed');
};
Copy the code

The listening notification message code is as follows

BroadcastChannel channel = BroadcastChannel('countService');
channel.onReceiveNoti = (dynamic content) {
  print('this will be invoked after receive notification');
};
Copy the code

Get the latest data

When users need to obtain data during data initialization, they can directly request to share data.

channel.request().then((value){
  setState(() {
    int counter = value as int;
    _counter = counter;
  });
  print('this will be invoked after data has received! ');
});
Copy the code

ImageCache Shared

Cache memory problem

Another issue to be aware of when using a lightweight engine is the cache in the engine, because creating additional engines can result in a multiple increase in the cache, and if this is not addressed, the memory benefits of a lightweight engine can be wiped out. While images occupy an absolute large proportion of the Flutter cache, image caching using a lightweight engine can cause the following problems:

Image memory is not shared, the same image in multiple engines need to be repeatedly decoded, repeated memory consumption
By default, each Engine has 100 MB of ImageCache. If the ImageCache is not shared, different engines may have different utilization rates. For example, some engines have too few images, and some engines have too many images, resulting in insufficient Cache utilization.

Picture status combing

Here’s a quick review of how Flutter loads images:

Use the Key of the Image to get the cache content, if the match is used directly, otherwise create a new ImageStreamCompleter;
ImageStreamCompleter creates the Codec inside, and the Codec triggers the decoding logic;
The internal MultiFrameCodec and SingleFrameCodec are decoded to obtain CanvasImage, which is bound to the Image on the Flutter side.
Image was obtained on the Flutter side for display

Program Core objectives

The core point to solve the above problems is that the C++ layer completes the reuse of CanvasImage and Codec to achieve the following state

Add a proxy mechanism to CavasImage and Codec. The first Engine that triggers image loading will actually trigger the creation and caching of CanvasImage and Codec. When the Engine triggers image loading, The agent is created based on the classes of CavasImage and Codec, which is equivalent to an empty shell and only plays the role of forwarding. All operations are forwarded to the real CavasImage and Codec for execution.

Concrete implementation scheme

In C++ side, Map CanvasImage and Codec used for cache creation were added. Reference to the cache was added when the proxy class was created, and reference to the cache was removed when the proxy class was destroyed.
Add the list record of ImageCacheKey, which is used to complete the LRU logic. When the Dart side accesses the image, the list will be notified to this list, and the list will migrate the corresponding Key. When the space is insufficient, the Engine Dart side will be notified to release the corresponding Key image. In order to avoid the new count logic, each Engine will not notify the list change when releasing, the list will first request each Engine is using the picture information, in order to clear in their own record but no Engine is using the picture, after the completion of clearing will carry out the relevant calculation and change;
New ImageCacheKey interface, implemented by each Object that is currently used as a Key, returns a String based on some eigenvalues in the Object, String is used as the ImageCacheKey of C++ side to judge the image equality.

In the process of solving the image caching problem, we also found some other aspects. For example, two engines display a Gif at the same time. After the main Engine is destroyed, the created Engine crashes. IOManager was destroyed after the main engine was destroyed, and an exception was raised when the second engine was used again. This issue was resolved by sharing IOManager directly with multiple engines. Github.com/flutter/eng…). In addition to the image cache, there are other cache elements that we are trying to reduce.

Page not visible release iOSSurface

As mentioned in the previous article, the memory increment of creating an additional card engine is ~180K. In the process of actual measurement, the memory increment of each additional engine created by iOS is 4-5m. On an Android machine, the process of creating Engine is essentially the same, so why the difference?

In the process of using Instrument to get memory growth details, the new lightweight engine interface is constantly pushed from the official Demo. It is clear that the highest percentage of memory usage is in the buffer generated during rendering. The size of the required chunk depends on the screen resolution and the ViewportMetrics used to create the FlutterView.

sk_sp<SkSurface> SkSurface::MakeFromCAMetalLayer(GrRecordingContext* rContext,
                         GrMTLHandle layer,
                         GrSurfaceOrigin origin,
                         int sampleCnt,
                         SkColorType colorType,
                         sk_sp<SkColorSpace> colorSpace,
                         const SkSurfaceProps* surfaceProps,
                         GrMTLHandle* drawable)
Copy the code

The idea here is that since the previous page is not displayed, it doesn’t matter if the memory is freed, and in theory it only needs to be restored when the page is displayed again. All we need to do on our side is find the owner of iOS_Surface and make sure that the iOS_Surface is released when the FlutterViewController disappears.

Ios_surface is held in two places, Rasterizer and PlatformView. In addition, the most direct reference relationship is FlutterView layer. Because iOS_Surface itself depends on Layer. In this relationship, the Shell creation and destruction cost is very high, and the holding relationship is very complex, which is basically equivalent to re-creating and destroying the Flutter context. Here, the Shell is not directly destroyed & created, and PlatformView and Rasterizer are handled separately.

Platfomview holds iOS_Surface, Since FlutterViewController in viewDidDisappear triggers surfaceUpdated to execute PlatformView’s NotifyDestroyed method, we can change it here. Ensure that references to iOS_Surface are removed.

After the above logic is completed, the memory usage after multiple pushes using Instrument is shown in the figure above. During the next Push, the memory usage of the previous page is greatly reduced. After using this scheme, the footprint of Surface in the current display page is removed. The memory increment for each new page is reduced from 5M to 500K. Since the previous page is destroyed for Sureface, creating a new Sureface for a new page will cause a temporary spike in memory, and it may be better to reuse the previous one without destroying &creating it.

FlutterView content is adaptive

The lightweight engine application scheme makes Flutter more convenient to be applied to list Item, Banner and other scenes. However, due to the limitation of the parent layout in the process of using FlutterView, Flutter content can only be filled with the parent layout, instead of adaptive layout according to the specific content. This makes the scheme problematic in some general scenarios.

Due to the diversity of sizes of mobile devices, the popover needs to be self-adaptive when it is displayed. Before any changes are made, the size of the popover can only be displayed in a fixed size, which also leads to the situation that the display of picture elements is not as expected.

The solution

When obtaining the whole Flutter layout, we need to modify the notification process of FlutterView Size change. First, a large enough Size is given to Dart side to ensure that Dart can measure the correct result during the Flutter layout.
Then monitor the Dart side layout to get the width and height notification to Native.

The approach used here is to wrap the RootWrapContentWidget for the outermost layer of the Widget, listening for the Layout process through a custom RenderObject, Also add a parent Widget to your own IntrinsicWidth or IntrinsicHeight so that the overall layout of the page is wrapped.

class RootWrapContentWidget extends StatelessWidget {
  /// constructor
  const RootWrapContentWidget(
      {Key? key,
      required this.child,
      this.wrapWidth = false.this.wrapHeight = false})
      : assert(child ! =null),
        assert(wrapWidth || wrapHeight),
        super(key: key);

  final Widget child;
  final bool wrapWidth;
  final bool wrapHeight;

  @override
  Widget build(BuildContext context) {
    Widget result = _RootSizeChangeListener(
      child: child,
    );
    if (wrapWidth) {
      result = IntrinsicWidth(child: result);
    }
    if (wrapHeight) {
      result = IntrinsicHeight(child: result);
    }
    returnresult; }}Copy the code

Picture size problem

If there are images on the page, the Dart side needs several layouts to get the exact width and height. You cannot change the size of the parent Layout until the final width and height is obtained. Otherwise, the size changes of the parent Layout will synchronize to the Dart side and affect the Dart side Layout. Either monitor the loading process of all pictures and use the measured value of Layout after all pictures are loaded as the Size of FlutterView, or try to get the accurate width and height at the first Layout. The code of monitoring the loading process of all pictures has a large change, so we finally decided to study on scheme 2.

Size _sizeForConstraints(BoxConstraints constraints) {
  // Folds the given |width| and |height| into |constraints| so they can all
  // be treated uniformly.
  constraints = BoxConstraints.tightFor(
    width: _width,
    height: _height,
  ).enforce(constraints);

  if (_image == null)
    return constraints.smallest;

  returnconstraints.constrainSizeAndAttemptToPreserveAspectRatio(Size( _image! .width.toDouble() / _scale, _image! .height.toDouble() / _scale, )); }Copy the code

Before retrieving the decoded _image information, the measurement logic is that if the ImageWidget has a set width and height, it uses the set width and height, and if it has no set width and height, it uses the minimum value. It seems that it is ok to ask the business side to specify the width and height of the image in the scenario of adaptive layout, but in real code writing, this is difficult to do, and in some layouts the width and height of the image is not available.

Finally, we asked the business side to write the aspect ratio, combined with the image aspect ratio and the constraints given by the parent layout in BoxConstraints, to figure out how big the ImageWidget should be without setting the width or decoding the data.

Fifth, summarize the outlook

In addition to the optimizations described above, we also address other lightweight engine-related issues such as ThreadMerge in PlatformView usage (PR: github.com/flutter/eng…). , deadlock problems in ThreadMerge (PR: github.com/flutter/eng…) And so on.

There has long been a need for the view-level use of Flutter. In the current era of existing apps, it is important to make Flutter better serve existing businesses. The use of view-level in cross-platform solutions is now a fundamental feature. This feature in Flutter is belated by official efforts, so we should make it faster and more accessible to solve business problems and expand business boundaries. From the current implementation effect, there are still areas to be improved. The authorities and the community are also continuing to optimize, and Byte will continue to improve the multi-engine solution based on actual business scenarios, and contribute relevant results to the community.

Refer to the link

The related documents

Flutter. Dev/docs/develo…

Mp.weixin.qq.com/s/6aW9vbith…

PR links

Github.com/flutter/eng…

# About bytedance Terminal Technology team

Bytedance Client Infrastructure is a global r&d team of big front-end Infrastructure technology (with r&d teams in Beijing, Shanghai, Hangzhou, Shenzhen, Guangzhou, Singapore and Mountain View), responsible for the construction of the whole big front-end Infrastructure of Bytedance. Improve the performance, stability and engineering efficiency of the company’s entire product line; The supported products include but are not limited to Douyin, Toutiao, Watermelon Video, Feishu, Guagualong, etc. We have in-depth research on mobile terminals, Web, Desktop and other terminals.

Now! Client/front-end/server/side intelligent algorithm/test development for global recruitment! Let’s change the world with technology. If you’re interested, please contact [email protected]. Email Subject Resume – Name – Job Objective – Desired city – Phone number.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Practice and optimization of the Flutter lightweight engine

The background,

2. Implementation of the scheme

EngineGroup

Memory footprint

startup

Third, business landing

Fourth, function optimization

Set page entry parameters

Multi-engine data communication

Data update broadcast

Monitoring data updates

Get the latest data

ImageCache Shared

Cache memory problem

Picture status combing

Program Core objectives

Concrete implementation scheme

Page not visible release iOSSurface

FlutterView content is adaptive

The solution

Picture size problem

Fifth, summarize the outlook

Refer to the link

Practice and optimization of the Flutter lightweight engine

The background,

2. Implementation of the scheme

EngineGroup

Memory footprint

startup

Third, business landing

Fourth, function optimization

Set page entry parameters

Multi-engine data communication

Data update broadcast

Monitoring data updates

Get the latest data

ImageCache Shared

Cache memory problem

Picture status combing

Program Core objectives

Concrete implementation scheme

Page not visible release iOSSurface

FlutterView content is adaptive

The solution

Picture size problem

Fifth, summarize the outlook

Refer to the link

Related Posts

JavaScript keyword instanceof, new implementation principle

React performance optimization methods and considerations

Build a community one day! Cloud development practice