This article mainly fixes the memory leak problem occurred when PlatformView memory leak was used in iOS version flutter1.12.x, and analyzes the principle of Platform from source code based on this, hoping readers can gain the following contents:

  1. Learn to solve other problems with Flutter Engine by yourself
  2. Understand the implementation of Flutter PlatformView

background

The official version of Flutter has now undergone a major evolution of 1.12. Since 1.9, this version has resolved 4,571 errors and incorporated 1,905 pr files. In practice, 1.12 has been greatly optimized for dart object memory release. Testing with DevTool repeatedly in and out of the same page found that 1.12 resolved the large number of Dart objects resident under 1.9. However, when the page is used in the PlatformView scenario, the page increases by up to 10M per entry and exit. Instrument analysis finds that the number of IOSurface will only increase, not decrease:

IOSurface is GL’s rendering canvas. It can be concluded that this is a leak in the underlying rendering of Flutter.

Debugging Flutter Engine

Before debugging the source code, you need to compile a Flutter Engine (Flutter. Framework) to replace the official library

Build the Flutter Engine

We need to stand on the shoulders of giants and make the best use of existing resources, so how to compile the Flutter Engine is no longer a burden. You can follow this guide to Build the Flutter Engine (juejin.cn/post/684490…).

1.1 Engine for building unoPT version

To make the code behave and execute sequentially during debugging, we need to build an unoptimized version of the Engine

Ninja -C out/ iOS_DEBUg_unopt // Debug for mobile devices Ninja -C out/ iOS_DEBUg_sim_unopt for emulatorsCopy the code

1.2 Replacing the official library

Copy will be compiled Flutter. The framework to your Flutter directory, specific path for ${you Flutter path} / bin/cache/artifacts/engine/ios, so when your application package, The app that uses Flutter. Framework is the library we just packed.

1.3 Breakpoint in project

Next we drag the products.xcodeProj generated during Engine compilation into our App project and break the entry to FlutterViewController.mm to start the project.

PlatformView implementation principle

According to the official documentation, there are two main steps to use PlatformView.

  1. Native registered an implementation with the Flutter FlutterPlatformViewFactory instance and of the agreement with an ID binding, ViewFactory method is mainly used in the agreement of the incoming a UIView to Flutter layers;
  2. Second, dart layer uses UiKitView to set its viewType property to the NATIVE registered ID value.

We know that the implementation of Flutter is a GL canvas (FlutterView), but how does the PlatformView we passed into the native display together with The FlutterView? To help you understand the process, we will start with the FlutterViewController and give you an overview of the core classes of Flutter.

2.1 FlutterEngine

Flutter’s application entry is in the FlutterViewController, but it is a wrapper around UIViewController, whose member variable FlutterEngine is the administrator of the DART runtime environment. In fact, not only can FlutterViewController be initialized independently of the FlutterViewController, but FlutterViewController can be switched at will.

The most important function of FlutterViewController is to provide a canvas (self.view) for FlutterEngine to draw, which is also the principle of the Idle Fish FlutterBoost library.

// FlutterViewController.mm // The first way to pass FlutterViewController into engine - (instancetype)initWithEngine:(FlutterEngine*)engine nibName:(nullable NSString*)nibName bundle:(nullable NSBundle*)nibBundle { NSAssert(engine ! = nil, @"Engine is required");
  self = [super initWithNibName:nibName bundle:nibBundle];
  if(self) { _engine.reset([engine retain]); // Reset engine logic, such as clean canvas... [enginesetViewController:self]; // Engine rebind FlutterViewController}returnself; } // the second way to initialize engine synchronously at FlutterViewController initialization - (instancetype)initWithProject:(nullable FlutterDartProject*)project nibName:(nullable NSString*)nibName bundle:(nullable NSBundle*)nibBundle { self = [super initWithNibName:nibName bundle:nibBundle];if(self) {// new an engine instance _engine.reset([[FlutterEngine alloc] initWithName:@"io.flutter"project:project allowHeadlessExecution:NO]); // create engine's scheduler shell instance [_engine.get() createShell:nil libraryURI:nil]; . }return self;
}
Copy the code

FlutterEngine has two core components, it is a Shell, the second is FlutterPlatformViewsController. Shell is active in the FlutterViewController call engine createShell, FlutterPlatformViewsController is is created when the engine is initialized.

// FlutterEngine.mm - (instancetype)initWithName:(NSString*)labelPrefix project:(FlutterDartProject*)project allowHeadlessExecution:(BOOL)allowHeadlessExecution { ... / / create FlutterPlatformViewsController _platformViewsController. Reset (new flutter: : FlutterPlatformViewsController ()); . }Copy the code

2.2 the Shell

Shell instances are also members of FlutterEngine. If FlutterEngine is the manager of Flutter operation environment, its member Shell is the brain of FlutterEngine, responsible for coordinating task scheduling. All four threads of Flutter are managed by Shell.

We all know that a Flutter has four threads inside it: Platform threads that communicate with native events such as EventChannel and Messagechannel GPU threads that draw UI elements on native canvas DART threads that execute DART code logic IO threads Because DART execution is single-threaded, wait time operations like I/O need to be placed in another thread

- (BOOL)createShell:(NSString*)entrypoint libraryURI:(NSString*)libraryURI {...if(flutter: : IsIosEmbeddedViewsPreviewEnabled ()) {/ / when flutter using PlatformView flutter: : TaskRunners task_runners(threadLabel.UTF8String, // label fml::MessageLoop::GetCurrent().GetTaskRunner(), // platform fml::MessageLoop::GetCurrent().GetTaskRunner(), // gpu _threadHost.ui_thread->GetTaskRunner(), // ui _threadHost.io_thread->GetTaskRunner() // io ); _shell = flutter::Shell::Create(std::move(task_runners), // task runners std::move(settings), // settings on_create_platform_view, // platform view creation on_create_rasterizer // rasterzier creation ); }else{ flutter::TaskRunners task_runners(threadLabel.UTF8String, // label fml::MessageLoop::GetCurrent().GetTaskRunner(), // platform _threadHost.gpu_thread->GetTaskRunner(), // gpu _threadHost.ui_thread->GetTaskRunner(), // ui _threadHost.io_thread->GetTaskRunner() // io ); _shell = flutter::Shell::Create(std::move(task_runners), // task runners std::move(settings), // settings on_create_platform_view, // platform view creation on_create_rasterizer // rasterzier creation ); }... }Copy the code

From the above code we can see that when the application identifies itself as using PlatformView, the platform thread and GPU thread share the same thread. Since The FlutterViewController is initialized in the main thread, it also shares the main thread of iOS. On this note, if your App uses other rendering related code, such as the live SDK, be careful not to have your GL code run on the main thread. If not, set GLContext(setCurrentContext) before calling, otherwise it will interfere with the GL state machine of the Flutter. Cause a blank screen or even crash.

2.3 Rasterizer

Rasterizer is a member variable of the shell. Each shell has only one Rasterizer and must work on the GPU thread. When the Dart code generates a layer_tree in the Dart thread, it calls back the shell’s proxy method OnAnimatorDraw(). At this point, the shell acts as the scheduling center, delivering UI configuration information to the GPU thread, and the Rasterizer takes the next step.

// shell. Cc void shell ::OnAnimatorDraw(FML ::RefPtr<Pipeline<flutter::LayerTree>> Pipeline) { Post UI configuration information to the GPU thread, Task_runners_.getgputaskrunner ()->PostTask([&waiting_for_first_frame = waiting_for_first_frame_, &waiting_for_first_frame_condition = waiting_for_first_frame_condition_, rasterizer = rasterizer_->GetWeakPtr(), Pipeline = STD :: Move (Pipeline)]() {// Pipeline is just a thread-safe container whose content LayerTree is an immutable UI description object computed from the Widget tree in DARTif(rasterizer) { rasterizer->Draw(pipeline); . }}); }Copy the code

Rasterizer has two core components: the Surface, which is an EGALayer wrapper that acts as a canvas for the home screen; The second is the CompositorContext instance, which holds all the draw-related information for LayerTree processing.

The Rasterizer: : DrawToSurface mainly do three things: 2. Call the Raster method of ScopedFrame to Raster layer_tree. 3. If PlatformView exists, call submitFrame for final processing

/// compositor_context.cc RasterStatus Rasterizer::DrawToSurface(flutter::LayerTree& layer_tree) { ... Auto Compositor_frame = Compositor_context_ ->AcquireFrame(compositor_context_) surface_->GetContext(), // skia GrContext root_surface_canvas, // root surface canvas external_view_embedder, // external view embedder root_surface_transformation, // root surface transformationtrue,                         // instrumentation enabled
      gpu_thread_merger_            // thread merger
  );

  if(compositor_frame) {// 2 Call the ScopedFrame::Raster method, Raster_status = Compositor_frame ->Raster(Layer_tree,false); .if(external_view_embedder ! External_view_embedder -> submitFrame (surface_->GetContext()));  }...return raster_status;
  }
  return RasterStatus::kFailed;
}

Copy the code

We all know that the underlying framework for drawing Flutter is SKCanva, and the DART code outputs the Flutter ::Layer object, so if you want to draw something on the screen, you need to preroll a transform object and then paint. The following code:

Layertree is a vertex-like result data object whose children are mapped from dart’s Widget object. Such as the Container corresponding to flutter in the dart: : ContainerLayer, whereas UiKitView corresponding flutter: : PlatformViewLayer. Layertree calls Preroll and Paint step by step from vertex to leaf nodes using a depth-first algorithm.

/// rasterizer.cc RasterStatus CompositorContext::ScopedFrame::Raster( flutter::LayerTree& layer_tree, Layer_tree. Preroll(*this, ignore_raster_cache); . Layer_tree. Paint(*this, ignore_raster_cache);return RasterStatus::kSuccess;
}

Copy the code
/// platform_view_layer.cc void PlatformViewLayer::Preroll(PrerollContext* context, const SkMatrix& matrix) { ... / / see the context - > 2.4 view_embedder - > PrerollCompositeEmbeddedView (view_id_, STD: : move (params)); } void PlatformViewLayer::Paint(PaintContext& Context) const {// see 2.4 SkCanvas* canvas = context.view_embedder->CompositeEmbeddedView(view_id_); context.leaf_nodes_canvas = canvas; }Copy the code

2.4 FlutterPlatformViewsController

FlutterPlatformViewsController instance is a member of the FlutterEngine, used to manage all PlatformView add remove, position and size, hierarchical order. Each UiKitView in the Dart layer corresponds to a PlatformView in the native layer, and the two are associated by holding the same viewiD. Each time a new PlatformView is created, the view ++ is used.

When PlatformViewLayer. Preroll, invoked FlutterPlatformViewsController instance PrerollCompositeEmbeddedView method, This method creates a new Skia object with view_id as key in the Picture_recorders_ dictionary, and puts view_id into the COMPOSItion_ORDER_ array, which records PlatformView hierarchy information.

/// FlutterPlatformViews.mm void FlutterPlatformViewsController::PrerollCompositeEmbeddedView( int view_id, STD ::unique_ptr<EmbeddedViewParams> params) {// Generate a SKIA object picture_recorders_[view_id] = based on view_id std::make_unique<SkPictureRecorder>(); picture_recorders_[view_id]->beginRecording(SkRect::Make(frame_size_)); picture_recorders_[view_id]->getRecordingCanvas()->clear(SK_ColorTRANSPARENT); // Record view_id to comPOSItion_order_ array comPOSItion_order_. push_back(view_id); . }Copy the code

When PlatformViewLayer. Paint, will call FlutterPlatformViewsController instance CompositeEmbeddedView method, according to the method before preroll generated skia object, Return an SKCanavs and assign it to the PaintContext’s leaf_nodes_canvas.

Note that the contents of the flutter::Layer are drawn on the leaf_nodes_canvas of the PaintContext. This means that when Paint is called on the PlatformViewLayer, the contents of a flutter::Layer will be drawn on the new SKCanvas if Paint is called on the PlatformViewLayer.

SkCanvas* FlutterPlatformViewsController::CompositeEmbeddedView(int view_id) {
   ...
   return picture_recorders_[view_id]->getRecordingCanvas();
}

Copy the code

Now let’s look at the iOS view hierarchy when Dart has two PlatformViews

As we know, when there is no PlatformView, there is only one FlutterView in the iOS View. However, for each additional UiKitView, there will be at least three more views in the iOS hierarchy, respectively:

1 PlatformView by FlutterPlatformViewFactory return of the native UIView

If 2 FlutterTouchInterceptingView PlatformView directly on FlutterView, according to the response chain order of iOS click, gestures events will fall on the PlatformView directly, The Flutter logic is all on dart, including click events, so PlatformView cannot be digested by itself. So here to add more FlutterTouchInterceptingView, as the father of PlatformView view, add to the FlutterView, FlutterTouchInterceptingView internal logic events will be forwarded to the FlutterViewController, unified by the dart processing ensure click gestures.

3 FlutterOverlayView serves as a mask for PlatformView, because if some view elements in DART need to be overlaid on UiKitView, those UI elements need to be drawn on the FlutterOverlayView. This is why PlatformViewLayer, after calling Paint, needs to switch the PaintContext’s leaf_nodes_canvas to a new canvas. Can draw the correct content on the FlutterOverlayView

At last, we look At the Rasterizer: : DrawToSurface finally SubmitFrame logic, this step is mainly to preroll and paint preparation before the closed loop.

bool FlutterPlatformViewsController::SubmitFrame(GrContext* gr_context,
                                                 std::shared_ptr<IOSGLContext> gl_context) {
  ...
  bool did_submit = true;
  for(int64_t view_id : Composition_order_) {// Initialize FlutterOverlayView, Create an OverlayView (EGALayer) for each PlatformView and place it in the Overlays_ dictionary EnsureOverlayInitialized(view_id, gl_context, gr_context); auto frame = overlays_[view_id]->surface->AcquireFrame(frame_size_);if(frame) {// key!! Copy the canvas contents of Picture_recorders_ [view_id] to overlays_[view_id]. SkCanvas* canvas = frame->SkiaCanvas(); canvas->drawPicture(picture_recorders_[view_id]->finishRecordingAsPicture()); canvas->flush(); did_submit &= frame->Submit(); } } picture_recorders_.clear();if(comPOSItion_order_ == active_comPOSItion_order_) {// Active_comPOSItion_order_ is the PlatformView hierarchy order after the last Submit // Composition_order_ is the PlatformView hierarchy order, If equal said hierarchical order unchanged FlutterPlatformViewsController Submit end composition_order_ operation. / / that the clear ();returndid_submit; UIView* flutter_view = flutter_view_.get();for(size_t i = 0; i < composition_order_.size(); i++) { int view_id = composition_order_[i]; // platform_view_root is PlatformView UIView* platform_view_root = root_views_[view_id].get(); Overlay UIView* overlay = overlays_[view_id]->overlay_view; overlay = overlays_[view_id]->overlay_view; / / below is the FlutterViewController. View addSubview logicif (platform_view_root.superview == flutter_view) {
      [flutter_view bringSubviewToFront:platform_view_root];
      [flutter_view bringSubviewToFront:overlay];
    } else{ [flutter_view addSubview:platform_view_root]; [flutter_view addSubview:overlay]; overlay.frame = flutter_view.bounds; Active_composition_order_.push_back (view_id);} // Save the local layer order. } composition_order_.clear();return did_submit;
}
Copy the code

Fix memory leak

Back to the memory leak we discussed at the beginning, it is Surface leak from Instrument, so far the only Surface canvas that can be created continuously is FlutterOverlayerView, let me see how it is created

Scoped_nsobject is the template class for Flutter, which makes an [obj release] of the content when it comes out of scope;

void FlutterPlatformViewsController::EnsureOverlayInitialized( int64_t overlay_id, std::shared_ptr<IOSGLContext> gl_context, GrContext* gr_context) { ... // init+retain reference count +2, Scoped_nsobject will only do -1 FML ::scoped_nsobject<FlutterOverlayView> overlay_view([[[FlutterOverlayView alloc] initWithContentsScale:contentsScale] retain]); std::unique_ptr<IOSSurface> ios_surface = [overlay_view.get() createSurface:std::move(gl_context)]; std::unique_ptr<Surface> surface = ios_surface->CreateGPUSurface(gr_context); overlays_[overlay_id] = std::make_unique<FlutterPlatformViewLayer>( std::move(overlay_view), std::move(ios_surface), std::move(surface)); overlays_[overlay_id]->gr_context = gr_context; }Copy the code

From the above code we can see that the code calls an extra retain when creating the FlutterOverlayView, which causes the FlutterOverlayView to end up referencing the technique 1 without releasing.

This is the biggest Memory Leak, and of course there are other code leaks, all of which are technical mistakes. I won’t explain the reasons in detail, so I’ll just change them

FlutterPlatformViews.mm

FlutterPlatformViews_Internal.mm

The above modification has been submitted to the official PR, you can also compare the modification on Github.

conclusion

To sum up, we can see that the cost of creating a PlatformView is not a small one, which is not the PlatformView itself, but the FlutterOverlayView. Because there is an extra Surface, it only needs to draw the FlutterOverlayView once for each refresh. Now you need to draw one more OverlayView, and maybe more than one. However, the benefits are also obvious, many audio and video SDKS provide a UIView or AndroidView to native, using PlatformView is not only easy to access, but also audio and video rendering performance is consistent with that of native.

The author

Levi