In the previous “Android PlatformView and keyboard problems” article introduced the implementation and problems of the hybrid development of Android PlatformView, originally on the Android platform in order to integrate such as WebView, MapView and other capabilities, Use the VirtualDisplays implementation.

The iOS PlatformView Hybrid Composition mode has been introduced in 1.20. In this article, we will introduce the application and principles of Hybrid Composition

Repeatedly, it is 1.20 not 1.2 ~ ~ ~

Old versions of VirtualDisplay

Prior to 1.20, the content rendered by AndroidView was drawn to VirtualDisplays in Flutter, and then in the corresponding memory of VirtualDisplay, the drawn screen can be retrieved from its Surface.

VirtualDisplay is similar to a VirtualDisplay area and needs to be invoked together with DisplayManager. It is generally used in secondary screen display or recording scenarios. VirtualDisplay renders the contents of the VirtualDisplay area on a Surface.

As shown above, simply put, the content of the native control is drawn into memory. Then the Flutter Engine gets its rendering data from the corresponding textureId and displays it.

The biggest problem with this implementation is that there are many problems to be dealt with in touch events, text input and keyboard focus. The method of the Flutter UI is not used in iOS, but is composed by dividing the Flutter UI into two transparent textures: one under the iOS platform view and one on top of it.

The advantage of this is that the Flutter UI that needs to be rendered below the iOS platform view will eventually be drawn onto the texture underneath it. The Flutter UI that needs to be rendered on top of the “platform” will eventually be drawn with the texture above it. They just need to be combined at the end.

This approach is usually better because it means that the Native View can directly participate in the UI hierarchy of the Flutter.

Add Hybrid Composition

As a result of official and community efforts, version 1.20 has added a Hybrid Composition PlatformView implementation to Android, which will solve most of the platformView-related problems previously found on Android. For example, the Web interface of Huawei mobile phone mysteriously disappears after the keyboard pops up.

To create Hybrid Composition, we need to use PlatformViewLink, AndroidViewSurface, and PlatformViewsService.

  • throughPlatformViewLink 的 viewTypeRegistered a registered name corresponding to the native layer, which is the same as the previous onePlatformViewRegister the same;
  • Then, insurfaceFactoryReturns aAndroidViewSurfaceHandle draw and receive touch events;
  • Finally, inonCreatePlatformViewMethods usingPlatformViewsServiceInitialize theAndroidViewSurfaceAnd initialization parameters, while using Engine to trigger the native layer display.
Widget build(BuildContext context) {
  // This is used in the platform side to register the view.
  final String viewType = 'hybrid-view-type';
  // Pass parameters to the platform side.
  final Map<String.dynamic> creationParams = <String.dynamic> {};return PlatformViewLink(
    viewType: viewType, 
    surfaceFactory:
        (BuildContext context, PlatformViewController controller) {
      return AndroidViewSurface(
        controller: controller,
        gestureRecognizers: const <Factory<OneSequenceGestureRecognizer>>{},
        hitTestBehavior: PlatformViewHitTestBehavior.opaque,
      );
    },
    onCreatePlatformView: (PlatformViewCreationParams params) {
      return PlatformViewsService.initSurfaceAndroidView(
        id: params.id,
        viewType: viewType,
        layoutDirection: TextDirection.ltr,
        creationParams: creationParams,
        creationParamsCodec: StandardMessageCodec(),
      )
        ..addOnPlatformViewCreatedListener(params.onPlatformViewCreated)
        ..create();
    },
  );
}
Copy the code

Next, we go to the Android native layer, where we inherit PlatformView and return the controls we want to render using the getView method.

package dev.flutter.example;

import android.content.Context;
import android.graphics.Color;
import android.view.View;
import android.widget.TextView;
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import io.flutter.plugin.platform.PlatformView;

class NativeView implements PlatformView {
   @NonNull private final TextView textView;

    NativeView(@NonNull Context context, int id, @Nullable Map<String.Object> creationParams) {
        textView = new TextView(context);
        textView.setTextSize(72);
        textView.setBackgroundColor(Color.rgb(255.255.255));
        textView.setText("Rendered on a native Android view (id: " + id + ")");
    }

    @NonNull
    @Override
    public View getView() {
        return textView;
    }

    @Override
    public void dispose() {}
}
Copy the code

We then inherit PlatformViewFactory to load and initialize the PlatformView using the create method.

package dev.flutter.example;

import android.content.Context;
import android.view.View;
import androidx.annotation.Nullable;
import androidx.annotation.NonNull;
import io.flutter.plugin.common.BinaryMessenger;
import io.flutter.plugin.common.StandardMessageCodec;
import io.flutter.plugin.platform.PlatformView;
import io.flutter.plugin.platform.PlatformViewFactory;
import java.util.Map;

class NativeViewFactory extends PlatformViewFactory {
  @NonNull private final BinaryMessenger messenger;
  @NonNull private final View containerView;

  NativeViewFactory(@NonNull BinaryMessenger messenger, @NonNull View containerView) {
    super(StandardMessageCodec.INSTANCE);
    this.messenger = messenger;
    this.containerView = containerView;
  }

  @NonNull
  @Override
  public PlatformView create(@NonNull Context context, int id, @Nullable Object args) {
    final Map<String.Object> creationParams = (Map<String.Object>) args;
    return newNativeView(context, id, creationParams); }}Copy the code

Finally in the MainActivity NativeViewFactory flutterEngine getPlatformViewsController to registration.

package dev.flutter.example;

import androidx.annotation.NonNull;
import io.flutter.embedding.android.FlutterActivity;
import io.flutter.embedding.engine.FlutterEngine;

public class MainActivity extends FlutterActivity {
    @Override
    public void configureFlutterEngine(@NonNull FlutterEngine flutterEngine) {
        flutterEngine
            .getPlatformViewsController()
            .getRegistry()
            .registerViewFactory("hybrid-view-type".new NativeViewFactory(null.null)); }}Copy the code

Of course, if you want to enable Hybrid Composition on Android, you also need to add the following code to androidmanifest.xml to enable configuration:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="dev.flutter.example">
    <application
        android:name="io.flutter.app.FlutterApplication"
        android:label="hybrid"
        android:icon="@mipmap/ic_launcher">
        <! -... -->
        <! -- Hybrid composition -->
        <meta-data
            android:name="io.flutter.embedded_views_preview"
            android:value="true" />
    </application>
</manifest>
Copy the code

In addition, Hybrid Composition works well on Android 10 and up, but the Flutter interface will appear slower on the screen in versions 10 and up. This overhead is due to the need for Flutter frames to be synchronized with the Android view system.

To mitigate this problem, avoid displaying native controls during Dart animations. For example, you can use placeholder for screenshots of native controls, and use the placeholder directly when those animations occur.

Iii. Characteristics and implementation principles of Hybrid Composition

To introduce the Hybrid Composition implementation, we have to introduce a new object: FlutterImageView.

A FlutterImageView is not an ImageView in the normal sense.

In fact, the layer compositing required to blend native controls on Hybrid Composition is done via FlutterImageView. The FlutterImageView itself is a plain native View that implements some of the capabilities of the FlutterSurfaceView by implementing the RenderSurface interface.

FlutterImageView mainly includes ImageReader, Image and Bitmap, among which:

  • ImageReaderCan be simply understood as being able to storeImageData objects, and can be providedSurfaceUsed to draw the receiving native layerImageThe data.
  • ImageThat is to includeByteBuffersPixel data, it andImageReaderGenerally used in the original asCameraRelated fields.
  • BitmapIs toImageConvert to a bitmap that can be drawn, and then inFlutterImageViewthroughCanvasLet me plot it.

It can be seen that FlutterImageView can provide Surface, and Image data of Surface can be read and drawn through Bitmap.

FlutterImageView provides background and overlay SurfaceKind, where:

  • Background applies to the default rendering mode of FlutterView, which is the default rendering mode of the main Flutter application. Therefore, FlutterView now has surface, Texture and Image rendermodes.

  • An overlay is just a pattern that you use to compose with PlatformView under Hybrid Composition.

There is also a little as you can see, in PlatformViewsController createAndroidViewForPlatformView and createVirtualDisplayForPlatformView two methods, This is one of the ways that Flutter officially provides Hybrid Composition while also embracing the VirtualDisplay default.

The Hybrid Composition Dart layer triggers the native PlatformViewsChannel create method via PlatformViewsService, After launching a PlatformViewCreationRequest will have usesHybridComposition judgment, if is go createAndroidViewForPlatformView behind are true.

thenHybrid CompositionmodeFlutterImageViewHow does it work?

Let’s start with the example in section 2 above, and open the Android phone’s layout boundary. You can see that a small white square containing the Re appears in the middle of the screen. As you can see from the layout boundaries, the Re white square is actually a native control.

I then use the same code to add a small white Re square in a different location. You can see that there is a small white Re square in the upper right corner of the screen with layout boundaries, so you can see that the PlatformView in Hybrid Composition mode is displayed using some kind of native control.

But then we wonder, what’s so new about putting native controls on Flutter? So this is layer composition, right? Then put the two small white squares of Re together, and instead of PlatformView on them draw a blue Re Text using the default Text.

See? Without PlatformView, the blue Re Text drawn by Text can be displayed on the white opaque white square of the native Re.

Some of you might say, what’s so unusual about that? But those of you who know how Flutter works in the first place should know that Flutter is a SurfaceView by default on the native layer, and the Engine renders all screen controls onto this Surface.

But now what do you see? The fact that the blue Re Text in our Dart layer Text can now appear on the small white square of the Re shows that Hybrid Composition is not just about putting native controls on a Flutter.

Then we found another strange problem. The blue Re Text drawn with Flutter default Text also has the native layout boundary displayed. So we added the yellow Re Text and the red Re Text with the default Text. You can see that only the Text that has an intersection with the PlatformView has layout boundaries.

After the yellow Re text is adjusted down, it can be seen that the layout boundary of the yellow Re text also disappears, so it can be determined that the Dart control under Hybrid Composition can be displayed above the original control. Because it’s redrawn in some native control when it intersects with PlatformView.

So we can see from the Layout Inspector that the overlapping Text controls are rendered through the FlutterImageView layer.

There is also an interesting phenomenon that when a Flutter has more than one default control displayed on a PlatformView area, the controls share a FlutterImageView.

If they are not in the same region, they each use their own FlutterImageView. Note also that the PlatformView accessed by default with Hybrid Composition is a FlutterMutatorView.

As a matter of fact, FlutterMutatorView is used to adjust the position of access to FlutterView and Matrix of native controls. Generally, PlatformView access relation under Hybrid Composition is as follows:

So PlatformView uses the FlutterMutatorView to add the native control addView to FlutterView, and then uses the ability of FlutterImageView to mix layers.

How does Flutter determine that a control needs to use FlutterImageView?

In fact, you can see that when Engine goes to SubmitFrame, it plans each view with current_frame_view_count, The native createOverlaySurface method is then triggered to create the FlutterImageView by determining whether the CreateSurfaceIfNeeded function is needed in the region.

for (const SkRect& overlay_rect : overlay_layers.at(view_id)) { std::unique_ptr<SurfaceFrame> frame = CreateSurfaceIfNeeded(context, // view_id, // pictures.at(view_id), // overlay_rect // ); if (should_submit_current_frame) { frame->Submit(); }}Copy the code

And the PlatformViewSurface at the Dart level is just adding PlatformViewLayer through PlatformViewRenderBox, Then add Layer information by calling Engine in the addPlatformView of UI.sceneBuilder. (This part can be seen in the full Analysis of Flutter Rendering.)

In fact, there are many implementation details, such as:

  • onDisplayPlatformViewMethod, which is to showPlatformViewIs calledflutterView.convertToImageViewMethod will berenderSurfaceSwitch to aflutterImageView;
  • ininitializePlatformViewIfNeededMethodPlatformViewsThe create is not initialized again;
  • FlutterImagaeView 在 createImageReader 和 updateCurrentBitmapAndroid 10 is capable of hardware acceleration via GPU, which is whyHybrid CompositionThe reason for the better performance on Android 10.

Hybrid Composition is now available on stable 1.20, which solved some of my keyboard problems. Of course, only time will tell if Hybrid Composition will stand the test

Resources to recommend

  • Making: github.com/CarGuo
  • Open Source Flutter complete project:Github.com/CarGuo/GSYG…
  • Open Source Flutter Multi-case learning project:Github.com/CarGuo/GSYF…
  • Open Source Fluttre Combat Ebook Project:Github.com/CarGuo/GSYF…