Recently, I have been learning untiy game engine. During the learning process, I came up with the idea that unity and Flutter can be drawn through OpengL and Vulkan. Is there a way to integrate the two interfaces, that is, to render the Interface of Flutter into unity objects? Or render the Unity interface onto the Flutter widget. Since these two renderings are roughly the same, let’s focus on how to render the Flutter interface into Unity. The first thing we thought of was to take a screenshot of the Flutter interface into a bitmap. We then interacted with the Bitmap and passed it to Unity to use in Unity, and we immediately noticed that the Screenshot function came with the Flutter. However, we can immediately see the drawbacks of this solution. In this solution, we first need to download the Flutter interface from the GPU to memory, then pass the Bitmap to Unity through Unity communication with Java, and finally upload the Bitmap to the GPU for use as a texture in Unity. This is a lot of flipping in memory, and bitmaps are often so large that passing them back and forth is inefficient. Is there a better way to solve this problem? Of course there is

Texture Shared

We know that OpengL context is usually bound to threads, different context environment is relatively independent, cannot share content, but in order to better work in multi-threaded environment, OpengL provides a way to share texture to compensate for the appeal problem, improve the efficiency of multi-threaded work. It is easy to create opengL environments by passing in an existing OpengL context and configs. In this way, you can share the same texture in both OpengL environments. As for how to implement Shared with the unity of the texture on the android, you can refer to the article blog.csdn.net/jnlws/artic… This article is more detailed, and the code is more perfect, you can read it carefully and implement it yourself, here I will briefly explain the implementation process of camera texture sharing (because the flutter sharing process is similar).

  1. Calls back Java methods in the Unity thread by communicating with Android
  2. Get the Unity thread’s OpengL context and parameter configuration in the Java method, and create a New Java thread for Java rendering thread, and create a new OpengL context in the Java thread, and pass unity’s OpengL context to achieve texture sharing
  3. Print android camera data to the Surfacetexture and bind it to a new textureid created in OpengL. Render the camera data to the new Textureid using the fBO off-screen render (since surfacetexture outputs textures in Android are android-specific texture format GL_TEXTURE_EXTERNAL_OES and cannot be used directly in Unity). Therefore, you need to convert it to a texture format that Unity can use through off-screen rendering) and return the new Textureid to Unity
  4. Unity receives the Textureid and renders it to the GameObject

The key code is as follows

Texture sharing is an OpengL approach that is not required for natural multi-threaded rendering pipelines such as Vulkan and Metal, but I’m not familiar with Vulkan so I won’t cover it here.

The Flutter interface is rendered into the texture

We’ve already analyzed how to share camera data with Unity via textures. We know that textures can be shared with Unity via Surfacetexture and FBO off-screen rendering, so we just need to find out how to render the Flutter interface into The Surfacetexture. You can integrate the Flutter and Unity interface. Let’s analyze the flutter source code. The rest of the source code is based on Flutter1.16. First of all, the new version of the flutter is taken out of the Flutter engine for easy blending, and we don’t have to render the interface directly. So all we need to do is create a flutter fragment without a view and put it into the Unity Activity. We will flutter fragments source copies and transformation, and the matching will need FlutterActivityAndFragmentDelegate, we follow the code analysis how the flutter interface rendering on the android, The flutter was first loaded into the Android interface via FlutterViewFlutterView is divided into two modes. We only need to analyze the Surface mode. We found that there is a FlutterSurfaceView that is associated with the flutter. We found that the Surface was passed into the Flutter via FlutterRender when the SurfaceView surface was created.All we need to do is create a surface, pass it to the Flutter via FlutterRender, export its data to the Surfacetexture, and export it to a texture ID that Unity can use via FBO off-screen rendering. The following code

 public void attachToFlutterEngine(FlutterEngine flutterEngine) {
        this.flutterEngine = flutterEngine;
        FlutterRenderer flutterRenderer = flutterEngine.getRenderer();

        flutterRenderer.startRenderingToSurface(GLTexture.instance.surface);
        flutterRenderer.surfaceChanged(GLTexture.instance.getStreamTextureWidth(), GLTexture.instance.getStreamTextureHeight());
        FlutterRenderer.ViewportMetrics viewportMetrics = new FlutterRenderer.ViewportMetrics();
        viewportMetrics.width = GLTexture.instance.getStreamTextureWidth();
        viewportMetrics.height = GLTexture.instance.getStreamTextureHeight();
        viewportMetrics.devicePixelRatio = GLTexture.instance.context.getResources().getDisplayMetrics().density;
        flutterRenderer.setViewportMetrics(viewportMetrics);
        flutterRenderer.addIsDisplayingFlutterUiListener(new FlutterUiDisplayListener() {
            @Override
            public void onFlutterUiDisplayed(a) {
                GLTexture.instance.setNeedUpdate(true);
                GLTexture.instance.updateTexture();
            }

            @Override
            public void onFlutterUiNoLongerDisplayed(a) {}}); GLTexture.instance.attachFlutterSurface(this);
    }
Copy the code

It is important to note that the transfer width and height of the flutter must be given, otherwise the flutter interface may not appear

The next step is to receive the texture in Unity and render it to the GameObject just like the camera. The final result is as follows, and we can see the Flutter interface rendered perfectly in UnityThe overall process is as follows. When redrawing, the data is inside the GPU, without copying back and forth. This is the benefit of texture sharing, which can achieve higher refresh rate.

Click on the event

The next thing we need to deal with is the flutter click event. We need to get the click event in Unity and pass it to Android and then pass it to the Flutter. The Unity click-processing code is as follows

void Update()
    {
        #if UNITY_ANDROID
        if(mGLTexCtrl.Call<bool> ("isNeedUpdate"))
            mGLTexCtrl.Call("updateTexture");
        if (Input.touches.Length > 0) {if(haveStartFlutter == 1) {// Pass touch information to Java, code omitted, specific can refer to Unity click processing and source code}}else{
                mFlutterApp.Call("startFlutter");
                haveStartFlutter = 1; }}#endif 
        
        if(Input.GetMouseButtonDown(1)){ Debug.Log(Input.mousePosition); }}Copy the code

Android passes the code to flutter as follows. Here we can find out how to transmit the click event in the Flutter source code, and here we do it with the Flutter source code. Okay

 public void onTouchEvent(int type, double x, double y) {
        ByteBuffer packet =
                ByteBuffer.allocateDirect(1 * POINTER_DATA_FIELD_COUNT * BYTES_PER_FIELD);
        packet.order(ByteOrder.LITTLE_ENDIAN);
        double x1, y1;
        x1 = GLTexture.instance.getStreamTextureWidth() * x;
        y1 = GLTexture.instance.getStreamTextureHeight() * y;
        Log.i("myyf"."x:" + x1 + "&y:" + y1 + "&type:" + type);
        addPointerForIndex(x1, y1, type + 4.0, packet);
        if(packet.position() % (POINTER_DATA_FIELD_COUNT * BYTES_PER_FIELD) ! =0) {
            throw new AssertionError("Packet position is not on field boundary");
        }

        flutterEngine.getRenderer().dispatchPointerDataPacket(packet,packet.position());

    }

	// Pass the click message to the flutter. The code is omitted. Refer to the internal code of the flutter
    // TODO(mattcarroll): consider creating a PointerPacket class instead of using a procedure that
    // mutates inputs.
    private void addPointerForIndex(
            double x, double y, int pointerChange, int pointerData, ByteBuffer packet) {}Copy the code

This allowed us to click on the Flutter interface in Unity and the final result looks like this

conclusion

Above, we analyzed how to render the Flutter interface to Unity by using OpengL texture sharing and Android Surface. Similarly, how to render the Unity interface to The Flutter is the same. We only need to customize UnityPlayer to export it to the texture. It can also be used in Flutter and for wider promotion, we can render the Android interface into Both Flutter and Unity in this way. The code for this article has been uploaded to Github and is available here (github.com/feiyin0719/…