After reading xianyu’s article “I never expected that A Flutter can have external textures”, we learned that a flutter provides a mechanism for sharing native textures with a flutter for rendering. However, because the data type of Flutter to obtain native texture is CVPixelBuffer, the native texture needs to go through the GPU->CPU->GPU conversion process and consume extra performance, which is unacceptable for real-time audio and video rendering.

Our solution is to modify the code of the Flutter engine to connect the GL environment of the Flutter and the native GL environment through ShareGroup, avoiding the need to go around the CPU memory for texture transfer between the two environments. This solution solves the memory copy performance problem, but exposing the GL environment of the flutter is a risky operation, which also adds complexity to future flutter rendering problem location. So, is there a perfect, easy solution? The answer is to take advantage of CVPixelBuffer’s shared memory mechanism.

Principle of flutter external texture

Let’s review the pre-knowledge and see how the official external texture mechanism works.

In the figure, the red block is the native code we wrote, and the yellow block is the internal code logic of the Flutter Engine. The whole process is divided into registering the texture, and the overall texture rendering logic.

Registered texture

  1. Create an object and implementFlutterTextureProtocol, which is used to manage specific texture data
  2. throughFlutterTextureRegistryTo sign up for step oneFlutterTextureGets a Flutter texture ID
  3. If the ID is sent to the DART side through the channel mechanism, the DART side can pass the idTextureThe widget is used to use the texture, and the parameter is id

Texture rendering

  1. Dart side declares oneTextureWidget, indicating that the widget actually renders a texture provided by Native
  2. Engine side get layerTree, layerTreeTextureLayerThe node is responsible for rendering the surrounding texture
  3. First go through the ID passed by the DART side to find the first registered oneFlutterTexture, the flutterTexture is realized by our own native code, and its core is realizedcopyPixelBuffermethods
  4. Flutter engine callscopyPixelBufferGet the specific texture data and hand it over to the underlying GPU rendering

CVPixelBuffer format analysis

The source of all the problems is here: CVPixelBuffer. According to the rendering process of the upper and external flutter textures, the data interaction between native textures and flutter textures is transmitted through copyPixelBuffer, whose parameter is CVPixelBuffer. The performance problems mentioned in the previous salty fish article come from the conversion between textures and CVPixelBuffer.

If CVPixelBuffer and OpenGL textures can share the same memory copy, will the GPU -> CPU -> GPU performance bottleneck be solved? CVPixelBuffer can be used to create textures in the Flutter Engine.

1234567891011121314Copy the code
 
void IOSExternalTextureGL::CreateTextureFromPixelBufferCVOpenGLESTextureRef texture; CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage( kCFAllocatorDefault, cache_ref_, buffer_ref_, nullptr, GL_TEXTURE_2D, GL_RGBA, static_cast<int>(CVPixelBufferGetWidth(buffer_ref_)), static_cast<int>(CVPixelBufferGetHeight(buffer_ref_)), GL_BGRA, GL_UNSIGNED_BYTE, 0, &texture);if(err ! = noErr) { FML_LOG(WARNING) <<"Could not create texture from pixel buffer: " << err;  } else {    texture_ref_.Reset(texture);  }}Copy the code

Flutter engine is to use CVOpenGLESTextureCacheCreateTextureFromImage this interface to from CVPixelBuffer object creation of OpenGL texture, then the interface actually do? Let’s take a look at the official documentation

This function either creates a new or returns a cached CVOpenGLESTextureRef texture object mapped to the CVImageBufferRef and associated parameters. This operation creates a live binding between the image buffer and the underlying texture object. The EAGLContext associated with the cache may be modified to create, delete, or bind textures. When used as a source texture or GL_COLOR_ATTACHMENT, the image buffer must be unlocked before rendering. The source or render buffer texture should not be re-used until the rendering has completed. This can be guaranteed by calling glFlush().

From the document, we learn several key points:

  1. The returned texture object is mapped directly into the memory of the CVPixelBufferRef object
  2. The buffer memory can be accessed by CPU and GPU at the same time. We just need to follow the following rules:
    • This parameter is used when the GPU is accessedCVPixelBufferCannot be in lock state.

      Pixelbuffer if you have used pixelBuffer before, the CPU must lock the pixelBuffer object first and then unlock it. So it’s easy to understand that when a GPU uses a texture, it must not be manipulated by the CPU at the same time.
    • CPU access, to ensure that the GPU has completed rendering, usually refers to inglFlush()After the call.

      When the CPU reads or writes the buffer, it must ensure that the associated texture is not being rendered by OpenGL.

We use instrument allocation to verify that:

The results of instrument can also validate the conclusions in the document. Memory is allocated only when pixelBuffer is created, and no new memory is allocated when textures are mapped.

This also confirms our conclusion that pixelBuffer is allocated when it is created, and no new memory is allocated when it is mapped to the texture.

Shared memory scheme

Now that we know that CVPixelBuffer objects can actually bridge an OpenGL texture, our overall solution comes naturally, as shown in the figure below

The key point is that you first need to create the pixelBuffer object and allocate memory. Then map a texture object to both the Native GL environment and the Flutter GL environment. Thus, in two separate GL environments, we each have our own texture object, but its memory is actually mapped to the same CVPixelBuffer. In the actual rendering process of each frame, the Native environment renders to the texture, while the Flutter environment reads data from the texture.

The Demo presentation

Here I wrote a small demo to verify the effect in action. The main logic of the demo is to render a rotated triangle onto a pixelBuffer mapped texture at 60FPS. Then, after each frame is drawn, inform the flutter side to read the pixelBuffer object for rendering.

The core code is shown below:

1234567891011121314151617181920212223242526272829303132333435363738394041Copy the code
 
- (void)createCVBufferWith:(CVPixelBufferRef *)target withOutTexture:(CVOpenGLESTextureRef *)texture { It's not the point CVReturn err = CVOpenGLESTextureCacheCreate (_context kCFAllocatorDefault, NULL, NULL, & _textureCache); // The core argument is this, Shared memory, this must be set kCVPixelBufferIOSurfacePropertiesKey CFDictionarySetValue (kCVPixelBufferIOSurfacePropertiesKey attrs, empty); // Allocate memory for pixelBuffer objects, CVPixelBufferCreate(kCFAllocatorDefault, _sie.width, _sie.height, kCVPixelFormatType_32BGRA, attrs, target); / / map above pixelBuffer object to a texture CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault _textureCache, * target, NULL, GL_TEXTURE_2D, GL_RGBA, _size.width, _size.height, GL_BGRA, GL_UNSIGNED_BYTE, 0, texture); CFRelease(empty); CFRelease(attrs); }- (CVPixelBufferRef)copyPixelBuffer { Each flutter directly reads the pixelBuffer object CVBufferRetain(_target) of our mapped texture;return_target; }- (void)initGL { _context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2]; [EAGLContextsetCurrentContext:_context]; [self createCVBufferWith:&_target withOutTexture:&_texture]; Glgenframebuffer (1, &_framebuffer); glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer); GlFramebufferTexture2D (GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(_texture), 0); / / slightly}Copy the code

The key code is commented, so I won’t analyze it here

We can see from the GIF above that the rendering process is very smooth, and finally the displayLink frame rate can reach 60FPS. This demo can be applied to other scenarios that require CPU and GPU shared memory.


Complete demo code:

flutter_texture

Original link:

www.luoyibu.cn/posts/9703/