SurfaceTexture for Android graphics

SurfaceTexture is the core of off-screen rendering and TextureView, and contains a BufferQueue that transforms Surface image streams into textures for further processing by the business. The entire architecture is shown below:

First, generate an image stream from Canvas, OpenGL, Camera or Video Decoder.
The image is then queued through the Surface to the BufferQueue and notified to the GLConsumer.
The GLConsumer then takes the image stream GraphicBuffer from the BufferQueue and converts it to a texture.
Finally, the business side can further manipulate the texture, such as rendering or screen rendering.

Let’s take a look at SurfaceTexture initialization and image data flow within the SurfaceTexture.

SurfaceTexture initialization

New SurfaceTexture(textureId) starts SurfaceTexture initialization. The core logic is as follows:

SurfaceTexture_init

static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached, jint texName, jboolean singleBufferMode, jobject weakThiz)
{
    // Create BufferQueueCore, BufferQueueProducer, BufferQueueConsumer
    sp<IGraphicBufferProducer> producer;
    sp<IGraphicBufferConsumer> consumer;
    BufferQueue::createBufferQueue(&producer, &consumer);

    if (singleBufferMode) { / / a single buffer
        consumer->setMaxBufferCount(1); // Double buffering and triple buffering refer to here
    }

    // The Java layer SurfaceTexture actually corresponds to the Native layer GLConsumer
    sp<GLConsumer> surfaceTexture;
    if (isDetached) {
        surfaceTexture = new GLConsumer(consumer,GL_TEXTURE_EXTERNAL_OES,true,! singleBufferMode); }else {
        surfaceTexture = new GLConsumer(consumer,texName,GL_TEXTURE_EXTERNAL_OES,true,! singleBufferMode); }// If the current context is protected, inform the producer.
    if (isProtectedContext()) {
        consumer->setConsumerUsageBits(GRALLOC_USAGE_PROTECTED);
    }
    / / for the Java layer SurfaceTexture mSurfaceTexture set GLConsumer address object
    SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
    // Set the Producer object address for the Java layer SurfaceTexture. MProducer
    SurfaceTexture_setProducer(env, thiz, producer);
    // SurfaceTexture jclass
    jclass clazz = env->GetObjectClass(thiz);

    / / weakThiz said Java object layer SurfaceTexture weak references, JNISurfaceTextureContext is JNI wrapper class, responsible for the callback Java layer SurfaceTexture. PostEventFromNative method
    sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz, clazz));
    // Set the callback for GLConsumer (callback to Java layer)
    surfaceTexture->setFrameAvailableListener(ctx); 
    / / for the Java layer SurfaceTexture mFrameAvailableListener set of CTX address object
    SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
}
Copy the code

SurfaceTexture after initialization, to set up a JNISurfaceTextureContext GLConsumer listener, the listener will be back to Java layer SurfaceTexture. PostEventFromNative method, The OnFrameAvailableListener listener registered to the SurfaceTexture is further called back to notify the business layer that a new GraphicBuffer has been enqueued. If the business layer is interested in the latest GraphicBuffer, it calls updateTexImage to update the GraphicBuffer to the texture, otherwise it does nothing and ignores some graph data.

The GLConsumer is the direct consumer of the BufferQueue and is responsible for converting graphicBuffers into textures. The indirect consumer is then notified to consume the texture using the listener class WP

mFrameAvailableListener. When the indirect consumer is SurfaceFlinger, the listening class is Layer, and the Layer further tells the SurfaceFlinger to synthesize all the layers and then screen. When the indirect consumer is SurfaceTexture, the listener class is JNISurfaceTextureContext, used to inform the Java layer that new image data is available.

SurfaceTexture Image data streams

This section mainly looks at the creation of the producer Surface, the process of the business layer receiving the notification of frame availability and updating the target texture.

Create a producer Surface based on SurfaceTexture

When SurfaceTexture is created based on the texture ID, the Surface is the producer and the Gletexture is the consumer. The consumer is responsible for converting the GraphicBuffer fetched from the BufferQueue into a texture, which can then be further processed by the business layer, such as on effect or screen.

The Surface, as the producer, writes GraphicBuffer to the BufferQueue via BufferQueueProducer. GLConsumer as consumer reads GraphicBuffer from BufferQueue via BufferQueueConsumer.

SurfaceTexture creates Surface logic at Native layer: Create a Surface object based on BufferqueueProducer held by the SurfaceTexture and bind the address of the object to the Java layer Surface. MNativeObject variable. The core code is as follows:

// Create Surface based on SurfaceTexture
public Surface(SurfaceTexture surfaceTexture) {
    synchronized (mLock) {
        mName = surfaceTexture.toString();
        // Save the Native layer Surface object addresssetNativeObjectLocked(nativeCreateFromSurfaceTexture(surfaceTexture)); }}// Create a Surface based on BufferqueueProducer held by the SurfaceTexture and return the object address
static jlong nativeCreateFromSurfaceTexture(JNIEnv* env, jclass clazz, jobject surfaceTextureObj) {
    // Get the BufferqueueProducer object address from the Java layer surfaceTexture. MProducer and create BufferqueueProducer.
    sp<IGraphicBufferProducer> producer(SurfaceTexture_getProducer(env, surfaceTextureObj));
   // Create Native Surface objects based on BufferqueueProducer
    sp<Surface> surface(new Surface(producer, true));
   // Return the Surface object address
    surface->incStrong(&sRefBaseOwner);
    return jlong(surface.get());
}
Copy the code

As you can see, the BufferqueueProducer parameter is required to create a Native layer Surface object. It is responsible for pulling queues from the BufferQueue and enqueuing the GraphicBuffer.

Once the Surface is created, you can draw image data to the Surface in various ways, such as: Canvas drawing, Camera output, video decoder rendering and OpenGL rendering.

Now, let’s look at how the graphics data is updated to the target texture in two steps.

The business layer receives notification of frame availability

Here, we take Canvas drawing as an example to look at the available callback process of the business layer receiving frames, as shown below:

sp<IConsumerListener> mConsumerListener

The business layer actively updates the target texture

After the Java layer OnFrameAvailableListener listener receives the callback, it calls updateTexImage from the OpenGL thread to update the GraphicBuffer to the texture. The texture here is the texture ID we passed in when creating SurfaceTexture. The update process looks like this: SurfaceTexture

OnFrameAvailableListener. OnFrameAvailable callback can occur in any thread, so can’t directly call in the callback updateTexImage, but need to switch to the OpenGL thread calls. Because the updateTexImage call chain involves an OpenGL operation, it must be in the GL thread.

The core code is GLConsumer: : updateTexImage:

First of all, throughBufferQueueConsumerGet the available from the BufferQueueBufferItem, which containsGraphicBuffer.
Then, based onGraphicBufferCreate EglImage and EGLImageKHR.
Finally, based onEGLImageKHRUpdate texture content.

Simply put, with updateTexImage, we update the latest graphics data to the texture, and how we use the texture is up to the business layer.

Update OES textures based on GraphicBuffer

The updateTexImage method eventually updates the GraphicBuffer to the target texture using EGLImageKHR and EGLImageKHR.

The texture target needs to be changed from GL_TEXTURE_2D to GL_TEXTURE_EXTERNAL_OES, such as glBindTexture, glTexParameteri, etc.
In the slice shader, declare OES extension: #extension GL_OES_EGL_image_external: require. Also, use samplerExternalOES instead of the sampler2D texture type.
EGLImageKHR is created with eglCreateImageKHR based on GraphicBuffer graph data.

Graphicbuffers can be retrieved from the BufferQueue, or they can be locked to the memory address and written to the graph. See GraphicBuffer.cpp for details.

throughglEGLImageTargetTexture2DOEStheEGLImageKHRFill it with texture data

glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, static_cast(EGLImageKHR));

Finally, useeglDestroyImageKHRThe destructionEGLImageKHR.

The function prototype for creating and destroying EGLImageKHR looks like this:

// Create EGLImageKHR. On Android, CTX can be EGL_NO_CONTEXT, target EGL_NATIVE_BUFFER_ANDROID, and buffer is created from GraphicBuffer.
EGLImageKHR eglCreateImageKHR(EGLDisplay dpy, EGLContext ctx, EGLenum target, EGLClientBuffer buffer.const EGLint *attrib_list)

/ / destroy EGLImageKHR
EGLBoolean eglDestroyImageKHR(EGLDisplay dpy, EGLImageKHR image)
Copy the code

Specific use can reference GLConsumer: : EglImage: : createImage.

GLConsumer encapsulates the EglImage class, which is responsible for the processing logic of GraphicBuffer, EGLImageKHR and OES textures. The core code is as follows:

// EglImage creates the EGLImageKHR from the GraphicBuffer, and then updates the texture with EGLImageKHR. GLConsumer is responsible for converting graphicBuffers from BufferQueue to textures.
class EglImage : public LightRefBase<EglImage>{
public:
    // a unique constructor that takes a GraphicBuffer argument
    EglImage(sp<GraphicBuffer> graphicBuffer);
    
    // If the parameters change, createImage is called to create an internal EGLImageKHR
    status_t createIfNeeded(EGLDisplay display, const Rect& cropRect, bool forceCreate = false);
    
    // Upload the EGLImageKHR bound GraphicBuffer to the target texture
    void bindToTextureTarget(uint32_t texTarget){
        glEGLImageTargetTexture2DOES(texTarget, static_cast<GLeglImageOES>(mEglImage));
    }
    
private:
    // Create an internal EGLImageKHR
    EGLImageKHR createImage(EGLDisplay dpy, const sp<GraphicBuffer>& graphicBuffer, const Rect& crop);
    
    // Create the GraphicBuffer for EGLImageKHR
    sp<GraphicBuffer> mGraphicBuffer;
    // EGLImageKHR created from GraphicBuffer
    EGLImageKHR mEglImage;
    // Parameters needed to create EGLImageKHR
    EGLDisplay mEglDisplay;
    // Clipping area to use when creating EGLImageKHR
    Rect mCropRect;
}
Copy the code

conclusion

SurfaceTexture is the core of off-screen rendering. For example, we can set the SurfaceTexture to the Camera to receive the Camera image data and convert it to OES textures. Then we can use OpenGL to do further special effects on OES textures, and finally display or record video. Therefore, understanding the underlying principles of SurfaceTexture can help with business layer development and troubleshooting, and hopefully this article will help you.

Reference documentation

Using GL_OES_EGL_image_external on Android
EGL_KHR_image_base.txt

SurfaceTexture initialization

SurfaceTexture Image data streams

Create a producer Surface based on SurfaceTexture

The business layer receives notification of frame availability

The business layer actively updates the target texture

Update OES textures based on GraphicBuffer

conclusion

Reference documentation

Related Posts

Android Coroutines series – In-depth understanding of Dispatchers Coroutines scheduling

Quick Start kotlin Programming [First line of code Kotlin study Notes]

Learning and using the XLog logging component