SurfaceTexture is the core of off-screen rendering and TextureView, and contains a BufferQueue that transforms Surface image streams into textures for further processing by the business. The entire architecture is shown below:
- First, generate an image stream from Canvas, OpenGL, Camera or Video Decoder.
- The image is then queued through the Surface to the BufferQueue and notified to the GLConsumer.
- The GLConsumer then takes the image stream GraphicBuffer from the BufferQueue and converts it to a texture.
- Finally, the business side can further manipulate the texture, such as rendering or screen rendering.
Let’s take a look at SurfaceTexture initialization and image data flow within the SurfaceTexture.
SurfaceTexture initialization
New SurfaceTexture(textureId) starts SurfaceTexture initialization. The core logic is as follows:
SurfaceTexture_init
static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached, jint texName, jboolean singleBufferMode, jobject weakThiz)
{
// Create BufferQueueCore, BufferQueueProducer, BufferQueueConsumer
sp<IGraphicBufferProducer> producer;
sp<IGraphicBufferConsumer> consumer;
BufferQueue::createBufferQueue(&producer, &consumer);
if (singleBufferMode) { / / a single buffer
consumer->setMaxBufferCount(1); // Double buffering and triple buffering refer to here
}
// The Java layer SurfaceTexture actually corresponds to the Native layer GLConsumer
sp<GLConsumer> surfaceTexture;
if (isDetached) {
surfaceTexture = new GLConsumer(consumer,GL_TEXTURE_EXTERNAL_OES,true,! singleBufferMode); }else {
surfaceTexture = new GLConsumer(consumer,texName,GL_TEXTURE_EXTERNAL_OES,true,! singleBufferMode); }// If the current context is protected, inform the producer.
if (isProtectedContext()) {
consumer->setConsumerUsageBits(GRALLOC_USAGE_PROTECTED);
}
/ / for the Java layer SurfaceTexture mSurfaceTexture set GLConsumer address object
SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
// Set the Producer object address for the Java layer SurfaceTexture. MProducer
SurfaceTexture_setProducer(env, thiz, producer);
// SurfaceTexture jclass
jclass clazz = env->GetObjectClass(thiz);
/ / weakThiz said Java object layer SurfaceTexture weak references, JNISurfaceTextureContext is JNI wrapper class, responsible for the callback Java layer SurfaceTexture. PostEventFromNative method
sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz, clazz));
// Set the callback for GLConsumer (callback to Java layer)
surfaceTexture->setFrameAvailableListener(ctx);
/ / for the Java layer SurfaceTexture mFrameAvailableListener set of CTX address object
SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
}
Copy the code
SurfaceTexture after initialization, to set up a JNISurfaceTextureContext GLConsumer listener, the listener will be back to Java layer SurfaceTexture. PostEventFromNative method, The OnFrameAvailableListener listener registered to the SurfaceTexture is further called back to notify the business layer that a new GraphicBuffer has been enqueued. If the business layer is interested in the latest GraphicBuffer, it calls updateTexImage to update the GraphicBuffer to the texture, otherwise it does nothing and ignores some graph data.
The GLConsumer is the direct consumer of the BufferQueue and is responsible for converting graphicBuffers into textures. The indirect consumer is then notified to consume the texture using the listener class WP
mFrameAvailableListener. When the indirect consumer is SurfaceFlinger, the listening class is Layer, and the Layer further tells the SurfaceFlinger to synthesize all the layers and then screen. When the indirect consumer is SurfaceTexture, the listener class is JNISurfaceTextureContext, used to inform the Java layer that new image data is available.
SurfaceTexture Image data streams
This section mainly looks at the creation of the producer Surface, the process of the business layer receiving the notification of frame availability and updating the target texture.
Create a producer Surface based on SurfaceTexture
When SurfaceTexture is created based on the texture ID, the Surface is the producer and the Gletexture is the consumer. The consumer is responsible for converting the GraphicBuffer fetched from the BufferQueue into a texture, which can then be further processed by the business layer, such as on effect or screen.
The Surface, as the producer, writes GraphicBuffer to the BufferQueue via BufferQueueProducer. GLConsumer as consumer reads GraphicBuffer from BufferQueue via BufferQueueConsumer.
SurfaceTexture creates Surface logic at Native layer: Create a Surface object based on BufferqueueProducer held by the SurfaceTexture and bind the address of the object to the Java layer Surface. MNativeObject variable. The core code is as follows:
// Create Surface based on SurfaceTexture
public Surface(SurfaceTexture surfaceTexture) {
synchronized (mLock) {
mName = surfaceTexture.toString();
// Save the Native layer Surface object addresssetNativeObjectLocked(nativeCreateFromSurfaceTexture(surfaceTexture)); }}// Create a Surface based on BufferqueueProducer held by the SurfaceTexture and return the object address
static jlong nativeCreateFromSurfaceTexture(JNIEnv* env, jclass clazz, jobject surfaceTextureObj) {
// Get the BufferqueueProducer object address from the Java layer surfaceTexture. MProducer and create BufferqueueProducer.
sp<IGraphicBufferProducer> producer(SurfaceTexture_getProducer(env, surfaceTextureObj));
// Create Native Surface objects based on BufferqueueProducer
sp<Surface> surface(new Surface(producer, true));
// Return the Surface object address
surface->incStrong(&sRefBaseOwner);
return jlong(surface.get());
}
Copy the code
As you can see, the BufferqueueProducer parameter is required to create a Native layer Surface object. It is responsible for pulling queues from the BufferQueue and enqueuing the GraphicBuffer.
Once the Surface is created, you can draw image data to the Surface in various ways, such as: Canvas drawing, Camera output, video decoder rendering and OpenGL rendering.
Now, let’s look at how the graphics data is updated to the target texture in two steps.
The business layer receives notification of frame availability
Here, we take Canvas drawing as an example to look at the available callback process of the business layer receiving frames, as shown below:
sp<IConsumerListener> mConsumerListener
The business layer actively updates the target texture
After the Java layer OnFrameAvailableListener listener receives the callback, it calls updateTexImage from the OpenGL thread to update the GraphicBuffer to the texture. The texture here is the texture ID we passed in when creating SurfaceTexture. The update process looks like this: SurfaceTexture
OnFrameAvailableListener. OnFrameAvailable callback can occur in any thread, so can’t directly call in the callback updateTexImage, but need to switch to the OpenGL thread calls. Because the updateTexImage call chain involves an OpenGL operation, it must be in the GL thread.
The core code is GLConsumer: : updateTexImage:
- First of all, through
BufferQueueConsumer
Get the available from the BufferQueueBufferItem
, which containsGraphicBuffer
. - Then, based on
GraphicBuffer
Create EglImage and EGLImageKHR. - Finally, based on
EGLImageKHR
Update texture content.
Simply put, with updateTexImage, we update the latest graphics data to the texture, and how we use the texture is up to the business layer.
Update OES textures based on GraphicBuffer
The updateTexImage method eventually updates the GraphicBuffer to the target texture using EGLImageKHR and EGLImageKHR.
-
The texture target needs to be changed from GL_TEXTURE_2D to GL_TEXTURE_EXTERNAL_OES, such as glBindTexture, glTexParameteri, etc.
-
In the slice shader, declare OES extension: #extension GL_OES_EGL_image_external: require. Also, use samplerExternalOES instead of the sampler2D texture type.
-
EGLImageKHR is created with eglCreateImageKHR based on GraphicBuffer graph data.
Graphicbuffers can be retrieved from the BufferQueue, or they can be locked to the memory address and written to the graph. See GraphicBuffer.cpp for details.
- through
glEGLImageTargetTexture2DOES
theEGLImageKHR
Fill it with texture data
glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, static_cast(EGLImageKHR));
- Finally, use
eglDestroyImageKHR
The destructionEGLImageKHR
.
The function prototype for creating and destroying EGLImageKHR looks like this:
// Create EGLImageKHR. On Android, CTX can be EGL_NO_CONTEXT, target EGL_NATIVE_BUFFER_ANDROID, and buffer is created from GraphicBuffer.
EGLImageKHR eglCreateImageKHR(EGLDisplay dpy, EGLContext ctx, EGLenum target, EGLClientBuffer buffer.const EGLint *attrib_list)
/ / destroy EGLImageKHR
EGLBoolean eglDestroyImageKHR(EGLDisplay dpy, EGLImageKHR image)
Copy the code
Specific use can reference GLConsumer: : EglImage: : createImage.
GLConsumer encapsulates the EglImage class, which is responsible for the processing logic of GraphicBuffer, EGLImageKHR and OES textures. The core code is as follows:
// EglImage creates the EGLImageKHR from the GraphicBuffer, and then updates the texture with EGLImageKHR. GLConsumer is responsible for converting graphicBuffers from BufferQueue to textures.
class EglImage : public LightRefBase<EglImage>{
public:
// a unique constructor that takes a GraphicBuffer argument
EglImage(sp<GraphicBuffer> graphicBuffer);
// If the parameters change, createImage is called to create an internal EGLImageKHR
status_t createIfNeeded(EGLDisplay display, const Rect& cropRect, bool forceCreate = false);
// Upload the EGLImageKHR bound GraphicBuffer to the target texture
void bindToTextureTarget(uint32_t texTarget){
glEGLImageTargetTexture2DOES(texTarget, static_cast<GLeglImageOES>(mEglImage));
}
private:
// Create an internal EGLImageKHR
EGLImageKHR createImage(EGLDisplay dpy, const sp<GraphicBuffer>& graphicBuffer, const Rect& crop);
// Create the GraphicBuffer for EGLImageKHR
sp<GraphicBuffer> mGraphicBuffer;
// EGLImageKHR created from GraphicBuffer
EGLImageKHR mEglImage;
// Parameters needed to create EGLImageKHR
EGLDisplay mEglDisplay;
// Clipping area to use when creating EGLImageKHR
Rect mCropRect;
}
Copy the code
conclusion
SurfaceTexture is the core of off-screen rendering. For example, we can set the SurfaceTexture to the Camera to receive the Camera image data and convert it to OES textures. Then we can use OpenGL to do further special effects on OES textures, and finally display or record video. Therefore, understanding the underlying principles of SurfaceTexture can help with business layer development and troubleshooting, and hopefully this article will help you.
Reference documentation
- Using GL_OES_EGL_image_external on Android
- EGL_KHR_image_base.txt