SurfaceTexture introduction: This article introduces the use and principles of SurfaceTexture, and focuses on internal implementations and EGLImage, including two ways to implement shared textures.

  • What is SurfaceTexture
  • Common applications of SurfaceTexture – cameras and video decoding
  • 3 Internal implementation of SurfaceTexture – EGLImageKHR
    • 3.1 How is SurfaceTexture created
    • 3.2 How does updateTexImage update data to a texture
  • 4 EGLImageKHR
    • 4.1 Permission Issues
    • 4.2 Important properties of EGLImageKHR
  • Two implementations of shared textures
    • 5.1 ShareContext
    • 5.2 EGLImageKHR
  • 6 conclusion

What is SurfaceTexture

SurfaceTexture is the core rendering component for Android, which is a combination of Surface and OpenGL ES textures that provide output to the GLES ‘Surface texture. From the Android rendering system, the SurfaceTexture is a consumer of a BufferQueue, and the onFrameAvailable() callback notifies the application when the producer enqueues a new buffer. The application then calls updateTexImage(), which frees the previously held buffer, fetches the new buffer from the queue, and performs the EGL call, allowing the GLES to use the buffer as an external texture.


The basic process is as follows:

  1. Generate image stream through Camera, Video Decoder and OpenGL.
  2. Images are queued through the Surface to the BufferQueue and notified to the GLConsumer.
  3. The GLConsumer gets the image stream GraphicBuffer from the BufferQueue and converts it to the EXTERNAL OES texture.
  4. Once you have an OES texture, the user can convert it to a normal texture and apply effects or screen it.

Common applications of SurfaceTexture – cameras and video decoding

SurfaceTexture is most commonly used as the output of a camera or video decoder, which is not described in detail here, but for cameras:

// SurfaceTexture works with GLSurfaceView to implement key code for rendering
// Initialize the Surface texture
fun initSurfaceTexture(textureCallback: (surfaceTexture: SurfaceTexture) - >Unit) {
  val args = IntArray(1)
  GLES20.glGenTextures(args.size, args, 0)
 surfaceTexName = args[0]  internalSurfaceTexture = SurfaceTexture(surfaceTexName)  textureCallback(internalSurfaceTexture) } GLSurfaceView is refreshed when the OnFrameAvailableListener callback is received cameraSurfaceTexture.initSurfaceTexture {  it.setOnFrameAvailableListener {  requestRender()  } cameraSurfaceTextureListener? .onSurfaceReady(cameraSurfaceTexture)} // Set the camera preview texture camera.setPreviewTexture(surfaceTexture) // Update texture in gl thread fun updateTexImage(a) {  internalSurfaceTexture.updateTexImage()  internalSurfaceTexture.getTransformMatrix(transformMatrix) } Copy the code

3 Internal implementation of SurfaceTexture – EGLImageKHR

SurfaceTexture is used in two main ways:

  • SurfaceTexture (int texName) // Create a SurfaceTexture
  • Void updateTexImage () // Updates the current image stream to the texture

3.1 How is SurfaceTexture created

The first step is to create a texture ID and pass it to the SurfaceTexture, for example

  // Create texture ID
    int[] tex = new int[1];
    GLES20.glGenTextures(1, tex, 0);
  // Create SurfaceTexture and pass Tex [0]
    mSurfaceTexture = new SurfaceTexture(tex[0]);
Copy the code

The Framework layer creates the SurfaceTexture code below

//frameworks\base\graphics\java\android\graphics
// constructor
public SurfaceTexture(int texName) {
    this(texName, false);
}
// constructor //singleBufferMode whether singleBufferMode is a single buffer. Default is false public SurfaceTexture(int texName, boolean singleBufferMode) {  mCreatorLooper = Looper.myLooper();  mIsSingleBuffered = singleBufferMode;  / / nativeInit native method  nativeInit(false, texName, singleBufferMode, new WeakReference<SurfaceTexture>(this)); } Copy the code

The framework layer calls the nativeInit method and ultimately the SurfaceTexture_init method.

//frameworks\base\core\jni\SurfaceTexture.cpp
//texName creates the texture name for the application
WeakThiz is a weak reference to the SurfaceTexture object
static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached,  jint texName, jboolean singleBufferMode, jobject weakThiz) {  sp<IGraphicBufferProducer> producer;  sp<IGraphicBufferConsumer> consumer;  // Create IGraphicBufferProducer and IGraphicBufferConsumer  BufferQueue::createBufferQueue(&producer, &consumer);   if (singleBufferMode) {  consumer->setMaxBufferCount(1);  }   sp<GLConsumer> surfaceTexture;  / / isDetached to false  if (isDetached) { . } else {  // Encapsulate consumer and texName as the GLConsumer class object surfaceTexture  surfaceTexture = new GLConsumer(consumer, texName,  GL_TEXTURE_EXTERNAL_OES, true, !singleBufferMode);  }  . // Set the surfaceTexture name  surfaceTexture->setName(String8::format("SurfaceTexture-%d-%d-%d". (isDetached ? 0 : texName),  getpid(),  createProcessUniqueId()));   // If the current context is protected, inform the producer.  consumer->setConsumerIsProtected(isProtectedContext());  // Save surfaceTexture to env  SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);  // Save producer to env  SurfaceTexture_setProducer(env, thiz, producer);   jclass clazz = env->GetObjectClass(thiz);  / / JNISurfaceTextureContext inherited GLConsumer: : FrameAvailableListener  sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz,  clazz));  //surfaceTexture sets the frame callback object CTX,  // The CTX ->onFrameAvailable method is triggered when frame data is received  surfaceTexture->setFrameAvailableListener(ctx);  SurfaceTexture_setFrameAvailableListener(env, thiz, ctx); } Copy the code

SurfaceTexture after initialization, to set up a JNISurfaceTextureContext GLConsumer listener, the listener will be back to Java layer SurfaceTexture. PostEventFromNative method, The OnFrameAvailableListener listener registered to the SurfaceTexture is further called back to notify the business layer that a new GraphicBuffer has been enqueued. The business layer can then call updateTexImage to update the GraphicBuffer to the texture.

3.2 How does updateTexImage update data to a texture

According to the document, OnFrameAvailableListener. OnFrameAvailable callback can occur in any thread, so can’t directly call in the callback updateTexImage, but need to switch to the OpenGL thread calls updateTexImage, So how does that work internally?

Application layer updateTexImage will eventually call into native GLConsumer: : updateTexImage () method.

//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::updateTexImage() {
.    / / mEglContext eglGetCurrentContext
    // Make sure the EGL state is the same as in previous calls.
 status_t err = checkAndUpdateEglStateLocked(); . BufferItem item;  // Acquire the next buffer.  // In asynchronous mode the list is guaranteed to be one buffer  // deep, while in synchronous mode we use the oldest buffer.  err = acquireBufferLocked(&item, 0); . // Update the Current GLConsumer state.  // Release the previous buffer.  err = updateAndReleaseLocked(item); . // Bind the new buffer to the GL texture, and wait until it's ready.  return bindTextureImageLocked(); } Copy the code

You can see that the method that gets the frame data is acquireBufferLocked.

//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::acquireBufferLocked(BufferItem *item,
        nsecs_t presentWhen, uint64_t maxFrameNumber) {
     // Get the current display of Consumer BufferItem
     // Get the camera preview frame data
 status_t err = ConsumerBase::acquireBufferLocked(item, presentWhen,  maxFrameNumber); . // If item->mGraphicBuffer is not null, this buffer has not been acquired  // before, so any prior EglImage created is using a stale buffer. This  // replaces any old EglImage with a new one (using the new buffer).  if(item->mGraphicBuffer ! = NULL) { int slot = item->mSlot;  // Generate an EglImage from the retrieved item->mGraphicBuffer and assign it to mEglSlots[slot]. MEglImage  mEglSlots[slot].mEglImage = new EglImage(item->mGraphicBuffer);  }   return NO_ERROR; } Copy the code

As you can see, the SurfaceTexture will eventually create an EglImage object for the GraphicBuffer, which holds the frame data.

So how does bindTextureImageLocked bind the frame data to the texture?

status_t GLConsumer::bindTextureImageLocked() {
.    GLenum error;
.    //mTexTarget GL_TEXTURE_EXTERNAL_OES texture created for the application
 glBindTexture(mTexTarget, mTexName); . // EGLImageKHR is generated by mGraphicBuffer  status_t err = mCurrentTextureImage->createIfNeeded(mEglDisplay,  mCurrentCrop); . // Bind the mCurrentTextureImage content to mTexTarget  mCurrentTextureImage->bindToTextureTarget(mTexTarget);  . // Wait for the new buffer to be ready.  return doGLFenceWaitLocked(); } Copy the code

See createIfNeeded method, it will eventually call to GLConsumer: : EglImage: : createImage

EGLImageKHR GLConsumer::EglImage::createImage(EGLDisplay dpy,
        const sp<GraphicBuffer>& graphicBuffer, const Rect& crop) {
    EGLClientBuffer cbuf =
            static_cast<EGLClientBuffer>(graphicBuffer->getNativeBuffer());
    const bool createProtectedImage =
 (graphicBuffer->getUsage() & GRALLOC_USAGE_PROTECTED) &&  hasEglProtectedContent();  EGLint attrs[] = {  EGL_IMAGE_PRESERVED_KHR, EGL_TRUE,  EGL_IMAGE_CROP_LEFT_ANDROID, crop.left,  EGL_IMAGE_CROP_TOP_ANDROID, crop.top,  EGL_IMAGE_CROP_RIGHT_ANDROID, crop.right,  EGL_IMAGE_CROP_BOTTOM_ANDROID, crop.bottom,  createProtectedImage ? EGL_PROTECTED_CONTENT_EXT : EGL_NONE,  createProtectedImage ? EGL_TRUE : EGL_NONE,  EGL_NONE,  }; . eglInitialize(dpy, 0.0);  Call eglCreateImageKHR to create the EGLImageKHR object  EGLImageKHR image = eglCreateImageKHR(dpy, EGL_NO_CONTEXT,  EGL_NATIVE_BUFFER_ANDROID, cbuf, attrs); . return image; } Copy the code

The key statement is eglCreateImageKHR, and as you can see, the EGLImageKHR object is created. Now that the EGLImageKHR image has been generated, let’s examine how the EGLImageKHR image is bound to the GL_TEXTURE_EXTERNAL_OES texture.

//frameworks\native\libs\gui\GLConsumer.cpp
void GLConsumer::EglImage::bindToTextureTarget(uint32_t texTarget) {
    // Update the texTarget data to mEglImage
    // equivalent to glTexImage2D or glTexSubImage2D
    glEGLImageTargetTexture2DOES(texTarget,
 static_cast<GLeglImageOES>(mEglImage)); } Copy the code

To summarize, the updateTexImage method is divided into these steps:

  • Generate texture content EGLImageKHR object, by ConsumerBase: : acquireBufferLocked GraphicBuffer gets the current display content (such as the camera frame data), An EGLImageKHR image is then generated from this GraphicBuffer.

  • The EGLImageKHR image through glEGLImageTargetTexture2DOES binding on GL_TEXTURE_EXTERNAL_OES type texture.

4 EGLImageKHR

SurfaceTexture can be internally SurfaceTexture by EGLImageKHR. EGLImageKHR is an extended format defined by EGL for sharing 2D image data. It can share data between various EGL client apis (such as OpenGL and OpenVG). The original intention is to share 2D image data, but the format and purpose of sharing data are not clearly defined. The EGLImageKHR creation function prototype is:

EGLImageKHR eglCreateImageKHR(
                            EGLDisplay dpy,
                            EGLContext ctx,
 EGLenum target,  EGLClientBuffer buffer,  const EGLint *attrib_list) Copy the code

In Android system, a Target called EGL_NATIVE_BUFFER_ANDROID is specially defined, which supports the creation of EGLImage object through ANativeWindowBuffer, and Buffer uses data when creating EGLImage object.

The EGLClientBuffer defined on Android is the GraphicBuffer class defined in native. To sum up, the basic usage flow of EGLImageKHR is as follows:

#define EGL_NATIVE_BUFFER_ANDROID 0x3140
#define EGL_IMAGE_PRESERVED_KHR   0x30D2

GraphicBuffer* buffer = new GraphicBuffer(1024.1024, PIXEL_FORMAT_RGB_565,
                                          GraphicBuffer::USAGE_SW_WRITE_OFTEN |
 GraphicBuffer::USAGE_HW_TEXTURE);  unsigned char* bits = NULL; buffer->lock(GraphicBuffer::USAGE_SW_WRITE_OFTEN, (void**)&bits);  // Write bitmap data into 'bits' here  buffer->unlock();  // Create the EGLImageKHR from the native buffer EGLint eglImgAttrs[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE }; EGLImageKHR img = eglCreateImageKHR(eglGetDisplay(EGL_DEFAULT_DISPLAY), EGL_NO_CONTEXT,  EGL_NATIVE_BUFFER_ANDROID,  (EGLClientBuffer)buffer->getNativeBuffer(),  eglImgAttrs);  // Create GL texture, bind to GL_TEXTURE_2D, etc.  // Attach the EGLImage to whatever texture is bound to GL_TEXTURE_2D glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, img); Copy the code

If the EGLClientBuffer data is in YUV format, you can also use the texture Target GL_TEXTURE_EXTERNAL_OES, which is also used to generate the texture from EGLImage.

4.1 Permission Issues

Android NDK does not expose GraphicBuffer interface, so if you use it directly, you need to download Android source code, compile and package it into a dynamic library. It should be noted that Android has prohibited the invocation of private native API after API26. However, after API 26, the Android NDK provides the Hardware Buffer APIs class to implement this functionality, which will be explained in more detail in this upcoming article. Please subscribe to 😁. For the most part, though, we can just use SurfaceTexture.

4.2 Important properties of EGLImageKHR

EGLImageKHR is designed to share 2D texture data, so when the driver is implemented at the bottom level, it often realizes CPU and GPU access to the same resource, so that data sharing without copy can be achieved, reducing power consumption and improving performance.

This is very important to understand on Android. IOS doesn’t use EGLImage because it uses EAGL instead of EGL, but has its own way of mapping data.

Two implementations of shared textures

As you can see from the above, there are two ways to implement shared textures:

  • One is the EGL ShareContext mechanism
  • One is shared memory, in this case EGLImageKHR

5.1 ShareContext

EGL ShareContext is a common way to ShareContext (EAGL for iOS is called ShareGroup).

/ * *share_context:
Specifies another EGL rendering context with which to share data, as defined by the client API corresponding to the contexts. Data is also shared with all other contexts with which share_context shares data. EGL_NO_CONTEXT indicates that no sharing is to take place.
* * /
EGLContext eglCreateContext( EGLDisplay display,  EGLConfig config,  EGLContext share_context,  EGLint const * attrib_list) Copy the code

When the share_context argument is passed to another EGL context, the two eGLContexts can share textures, vBos, etc.

Container Objects cannot be shared. For example:

  • Framebuffer objects
  • Vertex array objects
  • Transform feedback objects
  • Program pipeline objects

5.2 EGLImageKHR

This is actually a way of sharing memory to implement shared textures, most simply using SurfaceTexture directly, which I won’t go into detail here. It can also be implemented using HardwareBuffer and EGLImageKHR.

6 conclusion

In this article, SurfaceTexture will be introduced, and in the next article, SurfaceTexture and shared memory will be introduced.

Personal wechat official account, welcome to subscribe