SurfaceView, GLSurfaceView, SurfaceTexture, and TextureView are some of the more closely related classes in Android. This article is based on Android 5.0(Lollipop) code to clarify their basic principles, connections and differences.
SurfaceView
Since Android 1.0(API Level 1). It inherits from the class View, so it’s essentially a View. But unlike regular View, it has its own Surface. As we know, a typical Activity contains multiple views that form a tree of hierachy views, and only the top-level DecorView, the root View, is visible to THE WMS. This DecorView has a corresponding Windows State in THE WMS. Accordingly, the corresponding Layer in SF. The SurfaceView comes with a Surface, which has its own WindowState in WMS and its own Layer in SF. As shown below:
In other words,
While on the Client side (App) it is still in the View hierachy, on the Server side (WMS and SF) it is separate from the host window.The advantage of this is that rendering of the Surface can be done in a separate thread with its own GL context. This is useful for performance-related applications such as games, videos, etc., because it does not affect the main thread’s response to events. However, it also has disadvantages, because the Surface is not in the hierachy of the View, and its display is not controlled by the attributes of the View, so it cannot be translated, zooming and other transformations, nor can it be placed in other viewgroups, and some features in the View cannot be used.
While on the Client side (App) it is still in the View hierachy, on the Server side (WMS and SF) it is separate from the host window.The advantage of this is that rendering of the Surface can be done in a separate thread with its own GL context. This is useful for performance-related applications such as games, videos, etc., because it does not affect the main thread’s response to events. However, it also has disadvantages, because the Surface is not in the hierachy of the View, and its display is not controlled by the attributes of the View, so it cannot be translated, zooming and other transformations, nor can it be placed in other viewgroups, and some features in the View cannot be used.
GLSurfaceView
Added from Android 1.5(API Level 3) as a complement to SurfaceView. It can be seen as a typical usage pattern for SurfaceView. On top of SurfaceView, it adds EGL management and has its own rendering thread. In addition, it defines the Render interface that users need to implement, providing the flexibility to change specific Render behavior using Strategy patterns. As a Client of GLSurfaceView, you just need to set the implementation class of Renderer that implements the rendering function to GLSurfaceView. Such as:
The relevant class diagram is shown below. The SurfaceHolder in the SurfaceView mainly provides a bunch of interfaces to operate the Surface. The EglHelper and GLThread in GLSurfaceView do the aforementioned work of managing the EGL environment and rendering threads, respectively. GLSurfaceView users need to implement the Renderer interface.
SurfaceTexture
Joined from Android 3.0(API Level 11). And unlike SurfaceView,
The processing of image stream is not directly displayed, but converted to GL external texture, so it can be used for secondary processing of image stream data (such as Camera filter, desktop effects, etc.).Camera preview data, for example, can be presented to GLSurfaceView as a texture, or to TextureView as a hardware acceleration layer in View Heirachy via SurfaceTexture. First, the SurfaceTexture retrieves frame data from the image stream (from Camera preview, video decoding, GL rendering scene, etc.). When updateTexImage() is called, the GL texture object corresponding to the SurfaceTexture is updated based on the most recent image in the content stream. Next, You can manipulate it just like a normal GL texture. As you can see from the class diagram below, its core manages both the Consumer and Producer ends of a BufferQueue. The Producer side uses the source output data for the content stream, and the Consumer side uses the GraphicBuffer to generate textures. SurfaceTexture. OnFrameAvailableListener for SurfaceTexture so users know that there are new data arrives. JNISurfaceTextureContext is the JNI springboard for OnFrameAvailableListener from Native to Java. The SurfaceTexture attachToGLContext() and detachToGLContext() allow multiple GL Contexts to share the same content source.
The processing of image stream is not directly displayed, but converted to GL external texture, so it can be used for secondary processing of image stream data (such as Camera filter, desktop effects, etc.).Camera preview data, for example, can be presented to GLSurfaceView as a texture, or to TextureView as a hardware acceleration layer in View Heirachy via SurfaceTexture. First, the SurfaceTexture retrieves frame data from the image stream (from Camera preview, video decoding, GL rendering scene, etc.). When updateTexImage() is called, the GL texture object corresponding to the SurfaceTexture is updated based on the most recent image in the content stream. Next, You can manipulate it just like a normal GL texture. As you can see from the class diagram below, its core manages both the Consumer and Producer ends of a BufferQueue. The Producer side uses the source output data for the content stream, and the Consumer side uses the GraphicBuffer to generate textures. SurfaceTexture. OnFrameAvailableListener for SurfaceTexture so users know that there are new data arrives. JNISurfaceTextureContext is the JNI springboard for OnFrameAvailableListener from Native to Java. The SurfaceTexture attachToGLContext() and detachToGLContext() allow multiple GL Contexts to share the same content source.
Android 5.0 separates the core functionality of BufferQueue into a class called BufferQueueCore. BufferQueueProducer and BufferQueueConsumer are its producer and consumer implementation base classes (implementing the IGraphicBufferProducer and IGraphicBufferConsumer interfaces, respectively). They are created by createBufferQueue(), the static function of BufferQueue. Surface is the producer implementation class, provide the dequeueBuffer/queueBuffer hardware rendering interface, and lockCanvas/unlockCanvasAndPost software rendering interface, Enable the source of the content stream to fill the BufferQueue with a graphic buffer. GLConsumer inherits from ConsumerBase and is the consumer-side implementation class. It adds GL-related operations on top of the base class, such as converting content in a graphic Buffer to a GL texture. So far, a Surfacetexture-centric pipeline looks something like this:
TextureView
Introduced in 4.0(API Level 14). It can project content streams directly into a View and can be used for features like Live Preview. Unlike SurfaceView,
It does not create a separate window in THE WMS, but as a normal View in the View hierachy, so you can move, rotate, zoom, animate, and so on, just like any normal View. It’s worth noting
TextureView must be in the hardware Accelerated window.It displays content stream data that can come from the App process or a remote process. As you can see from the class diagram, TextureView inherits from the View, which is managed and drawn in hierachy like other views. TextureView overloads the Draw () method, which updates the image data received from the SurfaceTexture to the corresponding HardwareLayer as a texture. SurfaceTexture. OnFrameAvailableListener for inform TextureView content flow has a new image. The SurfaceTextureListener interface is used to let TextureView consumers know that SurfaceTexture is ready so that they can hand it to the appropriate content source. The Surface class implements the Producer interface for the BufferQueue, allowing producers to provide graphic buffers to the BufferQueue inside the SurfaceTexture via their software or hardware rendering interfaces.
It does not create a separate window in THE WMS, but as a normal View in the View hierachy, so you can move, rotate, zoom, animate, and so on, just like any normal View. It’s worth noting
TextureView must be in the hardware Accelerated window.It displays content stream data that can come from the App process or a remote process. As you can see from the class diagram, TextureView inherits from the View, which is managed and drawn in hierachy like other views. TextureView overloads the Draw () method, which updates the image data received from the SurfaceTexture to the corresponding HardwareLayer as a texture. SurfaceTexture. OnFrameAvailableListener for inform TextureView content flow has a new image. The SurfaceTextureListener interface is used to let TextureView consumers know that SurfaceTexture is ready so that they can hand it to the appropriate content source. The Surface class implements the Producer interface for the BufferQueue, allowing producers to provide graphic buffers to the BufferQueue inside the SurfaceTexture via their software or hardware rendering interfaces.
Below to VideoDumpView. Java (in the/frameworks/base/media/tests/MediaDump/SRC/com/android/MediaDump /) as an example using SurfaceTexture under analysis. The effect of this example is to take the video frame from MediaPlayer, display it on the screen, and dump the screen contents into the specified file. Since SurfaceTexture itself only produces textures, GLSurfaceView is needed to render the final output.
First, VideoDumpView is a descendant of GLSurfaceView. In the constructor VideoDumpView() we create VideoDumpRenderer, which is an instance of glSurfaceView.renderer, and then call setRenderer() to set it to the GLSurfaceView Renderer.
GLThread in GLSurfaceView is then started and onSurfaceCreated() in VideoDumpRenderer is called back after the EGL environment is created.
Here, we first create an external texture for GL using GLES. The external texture indicates that its real content is in the physical memory of the system allocated by ion, not in the GPU, where only its metadata is maintained. The SurfaceTexture is then created from the GL texture object created earlier. The process is as follows:
The SurfaceTexture argument is the TEXTURE object ID obtained by the GLES interface function glGenTexture(). SurfaceTexture_init() initializes the GLConsumer and corresponding BufferQueue, and places their Pointers to the Java layer object members of the SurfaceTexture via JNI.
Since the direct Listener is in the Java layer and the trigger is in the Native layer, callbacks are needed from the Native layer to the Java layer. This is a springboard through jnisurface ExtureContext. JNISurfaceTextureContext onFrameAvailable() acts as a bridge between Native and Java:
The fields. PostEvent is already initialized to the postEventFromNative() function of the SurfaceTexture_classInit(). This function puts a message into the message queue of the thread and asynchronously calls VideoDumpRenderer’s onFrameAvailable() function to notify VideoDumpRenderer of the arrival of new data.
Back to onSurfaceCreated(), next create the Surface class for external producers to use. One of the Surface constructors takes the argument SurfaceTexture.
It actually takes the Producer interface implementation class of BufferQueue created in The SurfaceTexture and creates the corresponding Surface class.
The Surface is the Producer end of the BufferQueue, and the GLConsumer from the SurfaceTexture is the Consumer end of the BufferQueue. When drawing from a Surface, the SurfaceTexture can bind the drawing result to the texture of GL by updateTexImage().
Back to the onSurfaceCreated() function, The next call setOnFrameAvailableListener () function will VideoDumpRenderer (SurfaceTexture. OnFrameAvailableListener interface) as the List of SurfaceTexture Ener, because it listens for new data on the content stream. The SurfaceTexture is then passed to MediaPlayer, because MediaPlayer is the producer and the SurfaceTexture is the consumer. The latter receives the Video frame output by the former. Thus, a chain of notifications is established through the Observer pattern: MediaPlayer -> SurfaceTexture -> VideDumpRenderer. In the onFrameAvailable() callback, set the updateSurface flag to true to indicate that a new image has arrived and the Surface needs to be updated. Why not update the texture right away, because it may not be in the render thread at the moment. SurfaceTexture objects can be created on any thread (callbacks are also called on that thread), but updateTexImage() can only be called in the thread of the GL context that contains the texture object. So normally you can’t call updateTexImage() directly in a callback.
At the same time, the GLThread in the GLSurfaceView is also running, which calls the drawframe () function of VideoDumpRenderer.
Here, SurfaceTexture’s updateTexImage() transforms the new image in the content stream into a texture in GL, followed by coordinate transformations. Bind the newly generated texture and paint it on the screen. The whole process is as follows:
Finally onDrawFrame() calls DumpToFile() to pour the contents of the screen into the file. In DumpToFile(), use glReadPixels() to store pixel data from the screen into a Buffer, and then use FileOutputStream to output it to a file.
SurfaceTexture was covered above, but let’s see how TextureView works. Again, let’s start with examples, Android’s official documentation about TextureView (http://developer.android.com/reference/android/view/TextureView.html) gives a simple example LiveCameraActiv Ity. It can display the contents of the Camera in a View. Create TextureView in onCreate () function first, then the Activity pass TextureView (realized TextureView. SurfaceTextureListener interface), is used to monitor the signal SurfaceTexture prepared.
The constructor for TextureView does not do the main initialization. The main initialization is done in getHardwareLayer(), which is called in draw() of its base class View. TextureView overrides this function:
Because TextureView is a hardware-accelerated layer (of type LAYER_TYPE_HARDWARE), it first creates the corresponding HardwareLayer class through HardwareRenderer, which is placed in the mLayer member. Then create the SurfaceTexture class, as described above. Then bind the HardwareLayer to the SurfaceTexture. The Native function nCreateNativeWindow is then called, which creates the Surface class from BufferQueueProducer in the SurfaceTexture. Note that Surface implements the ANativeWindow interface, which means it can be passed to the EGL interface as an EGL Surface for hardware rendering. Then setOnFrameAvailableListener () will register to SurfaceTexture mUpdateListener eavesdroppers. Thus, mUpdateListener’s onFrameAvailable() is called when a new image arrives on the content stream. Then you need to call to register in the TextureView SurfaceTextureListener onSurfaceTextureAvailable () callback function, inform users of TextureView SurfaceTexture ready. The whole process is as follows:
Note that DeferredLayerUpdater is created for TextureView instead of returning GLES20TextureLayer as in Android 4.4(Kitkat). Android 5.0(Lollipop) separated the render thread on the App side and put the rendering work into that thread. The thread can also receive VSync signals, so it can also process the animation itself. In fact, the creation of the DeferredLayerUpdater here is done synchronously in the render thread. DeferredLayerUpdater, as the name suggests, records the Layer’s update requests here and then does the actual work when the rendering thread actually wants to draw. SetSurfaceTexture() calls the HardwareLayer’s Native function nSetSurfaceTexture() to pass the SurfaceTexture member of the SurfaceTexture(type GLConsumer) to Def Erredlaydater, so that when the texture is to be updated later the DeferredLaydater will know where to start.
Mentioned in the initialization is called onSurfaceTextureAvailable () the callback function. In its implementation, TextureView consumers can pass SurfaceTexture prepared to the data source module for output from the data source. Such as:
Look at the implementation of setPreviewTexture(), which takes the GraphicBufferProducer created when initializing the SurfaceTexture and passes it to the Camera module.
At this point, everything is pretty much initialized. Next when content flows the new image is available, TextureView will be notified (via SurfaceTexture. OnFrameAvailableListener interface). SurfaceTexture OnFrameAvailableListener is SurfaceTexture callback interface come with new content. MUpdateListener in TextureView implements this interface:
As you can see, the updateLayer() function is called and the UI is updated via the invalidate() function. UpdateLayer () sets the mUpdateLayer flag bit. Thus, the next time VSync comes, Choreographer tells the App to redraw the View hierachy. In the UI redraw function performTranversals(), the draw() function of TextureView is called as a molecule of the View hierachy, The applyUpdate() and HardwareLayer’s updateSurfaceTexture() functions are called in succession.
UpdateSurfaceTexture () is actually called via JNI to the android_view_HardwareLayer_updateSurfaceTexture() function. The corresponding DeferredLayerUpdater flag bit, mUpdateTexImage, is set to indicate that the layer’s texture needs to be updated in the rendering thread.
As MENTIONED earlier,
Android 5.0 introduced rendering threadsIt is a larger topic beyond the scope of this article, and only the relevant parts are covered here. For background, only the relevant classes are drawn below. As you can see, ThreadedRenderer is the new hardwar renderreplacing the Gl20Renderer in Android 4.4. The key is the RenderProxy class, which sends tasks to the renderer thread when it needs to do something. The threadLoop() function is the main loop. It polls and waits on the thread’s Looper most of the time. When a synchronization request (or VSync signal) comes in, it wakes up. The tasks in the TaskQueue are then processed. A TaskQueue is a queue of RenderTasks that represent tasks in a rendering thread. For example, DrawFrameTask is one of RenderTask’s inherited classes, which is used to render the current frame. The DeferredLayerUpdater collection in DrawFrameTask holds previous requests to update the hardware acceleration layer.
Android 5.0 introduced rendering threadsIt is a larger topic beyond the scope of this article, and only the relevant parts are covered here. For background, only the relevant classes are drawn below. As you can see, ThreadedRenderer is the new hardwar renderreplacing the Gl20Renderer in Android 4.4. The key is the RenderProxy class, which sends tasks to the renderer thread when it needs to do something. The threadLoop() function is the main loop. It polls and waits on the thread’s Looper most of the time. When a synchronization request (or VSync signal) comes in, it wakes up. The tasks in the TaskQueue are then processed. A TaskQueue is a queue of RenderTasks that represent tasks in a rendering thread. For example, DrawFrameTask is one of RenderTask’s inherited classes, which is used to render the current frame. The DeferredLayerUpdater collection in DrawFrameTask holds previous requests to update the hardware acceleration layer.
When the main thread is ready to render data, it synchronously lets the render thread finish rendering. ProcessLayerUpdate () is called to update all the attributes in the hardware acceleration layer, and then the Apply () function of DeferredLaydater is called, where the flag bit mUpdateTexImage is detected to be set, This calls doUpdateTexImage() to actually update the GL texture and transform coordinates.
Finally, summarize the differences and connections between these several. To put it simply:
SurfaceView is a View that has its own Surface. Its rendering can be placed in a separate thread rather than the main thread. The disadvantage is that it cannot be morphed or animated.
SurfaceTexture can be used as a stream of content that is not directly output, thus providing opportunities for secondary processing.There is a delay of several frames compared to SurfaceView direct output. And since it manages the BufferQueue itself, so
memoryThe cost is also slightly higher.
memoryThe cost is also slightly higher.
TextureView is a View that outputs a stream of content as an external texture, which itself needs to be a hardware acceleration layer.
In fact, TextureView itself also includes SurfaceTexture, which performs a similar function compared to SurfaceView+SurfaceTexture (turning images on the content stream into textures, The difference is that TextureView is drawn in the View hierachy, so it is normally done on the main thread (it was done in the render thread after Android 5.0 introduced the render thread). While SurfaceView+SurfaceTexture does its rendering on a separate Surface, it can be a user-supplied thread, rather than the main or render thread of the system. Another advantage over TextureView is that it can be displayed using Hardware overlays.
From: www.bubuko.com/infodetail-…