The Android framework provides a variety of apis for 2D and 3D graphics rendering that interact with manufacturers’ graphical driver implementations. Application developers can draw images to the screen in three ways: Canvas, OpenGL ES, or Vulkan.

Android graphics components

Whatever rendering API the developer uses, everything will be rendered to the Surface. The Surface represents the producer in the buffer queue, which is normally consumed by SurfaceFlinger. Every window created on the Android platform is supported by Surface. All rendered visible surfaces are composited into the display by SurfaceFlinger.

The following figure shows how the key components work together:

The main components are as follows:

Image stream producer

An image stream producer can be anything that generates a graphics buffer for consumption. Examples include OpenGL ES, Canvas 2D, and Mediaserver video decoders.

WindowManager

WindowManager controls Window objects, which are containers that hold View objects. Window objects are always supported by Surface objects. WindowManager oversees the life cycle, input and focus events, screen orientation, transitions, animation, positioning, deformation, Z-axis order, and many other aspects of Windows. WindowManager sends all window metadata to SurfaceFlinger so that SurfaceFlinger can use this data to compose the Surface on the screen.

Image flow consuming party

The most common consumer of image streams is SurfaceFlinger, but other OpenGL ES applications can also consume image streams, such as camera apps that consume camera preview image streams. Non-gl applications can also be users, such as the ImageReader class.

SurfaceFlinger

The system service consumes the currently visible Surface and composes it into the display using the information provided in the window manager. SurfaceFlinger takes buffers of data from multiple sources, composts them and sends them to the display. SurfaceFlinger is the only service that can modify parts of what is displayed. SurfaceFlinger uses OpenGL and Hardware Composer to compose a set of surfaces.

The SurfaceFlinger accepts buffers in two ways: through BufferQueue and SurfaceControl.

One way SurfaceFlinger accepts buffers is through BufferQueue and SurfaceControl. When the application comes to the foreground, it requests a buffer from WindowManager. WindowManager then requests the layer from SurfaceFlinger. Layers are a combination of Surface (which contains BufferQueue) and SurfaceControl (which contains layer metadata such as screen frames). SurfaceFlinger creates the layer and sends it to WindowManager. WindowManager then sends the Surface to the application, but leaves the SurfaceControl in place to control how the application looks on the screen.

While an app can submit a buffer at any time, the SurfaceFlinger can only wake up when the screen is between refreshes to receive the buffer, which varies from device to device. This minimizes memory usage and avoids the visible tearing on the screen that can occur if the display is updated during a refresh.

The screen sends a VSYNC signal to the SurfaceFlinger when the screen is between refreshes. The VSYNC signal indicates that the screen can be refreshed without tearing. When the SurfaceFlinger receives the VSYNC signal, the SurfaceFlinger traverses its layer list to find a new buffer. If SurfaceFlinger finds a new buffer, SurfaceFlinger gets the buffer; Otherwise, SurfaceFlinger will continue to use the same buffer that was acquired last time. SurfaceFlinger must always display content, so it keeps a buffer. If there is no commit buffer on a layer, that layer is ignored.

After SurfaceFlinger collects all the buffers in the visible layer, it asks the hardware blending renderer (HWC) how it should compose. If HWC marks the layer composition type as client composition, SurfaceFlinger will compose these layers. SurfaceFlinger then passes the output buffer to the HWC.

layer

The layer is the most important unit of synthesis. Layers are a combination of Surface and SurfaceControl. Each layer has a set of properties that define how it interacts with other layers. The layer attributes are:

attribute instructions
positioning Defines where a layer is displayed on its screen. This includes information such as the position of layers’ edges and their Z-order relative to other layers (indicating whether the layer comes before or after the other layers).
content Defines how the content displayed on the layer should be rendered within the boundaries of the positioning property definition. This includes information such as clipping (to extend a portion of the content to fill the boundaries of the layer) and transformation (to show rotated or flipped content).
synthetic Defines how layers should be composed with other layers. Includes information such as blending modes and full-layer Alpha values for Alpha synthesis.
To optimize the Provides information that is not absolutely necessary for the correct composition layer but can be used by hardware blending renderer (HWC) devices to optimize the composition execution. This includes information such as the visible area of the layer and which part of the layer has been updated since the last frame.

Hardware Hybrid Renderer (HWC)

Displays the hardware abstract implementation of the subsystem. SurfaceFlinger can delegate some composition work to Hardware Composer to share the workload on OpenGL and GPU. SurfaceFlinger just acts as another OpenGL ES client. Therefore, SurfaceFlinger uses OpenGL ES as it composes one or two buffers into a third buffer. This makes the resulting power consumption much lower than performing all the calculations through the GPU.

Hardware Composer HAL does the other half of the work and is at the heart of all Android graphics rendering. Hardware Composer must support events, one of which is VSYNC (the other is hot plug for plug-and-play HDMI).

Hardware Hybrid Renderer (HWC) HAL is used to determine the most efficient way to compose buffers from available hardware. As HAL, its implementation is device-specific and is usually done by screen hardware original equipment manufacturers (Oems).

When you consider using overlay planes, it’s easy to see the benefits of this approach, which synthesizes multiple buffers in the display hardware (rather than the GPU). For example, suppose you have a normal Android phone with the screen facing up, the status bar at the top, the navigation bar at the bottom, and other areas showing app content. The contents of each layer are in separate buffers. You can handle composition in any of the following ways:

  • Render the application content into the staging buffer, then render the status bar on it, then render the navigation bar on it, and finally pass the staging buffer to the display hardware.
  • Pass all three buffers to the display hardware and instruct it to read data from different parts of the screen from different buffers.

The latter approach can significantly improve efficiency.

Display processor functions vary widely. The number of stacking layers (whether or not the layers can be rotated or blended) and the limitations on positioning and stacking are difficult to express through the API. To accommodate these options, HWC performs the following calculations:

  • SurfaceFlinger provides HWC with a complete list of layers and asks “What do you want to do with these layers?”
  • HWC responds by marking each layer as a device or client composite.
  • SurfaceFlinger handles all clients, passes the output buffer to HWC, and lets HWC handle the rest.
  • Because hardware vendors can customize the decision code, the best performance can be achieved on each device.

When nothing changes on the screen, the superposition plane may be less efficient than GL compositing. This is especially true when the overlay content has transparent pixels and the overlay is mixed together. In such cases, HWC can request the GLES composition for some or all of the layers and retain the synthesized buffer. If SurfaceFlinger requires the same set of buffers to be synthesized, HWC can display the previously synthesized temporary buffers. This can extend the battery life of idle devices.

Android devices typically support four overlay planes. Attempting to compose more layers than stacking can cause the system to use GLES composition for some of them, meaning that the number of layers used by the application can have a significant impact on power consumption and performance.

HWC can synthesize surfaces by abstracting objects such as overlay layers and 2D bit-block transmitters, and communicate with specialized window composition hardware to synthesize Windows. Use HWC to compose Windows instead of SurfaceFlinger and GPU. Most Gpus are not optimized for composition, and when the GPU composes layers from SurfaceFlinger, the application cannot render itself using the GPU.

HWC implementations should support:

  • At least 4 stacking layers:

    • The status bar
    • The system bar
    • application
    • Wallpaper/Background

  • Layers larger than the screen (e.g. wallpaper)

  • Simultaneously premultiply Alpha per pixel blending and Alpha per plane blending

  • Hardware path for playing protected videos

  • RGBA package order, YUV format, and tiling, rearrangement, and stride attributes

Gralloc

A graphics memory allocator (Gralloc) is required to allocate the memory requested by the image producer. The Gralloc memory allocator implemented through the vendor-specific HAL interface will be used to perform buffer allocation tasks.

The screen

The system can have multiple screens, and screens can be added or removed during normal system operation. Screens are added/removed at the request of HWC or the framework. When the external screen is connected to or disconnected from the device (called hot swap), the HWC device requests to add or remove the screen. The client requests the virtual screen and its contents are rendered to the off-screen buffer (instead of the physical screen).

Virtual screen

The SurfaceFlinger supports an internal screen (built into a phone or tablet), an external screen (such as a TV connected via HDMI), and one or more virtual screens that make the synthesized output available within the system. The virtual screen can be used to record screen information or send screen information over the network. Frames generated for the virtual screen are written to the BufferQueue.

A virtual screen can share the same set of layers (stack) as the home screen, or it can have its own set of layers. Virtual screens do not have VSYNC, so VSYNC for internal screens is used to trigger compositing for all screens.

In hardware hybrid Renderer (HWC) implementations that support virtual screens, virtual screens can be composed with OpenGL ES (GLES), HWC, or GLES and HWC. In implementations that do not support virtual screens, virtual screens are always composed using GLES.

VSYNC

VSYNC signal can display pipe synchronously. The display pipeline consists of application rendering, SurfaceFlinger compositing, and a hardware hybrid renderer (HWC) for displaying images on screen. VYSNC synchronizes the time the application wakes up to start rendering, the time the SurfaceFlinger wakes up to compose the screen, and the screen refresh cycle. This synchronization eliminates lag and improves the visual representation of graphics.

SurfaceFlinger controls whether HWC generates VSYNC events by calling setVsyncEnabled. SurfaceFlinger enables setVsyncEnabled to generate VSYNC events so that it can be synchronized with the screen’s refresh cycle. When SurfaceFlinger syncs to the screen refresh cycle, SurfaceFlinger disables setVsyncEnabled to prevent HWC from generating VSYNC events. If SurfaceFlinger detects a difference between the actual VSYNC and the VSYNC it previously created, SurfaceFlinger restarts the VSYNC event generation process.

VSYNC migration

Sync application and SurfaceFlinger render loop should be synchronized to hardware VSYNC. In the VSYNC event, the screen starts displaying frame N, and SurfaceFlinger starts compositing the window for frame N + 1. The application processes the waiting input and generates frame N+2.

Synchronization with VSYNC achieves a consistent delay time. It reduces errors in the app and SurfaceFlinger, and minimizes the offset between the screen and the outside of the phase. This assumes that the frame-to-frame times for apps and SurfaceFlinger don’t vary much. The delay is at least two frames.

To solve this problem, you can take advantage of the VSYNC offset to reduce the input device to the screen delay by correlating the application and compositing signals to the hardware VSYNC. This is possible because it usually takes less than 33 milliseconds to apply compositing.

The result of VSYNC offset is three signals with the same period and offset phase:

  • HW_VSYNC_0 – The screen starts displaying the next frame.
  • VSYNC – The application reads the input and generates the next frame.
  • Sf_vsync-surfaceflinger starts composition for the next frame.

With VSYNC offset, SurfaceFlinger receives the buffer and composes the frame, while the application processes the input and renders the frame.

DispSync

DispSync maintains the screen’s model of periodic VSYNC events based on hardware and uses this model to perform callbacks at specific phase offsets of hardware VSYNC events.

DispSync is a software phase-locked loop (PLL) that can generate VSYNC and SF_VSYNC signals used by Choreographer and SurfaceFlinger, even without offsets from hardware VSYNC.

DispSync has the following characteristics:

  • See -hw_vsync_0.
  • Output -vsync and SF_VSYNC.
  • Feedback – The exit fence from the hardware hybrid renderer has a signal status timestamp.

Second, data flow

For a description of the Android graphics pipeline, see the following figure:

The objects on the left are renderers that generate graphic buffers, such as the home screen, the status bar, and the system interface. The SurfaceFlinger is the synthesizer, while the hardware blending renderer is the builder.

BufferQueue

BufferQueues are the glue between Android graphics components. They are a pair of queues that mediate a fixed period of the buffer from the producer to the consumer. Once the producer hands over its buffer, SurfaceFlinger takes care of synthesizing everything into the display.

The consumer creates and owns the BufferQueue data structure and can exist in a different process from its producer. When the producer needs a buffer, it requests an available buffer from the BufferQueue by calling dequeueBuffer(), specifying the buffer’s width, height, pixel format, and usage tag. The producer then populates the buffer and returns it to the queue by calling queueBuffer(). Next, the user acquireBuffer() takes the buffer and uses its contents. When the consumer is done, it returns the buffer to the queue by calling releaseBuffer(). The synchronization framework controls how buffers move through the Android graphics pipeline.

Some characteristics of a BufferQueue, such as the maximum number of buffers that can be held, are determined jointly by producers and consumers. However, BufferQueue allocates buffers as needed. Buffers are retained unless features change; For example, if a producer requests buffers of different sizes, the system frees the old buffers and allocates new buffers as needed.

BufferQueue never copies the contents of the buffer, because moving so much data is a very inefficient operation. Instead, buffers are always passed through a handle.

For the BufferQueue communication process, see the following figure.

The BufferQueue contains the logic that combines the producers of the image stream with the consumers of the image stream. Some examples of image producers include camera previews generated by camera HAL or OpenGL ES games. Some examples of image consumers include SurfaceFlinger or another application that displays OpenGL ES streams, such as the Camera app that displays the camera viewfinder.

A BufferQueue is a data structure that combines buffer pools with queues and uses Binder IPC to pass buffers between processes. The producer interface, or something you pass to someone who wants to generate a graphics buffer, is IGraphicBufferProducer (part of the SurfaceTexture). BufferQueue is typically used to render to Surface and consume content along with GL consumers and other tasks. BufferQueue can run in three different modes:

Class-synchronous mode – By default, BufferQueue runs in class-synchronous mode, where every buffer entered from the producer exits at the consumer. No buffers are discarded in this mode. If the producer is too fast and creates buffers faster than it consumes them, it will block and wait for available buffers.

Non-blocking mode – BufferQueue can also run in non-blocking mode, in which case it generates an error rather than waiting for the buffer. Buffers are also not discarded in this mode. This helps avoid potential deadlocks in applications that may not understand the complex dependencies of the graphical framework.

Discard mode – Finally, BufferQueue can be configured to discard old buffers rather than generate errors or wait. For example, if a GL rendering is performed on a texture view and is drawn as quickly as possible, the buffer must be discarded.

To perform most of this work, SurfaceFlinger works just like another OpenGL ES client. For example, when SurfaceFlinger is actively synthesizing a buffer or two buffers into a third buffer, it uses OpenGL ES.

Hardware Composer HAL does the other half of the work. This HAL acts as the central point for all Android graphics rendering.

References:

  1. Source. The android. Google. Cn/devices/gra…