Android rendering mechanism — (Display principle process analysis)

Frame rate

Frames per second (FPS) or frame rate is the number of consecutive times (rate) a bitmap image appears on the display in frames per second. It’s basically how many images are displayed on the screen per second.

Why 60fps?

The interface refresh rendering relies on the underlying VSYNC signal, which is sent to the upper layer at a rate of 60 times per second.

The performance goal for app development is to maintain 60 FPS, which means you only have 1000/60=16.67ms per frame to process all the tasks. ** if there is no way to complete the task within 16ms, a frame will be lost. ** The more frames are lost, the worse the user will feel the lag.

OpenGL ES

Is a 3D API for handheld embedded devices, a cross-platform, fully functional 2D and 3D graphics API, with a fixed set of rendering pipeline.

DisplayList

On Android, the XML layout file is converted into an object that the GPU can recognize and draw. This is done with the help of DisplayList.

DisplayList holds all the data information that will be given to the GPU to draw onto the screen.

Frequency and performance

The main cause of performance problems such as lag perceived by most users is the rendering performance, that is, there is no way to complete the task of this frame within 16ms, and the phenomenon of frame loss occurs. From the perspective of designers and products, they want the App to have more fashionable elements such as animations and pictures to achieve a smooth user experience. But the Android system probably won’t be able to do the complicated interface rendering in time. The Android system signals VSYNC every 16ms to trigger a UI rendering. If each rendering is successful, this will achieve the 60fps required for smooth graphics. In order to achieve 60fps, this means that most of the application’s operations must be done in 16ms.

If an operation of the App takes 24ms, the system will not be able to render properly when it gets the VSYNC signal, and thus frame loss occurs. Then the user will see the same frame for 32ms.

Generally, the more complex the UI, the more likely it is to cause stutter. For example, the user is easy to perceive stutter when the UI executes animation or slides ListView, because the operation here is relatively complex and easy to lose frames, thus feeling stutter. Frames can be lost for many reasons. Maybe your layout is too complex to render in 16ms, maybe your UI has too many layers on top of each other, or maybe the animation is executed too many times. The CPU or GPU is overloaded.

The relationship between rendering and CPU and GPU

Due to the different design of CPU and GPU, CPU is better at complex logic control, while GPU is better at math operation thanks to a large number of ALU and parallel structure design. In Android, pages are made up of a variety of base elements (DisplayList) and render requires a lot of floating point arithmetic, which is better left to the GPU. Android introduced hardware acceleration in Android 3.0 to improve the performance of view rendering. As a result, Android has two drawing models: software-based drawing model and hardware accelerated drawing model.

In Android, CPU and GPU are divided differently:

  • CPU is mainly responsible for calculation operations including Measure, Layout, Record and Execute.

  • The GPU is primarily responsible for **Rasterization ** operations.

Rasterization refers to the process of converting an image represented in vector graphics format into a bitmap (pixel) for display device output. In simple terms, we convert the view we want to display into a format represented in pixels. Rasterization is a very time consuming task.

Comparison of CPU and GPU responsibilities

To briefly illustrate the CPU and GPU responsibilities, take a look at the following figure:

The figure briefly illustrates the responsibilities of the CPU and GPU, as well as possible problems and solutions.

CPU problems: unnecessary layouts and failures

On the CPU side, the most common performance issues are unnecessary layouts and failures that must be measured, cleaned up, and recreated in the view hierarchy. There are two common causes of this problem:

  • Too many times to rebuild the display list. (DisplayList)

  • Spending too much time invalidating view hierarchies and unnecessarily redrawing.

These two causes cause the CPU to overwork when updating display lists or other cached GPU resources.

Gpu-generated issues: Overdraw

On the GPU side, the most common problem is what we call overdraw, which is wasting GPU processing time during the pixel coloring process while doing late coloring through other tools. Let’s optimize for two major issues that arise from the GPU and CPU.

The drawing process of GPU is the same as that of wall painting, which is carried out one layer at a time of 16ms. In this way, layer coverage will be caused, that is, useless layers are also drawn at the bottom layer, resulting in unnecessary waste.

Overdraw viewer tools

In the mobile developer options, there are OverDraw detection tools, modulated GPU OverDraw tools,

The colors represent the layers rendered, representing 1 layer, 2 layer, 3 layer and 4 layer overlay respectively.

Sometimes it’s because your UI layout has a lot of overlap, and sometimes it’s because you don’t have to have overlapping backgrounds. For example, an Activity has a background, and a Layout inside it has its own background, while sub-views have their own background. This reduces a lot of the red Overdraw area and increases the proportion of the blue area, simply by removing the background image that is not necessary. This can significantly improve application performance.

Calculate the rendering time

Any time a View’s drawing changes, it will re-create the DisplayList, render the DisplayList, update it to the screen, and so on. The performance of this process depends on the complexity of your View, the state changes of the View, and the performance of the render pipeline.

To optimize the

On Android, theme-provided resources, such as Bitmaps and Drawables, are packaged together into a uniform Texture and then passed to the GPU, which means that every time you need to use these resources, you get the rendering directly from the Texture.

Flat processing to prevent over drawing

When the background is unavoidable, use color.transparent

Since color.transparent is not rendered, it is TRANSPARENT.



/ / before optimization
// Before optimization: When the image is not empty,ImageView loads the image and sets the background uniformly
Bean bean=list.get(i);
 if (bean.img == 0) {
            Picasso.with(getContext()).load(bean.img).into(holder.imageView);
        }
        chat_author_avatar.setBackgroundColor(bean.backPic);


/ / after optimization
// Optimized: When the image is not empty,ImageView loads the image and sets the background to TRANSPARENT;
// When the image is empty, the ImageView loads TRANSPARENT and sets the background to no photo background
Bean bean=list.get(i);
 if (bean.img == 0) {
            Picasso.with(getContext()).load(android.R.color.transparent).into(holder.imageView);
            holder.imageView.setBackgroundColor(bean.backPic);
        } else {
            Picasso.with(getContext()).load(bean.img).into(holder.imageView);
            holder.imageView.setBackgroundColor(Color.TRANSPARENT);
        }



Copy the code

Where do mobile phone pixels come from?

Frame Buffer and Back Buffer

The screen of a mobile phone is made up of many pixels, which can be combined into a variety of images by having each pixel display a different color. Where does the color data for these pixels come from?

In a Buffer controlled by the GPU, this Buffer is called a Frame Buffer. It can be simply understood as a two-dimensional array. Each element in the array corresponds to a pixel on the mobile phone screen, and the value of the element represents the color to be displayed by the corresponding pixel on the screen.

The optimized screen is constantly changing, requiring this buffer to constantly update data. A FramBuffer is too much to handle.

So in addition to the Back Buffer, which you hand to the phone screen for drawing, the GPU also has a Buffer called the Back Buffer (which you hand to your app) that you fill with data.

The GPU will periodically swap the Back Buffer and Frame Buffer, that is, turn the Back Buffer into a Frame Buffer for the screen to draw, Turn the original Frame Buffer into a Back Buffer for your application to draw. The switching rate is also 60 times per second, which keeps pace with the refresh rate of the screen hardware circuit.

GPU

How does Android use the GPU for rendering?

One straightforward question is: How is the Activity’s image drawn to the screen? How can complex XML layout files be identified and plotted?

Resterization is the basic operation for drawing buttons, Shapes, paths, Strings, bitmaps, etc. It splits those components into different pixels for display. This is a time-consuming operation, and the GPU was introduced to speed up rasterization.

CPU

The CPU computes the UI components into Polygons, textures, and hands them over to the GPU for rasterization.

Here’s a question: how does the CPU pass data to the GPU?

Moving textures from the CPU to the GPU is a hassle, but OpenGL ES can Hold those textures in GPU Memory for rendering the next time. So if you update the texture content held by the GPU, the previously saved state is lost.

On Android, theme-provided resources such as Bitmaps and Drawables are bundled together into a uniform Texture that is then passed to the GPU, which means that every time you need to use these resources, you get the rendering directly from the Texture. Of course, as UI components get richer, there are more forms of evolution. For example, when a picture is displayed, it needs to be loaded into memory after calculation by the CPU, and then transferred to the GPU for rendering. The text display is complicated. It needs to be converted into texture by CPU, and then rendered by GPU. When the CPU draws a single character, the content rendered by GPU is re-referenced. Animation is a more complex process.

Rendering flow line:

Android Rendering

The Android system uses a UI architecture called Surface to provide the user interface for applications.

In Android applications, each Activity component is associated with one or more Windows, and each window should have a Surface. With the Surface, applications can render Windows’ UI on top of it.

Finally, these completed surfaces will be submitted to SurfaceFlinger, the Surface management service, for synthesis, and finally displayed on the screen.

Both applications and SurfaceFlinger can use hardware such as gpus for UI rendering to achieve smoother UI.

Summary: The Android application calls the SurfaceFlinger service to render the measured, laid out, and drawn Surface onto the display screen.

Key members:

  • ViewRootImpl: Used to control window rendering and communicate with WindowManagerService and SurfaceFlinger.

  • WindowManager: WindowManager controls window objects, which are containers used to hold view objects. Window objects are always supported by Surface objects. WindowManager oversees the life cycle, input and focus events, screen orientation, transitions, animation, positioning, deformation, Z-axis order, and many other aspects of Windows.

    WindowManager sends all window metadata to SurfaceFlinger so that SurfaceFlinger can use this data to compose the Surface on the screen.

  • Surface: Each window of an Android application corresponds to a Canvas, which is a Surface, which can be understood as a window of an Android application. Surface is an interface for exchanging buffers between producers and consumers.

  • SurfaceView: The SurfaceView is a component that can be used to embed additional composition layers in the View hierarchy. The SurfaceView takes the same layout parameters as other views, so you can operate on it just like any other View, but the SurfaceView’s contents are transparent. When the View component of the SurfaceView is about to become visible, the framework asks the SurfaceControl to request a new Surface from the SurfaceFlinger.

  • BufferQueue: The BufferQueue class connects the component (producer) that can generate a buffer of graphical data to the component (consumer) that receives the data for display or further processing. Almost everything that moves the graphical data buffer in the system depends on the BufferQueue.

  • SurfaceFlinger: Android system service that manages the Frame buffer of the Android system, i.e. the display screen. EGLSurface and OpenGL ES: OpenGL ES (GLES) defines the graphics rendering API used in conjunction with EGL. EGI is a library that specifies how Windows are created and accessed from the operating system (to draw texture polygons, use the GLES call; To bring the render to the screen, use the EGL call.

  • Vulkan: Vulkan is a low-overhead, cross-platform API for high-performance 3D graphics. Like OpenGL ES, Vulkan provides tools for creating high-quality real-time graphics in your applications.

Android graphic architecture diagram

Android graphical architecture diagram, showing how key components work together:

The graphical system uses a producer-consumer model. According to the block diagram, the basic operation flow can be seen:

The graphics producer creates a Surface and draws the graphics data onto the Surface.

The graphics buffer used by Surface is allocated via Gralloc.

The graph producer’s window information is managed by Windows Manager, which sends window metadata to SurfaceFlinger for composition.

Graphic consumers process data generated by graphic producers. When graphical data is used for on-screen display, SurfaceFlinger uses window metadata to compose graphical buffers onto the display.

The final rendering of graphic data to the display device is done by Hardware Composer. Hardware generally supports multiple display layers. If the display layer meets Surface requirements, all visible surfaces can be sent to Hardware Composer for synthesis. You can also use the GPU to complete some Surface data synthesis work in advance, and then send it to Hardware Composer.

Image stream producer

An image stream producer can be anything that generates a graphics buffer for consumption. Examples include OpenGL ES, Canvas 2D, and Mediaserver video decoders.

Image flow consuming party

The most common consumer of image streams is SurfaceFlinger, a system service that consumes currently visible surfaces and synthesizes them into the display using information provided in the window manager. SurfaceFlinger is the only service that can modify parts of what is displayed. SurfaceFlinger uses OpenGL and Hardware Composer to compose a set of surfaces.

Other OpenGL ES applications can also consume image streams, such as camera apps that consume camera preview image streams. Non-gl applications can also be users, such as the ImageReader class.

Hardware hybrid renderer

Displays the hardware abstract implementation of the subsystem. SurfaceFlinger can delegate some composition work to Hardware Composer to share the workload on OpenGL and GPU. SurfaceFlinger just acts as another OpenGL ES client. Therefore, SurfaceFlinger uses OpenGL ES as it composes one or two buffers into a third buffer. This makes the resulting power consumption much lower than performing all the calculations through the GPU.

Hardware Composer HAL does the other half of the work and is at the heart of all Android graphics rendering. Hardware Composer must support events, one of which is VSYNC (the other is hot plug for plug-and-play HDMI).

Gralloc is used to allocate memory requested by the graphics producer.

Gralloc

A graphics memory allocator (Gralloc) is required to allocate the memory requested by the image producer. Typically SurfaceFlinger requests and stores the display content in a cache for HAL to use.

Next, let’s look at how Android renders an XML layout to the screen. We can simply divide the rendering process into two parts: application side rendering and system side rendering.

Application side drawing

An Android application window contains many UI elements organized in a tree structure. The parent view contains child views. The root view of a window is a DecorView object, and the ViewRootImpl is responsible for communicating with system processes.

Before drawing the UI of an Android application window, we first need to determine the size and position of the child UI elements within the parent UI element. The process of determining the size and position of each child UI element within the parent UI element is also known as the measurement process and layout process. Therefore, the UI rendering process of Android application window can be divided into three stages: measurement, layout and drawing, and finally generate Display List.

As shown below:

  1. Measurement: Recursively (depth-first) determine the size (height, width) of all views
  2. Layout: Recursively (depth-first) positions all views (upper-left coordinates)
  3. Draw: Draws all the views of the application window on the canvas
  4. Generate the Display List data.

Measurement and layout determine the size and location of each view component, followed by drawing.

Android has two drawing models: software-based drawing model and hardware-accelerated drawing model. See also: Android Rendering Mechanics — Drawing Models.

Android needs to convert the XML layout file into an object that the GPU can recognize and draw. This is done with the help of DisplayList. Display List holds all the data information that will be given to the GPU to draw onto the screen.

For details on Display List, please refer to the previous article: Android Rendering — Display List.

System side rendering

1. Submit Display List data to GPU for rendering

Android needs to convert the XML layout file into an object that the GPU can recognize and draw. This is done with the help of DisplayList. Display List holds all the data information that will be given to the GPU to draw onto the screen.

Display List is a Buffer that caches drawing commands. The essence of Display List is a Buffer that records the sequence of drawing commands to be executed. Render Display List, which occurs in the Render Thread of the application process. The Render Thread Thread is also added to prevent the UI Thread from being overloaded to improve rendering performance.

These draw commands are eventually converted into Open GL commands to be executed by the GPU. This means that when we call the Canvas API to draw the UI, we actually just record the Canvas API call and its parameters in the Display List, and then wait until the next VSYNC signal arrives, Draw commands recorded in the Display List will be converted to Open GL commands and executed by the GPU.

2. GPU rendering processing

Android uses the OpenGL ES (GLES) API to render graphics. After the GPU rasterizes the view, Surface is generated. The GPU, as the producer of the image stream, will display the content and finally send it to the SurfaceFlinger, the consumer of the image stream.

Most clients render to Surface using OpenGL ES or Vulkan (hardware accelerated, using GPU rendering). However, some clients use canvas rendering to Surface (without hardware acceleration).

3. Generate the Surface and store it in the BufferQueue

The Surface object enables the application to render the image to be displayed on the screen. Through the SurfaceHolder interface, apps can edit and control the Surface. Surface is an interface for exchanging buffers between producers and consumers.

SurfaceHolder is the interface that the system uses to share Surface ownership with applications. Some clients that work with Surface require a SurfaceHolder because the API for getting and setting Surface parameters is implemented through the SurfaceHolder. A SurfaceView contains a SurfaceHolder. Most of the components that interact with a View involve a SurfaceHolder. If you’re developing a camera-related app, there will be apis involved.

The buffer queues typically used by SurfaceFlinger are generated by the Surface, and when rendered to the Surface, the results will eventually appear in the buffer passed to the consumer. The Canvas API provides a software implementation (with hardware acceleration) for drawing directly on the Surface (a low-level alternative to OpenGL ES). Anything related to the view refers to the SurfaceHolder, whose API can be used to get and set Surface parameters such as size and format.

BufferQueue is the BufferQueue used by SurfaceFlinger, and Surface is the producer of BufferQueue. The BufferQueue class connects a component (the producer) that can generate a buffer of graphical data to a component (the consumer, such as SurfaceFlinger) that receives the data for display or further processing. Almost everything that moves the graphical data buffer in the system depends on the BufferQueue.

The BufferQueue used to display the Surface is usually configured as triple buffering. Buffers are allocated on demand, so if the producer generates buffers slowly enough (for example, at 30 FPS on a 60 FPS display), there may be only two allocated buffers in the queue. Allocating buffers on demand helps minimize memory consumption.

Data processing process:

The figure above depicts the flow of a graphics pipeline. Graph producers on the left and graph consumers on the right, connected by BufferQueues in the middle. In the figure, the main screen, status bar and system interface are rendered by GPU to generate graphics buffers, which are passed to BufferQueues as producers. The SurfaceFlinger, as the consumer, receives the notification from the BufferQueues and takes out the usable graphics buffer and sends it to the display end. In the example figure, the graphics buffers of the status bar and system interface are sent to the GPU for synthesis. After the new graphics buffers are generated, the new graphics buffers are sent to the hardware mixture renderer via BufferQueues.

Note: SurfaceFlinger creates and owns the BufferQueue data structure and can exist in a different process from its producer.

4. SurfaceFlinger sends display data to the display

SurfaceFlinger takes buffers of data from multiple sources, composts them and sends them to the display.

WindowManager provides buffers and window metadata to SurfaceFlinger, which SurfaceFlinger can use to compose the Surface onto the screen.

SurfaceFlinger accepts buffers in two ways: through BufferQueue and SurfaceControl, or through ASurfaceControl.

In the previous section, we introduced BufferQueue. ASurfaceControl is new to Android 10, which is another way SurfaceFlinger accepts buffers.

ASurfaceControl combines Surface and SurfaceControl into a transaction package that is sent to SurfaceFlinger. ASurfaceControl is associated with a layer that applications can update through ASurfaceTransactions. The application can then get information about ASurfaceTransactions via a callback (used to pass ASurfaceTransactionStats that contains information about the lock time, fetch time, and so on).

The interaction between SurfaceFlinger and WMS

One way SurfaceFlinger accepts buffers is through BufferQueue and SurfaceControl.

When the application comes to the foreground, it requests a buffer from WindowManager.

WindowManager then requests the layer from SurfaceFlinger. A combination of Surface (which contains BufferQueue) and SurfaceControl (which contains layer metadata such as screen frames).

SurfaceFlinger creates the layer and sends it to WindowManager.

WindowManager then sends the Surface to the application, but leaves the SurfaceControl in place to control how the application looks on the screen.

The screen sends a VSYNC signal to the SurfaceFlinger when the screen is between refreshes. The VSYNC signal indicates that the screen can be refreshed without tearing. When the SurfaceFlinger receives the VSYNC signal, the SurfaceFlinger traverses its layer list to find a new buffer. If SurfaceFlinger finds a new buffer, SurfaceFlinger gets the buffer; Otherwise, SurfaceFlinger will continue to use the same buffer that was acquired last time. SurfaceFlinger must always display content, so it keeps a buffer. If there is no commit buffer on a layer, that layer is ignored.

After SurfaceFlinger collects all the buffers in the visible layer, it asks the hardware blending renderer (HWC) how it should compose. If HWC marks the layer composition type as client composition, SurfaceFlinger will compose these layers. SurfaceFlinger then passes the output buffer to the HWC.

Note: For a more detailed explanation of SurfaceFlinger, check out the Android Rendering mechanism — SurfaceFlinger.

conclusion

  1. Frame rate is the number of consecutive times (rate) a bitmap image appears on the display in frames per second. It’s basically how many images are displayed on the screen per second.

  2. When Android is designed, the frame rate is limited to 60 frames per second. When the frame rate of our APP is 60 FPS, the picture will be very smooth.

  3. The interface refresh rendering relies on the underlying VSYNC signal, which is sent to the upper layer at a rate of 60 times per second, and a frame rate higher than 60 FPS is not necessary because the collaboration between the human eye and the brain cannot perceive updates beyond 60 FPS.

  4. The main cause of performance problems such as lag perceived by most users is the rendering performance, that is, there is no way to complete the task of this frame within 16ms, and the phenomenon of frame loss occurs.

  5. Due to the different design of CPU and GPU, CPU is better at complex logic control, while GPU is better at math operation thanks to a large number of ALU and parallel structure design. In The Android system, CPU and GPU are divided differently. CPU is mainly responsible for calculation operations including Measure, Layout, Record and Execute, while GPU is mainly responsible for Rasterization.

  6. Android introduced hardware acceleration in Android 3.0 to improve the performance of view rendering.

  7. In Android applications, each Activity component is associated with one or more Windows, and each window should have a Surface. With the Surface, applications can render Windows’ UI on top of it. Finally, these completed surfaces will be submitted to SurfaceFlinger, the Surface management service, for synthesis, and finally displayed on the screen. Both applications and SurfaceFlinger can use hardware such as gpus to render UI for smoother UI.

  8. To summarize the process of displaying an Android application, an Android application calls the SurfaceFlinger service to render the measured, laid out, and drawn Surface onto the display screen.

  9. The graphics producer creates a Surface and draws the graphics data onto the Surface. The graphics buffer used by Surface is allocated via Gralloc. The graph producer’s window information is managed by Windows Manager, which sends window metadata to SurfaceFlinger for composition. Graphic consumers process data generated by graphic producers. When graphical data is used for on-screen display, SurfaceFlinger uses window metadata to compose graphical buffers onto the display. The final rendering of graphic data to the display device is done by Hardware Composer.

  10. Android needs to convert the XML layout file into an object that the GPU can recognize and draw. This is done with the help of DisplayList. Display List holds all the data information that will be given to the GPU to draw onto the screen.

  11. Display List is a Buffer that caches drawing commands. The essence of Display List is a Buffer that records the sequence of drawing commands to be executed. Render Display List, which occurs in the Render Thread of the application process. The Render Thread Thread is also added to prevent the UI Thread from being overloaded to improve rendering performance.

  12. Android uses the OpenGL ES (GLES) API to render graphics. After the GPU rasterizes the view, Surface is generated. The GPU, as the producer of the image stream, will display the content and finally send it to the SurfaceFlinger, the consumer of the image stream.

  13. BufferQueue is the BufferQueue used by SurfaceFlinger, and Surface is the producer of BufferQueue. The BufferQueue class connects a component (the producer) that can generate a buffer of graphical data to a component (the consumer, such as SurfaceFlinger) that receives the data for display or further processing.

  14. SurfaceFlinger takes buffers of data from multiple sources, composts them and sends them to the display. WindowManager provides buffers and window metadata to SurfaceFlinger, which SurfaceFlinger can use to compose the Surface onto the screen.