SF’s first law of rendering:

SF is the core process for rendering the entire Android system. The rendering logic of all applications will be processed in SF eventually, and the processed image data will be handed over to CPU or GPU for drawing.

Let’s take this as the first law of Android rendering. SF does not play the role of rendering in the entire Android system, but acts as a pixel projector, processing the image metadata transmitted by all application processes and handing it to CPU and GPU for real rendering.

SF’s second law of rendering:

In each application, Surface is used as a pixel transfer unit to transfer graph metadata to SF server.

This is the second law of Android rendering. Put these two laws together and you get the following simple diagram.

SF third Law of Rendering:

SF takes producer and consumer as the core design idea. Every application process is stored in SF’s graph queue as a producer, and SF, as a consumer, stores the producer in SF’s queue according to certain rules.

It is shown as follows:

The fourth Law of SF rendering:

In order to transfer large volume of graph metadata across processes, anonymous shared memory memory is used as a tool to transfer graph metadata to SF for processing.

As we all know, we need to transfer the graph metadata from the application process to the SF process for processing, which requires cross-process communication. Socket, for example, is not a good choice because of its efficiency and two copies of data (from the physical memory page level). One that copies a large amount of data itself is a problem. Binder is the first one to come to mind. However, with Binder interprocess communication, especially with applications where the total amount of communication data is less than 1M plus other applications, there is bound to be a problem.

To solve this problem, Android uses shared memory, using anonymous shared memory (Ashmem). Anonymous shared memory is also a copy-once interprocess communication method whose core is closer to Linux’s concept of shared memory than binder’s complex MMAP.

The fifth Law of SF system rendering:

There is a time clock at the bottom of SF in a continuous loop, either emitted from hardware interrupts or from software simulations, and every once in a while the pixel queue in SF will be captured and drawn on the screen by CPU/GPU.

The birth of the fifth law actually conforms to the design situation of the Android system. In addition to the requirement that the Android application has a way to inform SF of the mode to be rendered, SF of course needs to constantly call back its own behavior of drawing primions to the screen, SF constantly draws the graph metadata in SF.

EventThread plays an extremely important role and is designed as follows in SF:

The introduction of Vsync

There is a new rank called VSync, which is what we call a VSync signal when we play the game. When I used to play games with a lousy computer, I often lost frames. After turning off the vertical signal, I felt better, which made me think it was a bad thing for a while.

Here’s a look at Android’s iterative efforts to make the UI experience better: Project Butter. The butter project was named so that the UI of the system would behave as smoothly as the butter surface. This led to two important concepts, Vsync and Triple Buffer, vertical signal and Triple Buffer.

The concept of double buffering, which you’re probably all familiar with, and I explained in OpenGL, double buffering is rendering the first frame while drawing the second frame, and then displaying it after the second frame is drawn. The advantage of doing this is obvious. If one frame is drawn before the next frame is drawn, there must be a computational process that causes UI interaction to slow down.

In this way, when displaying the previous frame, draw the next frame in advance and put it behind you to wait for the opportunity to swap, so that the sensory flow is much smoother.

This is ideal for display, but how to find the right time to swap the two frames is a problem? If you’re wondering, you can refresh the screen at a rate of 60fps, which is about 16ms.

The ideal is very rich, but the reality is very backbone, so there seems to be no problem, let’s think about it deeply, in fact, there are two variables in this process, one is the drawing speed, the other is the display speed. Even graphics speed is divided into CPU and GPU drawing speed.

Here’s a diagram of Google’s butter project. Let’s take a look at the diagram that works without buffering:

The best case is shown in the picture above, when displaying frame 0, the CPU/GPU composite drawing completes frame 1 in 16ms, and when the Vsync signal comes, the first frame is switched to the display.

What is Vsync? Vsync is one of the things you see a lot when you play games. Its purpose is to tell the system when the screen should be refreshed through screen hardware interrupts. In this way, it takes roughly 16ms to send an interrupt to refresh the system.

However, it is likely that the CPU is too busy to render the second frame after the first frame is displayed. Even if SF receives the signal from Vsync, it can only render the first frame on the screen. This repeats the first frame, which the Google development team calls Jank.

I can see that I’m showing frame 1 because I’m not ready for frame 2, so I have to repeat frame 1.

Let’s look at the workflow with multiple buffering:

You can see that instead of just frame 1 and frame 2, there’s A buffer, there’s A buffer. You can see that under normal circumstances, the A buffer is displayed first and the B buffer is prepared. When everything is ok, the B buffer should be ready before the next vsync comes. Once vsync arrives, the B buffer is displayed and the A buffer goes back to the background and continues drawing.

So what happens when this approach meets Jank?

If double buffering seems to be no problem, but once there is janK, then the display will continuously appear JanK. If buffer A is displayed and the preparation time of buffer B is more than 16ms, buffer A will be displayed repeatedly. When buffer B is displayed, buffer A may also have less than 16ms preparation time, so the drawing cannot be completed and only the contents of buffer B can be displayed repeatedly.

This approach is more dangerous, and to address this problem, Google introduced triple buffering.

When triple buffering is handled jank’s principle flow chart:

You can see that in order to avoid chain-type errors later on, triple buffering was introduced to allow more to be done with the idle wait time. After the double buffering meets Jank, once the TIME of B buffering CPU+GPU exceeds the time of the next vsync, it can be found that in fact, the CPU and GPU have nothing to do for a period of time. Only after the arrival of the next vsync will the whole system’s drawing chain appear JanK.

With the appearance of the triple buffer, the CPU will not wait for the buffer A to be displayed repeatedly, but will prepare the pixel of the C buffer, and then can connect the C buffer. This is where Google’s triple buffer comes in.

But for the most part, the buffer strategy is determined by the SF system itself, and it’s what we call double buffering, triple buffering.

In fact, this way can also be used in audio and video programming optimization, the common buffer design and here also have the same wonderful, but not so extreme system. If you have read the videoView source code of the system, you can see that NullPlayer is essentially using the Surface pixel buffer to achieve the ultimate experience, but videoView also has unreasonable design, after reading the Android rendering system, let’s analyze the source code.

But this part of knowledge is not enough to understand law 5. In fact, every time Vsync comes from hardware/software, Dispsync will try to send SF and APP to Dispsync. There is no problem at all.

In fact, this is the clever design of the system. What will happen if we inform APP and SF of the signal at the same time?

If at this time app returns primiprimies later, but SF has performed the behavior of refreshing composite drawing (it is very likely, because the transfer speed of primiprimies from APP to SF must be slower than sf notifits itself), then jank-like problems will occur, and the next Vsync will still display the current frame number, so the following time difference is required. Notify APP first and then SF, as shown below:

summary

These five laws are the core ideas that guide SF design and have not changed much from Android4.1 to 9.0. As long as we grasp these five core ideas, it will be much easier for us to read SF.

Let’s break it down by character:

  1. Framework for developers all View is easy to develop controls, which only provides the current View of various attributes and functions.
  2. The Android is the underlying Skia Android for the brush on the screen, after the View drawing process of ontouch method callback, the need to draw something by Skia map into pixels yuan saved
  3. SF is the last to accept the drawing result of Skia and draw it on the screen.

So it is true that Skia is the core of Android rendering, but in the end, it is Skia and the system that provide a complete Android rendering system.

This layer of masking allows developers to draw well without any understanding of Android’s underlying rendering system.

The result is finally transferred to the screen.

plan

    1. Figure core transfer tool, anonymous shared memory Ashmem driver core principle,ashmem schematic diagram is roughly as follows:

However, in higher versions of Android, ashemem has been dropped in favor of ion drivers. The schematic diagram of ion is as follows:

Ion actually generates DMA direct access to memory. The original Ashmem approach required access from the GPU to the CPU and then to the address in memory. But in this case, the GPU can directly modify DMA, and the CPU can also modify DMA directly. That’s the big change.

    1. Launch of SurfaceFlinger.

    1. There is no Activity after boot, so we can only use SF mechanism and OpenGL es to display BootAnimation. Let’s look at BootAnimation from Linux BootAnimation to Android BootAnimation.

    1. Understand how application processes relate to SF. \

SF knows the draw cycle by listening to VSync through a single agent called Choreographer. The schematic diagram is roughly as follows:

    1. Understand and operate the SF hardware abstraction layer Hal and understand how SF relates to the underlying HWC/ FB drivers.

The correlation between underlying hardware callback and SF is shown as follows:

    1. SF is how to connect to DisplayManagerService and simply get screen information from SurfaceFlinger and put it in the Framework layer.
    1. Android end in opengL ES core principle, look at Android on OpengL ES do what encapsulation. This module is divided into two parts: one is the normal Process of using OpenGL ES, software simulation of each key step of the working principle, Android in which what optimization.

One of the key data structures is shown in the UML diagram below:

A texture in OpenGL ES is how to synthesize and draw, and Android local texture optimization, the whole OpenGL ES drawing principle is as follows:

    1. How primitives produce graph metadata through Hal layer; How to get the application’s graph metadata into SurfaceFlinger’s buffer queue.

The schematic diagram of simultaneous operation is as follows:

    1. How the graph metadata of the application is consumed. Swap buffer drawing parameters, essentially taking out a GraphicBuffer stored in the buffer queue and the current time is expected to display the closest one, rendering to the screen. Meanwhile, the GraphicBuffer from the previous frame is placed in the idle queue.

We need to remember the following data structure for the buffer queue design in SF:

    1. SF is how to combine layers through HWC, how to combine layers, output to Opengles processing. It can be roughly divided into the following 7 steps: 2. RebuildLayerStacks rebuild the Layer stack 3. SetUpHWComposer HWC rendering or preparation of these three steps, I call rendering preparation, For details, see Figure Composition (1). Preparing for drawing During preparing for drawing, it is important to distinguish the following drawing modes and store related data in THE Hal layer of HWC.
Type of Composition Layer hasClientComposition hasDeviceComposition rendering
HWC2::Composition::Client true – OpenGL es
HWC2::Composition::Device – true HWC
HWC2::Composition::SolidColor – true HWC
HWC2::Composition::Sideband – true HWC or OpenGL ES

DoDebugFlashRegions open the debug drawing mode. 5. DoTracing tracing and printing. 6.

For the next four steps, we only need to focus on the last two. See the composition of primitives (part 2)

The schematic diagram of the whole process from consumption to synthesis is roughly as follows:

During the composition process, HWC and OpenGL ES are divided into two types. Their roles are as follows:

Of course, there’s more to Android rendering than just a pair of producer-consumer models:

    1. There are three types of Vsync sending cycles: hardware sending Vsync cycle, software sending Vsync cycle, APP processing Vsync cycle, and SF processing Vsync cycle. For details, see Vsync Synchronization signal Principles

For convenience, Android temporarily regards the whole cycle as a function of period continuity, and the calculation principle is as follows:

In fact, it is to obtain the phase of each sampling point, and calculate the average of the phase of the sampling point is the ideal phase. Similarly, the period is calculated as the average period of the sampling points to calculate an appropriate software send VSync axis.

Finally, on the basis of software rendering, VSync of APP and VSync of SF carry out delay acceptance processing respectively to avoid timing conflict of law, as shown in the blue picture above.

    1. To understand the fence, it is necessary to understand the state changes of GraphicBuffer: Dequeue (queue to draw in application), Queue (queue to SF buffer for consumption), acquire(select rendered GraphicBuffer), free(wait for dequeue after consumption)

The state of Fence is simpler, with acquire, release, and retried state flows as follows:

Retried is that every time a Fence is drawn, it is combined and recorded in a different Fence.

In summary, the acquire state of a Fence blocks when it can be consumed and when it can be rendered to the screen; The release state of a Fence controls when it can be lined up for application drawing and when it can be mapped to memory.

Only by understanding these 12 points can you say that you understand SF, but you can’t say that you are proficient. After all, you can’t type it out blindly.

After all these 12 points are understood, we will start a new chapter of Skia, talking about how Skia works and source code parsing. Finally, we will go back to the roots and talk about the View drawing process and WMS.

This is just a guide, hidden behind the 12 knowledge points a lot of things, I hope a general outline can let a person have a general grasp, not lost in the source code.

Author: yjy239 links: www.jianshu.com/p/c954bcceb… The copyright of the book belongs to the author. Commercial reprint please contact the author for authorization, non-commercial reprint please indicate the source.