1 introduction

When it comes to Android rendering, you might think of measurement, layout, and drawing. But how does our view actually get to the screen step by step? What exactly is CPU/GPU rendering for App? What is OpenGL/Vulkan/ Skia? What about surfaceFlinger and HAL?

With these questions in mind, let’s go deep into the entire process of Android drawing today.

Using the layering idea, we will roughly divide the entire render into App layer and SurfaceFlinger layer. We will first talk about what each layer does and then connect the two.

2 Related Concepts

2.1 Vsync signal

Generated by the system device. Assume that on a 60HZ screen, the screen will scan every 16ms, and there will be an interval between the two scans. At this time, the system will send Vsync signal to notify APP (vsync-App) for rendering, and SurfaceFlinger (vsync-SF) for swap buffer display. Therefore, as long as the App’s rendering process (CPU calculation +GPU drawing) does not exceed 16ms, the image will look smooth.

Description:

  • Vysnc signals are generated by hardware if the system detects hardware support, otherwise by software simulation. This understanding can be.
  • Vsync offset mechanism: vsync-app and vsync-sf are not notified at the same time, vsync-sf will be relatively late, but for us app developers, it can be considered as approximately simultaneous.

2.2 OpenGL, Vulkan, Skia

  • OpenGL is a cross-platform 3D graphics rendering specification interface. OpenGL EL is optimized for embedded devices such as mobile phones.
  • Vulkan: Has the same functionality as OpenGL, but it supports 3D and 2D at the same time. It is lighter and has higher performance than OpenGL.
  • Skia: Skia is the image rendering library, 2D graphics can be done by themselves. 3D effects (hardware dependent) supported by OpenGL, Vulkan, Metal. It supports not only 2D and 3D, but also CPU software drawing and GPU hardware acceleration. Android and Flutter use it to draw.

2.3 the GPU and OpenGL

OpenGL is the specification, and GPU is the specific device implementer of the specification.

2.4 Surface and Graphic Buffer

Insert a question: How many Windows does an Android application have?

A: Activity corresponds to an application Window, dialog corresponds toa child window, and toast corresponds toa system window. Therefore, there could theoretically be an infinite number of Windows.

In addition, the phone’s top statusBar and bottom menu bar (statusBar+menu) also correspond to a window.

An Android Window corresponds to a Surface, and a Surface corresponds to a BufferQueue. But: A surface doesn’t necessarily correspond to a Window. For example, the surfaceView encapsulates the Surface, and the drawing operation is in the child thread, but it belongs to the View.

Therefore, an application can have multiple surfaces.

Canvas is obtained by surface.lockCnavas (eventually, the surface.lock method of JNI framework layer is called to obtain the graphic buffer).

The surface gets the graphic buffer from the Dequeue and renders it. After rendering, the surface returns to the BufferQueu queue and finally notifts surfaceFlinger to consume it.

2.5 What is SurfaceFlinger?

You can think of it as the coordinator for coordinating buffer data and device display. Vsync signals, triple buffering, and buffer synthesis are controlled by it.

3 Android rendering evolution

It’s helpful to understand the history of Android’s constant optimization of rendering.

3.1 the Android 4.1

Introduced Project Butter: Vsync, triple Buffering, Choreography Dancer.

3.2 the android 5.0

The RenderThread thread (which is maintained by the system at the framework level) was introduced, leaving the previously direct rendering instructions from the CPU (OpenGL/ Vulkan/Skia) to a separate rendering thread. Reduce main thread work. Even if the main thread is stuck, rendering is not affected.

3.3 the Android 7.0

Vulkan support was introduced. OpenGL is a 3D rendering API, VulKan is used to replace OpenGL. It supports BOTH 3D and 2D and is much more lightweight.

4. What does App do (key points)

4.1 How is an Activity displayed?

The display of activities is divided into root activities (cold start) and normal activities.

The root Activity is initiated by the Luancher desktop application. Normal activities are initiated by the current application and eventually call the startActivity() method in the Activity.

The start of the root Activity is relatively complex and involves communication between processes. This analysis of the root Activity start, normal Activity start actually included in the inside.

Brief process of App startup:

4.1.1. What does clicking the desktop App icon Launcher do

When the App icon is clicked on the desktop, the Laucher process calls startActivitySafely(), then activityForResult (), To instrumentation. ExecStartActivity (),

public ActivityResult execStartActivity( Context who, IBinder contextThread, IBinder token, Activity target, Intent intent, int requestCode, Bundle options) { //... Omitting // Cross-process communication to AMS with binder // this carries information, AMS told me to start the Activity in the process of information, the package name int result = ActivityManager. GetService () startActivity (whoThread, who.getBasePackageName(), intent, intent.resolveTypeIfNeeded(who.getContentResolver()), token, target ! = null ? target.mEmbeddedID : null, requestCode, 0, null, options); checkStartActivityResult(result, intent); }Copy the code

At this point, the calling process leaves the Launcher process and enters AMS in the SystemServer process.

4.1.2, the Launcher to AMS

AMS internal calls to ActivityStackSupervisor. StartSpecificActivityLocked () :

void startSpecificActivityLocked(ActivityRecord r, boolean andResume, boolean checkConfig) { // Is this activity's application already running? ProcessRecord app = mService.getProcessRecordLocked(r.processName, r.info.applicationInfo.uid, true); r.getStack().setLaunchTime(r); If the target process is already started, start the Activity directly. if (app ! = null && app.thread ! = null) { try { if ((r.info.flags&ActivityInfo.FLAG_MULTIPROCESS) == 0 || !" android".equals(r.info.packageName)) { // Don't add this if it is a platform component that is marked // to run in multiple processes, because this is actually // part of the framework so doesn't make sense to track as a // separate apk in the process. app.addPackage(r.info.packageName, r.info.applicationInfo.versionCode, mService.mProcessStats); } realStartActivityLocked(r, app, andResume, checkConfig); return; } catch (RemoteException e) { Slog.w(TAG, "Exception when starting activity " + r.intent.getComponent().flattenToShortString(), e); } // If a dead object exception was thrown -- fall through to restart the application. Here goes back to the AMS class startProcessLocked mService () method. The startProcessLocked (r.p rocessName, r.i show nfo. ApplicationInfo, true, 0, "activity", r.intent.getComponent(), false, false, true); }Copy the code

When the root Activity is started, the process must not exist. (I gather, you leave me? When will you be calling realStartActivityLocked?

ActivityManagerService.java

private final void startProcessLocked(ProcessRecord app, String hostingType, String hostingNameStr, String abiOverride, String entryPoint, String[] entryPointArgs) { // ... StartResult = process. start(entryPoint, app.processName, uid, uid, gids, debugFlags, mountExternal, app.info.targetSdkVersion, seInfo, requiredAbi, instructionSet, app.info.dataDir, invokeWith, entryPointArgs); / /... Omit code}Copy the code

4.1.3 AMS Starts application processes

Process.start will exit the application Process from the Zygote incubation Process and finally execute the static main method of the application Process’s Java class activityThread. Java. Note that this is done through socket communication.

4.1.4 Initialization of the target App process

In the main method, the main thread looper is created and started. Create an ActivityThread object and call ActivityThread.attach() to bind ApplicationThread to AMS, telling AMS that the application process has started.

private void attach(boolean system) { final IActivityManager mgr = ActivityManager.getService(); Try {// tell AMS that the application and main threads are initialized // ApplicationThread is the communication link between application and SystemServer. AttachApplication (mAppThread); } catch (RemoteException ex) { throw ex.rethrowFromSystemServer(); }}Copy the code

4.1.5, AMS calls back the App process lifecycle method

AMS attachApplication(IApplicationThread thread)

@Override public final void attachApplication(IApplicationThread thread) { synchronized (this) { int callingPid = Binder.getCallingPid(); final long origId = Binder.clearCallingIdentity(); attachApplicationLocked(thread, callingPid); Binder.restoreCallingIdentity(origId); }}Copy the code

The attachApplicationLocked () method does the following:

  1. Callback of the application processApplicationThread.bindApplication()Handler sends a message to the ActivityThread, which is eventually calledApplication. The onCreate ()Methods.
  2. Check whether the process needs to start the Activity, and finally we are back to the Activity start… It’s too difficult!
private final boolean attachApplicationLocked(IApplicationThread thread, int pid) { //.. //1. Call the application.onCreate () method if (app.instr! = null) { thread.bindApplication(processName, appInfo, providers, app.instr.mClass, profilerInfo, app.instr.mArguments, app.instr.mWatcher, app.instr.mUiAutomationConnection, testMode, mBinderTransactionTrackingEnabled, enableTrackAllocation, isRestrictedBackupMode || ! normalMode, app.persistent, new Configuration(getGlobalConfiguration()), app.compat, getCommonServicesLocked(app.isolated), mCoreSettingsObserver.getCoreSettingsLocked(), buildSerial); } else { thread.bindApplication(processName, appInfo, providers, null, profilerInfo, null, null, null, testMode, mBinderTransactionTrackingEnabled, enableTrackAllocation, isRestrictedBackupMode || ! normalMode, app.persistent, new Configuration(getGlobalConfiguration()), app.compat, getCommonServicesLocked(app.isolated), mCoreSettingsObserver.getCoreSettingsLocked(), buildSerial); // See if the top Visible activity is waiting to run in this process... If (normalMode) {try {** if (normalMode) {try {** ** if (mStackSupervisor.attachApplicationLocked(app)) { didSomething = true; } } catch (Exception e) { Slog.wtf(TAG, "Exception thrown launching activities in " + app, e); badApp = true; }}}Copy the code

4.1.6, AMS starts the Activity

Application has been launched, the AMS follow-up will ApplicationThread. Calling through the Binder scheduleLaunchActivity (), thus the callback to the main thread of performLaunchActivity (), It calls the activity’s onCreate () and onResume () methods.

4.1.7 App process displays Activity

. Eventually, the Activity will pass WindowMnagerGlobal addView () method combined decorView added to the list, then call ViewRootImpl. SetView (view) method, Finally, call requestLayout() to complete the measurement, layout, and drawing process.

4.2 Render Entry

From the summary above, we know that the display of the Activity will eventually call the requestLayout () method.

When we want to redraw a view, we call the invalidate () method (which triggers the view.ondraw () method on the next frame when the onVsync signal arrives).

Invalidate () invalidates the drawing cache, which is called dirty, so it needs to be redrawn.

4.2.1 Relationship between View and ViewRootImpl

ViewRootImpl binds itself to the View’s ViewParent member when it calls the setView () method

ViewRootImpl

/** * We have one child */ public void setView(View view, WindowManager.LayoutParams attrs, View panelParentView) { //... // Finish binding view.assignparent (this); mAddedTouchMode = (res & WindowManagerGlobal.ADD_FLAG_IN_TOUCH_MODE) ! = 0; mAppVisible = (res & WindowManagerGlobal.ADD_FLAG_APP_VISIBLE) ! = 0; if (mAccessibilityManager.isEnabled()) { mAccessibilityInteractionConnectionManager.ensureConnection(); }}Copy the code

View.java

void assignParent(ViewParent parent) { if (mParent == null) { mParent = parent; } else if (parent == null) { mParent = null; } else { throw new RuntimeException("view " + this + " being added, but" + " it already has a parent"); }}Copy the code

When the View’s invalidate () method is called, the invalidateChild () method of the viewParent member variable is called, which eventually switches to the ViewRootImpl invalidate () ->scheduleTraversals () method.

// Propagate the damage rectangle to the parent view. final AttachInfo ai = mAttachInfo; final ViewParent p = mParent; if (p ! = null && ai ! = null && l < r && t < b) { final Rect damage = ai.mTmpInvalRect; damage.set(l, t, r, b); p.invalidateChild(this, damage); }Copy the code

So, let’s take a look at methods: ViewRootImpl. ScheduleTraversals () :

ViewRootImpl.java

void invalidate() {
    mDirty.set(0, 0, mWidth, mHeight);
    if (!mWillDrawSoon) {
        scheduleTraversals();
    }
}
Copy the code
@Override
public void requestLayout() {
    if (!mHandlingLayoutInLayoutRequest) {
        checkThread();
        mLayoutRequested = true;
        scheduleTraversals();
    }
}
Copy the code
void scheduleTraversals() { if (! mTraversalScheduled) { mTraversalScheduled = true; mTraversalBarrier = mHandler.getLooper().getQueue().postSyncBarrier(); // Want the choreographer to send a callback, in a frame callback. mChoreographer.postCallback( Choreographer.CALLBACK_TRAVERSAL, mTraversalRunnable, null); if (! mUnbufferedInputDispatch) { scheduleConsumeBatchedInput(); } notifyRendererOfFramePending(); pokeDrawLockIfNeeded(); }}Copy the code

The viewRootImpl invalidate() method postcalbacks to the Choreography class.

Choreography registers the vsync signal for the listening system when the View arotimPL is created.

When onVsync calls back to the next frame, choreography.doframe () will be executed, and then callback will be executed, calling viewRootImpl’s performTraversal()–doTraversal() method, Thus onMeasure(), onLayout() and onDraw() are implemented.

4.3 What does the UI thread’s draw() method actually do

Because both performMeasure() and performLayout() are still calculating the size of the view and the position of the layout from the CPU, the actual drawing starts with perfomDraw ().

ViewRootImpl’s draw () method, which calls drawSoftware () :

private boolean drawSoftware(Surface surface, AttachInfo attachInfo, int xoff, int yoff, boolean scalingRequired, Rect dirty) {// omit code.. // Draw with software renderer. final Canvas canvas; try { final int left = dirty.left; final int top = dirty.top; final int right = dirty.right; final int bottom = dirty.bottom; // Get canvas from surface and start drawing canvas = msurface.lockCanvas (dirty); // The dirty rectangle can be modified by Surface.lockCanvas() //noinspection ConstantConditions if (left ! = dirty.left || top ! = dirty.top || right ! = dirty.right || bottom ! = dirty.bottom) { attachInfo.mIgnoreDirtyState = true; } // TODO: Do this in native canvas.setDensity(mDensity); } catch (Surface.OutOfResourcesException e) { handleOutOfResourcesException(e); return false; } catch (IllegalArgumentException e) { Log.e(mTag, "Could not lock surface", e); // Don't assume this is due to out of memory, it could be // something else, and if it is something else then we could // kill stuff (or ourself) for no reason. mLayoutRequested = true; // ask wm for a new surface next time. return false; } // omit code.. try { canvas.translate(-xoff, -yoff); if (mTranslator ! = null) { mTranslator.translateCanvas(canvas); } canvas.setScreenDensity(scalingRequired ? mNoncompatDensity : 0); attachInfo.mSetIgnoreDirtyState = false; // Take the canvas, pass it, and start drawing. This is all on the main thread, so you can't do time-consuming operations. mView.draw(canvas); drawAccessibilityFocusedDrawableIfNeeded(canvas); } finally { if (! attachInfo.mSetIgnoreDirtyState) { // Only clear the flag if it was not set during the mView.draw() call attachInfo.mIgnoreDirtyState = false; }} // omit code.. } finally {try {/ / end to draw the data through the JNI to the engine surface. The CPP surface. UnlockCanvasAndPost (canvas); } catch (IllegalArgumentException e) { Log.e(mTag, "Could not unlock surface", e); mLayoutRequested = true; // ask wm for a new surface next time. //noinspection ReturnInsideFinallyBlock return false; } if (LOCAL_LOGV) { Log.v(mTag, "Surface " + surface + " unlockCanvasAndPost"); }}Copy the code

The above code does three things:

  1. throughsurface.lockCanvas()Bind canvas object to Native side and native layerdequeue()Create a graphic Buffer and bind canvas to graphic Buffer. Therefore, Canvas is actually the representation of graphic Buffer in Java layer.
  2. After canvas binding, start the View’s draw process.
  3. After the drawing is complete, the graphic Buffer is finally added to the teamenqueue(), and unbind the canvas.

4.4 Evolution of drawing methods

However, because Android4.0 is software drawing, hardware drawing is enabled by default after 4.0, so the third point is different.

  • If it is software drawing (prior to Android3.0), it will be done directly by the CPU, which will inevitably cause the UI thread to stall or ANR.
  • Hardware drawing is to hand over this process to THE GPU device to call OpenGL to complete, bearing part of the WORK of the CPU, but it is still completed in the main thread, the main thread not only to complete the update and maintenance of the display list, but also to convert the display list into OpenGL drawing instructions to participate in the drawing.
  • After Android5.0, RenderThread is opened in native layer, which is used to call OpenGL interface to complete drawing. The UI thread only needs to keep track of the updated list of the View. After the display list is updated, the RenderThread thread is notified to draw without actually participating in the drawing details, greatly reducing the work of the UI thread.

4.5 Data transfer between UI thread, RenderThread thread, and SurfaceFlinger

How does the UI thread interact with the RenderThread? When do you hand over the plotted data to SurfaceFlinger?

OnMeasure () and onLayout() calculate the size and placement of the view, which is what the UI thread does.

  1. Draw in the draw() method without actually drawing. Instead, we encapsulate the drawn instructions as displayList and further as RendNode, which are synchronized to the RenderThread.
  2. RenderThread throughDequeue ()Get the graphic buffer (surfaceFlinger buffer), operate the Drawing interface of OpenGL directly according to the drawing instruction, and finally render the drawing instruction to the off-screen buffer graphic buffer through THE GPU device.
  3. After rendering, the buffer is returned to SurfaceFlinger’s BufferQueue. SurfaceFlinger performs layer composition through hardware devices and finally displays it on the screen.

The above processes also reflect the producer and consumer model:

Producer: APP, which goes further is Canvas -> Surface.

Consumer: SurfaceFlinger

The size of a BufferQueue is usually 3.

  • A buffer is used to be presented to the device by SurfaceFlinger
  • One is used to draw buffer data for App
  • In addition, if the App draws more than one frame 16ms, the next frame vsync comes, two of them have been occupied, so the third one should be used to avoid the vsync signal CPU and GPU being idle (because if it is idle, jank will appear in the next frame).

5 What did SurfaceFlinger do

SurfaceFlinger is a display synthesis system. When an application requests a surface, SurfaceFlinger creates a Layer. Layer is the basic unit of SurfaceFlinger operation composition. So, one surface corresponds to one Layer.

After the application puts the GraphicBuffer data into the BufferQueue, the SurfaceFlinger does the rest.

Description:

The system will have multiple applications, and one application will have multiple BufferQueues. SurfaceFlinger is used to determine when and how to manage and display queues.

SurfaceFlinger requests HAL hardware to decide whether buffers should be composed by hardware or by OpenGL itself.

Finally, the synthesized buffer data is displayed on the screen.

Official full render architecture:

Description:

  • Image stream produceers: Producers of render data. For example, the App’s Draw method passes drawing instructions to the Framework’s RenderThread thread through the Canvas.

  • The thread gets the buffer graphic bufer from the surface.dequeue, and then uses OpenGL to execute the actual rendering command on it. We’re giving the buffer back to the BufferQueue.

  • Image Stream Consumers: surfaceFlinger gets the data from the queue, completes the layer composition with HAL, and finally presents it to HAL.

  • HAL: Hardware abstraction layer. Display graphical data to the device screen

Reference:

Source. The android. Google. Cn/devices/gra…

zhuanlan.zhihu.com/p/351743856

Juejin. Cn/post / 684490…

Testerhome.com/topics/2336…

Androidperformance.com/2019/10/22/…