1. Android Refresh mechanism
  2. SurfaceView understand

First, Android screen refresh mechanism

  1. First, you need to understand some basic concepts

    • In a Display system, there are generally three parts: CPU, GPU and Display. The CPU is responsible for computing data and gives the data of the calculation number to the CPU. The GPU will render the graph data and store it in buffer after rendering. The Display (screen or Display) is then responsible for rendering the data in the buffer onto the screen. In simple terms, the Display process is that CPU/GPU prepares data and stores it into buffer. Display takes data from buffer at regular intervals and displays it. The frequency of Display reads is fixed, such as 16ms at a time, but the CPU/GPU writes data irregularly.
    • The CPU computed data refers to the process of drawing the View tree. That is, the Activity tree traverses the View starting from the root layout DecorView, measuring, laying out, and drawing. The 16.6ms screen refresh is the underlying display of screen data in the buffer at a fixed frequency.

  2. In a couple of previous posts, For example, Windows and WindowManager (6) The link between the DecorView and setContentView and Window shows that the ViewRootImpl is created only when onResume is used. As with any View call to a refresh such as invalidate, the scheduleTraversals() method in ViewRootImpl is followed by a call to Choreographer’s postCallBack method

        void scheduleTraversals() {
        if(! MTraversalScheduled) {// ref 1 mTraversalScheduled =true; MTraversalBarrier = mhandler.getLooper ().getQueue().postsyncbarrier (); / / number 3 mChoreographer postCallback (Choreographer. CALLBACK_TRAVERSAL mTraversalRunnable, null);if (!mUnbufferedInputDispatch) {
                scheduleConsumeBatchedInput();
            }
            notifyRendererOfFramePending();
            pokeDrawLockIfNeeded();
        }
    }
Copy the code

  3. Choreographer passed a mTraversalRunnable into postCallBack, as you can see below, the run method calls the doTraversal method

        final class TraversalRunnable implements Runnable {
        @Override
        public void run() {
            doTraversal();
        }
    }
    
   void doTraversal() {
      if(mTraversalScheduled) {// mTraversalScheduled =false; Mhandler.getlooper ().getQueue().removesyncBarrier (mTraversalBarrier);if (mProfile) { 
              Debug.startMethodTracing("ViewAncestor"); } // Here we begin to measure, arrange, and render performTraversals();if (mProfile) {
              Debug.stopMethodTracing();
              mProfile = false; }}}Copy the code

  4. A problem arises here. After calling the scheduleTraversals method, the code simply passes Runnable to Choreographer’s postCallback method. To call the doduleTraversal method, There has to be a Runnable somewhere. To understand the execution logic, look at Choreographer’s postCallback code

        public void postCallback(int callbackType, Runnable action, Object token) {
        postCallbackDelayed(callbackType, action, token, 0);
    }
    public void postCallbackDelayed(int callbackType,
        Runnable action, Object token, long delayMillis) {
     if (action == null) {
         throw new IllegalArgumentException("action must not be null");
     }
     if (callbackType < 0 || callbackType > CALLBACK_LAST) {
         throw new IllegalArgumentException("callbackType is invalid");
     }
     postCallbackDelayedInternal(callbackType, action, token, delayMillis);
 }
 private void postCallbackDelayedInternal(int callbackType,
            Object action, Object token, long delayMillis) {
    if (DEBUG_FRAMES) {
        Log.d(TAG, "PostCallback: type=" + callbackType
                    + ", action=" + action + ", token=" + token
                    + ", delayMillis="+ delayMillis); } synchronized (mLock) { final long now = SystemClock.uptimeMillis(); final long dueTime = now + delayMillis; Queues[callbackType]. AddCallbackLocked (dueTime, Action, token);if (dueTime <= now) {
            scheduleFrameLocked(now);
        } else {
            Message msg = mHandler.obtainMessage(MSG_DO_SCHEDULE_CALLBACK, action);
            msg.arg1 = callbackType;
            msg.setAsynchronous(true); mHandler.sendMessageAtTime(msg, dueTime); }}}Copy the code

    It will eventually call postCallbackDelayedInternal method, and passed delayMillis is zero, so the dueTime = now, so call is scheduleFrameLocked method, then look at its calling code

    // private void scheduleFrameLocked(long now) {if(! mFrameScheduled) { mFrameScheduled =true;
            if (USE_VSYNC) {
                if (DEBUG_FRAMES) {
                    Log.d(TAG, "Scheduling next frame on vsync."); } // Whether it is in the main threadif (isRunningOnLooperThreadLocked()) {
                    scheduleVsyncLocked();
                } else {
                    Message msg = mHandler.obtainMessage(MSG_DO_SCHEDULE_VSYNC);
                    msg.setAsynchronous(true); mHandler.sendMessageAtFrontOfQueue(msg); }}else {
                final long nextFrameTime = Math.max(
                        mLastFrameTimeNanos / TimeUtils.NANOS_PER_MS + sFrameDelay, now);
                if (DEBUG_FRAMES) {
                    Log.d(TAG, "Scheduling next frame in " + (nextFrameTime - now) + " ms.");
                }
                Message msg = mHandler.obtainMessage(MSG_DO_FRAME);
                msg.setAsynchronous(true); mHandler.sendMessageAtTime(msg, nextFrameTime); }}} // method private voidscheduleVsyncLocked() { mDisplayEventReceiver.scheduleVsync(); // Method DisplayEventReceiver public voidscheduleVsync() {
      if (mReceiverPtr == 0) {
          Log.w(TAG, "Attempted to schedule a vertical sync pulse but the display event "
                    + "receiver has already been disposed.");
      } else{ nativeScheduleVsync(mReceiverPtr); }} / / method FrameDisplayEventReceiver class private static native void nativeScheduleVsync (long receiverPtr);Copy the code

    If you put Runnable in the CallbackQueue, see 6 in point 4, then the timing of the call must correspond to getting Runnable from the CallbackQueue. Namely CallbackQueue extractDueCallbacksLocked method, after the search, can draw is doCallbacks method calls the method (CallbackQueue Choreographer of the inner class), It’s the doFrame method that calls the doCallbacks method. (Here you can see a callbackType corresponding to a Runnable queue, mCallbackQueue[callbackType])

        void doCallbacks(int callbackType, long frameTimeNanos) {
        CallbackRecord callbacks;
        synchronized (mLock) {
            // We use "now" to determine when callbacks become due because it's possible // for earlier processing phases in a frame to post callbacks that should run // in a following phase, such as an input event that causes an animation to start. final long now = System.nanoTime(); callbacks = mCallbackQueues[callbackType].extractDueCallbacksLocked( now / TimeUtils.NANOS_PER_MS); . } } void doFrame(long frameTimeNanos, int frame) { final long startNanos; synchronized (mLock) { ...... try { Trace.traceBegin(Trace.TRACE_TAG_VIEW, "Choreographer#doFrame"); AnimationUtils.lockAnimationClock(frameTimeNanos / TimeUtils.NANOS_PER_MS); mFrameInfo.markInputHandlingStart(); doCallbacks(Choreographer.CALLBACK_INPUT, frameTimeNanos); .Copy the code

    So where is the doFrame method called? Is obtained from previous calls after FrameDisplayEventReceiver native methods as not bottom go to, then enter the class can be found in its run method on the call doFrame method, OnVsync method Called when a vertical sync pulse is received

            @Override
        public void onVsync(long timestampNanos, int builtInDisplayId, int frame) {
            if(builtInDisplayId ! = SurfaceControl.BUILT_IN_DISPLAY_ID_MAIN) { Log.d(TAG,"Received vsync from secondary display, but we don't support "
                        + "this case yet. Choreographer needs a way to explicitly request "
                        + "vsync for a specific display to ensure it doesn't lose track "
                        + "of its scheduled vsync.");
                scheduleVsync();
                return;
            }
            // Post the vsync event to the Handler.
            long now = System.nanoTime();
            if (timestampNanos > now) {
                timestampNanos = now;
            } 
            if (mHavePendingVsync) {
                Log.w(TAG, "Already have a pending vsync event. There should only be "
                        + "one at a time.");
            } else {
                mHavePendingVsync = true; } mTimestampNanos = timestampNanos; mFrame = frame; Message MSG = message. obtain(mHandler, this); msg.setAsynchronous(true);
            mHandler.sendMessageAtTime(msg, timestampNanos / TimeUtils.NANOS_PER_MS);
        }
        
        @Override
        public void run() {
            mHavePendingVsync = false;
            doFrame(mTimestampNanos, mFrame);
        }
Copy the code

    If a Runnable is passed, its run method will be executed. If a Runnable is passed, your Callback will be executed. If no, handleMessage will be called. We pass this to Message, we execute our run method, we execute our doFrame method, we execute the doCallbacks, we execute the Run method of the Runnable object, we execute the doTraversals within the run method, So let’s start refreshing the view

  5. The subtotal, FrameDisplayEventReceiver inheritance DisplayEventReceiver VSync signal began to accept the bottom handle UI process, VSync signal sent by SurfaceFlinger realize and timing. FrameDisplayEventReceiver after receiving the signal, call onVsync method group message is sent to the main thread processing. The main content of this message is the doFrame in the run method. FrameDsiplayEventReceiver can receive signals, the callback onVsync method, can be understood as the APP layer to the bottom when call native methods nativeScheduleVsync registered a screen refresh signal to monitor events, How else would the bottom layer know that the APP needs to refresh data? And APP is hidden from the bottom layer, the bottom layer does not know the existence of APP.

    To clarify, the APP registers the screen refresh interface of the next frame with the bottom layer through native method, and then it will call onVsync to refresh the screen every 16.6ms frame signal

  6. So the question is, can I register this listener indefinitely in 16.6ms or before the screen refreshes, that is, can I register many repeat listeners in one frame

    • Let’s start with some code

          void scheduleTraversals() {
        if(! mTraversalScheduled) { mTraversalScheduled =true; MTraversalBarrier = mhandler.getLooper ().getQueue().postsyncbarrier (); mChoreographer.postCallback( Choreographer.CALLBACK_TRAVERSAL, mTraversalRunnable, null); . } voiddoTraversal() {
        if (mTraversalScheduled) {
            mTraversalScheduled = false; Mhandler.getlooper ().getQueue().removesyncBarrier (mTraversalBarrier); . }Copy the code

    • First, the scheduleTraversals method, which has a mTraversalScheduled variable that defaults to false before registering with the underlying listener, calls the scheduleTraversals method multiple times, The registration is repeated only when mTraversalScheduled is false, and from the doTraversal method you can see that this variable is reset to false only when a frame is received. So why is it designed this way? If you look at the performTraversals method, you will see that it is invoked by the interface refresh, which traverses the View tree for every View that needs to be refreshed, so there is no need for repeated registration

  7. The next thing you might want to ask is the code at number 7 in the onVsync method, which encapsulates the refresh Message into the Message in the main thread MessageQueue, which is the main thread that handles the messages in the MessageQueue, Only one Message can be processed at a time. What if other messages take time and the refresh Message is not processed for 16.6ms

    • This is inevitable, and you have to try not to do too many time-consuming operations in it
    • The Android system has an optimization for this, which is code 8 and code 9, which can be understood as the identification of setting and removing barrier messages. First, the code at no. 8 will send a synchronization barrier to the queue, and the main Looper will execute it by next continuously retrieving the Message of the queue header. If the queue header is a synchronization barrier Message, it will walk through the whole queue, looking for the Message with the asynchronous identifier set, and fetch the Message to execute. Otherwise, the next method gets stuck in a blocking phase. So all synchronous messages behind the barrier message are intercepted until the barrier message is removed from the queue using the doTraversal method. This ensures that the refresh Message is executed first, but the Message retrieved earlier will still be executed, so if the contents of the Message are time-consuming, it will still affect the refresh Message execution.

  8. summary

    • The scheduleTraversals() method of ViewRootImpl is invoked for any View refresh request, which filters repeated traversals within a frame to ensure that the View tree is refreshed only once within a frame
    • When the scheduleTraversals() method is called, a synchronization barrier is sent to the message queue of the main thread, blocking all synchronization messages after that point until a refresh interface message with asynchronous identity generated after the underlying onVsync callback is executed
    • The refresh interface, the doTraversals method, will then be placed into the Runnable, and Choreographer’s postCallback method will be placed into the queue as a timestamp. If the current thread is the main thread, the native method will be called directly. If it is not the main thread, Message will be placed on the main thread with the highest priority, ensuring that the native method is executed as first as possible
    • This method is used by Native to register with the underlying layer to listen for the next screen refresh signal, and when the next screen refresh signal is emitted, the underlying layer calls back to Choreographer’s onVsync method
    • After the onVsync method is called, because it is a method with asynchronous identification, it will not be blocked by the synchronization barrier, so it can ensure that the interface is retrieved and refreshed in the first time, and the synchronization barrier is removed
    • The last step is to perform the View traversal refresh
    • If the interface stays the same, no listener events are registered, but the underlying interface still switches frames at a fixed rate of 16.6ms, but the final frames are the same.
    • The bottom layer switches the screen at a fixed frequency, and even when the CPU has finished calculating the data, which means measuring, layout, drawing, and so on, it does not display it immediately until the signal arrives.

Second, the SurfaceView

  1. Let’s start with the most common differences with Views
    • View drawing efficiency is low, mainly used for animation changes less programs, must be updated in the main thread
    • SurfaceView drawing efficiency, used for interface updates frequently, generally in the child thread update
    • SurfaceView has a separate Surface(drawing Surface), that is, it does not share the same Surface with its host window
    • SurfaceView uses double buffering to make video playback smoother
    • Each window has a Layer in the SurfaceFlinger service that describes its drawing surface. For Windows with a SurfaceView, each SurfaceView in the SurfaceFlinger service also corresponds to a separate Layer or LayerBuffer that describes its drawing surface separately from the drawing surface of its host window. So the UI of the SurfaceView can be drawn in a separate thread without using resources from the main thread, which is also created to cope with time-consuming operations such as Camera X.
  2. Part of the concept
    • Canvas is a data structure constructed by Java layer and used by View. ViewGroup will split Canvas into sub-views and draw the graph data on the Canvas obtained by it in onDraw method
    • Surface is a data structure built by Native layer and a canvas for SurfaceFlinger. It is a data structure that is directly used to draw on the screen
    • Generally, the views used by developers are drawn on Canvas, and the data information of the Canvas of the topmost View will be converted to a Surface. SurfaceFlinger will synthesize the Surface of each application window and then draw it on the screen.
  3. Double buffering mechanism
    • The SurfaceView updates the view using two canvases, one frontCanvas and one backCanvas. The frontCanvas is actually displayed each time, and the backCanvas stores the view that was changed last time. When you play this frame, it has already loaded the next frame for you, so it plays smoothly.
    • When we use lockCanvas() to get the canvas, we actually get the backCanvas instead of the frontCanvas that is being displayed. Then we draw a new view on the backCanvas, using unlockCanvasAndPost(canvas), Then the uploaded Canvas will replace the original frontCanvas as the new frontCanvas, and the original frontCanvas will switch to the background as the backCanvas. Equivalent to multiple threads alternately parsing and rendering each frame of video data
  4. The SurfaceView is zOrd-sorted and by default comes after the host Window. The SurfaceView “burrows” above the host Window.
  5. SurfaceView is a View that has its own Surface. Its rendering can be placed in a separate thread rather than the main thread. The disadvantage is that it cannot be morphed or animated
  6. SurfaceTexture can be used as a stream of content that is not directly output, thus providing the opportunity for secondary processing with several frames of delay and higher memory consumption than SurfaceView direct output
  7. TextureView is drawn in the View hierachy, so it is usually done on the main thread
  8. SurfaceTexture, TextureView, SurfaceView and GLSurfaceView are available for reference