IOS Interface rendering and Optimization (PART 1) – What the CPU and GPU are doing

Recently, I have been studying interface optimization and have seen a lot of content related to iOS interface rendering. Here is a simple summary.

1. Computer rendering principle

Refer to the OpenGL rendering, or refer to the following article:

IOS Rendering full parsing (Juejin.cn)

ObjC China – Draw Pixels to screen (objccn.io)

2. Imaging principle and screen lag

So when the rendered content is in the framebuffer, the video control is needed to bring the framebuffer to the screen. The following two processes are required for each frame not shown in front of the bounds:

  1. View creation, layer-tree deletion (layer-tree adjustment), layout frame calculation, image decoding, text drawing, and so on in the CPU, essentially manipulating the layer tree, and storing the results in the CommandBuffer
  2. The CPU processing result CommandBuffer is submitted to the GPU, and the GPU performs rendering

As shown in the following figure, if either CPU or GPU takes a long time, frames may be lost, resulting in lag.

The principle of specific rendering can be referenced

IOS event processing mechanism and image rendering process (QQ.com),

Keep the iOS interface and fluent skills | Garan no dou (ibireme.com)

3. The process of triggering rendering in Runloop

Here is an excerpt from the reference article:

The iOS display is powered by the VSync signal, which is generated by the hardware clock and emits 60 times per second (depending on the device’s hardware, for example, it’s usually 59.97 on an iPhone). After receiving the VSync signal, the iOS graphics service will notify the App through IPC. After startup, the Runloop of the App registers the corresponding CFRunLoopSource to receive the clock signal notification through mach_port, and then the callback of the Source drives the animation and display of the entire App.

In iOS, the notification listening and subsequent processing in Runloop is done through the CoreAnimation framework. CoreAnimation’s name is misleading, and on MAC it’s called LayerKit.

The CoreAnimation framework registers an Observer in RunLoop that listens for BeforeWaiting and Exit events. The priority of this Observer is 2 million, which is lower than other common observers. When a touch event arrives, the RunLoop wakes up and the code in the App performs some actions called handleEvents. The common actions are the ones we implemented in our development, such as:

  1. Create and adjust view hierarchies such as addSubView, removeSubView, etc
  2. Set the FRAME of the UIView, modulate the Autolayout constraint, etc
  3. Modified CALayer transparency
  4. Add an animation to the view
  5. Other operations that may cause changes to calayer-Tree

The above action will actually generate a hermit’s CATransaction:

Hermit [CATransaction begin]; CALayer layer change hermit [CATransaction Commit];Copy the code

All of these operations are eventually captured by the CALayer and saved to an intermediate state via the CATransaction. When the RunLoop is about to go to sleep (or exit), Any Observer concerned about the event is notified (this Observer has a very low priority and is executed last). The Observer registered by the CoreAnimation will then merge all the intermediate states and submit them to the GPU in the callback. If there is animation here, the CA will trigger the process several times through mechanisms such as DisplayLink.

Implicit animation is done automatically by the system framework.

Core Animation automatically starts a new transaction each runloop cycle. Even if you don’t explicitly start a transaction with [CATransaction Begin], any property changes that occur during a runloop cycle are aggregated and animated once for a quarter of a second.

In ios, apple to add a UIView animation method based on block: + animateWithDuration: animations:. It’s syntactically easier to write right and do a bunch of property animations, but essentially they’re all doing the same thing. CATransaction + begin and + commit method in + animateWithDuration: animations: internal calls automatically, so that all the properties in the block contains the change will be firm.

The above procedure in the CPU call stack might be as follows:

_ZN2CA11Transaction17observer_callbackEP19__CFRunLoopObservermPv()
    QuartzCore:CA::Transaction::observer_callback:
        CA::Transaction::commit();
            CA::Context::commit_transaction();
                CA::Layer::layout_and_display_if_needed();
                    CA::Layer::layout_if_needed();
                          [CALayer layoutSublayers];
                          [UIView layoutSubviews];
                    CA::Layer::display_if_needed();
                          [CALayer display];
                          [UIView drawRect];
Copy the code

4. What the CPU does before rendering the interface

WWDC 2014 -Advanced Graphics and Animations for iOS Apps

This session has been deleted by Apple. Someone posted this video on b

Use another diagram to show the overall process from CPU -> GPU as follows:

  1. CoreAnimation submits the session, including the layout state of itself and its subtree (View hierarchy); (This step is done in the APP and the CA information is submitted to the CA through the IPC framework)

  2. RenderServer parses the submitted subtree state and generates the draw instruction; (This step is in the Render Server, not inside the APP)

  3. The GPU executes the drawing instruction

  4. Display the rendered data

The CPU goes through the following steps. The CA actually goes through the following four steps before committing a Transaction:

  1. Layout – Builds a view

    • Call the layoutSubviews method
    • Call the addSubview: method
    • Text calculation (size), etc
    • AutoLayout calculates the frame of each view according toLayout Constraint

  2. Display – Draws a view

    • At this stage the program will create the backing image of the layer, either passing an image to the layer with setContents, or drawing it with drawRect: or drawLayer: inContext:.

      So functions such as drawRect: are called at this stage.

      Be careful not to confuse the Display here with the final Display, and this is done in the CPU

  3. Prepare – Prepare before submitting

    • Image Copy + Image decode – more on this later
    • Try to use a format supported by the GPU; Apple recommends JPG and PNG

  4. Commit – Pack layers to Render-Server via IPC

    • Package layers and send them to Render-server
    • — Layers recursively commit subtree layers (if the view has too many layers, you will see a large number of methods in the call stack:CA::Layer::commit_if_needed)

After the CPU finishes preparing layers and submits them to the Render-server, the real content is left to the GPU!

Here is an additional content about the Animation:

Apple’s approach is very simple and direct, for each Animation, that the preparation stage and the separate image to submit basically the same (Layout/Display/Prepare/Commit). However, when rendering, the Render Server will automatically calculate each frame image required for Animation according to Animation parameters of Core Animation, and then Render and display frame by frame, finally rendering is the Animation effect. This means that the App only needs to tell the Start and end states of the animation to the Render Server, and it will automatically calculate the intermediate process and automatically Render the animation.

Small tips:

Therefore, it can be seen from here that the optimization points for the content processing in the CPU are as follows (the common optimization methods are specially summarized in the following):

  1. The operations in the handleEvent process should be as few as possible, and no time-consuming operations should be done
  2. Layout calculations, text calculations are likely to be less, and Autolayout is more performance consuming than Frame
  3. Images need memory alignment, and the image decoding process is done ahead of time in the child thread!! And uiImageView. size is consistent with the image size!

Small tips2:

How to create an Animation

We will introduce three types of layer-tree concepts later, which will help us understand animation more deeply

To put it simply: you only need to operate on the start and end state modal Layer properties and Render Server will do all the intermediate calculations in the Presentation Layer

5. GPU rendering process

At present, mobile devices are all using Tiled-based rendering. Here are the general rendering process and an off-screen rendering process:

5.1 Ordinary Tile-based rendering process is as follows (normal rendering):
  1. CommandBuffer, which receives render instructions after OpenGL ES has finished processing;
  2. Tiler, the vertex shader is called, the vertex data is divided into blocks (Tiling);
  3. ParameterBuffer accepts tiles that have been partitioned and corresponding render parameters.
  4. The Renderer, which calls the slice shader, renders pixels;
  5. RenderBuffer, which stores the rendered pixels;

5.2 Off-screen rendering — MASK

When the Mask is used, after the pass1 procedure is completed, save the pass1 result, save the pass2 result, and merge it into the pass3 result (there is only one storage result in the normal mode).

  1. Render layer mask texture, same as the basic Tile-Based rendering logic;
  2. The render layer content texture, same as the basic Tile-Based rendering logic;
  3. Compositing operation, merge textures 1 and 2;

Other effects, such as THE UIVisiualEffectView effect, are much more performance intensive and are called expensive effects!! WWDC 2014 -Advanced Graphics and Animations for iOS Apps

6. Performance optimization questions

1. What is the frame rate? 2. Is there a CPU and GPU bottleneck? 3. Extra CPU used for rendering? 4. Is there too much off-screen rendering? 5. Do you render too many views? 6. Using strange image formats and sizes? Use expensive special effects? Unnecessary elements in the view tree?

Reference article:

WWDC2011 121: understanding uikit rendering

WWDC2012 211: building concurrent user interfaces on ios

WWDC2012 235: iOS App Performance: Responsiveness

WWDC2012 242: iOS App Performance: Memory

WWDC 2012: iOS App Performance: Graphics and Animations

WWDC 2014 -Advanced Graphics and Animations for iOS Apps

WWDC2018 Image and Graphics Best Practices

WWDC2018 iOS Memory Deep Dive

IOS View – Animation rendering mechanism exploration – CocoaChina_ one-stop developer growing community

IOS Rendering full parsing (Juejin.cn)

IOS image loading speed limit optimization – FastImageCache parsing

Getting Pixels onto the Screen

Offscreen Render

Design for iOS: Graphics and performance

IOS Performance Optimization series “List Fluency Optimization”

IOS image display principle and lag optimization

IOS performance optimization summary

ios-rounded-corner

Falling Shadows — iOS performance optimization — Image loading and processing — Cloud + Community — Tencent Cloud (Tencent.com)

IOS Off-screen rendering optimization (with DEMO)

[[turn] iOS event handling mechanism and image rendering process] (www.cnblogs.com/linganxiong)…

IOS 2D Graphic (1) — Concept Indicates the basic concepts and principles

Image usage, memory comparison, and best practices in iOS (juejin.cn/post/684490…)

IOS Interface Rendering Process Analysis – Cloud + Community – Tencent Cloud (Tencent.com)

How to build iOS Efficient picture IO frameworkiOSDevelopment -CSDN blog

Image memory management and Performance optimization for iOS – Jianshu.com

5 Image thumbnails in iOS and their performance

Md at master · SunshineBrother/JHBlog (github.com)

Extracting images to render in iOS – Jianshu.com

IOS Graphics performance optimization (juejin.cn)