In-depth understanding of iOS Rendering Process

preface

IOS, originally named iPhone OS, is an operating system specially developed by Apple for its hardware devices. It was first launched with the first generation oF iPhone in 2007, and later expanded to support other hardware devices owned by Apple, such as iPod and iPad.

As an iOS Developer, I’m sure most of you have written code that causes iOS devices to lag, and have had the experience of trying to optimize the code for lag.

This article will introduce the concept of iOS Rendering Process and all the processes of the whole underlying Rendering pipeline from the perspective of OpenGL combined with some official information provided by Apple.

We believe that after understanding the underlying stages of iOS Rendering Process, we can write higher-performance code in daily development work and have more ideas to solve the problem of display stutter with low frame rate

The index

IOS Rendering Process concept
IOS Rendering technical framework
OpenGL main rendering steps
OpenGL Render Pipeline
Core Animation Pipeline
Commit Transaction
Animation
The full text summary
Further reading

IOS Rendering Process concept

IOS Rendering Process (iOS Rendering Process) is the iOS device Rendering Process, from setting up the image metadata to be displayed to Rendering the image on the device screen.

Before we start dissecting the iOS Rendering Process, we need to have a basic understanding of iOS Rendering concepts:

Tile-based rendering

The screen of iOS devices is divided into N * N pixel blocks, each of which is suitable for SoC cache, and the geometry is broken up in large numbers within the blocks, which can only be rasterized after all the geometry has been committed.

Note: Rasterization here refers to the process of rendering a large amount of separated geometry on the screen as pixels.

IOS Rendering technical framework

In fact, the hierarchy associated with iOS rendering is as follows:

UIKit

As an iOS Developer, we are familiar with UIKit. The user interaction components used in our daily development are all from UIKit Framework. We complete daily interface painting by setting UIKit component Layout and BackgroundColor and other properties.

In fact, UIKit Framework itself does not have the ability to image images on the screen. It is mainly responsible for responding to user operation events, and the transmission of event responses is basically realized by traversing the view tree layer by layer.

So why are UIKit components that we write everyday displayed on the screen of an iOS device?

Core Animation

Core Animation is actually a misleading name. You might think it’s just for animating, but it’s actually evolved from a name called Layer Kit that doesn’t have much to do with Animation, so animating is just the tip of the iceberg for Core Animation.

Core Animation is essentially a composite engine designed to combine different on-screen displays as quickly as possible. These displays are broken down into separate layers called Calayers, which are the basis of everything you see on screen.

So you can get the Backing Layer from a CALayer in UIKit. Because of the one-to-one correspondence, CALayer is also a tree structure, which we call a layer tree.

The view’s responsibility is to create and manage this layer to ensure that when subviews are added or removed from the hierarchy, their associated layers also have the same action in the hierarchy tree.

But why does iOS provide two parallel hierarchies based on UIView and CALayer? Why not have a simple hierarchy for everything?

The reason for this is separation of responsibilities, which also avoids a lot of duplicate code. Events and user interaction are different in a lot of ways on iOS and Mac OS X. There’s a fundamental difference between a multi-touch based user interface and a mouse and keyboard based interaction. That’s why iOS has UIKit and UIView. Mac OS X has AppKit and NSView. They are similar in function, but there are significant differences in implementation.

Note: Actually, there are not two hierarchies, but four, each playing a different role. In addition to view trees and layer trees, there are also render trees and render trees.

OpenGL ES & Core Graphics

OpenGL ES

OpenGL ES is a subset of OpenGL that renders 2D and 3D computer graphics, such as those used in video games, for graphics hardware accelerated processing units (Gpus) **.

OpenGL ES is designed for embedded systems such as smartphones, tablets, video game consoles and PDAs. OpenGL ES is “the most widely used 3D graphics API in history”.

Core Graphics

Core Graphics Framework is based on the Quartz advanced Graphics engine. It provides low-level lightweight 2D rendering with unmatched output fidelity. You can use this framework to handle path-based drawing, transformation, color management, off-screen rendering, patterns, gradients and shadows, image data management, image creation and image masking as well as PDF document creation, display and analysis.

Note: In Mac OS X, Core Graphics also includes services for handling display hardware, low-level user input events, and windowing systems.

Graphics Hardware

IOS devices also have their own Graphics Hardware, which is often referred to as the GPU.

A graphics processing unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory to speed up the creation of images in the frame buffer used for output to a display device. Gpus are used in embedded systems, mobile phones, personal computers, workstations and game consoles. Modern Gpus are very efficient at processing computer graphics and images, and their highly parallel structure makes them more efficient than general-purpose cpus in algorithms for parallel processing of large chunks of data.

OpenGL main rendering steps

OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector Graphics. OpenGL can access the GPU directly for hardware-accelerated rendering.

An OpenGL program for rendering images can be roughly divided into the following steps:

Set the graph metadata
Shader computes graph metadata (position · color · other)
Rasterization – Rasterization renders to pixels
Fragment shader, determine the final image
Other operations (Show, hide, merge)

Note: There are also some non-essential steps that are not relevant to the topic of this article.

We use UIKit layout view controls in our daily development, and setting transparency and so on are all part of setting up the graph metadata, which is the main part of our daily development that can affect OpenGL rendering.

OpenGL Render Pipeline

If some students have seen WWDC some speeches or contact with some OpenGL knowledge, should be the Render Pipeline this professional term is not strange.

But Render Pipeline is a first time not easy to understand the word, it translates to Render Pipeline, also has translated to Render Pipeline…

Render Pipeline refers to a series of data processing processes from application data to the final rendered image.

Just like the main rendering step in OpenGL mentioned above, we set the layout, background color, transparency, and shadows of the view controls in the setting metadata step when developing the application.

Take OpenGL 4.5 Render Pipeline as an example:

This graph metadata flows into OpenGL, passes it to the Vetex shader, and then the vertex shader processes it inside the shader before it flows out. After that, it may enter the tessellation shading stage, which may be divided into two parts: the segmentation control shader and the segmentation assignment shader, and the geometry shading stage, from which data is transferred. In the end, they both go through the fragment shading stage.

Note: The graph metadata is sent to the Shader as a copy, and the Shader usually receives the data in a special global-like form.

OpenGL also goes through a phase called the Compute Shaing stage, where OpenGL calculates the location and color of the pixels that are most important to image on the screen, If CALayer was used before submitting code, it would cause blending’s display effects (such as Shadow) or the alpha channel of view colors or content images to be turned on, which would increase OpenGL’s workload at this stage.

Core Animation Pipeline

As mentioned above, iOS devices can image not because of UIKit but because of LayerKit, which is Core Animation.

The Core Animation layer, CALayer, contains a property contents that we can assign to control what CALayer images. This property has a type id, and it will compile regardless of what type of value we give contents when we compile the program. But in practice, if contents assignment type is not CGImage, then you will get a blank layer.

Note: The reason for the strange behavior of contents property is the historical baggage of Mac OS X. It is defined as an ID type because in Mac OS X this property works on both CGImage and NSImage values. But in iOS, if you give a UIImage property value, you just get a blank layer.

After finishing the contents property of Core Animation, the following introduces Core Animation Pipeline in iOS:

Lay out the UIKit view controls in Application and associate them indirectly with the Core Animation layer
The data related to the Core Animation layer is submitted to the iOS Render Server, OpenGL ES & Core Graphics
Render Server will communicate with GPU to transfer data to GPU after processing
The GPU calls the current iOS device to render the related graphics device Display

Note: Since the current display on iOS devices supports a maximum refresh rate of 60 FPS, each processing interval is 16.67ms.

As you can see from the Commit Transaction our graph metadata will be sent by the Application to the underlying Render Server on the next RunLoop, The underlying Render Server directly faces THE GPU and passes the processed data to the GPU through some columns of data processing, and then the GPU is responsible for rendering. Image pixel position is calculated according to the screen of the current iOS device, and pixel alpha channel color mixing calculation is performed to render the image in the display of the current iOS device.

Render Server OpenGL ES & Core Graphics OpenGL ES rendering can be understood by referring to OpenGL Render Pipeline above.

Commit Transaction

In the whole Pipeline of Core Animation Pipeline, the scope that iOS normal development can affect is only the layout of UIKit view controls in Application and the indirect association of Core Animation layer. That is, operations before a Commit Transaction.

What are the common things to do before committing a Transaction?

Layout, build the view
Display, draw a view
Prepare, additional Core Animation work
Commit, package the layers and send them to Render Server

Layout

What we can do in the Layout stage is to write constraints as efficiently as possible. IOS Layout constraint is similar to Android Relative Layout.

Note: Emmmmm… It is observed that iOS Layout Constraint should be written as little as possible depending on sibling view nodes in the view tree, which will slow down the Layout calculation process of the entire view tree.

In this stage, the Layout calculation work is completed in CPU, including the overload of layoutSubviews method, addSubview: method to fill the subview, etc

Display

In fact, Display here is just the metadata that we set up for the final image of the iOS device. Reload the drawRect: method to customize the Display of UIView by drawing a bitmap inside the drawRect: method.

Note: Overload the drawRect: method to draw a bitmap using CPU and memory.

Therefore, overloaded drawRect: Improper use will cause CPU overload, App memory surge and other problems.

Prepare

This step is an additional step that generally deals with decoding & converting the image.

Commit

The Commit step means packaging layers and sending them to Render Server.

Note: The Commit operation is performed recursively, and since layers, like views, have a tree structure, the Commit operation can be very expensive if the layer tree is too complex.

CATransaction

CATransaction is the mechanism used in Core Animation to assign multiple layer tree operations to atomic updates of the render tree. Every modification to the layer tree must be part of a transaction.

The CATransaction class has no properties or instance methods, and it cannot be created using the +alloc and -init methods. We can only use the class methods +begin and + COMMIT to push and unload, respectively.

In fact, any animationable layer attributes will be added to the transaction at the top of the stack. You can use the +setAnimationDuration: method to set the animationDuration of the current transaction, or use the +animationDuration method to get the duration (default: 0.25 seconds).

Core Animation automatically starts a new transaction within each RunLoop cycle, even if you don’t explicitly start a transaction with [CATransaction Begin], any property changes within a particular RunLoop loop are collected. Then do a 0.25 second animation (CALayer implicit animation).

Note: CATransaction supports nesting.

Animation

The most obvious work to improve App user interaction experience is the use of animation, so how does iOS deal with the rendering process of animation?

In daily development, we usually use UIView Animation for non-complicated animations. IOS divides the processing process of UIView Animation into the following three stages:

callanimateWithDuration:animations:methods
Perform Layout, Display, Prepare, Commit in the Animation Block
The Render Server renders frame by frame according to the Animation

Note: principle is animateWithDuration: animations: internal used CATransaction to the entire Animation Block of code as atomic operation commit to the RunLoop.

Based on CATransaction to achieve chain animation

In fact, most Animation interactions have an Animation execution order, and although UIView Animation is very powerful, But in order to write some Animation to use when UIView Animation only in + (void) animateWithDuration: delay: options: animations: completion: The completion block of the method is nested in layers, creating blocks upon blocks of code that are difficult to read, let alone maintain.

When we see that UIView Animation uses a CATransaction, we can’t help but wonder if the Completion Block is also implemented based on a CATransaction.

Bingo! CATransaction has +completionBlock and +setCompletionBlock: methods that correspond to the writing of a Completion block in UIView Animation.

Note: One of my open source libraries, LSAnimator, also uses CATransaction for sequential animation links.

The full text summary

It is not difficult to tease out the basic and complete Rendering pass of iOS based on context.

Performance Detection

Based on the content of the whole article, we can summarize the ideas of detecting problematic code when we encounter performance problems in daily development work:

The problem	advice	Detection tools
The target frame rate	60 FPS	Core Animation instrument
CPU or GPU	Reduce utilization rate and save energy consumption	Time Profiler instrument
Unnecessary CPU rendering	GPU rendering is better, but be clear about when CPU rendering makes sense	Time Profiler instrument
Too many offscreen passes	The less the better	Core Animation instrument
Too much of the blending	The less the better	Core Animation instrument
Strange image format or size	Avoid real-time conversions or resizing	Core Animation instrument
Expensive views or effects	Understand the overhead costs of the current solution	Xcode View Debugger
Unimaginable hierarchies	Understand the actual view hierarchy	Xcode View Debugger

The article is written carefully (it is my personal original article, please indicate lision. Me /). If any mistake is found, we will give priority to… Personal blog updates. If you have any questions, please feel free to contact me on my Weibo @lision

I hope my article can bring you value ~