CPU and GPU

  • The Central Processing Unit (CPU) is one of the three core parts of a modern computer. It is the computing and control Unit of the entire system. The pipeline structure of THE CPU enables it to have a certain degree of parallel computing capability.
  • Graphics Processing Unit (GPU) : a dedicated microprocessor that can perform Graphics computing. The GPU can generate 2D/3D graphics images and videos, thus supporting window-based operating systems, graphical user interfaces, video games, visual image applications and video playback. GPU has very strong parallel computing capability.

The fundamental reason for using GPU to render graphics is speed. GPU’s excellent parallel computing capability enables it to quickly calculate and display graphic results in all pixels of the screen.

Display principle of screen image

To introduce the principle of screen image display, we need to start from the principle of CRT display, as shown in the figure below. The CRT electron gun scans the screen line by line from top to bottom, rendering a single frame. The gun then returns to its original position for the next scan. To synchronize the display process with the system’s display controller, the display generates a series of timing signals using a hardware clock. When the gun switches lines to scan, the monitor emits a horizontal synchronization signal, or HSync; When a frame is drawn and the gun returns to its original position, the display sends a vertical synchronization signal, or VSync, before it is ready to draw the next frame. The monitor usually refreshes at a fixed rate, which is the frequency at which the VSync signal is generated. Although today’s displays are basically LCD screens, the principle is basically the same.

The following figure shows the common working modes of CPU, GPU and monitor. The CPU calculates the display content and submits it to THE GPU. After the GPU rendering is completed, the rendering result is stored in the frame buffer. The display controller will read the data in the frame buffer frame by frame according to the VSync signal, and finally display it by the display after data conversion.

Double buffering mechanism

Therefore, displaying a screen requires two steps:

  • The CPU calculates the screen data to be displayed
  • The monitor displays the data

Both steps take time and can be performed in parallel because the hardware that performs them is independent of each other (CPU/ graphics card and display controller). But the two jobs take a different amount of time. The number of images a CPU and graphics card can compute per second is determined by hardware performance. But the monitor refreshes at a fixed rate per second (typically 60hz, so it refreshes every 16.667ms).

The concept of a FrameBuffer was introduced because of the problem of inconsistent rates on both sides.

In the simplest case, there is only one frame buffer. At this point, the reading and flushing of the frame buffer will be relatively efficient. To address efficiency issues, gpus typically introduce two buffers, known as double buffering. In this case, the GPU will pre-render a frame into a buffer that displays the controller’s read. After the next frame is rendered, the GPU points the pointer to the display controller directly to the second buffer.

According to Apple’s documentation, iOS devices will always use Vsync + Double Buffering.

Screen tearing

Double buffering solves the efficiency problem, but it introduces a new one. When the display controller has not finished reading, that is, when the screen content is just half displayed, THE GPU submits a new frame content to the frame buffer and exchanges the two buffers, the display controller will display the lower part of the new frame data on the screen, causing the picture tearing phenomenon, as shown below:

To solve this problem, gpus usually have a mechanism called VSync (also known as V-sync). When VSync is enabled, the GPU will wait for a VSync signal from the display before performing a new frame rendering and buffer update. This will solve the problem of tearing and increase the smoothness of the picture, but it will consume more computing resources and cause some latency.

Frame drop

With vSYNC enabled, the CPU and GPU can ideally process each frame in 16ms. However, if the graphics card’s frame rate is less than the screen’s refresh rate, and the CPU and GPU take more than 16ms to process a frame, frame loss will occur. That frame is discarded until the next opportunity to display it, and the display remains unchanged. That’s why the interface gets stuck.

The controller occupies one Buffer and the GPU occupies one Buffer. Both buffers are occupied, causing the CPU to be idle and wasting resources. Due to VSync, the CPU cannot trigger drawing until the VSync signal is reached.

Three buffer mechanism

Starting with Android4.1, we introduced triple buffering + vsync. Due to the addition of a Buffer, CPU and GPU can be parallel, so that only one frame can be dropped at the beginning, but no subsequent frame can be dropped. Double buffering makes full use of 16ms to achieve low delay, and triple buffering ensures its stability.

IOS rendering framework

Graphics rendering of iOS App uses frameworks such as Core Graphics, Core Animation and Core Image to draw visual content, and these software frameworks are also dependent on each other. These frameworks all need to use OpenGL to call the GPU for drawing, and finally display the content on the screen.

UIKit

UIKit is the most commonly used framework for iOS developers. You can draw interfaces by setting the layout of UIKit components and related properties.

In fact, UIKit itself does not have the ability to image images on the screen. It is mainly responsible for responding to user operation events (UIView inherits from UIResponder), and the transmission of event responses is basically realized by traversing the view tree layer by layer.

Core Animation

Core Animation is derived from Layer Kit, and Animation is just the tip of the iceberg.

Core Animation is a composite engine whose job is to combine different on-screen visuals as quickly as possible. These visuals can be broken down into separate layers (calayers) that are stored in a system called layer trees. In essence, CALayer is the foundation of everything a user can see on screen.

Core Graphics

Core Graphics is based on the Quartz advanced Graphics engine and is primarily used to draw images at runtime. Developers can use this framework to handle path-based drawing, transformation, color management, off-screen rendering, patterns, gradients and shadows, image data management, image creation and image masking, and PDF document creation, display and analysis.

When developers need to create images at run time, they can use Core Graphics to draw them. The opposite is to create images before they run, such as using Photoshop to create images in advance and import them directly into the app. Instead, we need Core Graphics to compute and draw a series of image frames in real time to animate them.

Core Image

Core Image is the opposite of Core Graphics, which is used to create images at run time, and Core Image, which is used to process images created before run. The Core Image framework has a series of ready-made Image filters to efficiently process existing images.

In most cases, Core Image will do the work on the GPU, but if the GPU is busy, it will use the CPU for processing.

OpenGL ES

OpenGL ES (GLES) is a subset of OpenGL. OpenGL is a set of third-party standards. The internal implementation of functions is developed and implemented by the corresponding GPU manufacturer.

Metal

Metal is similar to OpenGL ES and is a third-party standard implemented by Apple. Most developers don’t use Metal directly, but virtually all developers use Metal indirectly. Core Animation, Core Image, SceneKit, SpriteKit and other rendering frameworks are all built on top of Metal.

When you debug the OpenGL program on a real machine, the console prints a log of Metal being enabled. From this point you can assume that Apple has implemented a mechanism to seamlessly bridge OpenGL commands to Metal and let Metal do the actual hardware interaction.

UIView and CALayer

CALayer is the foundation of everything a user can see on the screen, and is used to hold bitmaps. Every UIView in UIKit has a backing layer attached to it, called a CALayer.

Because of this one-to-one correspondence, the UIView level has the tree structure of the view tree, and the corresponding CALayer level has the tree structure of the layer tree.

The UIView’s job is to create and manage layers to ensure that when a subview is added or removed from a hierarchy, its associated layers do the same in the layer tree, ensuring that the view tree and layer tree are structurally consistent.

So why does iOS provide two parallel hierarchies based on UIView and CALayer?

The reason for this is separation of responsibilities, which also avoids a lot of duplicate code. Events and user interaction are different in a lot of ways on iOS and Mac OS X. There’s a fundamental difference between a multi-touch based user interface and a mouse and keyboard based interaction. That’s why iOS has UIKit and UIView. Mac OS X has AppKit and NSView. They are similar in function, but there are significant differences in implementation.

CALayer

In calayer.h, CALayer has this property contents

/** Layer content properties and methods. **/

/* An object providing the contents of the layer, typically a CGImageRef, * but may be something else. (For example, NSImage objects are * supported on Mac OS X 10.6 and later.) Default value is nil. * Animatable

@property(nullable.strong) id contents;
Copy the code

Contents provides the contents of layer and is a pointer type, which in iOS is CGImageRef (or NSImage in OS X). The contents property in CALayer holds the bitmap rendered by the device rendering pipeline (also known as the backing Store), and when the device screen is refreshed, the generated bitmap is read from the CALayer and rendered to the screen.

Graphics rendering pipelines support drawing from vertices (where vertices are processed to produce textures), as well as rendering directly with textures (images). Accordingly, in the actual development, there are two ways to draw the interface: one is manual drawing; The other is to use pictures.

  • Use image: Contents Image
  • Manual drawing: Custom Drawing

Contents Image Contents Image means to configure the Image through the Contents property of CALayer. However, the contents property is of type ID. In this case, you can give the contents property any value you want, and your app will still compile. But in practice, if the content value is not CGImage, the resulting layer will be blank.

Essentially, the contents property points to an area of cache, called the backing Store, where bitmap data can be stored.

Custom Drawing refers to Drawing boarding diagrams directly using Core Graphics. In practice, you typically customize your drawing by inheriting UIView and implementing the -drawRect: method.

Although -drawRect: is a UIView method, it’s actually the underlying CALayer that does the redrawing and saves the resulting image. The diagram below shows -drawRect: Drawing the basic principle that defines the boarding diagram.

  • UIViewThere is an association layer, i.eCALayer.
  • CALayerThere’s an optional onedelegateProperty, implementedCALayerDelegateThe agreement.UIViewAs aCALayerThe proxy is implementedCALayerDelegaeThe agreement.
  • Called when a redraw is required-drawRect:.CALayerAsk its agent for a boarding chart to display.
  • CALayerCall will be tried first-displayLayer:Method, where the proxy can be set directlycontentsProperties.
- (void)displayLayer:(CALayer *)layer;
Copy the code
  • If the proxy is not implemented-displayLayer:Method,CALayerAttempts to call-drawLayer:inContext:Methods. Before calling this method,CALayerWill create an empty boarding diagram (size bybounds 和 contentScaleDecide) and oneCore GraphicsTo prepare for rendering the homestay map, ascontextParameter passed in.
- (void)drawLayer:(CALayer *)layer inContext:(CGContextRef)ctx;
Copy the code
  • Finally, byCore GraphicsThe generated boarding map will be savedbacking store.
UIView

UIView is the basic structure in your app, and it defines some uniform specifications. It takes care of rendering the content and handling the interaction events.

  • Drawing and animation
  • Layout and subview Management: Layout and subview management
  • Event Handling: Click Event handling

CALayer is one of the UIView properties responsible for rendering and animating and providing visual presentation of content. UIView provides an encapsulation of some of the functions of CALayer and is also responsible for handling interactive events.

  • The same hierarchy: We are familiar with the hierarchy of UIView, since each UIView corresponds to a CALayer that is responsible for drawing pages, so CALayer also has a corresponding hierarchy.

  • Partial effect Settings: Because UIView only encapsulates part of CALayer’s functions, other effects such as rounded corners, shadows, borders, etc. need to be set by calling layer properties.

  • Whether to respond to click events: CALayer is not responsible for click events, so it does not respond to click events, whereas UIView does.

  • Different inheritance relationships: CALayer inherits from NSObject, UIView inherits from UIResponder because it’s responsible for interaction events.

Core Animation

Core Animation pipeline

In fact, the app itself is not responsible for rendering, rendering is handled by a separate process, the Render Server process.

App submits the rendering task and related data to Render Server through IPC. After the Render Server processes the data, it passes it to the GPU. Finally, GPU invokes iOS image device for display.

  • First of all, App handles Events, such as user click operations. During this process, APP may need to update view tree, and accordingly, layer tree will also be updated.
  • Secondly, APP completes the calculation of display content through CPU, such as view creation, layout calculation, picture decoding, text drawing, etc. After completing the calculation of the display, the app packs the layers, and the next timeRunLoopSend it toRender Server, that is, done onceCommit TransactionOperation.
  • Render ServerThe main executionOpen GL,Core GraphicsRelated procedures, and call the GPU.
  • The GPU completes the rendering of the image on the physical layer.
  • Finally, the GPU passesFrame Buffer,Video controllerAnd other related components, display the image on the screen.

The above steps are cascaded in succession and take far more than 16.67 ms to execute, so to support a screen refresh rate of 60 FPS, these steps need to be broken down and pipelined in parallel, as shown in the figure below.

The layer tree

CoreAnimation is a composite engine that combines different view layers on the screen and stores them in a layer tree, showing us everything on the screen.

The whole process actually goes through three tree structures before being displayed on the screen: model tree > Render tree > Render tree

In addition to view trees and layer trees, there are also render trees and render trees. Each of them has his own duty.

  • Render tree: We can go throughCALayerthe-presentationLayerMethod to access the corresponding render tree layer. Note that the render layer is only created when the layer is first submitted (the first time it is shown on screen), so it is called before that-presentationLayerIt’s going to return nil.
- (nullable instancetype)presentationLayer;
Copy the code
  • The model tree: called on the render layer- modelLayerWill return what it is rendering onCALayer. Usually called on a layer-modelLayerReturns theselfThe original layer we have created is actually a data model.
- (instancetype)modelLayer;
Copy the code

Usually, we operate the model tree modelLayer, and at the end of the redraw cycle, we serialize the model tree related content (hierarchy, layer properties and animation) and pass it to the renderer process that is responsible for screen rendering via IPC. The renderer takes the data and deserializes the tree structure — the rendering tree. This rendering layer is actually a copy of the model layer, but its property values represent the current appearance at any given moment. In other words, you can get the actual value displayed on the current screen by rendering the value of the layer.

When the model tree has animation features on the modelLayer, after submitting to the rendering process, the rendering process will constantly modify the layer properties on the presentationLayer of the rendering tree according to the animation features, and constantly render on the screen at the same time, so that we can see the animation.

If you want the animation layer to respond to user input, you can use the -hittest: method to determine whether the specified layer is touched. It makes more sense to call -hittest: on the rendering layer instead of the model layer, because the rendering layer represents the position the user is currently seeing, not the position after the current animation.

ModelLayer is responsible for storing and retrieving data, and presentationLayer is responsible for displaying it. The presentationLayer is synchronized with the modelLayer state every time the screen is refreshed.

When CAAnimation is added to the Layer, the presentationLayer will ask the CAAnimation and synchronize the state each time it is refreshed. The CAAnimation controls the presentationLayer from fromValue to toValue to change the value, and when the animation is done, the CAAnimation is removed from the layer. When the screen refreshes, the presentationLayer synchronizes the state of the modelLayer. The modelLayer does not change, so it is back to the starting point. Of course we can continue to influence the state of the presentationLayer by setting it.

The Core Animation Animation

Core Animation, or transaction-based Animation, is the most common Animation implementation. The animator is the rendering process that is responsible for rendering and operates on the rendering tree. We should use Core Animation to control animations as much as possible because Core Animation is fully optimized:

In the process of layer-based drawing, Core Animation operates bitmap (transformation, combination, etc.) by hardware, which generates Animation much faster than software.

The drawRect: method is triggered when the View is changed to redraw the bitmap, but this method needs to be executed by the CPU in the main thread, which is time-consuming. While Core Animation operates the cached bitmaps in hardware as much as possible to achieve the same effect, thus reducing resource consumption.

Non-core Animation

The non-corea Nimation animation executor is the current process and operates on the model tree. Common are timer animation and gesture animation. The timer animation modifies the layer properties of the model tree when the timer cycle is triggered. Gesture animation modifies the layer properties of the model tree when the gesture event is triggered. Both allow the view to change over time, enabling animation.

In fact, the model tree is constantly changed during the Animation process, while the rendering tree only becomes a copy of the model tree, and its state is consistent with that of the model tree. In the whole process, CPU constantly adjusted layer attributes, calculated layout and submitted data in the main thread, which failed to make full use of the powerful Animation control function of Core Animation.