The principle of screen display image

Let’s start with the principle of CRT display in the past. CRT electron gun scans from top to bottom line in the above way. After scanning, the display displays a frame, and then the electron gun returns to the initial position to continue the next scan. To synchronize the display with the system’s video controller, the display (or other hardware) generates a series of timing signals using a hardware clock. When the gun moves to a new line and is ready to scan, the monitor emits a horizontal synchronization signal, or HSync; When a frame is drawn and the gun returns to its original position, the display sends a vertical synchronization signal, or VSync, before it is ready to draw the next frame. The monitor usually refreshes at a fixed rate, which is the frequency at which the VSync signal is generated. Although today’s devices are mostly LCD screens, the principle remains the same.

Generally speaking, the CPU, GPU, and display in a computer system work together in this way. The CPU calculates the display content and submits it to THE GPU. After the GPU finishes rendering, the rendering result is put into the frame buffer. Then the video controller will read the data of the frame buffer line by line according to the VSync signal and transmit it to the display through possible digital-to-analog conversion.

In the simplest case, there is only one frame buffer, and reading and flushing of the frame buffer can be inefficient. To solve the efficiency problem, the display system usually introduces two buffers, that is, double buffering mechanism. In this case, the GPU will pre-render one frame into a buffer for the video controller to read, and after the next frame is rendered, the GPU will point the video controller’s pointer directly to the second buffer. So there’s a huge increase in efficiency.

Double buffering solves the efficiency problem, but it introduces a new one. When the video controller has not finished reading, that is, the screen content is just half displayed, THE GPU submits a new frame content to the frame buffer and exchanges the two buffers, the video controller will display the lower part of the new frame data on the screen, causing the picture tearing phenomenon, as shown below:To solve this problem, gpus usually have a mechanism called V-sync. When VSync is enabled, the GPU will wait for the VSync signal from the display before performing a new frame rendering and buffer update. This will solve the problem of tearing and increase the smoothness of the picture, but it will consume more computing resources and cause some latency.

Causes and solutions of caton



After the arrival of VSync signal, the graphics service of the system will notify the App through CADisplayLink and other mechanisms, and the main thread of the App will start to calculate the display content in the CPU, such as view creation, layout calculation, picture decoding, text drawing, etc. Then THE CPU will submit the calculated content to the GPU, which will transform, synthesize and render. The GPU then submits the rendering result to the frame buffer and waits for the next VSync signal to be displayed on the screen. Due to the VSync mechanism, if the CPU or GPU does not complete the content submission within a VSync period, the frame will be discarded and displayed at the next opportunity, while the display will keep the previous content unchanged. That’s why the interface gets stuck.

As you can see from the above figure, whichever CPU or GPU blocks the display process will cause frame drops. Therefore, it is necessary to evaluate and optimize CPU and GPU pressure respectively during development.

Causes and solutions of CPU resource consumption

Object creation

The creation of an object allocates memory, adjusts properties, and even reads files, consuming CPU resources. You can optimize performance by replacing heavy objects with lightweight objects. For example, CALayer is much lighter than UIView, so controls that do not need to respond to touch events are better displayed with CALayer. If the object does not involve UI operations, try to create it in the background thread, but unfortunately controls that contain CALayer can only be created and manipulated in the main thread. Creating view objects in Storyboard is much more expensive than creating them directly in code, and Storyboard is not a good technology choice for performance-sensitive interfaces.

Delay object creation as long as possible and spread it out over multiple tasks. Although this is a bit of a hassle to implement and doesn’t offer many advantages, if you can do it, try it. If objects can be reused and the cost of reuse is less than releasing and creating new objects, then such objects should be reused in a cache pool whenever possible.

Object to adjust

Object tuning is also a frequent CPU drain. Here’s a special word for CALayer: CALayer has no properties inside it. When a property method is called, it temporarily adds a method to the object through the runtime resolveInstanceMethod and stores the corresponding property value in an internal Dictionary. It also notifies the delegate, creates animations, and so on, which is very resource-intensive. UIView’s display-related properties (frame/bounds/ Transform, for example) are actually mapped from the CALayer properties, so adjusting these UIView properties consumes far more resources than normal properties. You should minimize unnecessary property changes in your application.

There are a lot of method calls and notifications between UIView and CALayer when view hierarchies are adjusted, so you should avoid adjusting view hierarchies, adding and removing views when optimizing performance.

Object is destroyed

The destruction of objects consumes a small amount of resources, but it adds up to a significant amount. Usually when a container class holds a large number of objects, the resource cost of their destruction becomes apparent. Similarly, if an object can be released in a background thread, move it to the background thread. Here’s a little Tip: Capture an object in a block, throw it on a background queue, send a random message to avoid compiler warnings, and let the object be destroyed in the background thread.

NSArray *tmp = self.array;
self.array = nil;
dispatch_async(queue, ^{
    [tmp class];
});
Copy the code

Layout calculation

View layout calculation is the most common CPU drain in an App. If the view layout is precomputed in the background thread and cached, there are few performance issues in this place.

Regardless of the technology to the view layout through, it will eventually fall for UIView. The frame/bounds/center on the adjustment of the property. As mentioned above, adjusting these attributes is very expensive, so try to calculate the layout in advance and adjust the corresponding attributes at once if necessary, rather than multiple, frequent calculation and adjustment of these attributes.

Autolayout

Autolayout is a technology advocated by Apple itself and can be a great way to improve development efficiency in most cases, but Autolayout can often cause serious performance problems for complex views. As the number of views grows, the CPU consumption from Autolayout increases exponentially. For more information, see pilky.me/36/. If you don’t want to manually adjust the frame and other attributes, you can use some tools to replace (such as common left/right/top/bottom/width/height and quick properties), or use ComponentKit, AsyncDisplayKit framework, etc.

The text calculated

If an interface contains a large amount of text (such as weibo and wechat moments, etc.), text width and height calculation will occupy a large portion of resources and is inevitable. If you have no special requirements for text display, you can refer to the internal implementation of UILabel: With [NSAttributedString boundingRectWithSize: options: context:] to calculate the text width is high, With [NSAttributedString drawWithRect: options: context:] to draw text. Although these two methods perform well, they still need to be put into background threads to avoid blocking the main thread.

If you draw text in CoreText, you can create a CoreText typeset object and do the calculation yourself, and the CoreText object can be saved for later drawing.

Text rendering

All text content controls that can be seen on the screen, including UIWebView, are at the bottom of the page formatted and drawn as bitmaps through CoreText. Common text controls (UILabel, UITextView, etc.), its typesetting and drawing are carried out in the main thread, when the display of a large number of text, the CPU pressure will be very large. There is only one solution, and that is a custom text control that asynchronously draws text using TextKit or the low-level CoreText. Once the CoreText object is created, it can directly obtain the width and height of the text, avoiding multiple calculations (once when the UILabel size is adjusted, and again when the UILabel is drawn). CoreText objects take up less memory and can be cached for later multiple renders.

Image decoding

When you create an image using UIImage or CGImageSource methods, the image data is not immediately decoded. The image is set to UIImageView or calayer.contents, and the data in the CGImage is decoded before the CALayer is submitted to the GPU. This step occurs on the main thread and is unavoidable. If you want to get around this mechanism, it is common to draw images into CGBitmapContext in the background line and then create images directly from the Bitmap. At present, the common network photo library has this function.

Image drawing

Drawing an image usually refers to the process of drawing an image onto a canvas using methods that begin with CG, and then creating and displaying the image from the canvas. The most common place to do this is inside [UIView drawRect:]. Since CoreGraphic methods are usually thread-safe, drawing images can easily be put into background threads. The process for a simple asynchronous drawing looks something like this (it’s much more complicated than this, but the principle is the same) :

- (void)display { dispatch_async(backgroundQueue, ^{ CGContextRef ctx = CGBitmapContextCreate(...) ; // draw in context... CGImageRef img = CGBitmapContextCreateImage(ctx); CFRelease(ctx); dispatch_async(mainQueue, ^{ layer.contents = img; }); }); }Copy the code

Causes and solutions of GPU resource consumption

Compared to the CPU, the GPU can do a single thing: take the submitted Texture and vertex description, apply the transform, mix and render, and then print it to the screen. The main things you can see are textures (pictures) and shapes (vector shapes for triangle simulation).

Texture rendering

All bitmaps, including images, text and rasterized content, are eventually committed from memory to video memory and bound to the GPU Texture. Both the process of submitting to video memory and the process of GPU adjusting and rendering Texture consume a lot of GPU resources. When a large number of images are displayed in a short period of time (such as when the TableView has a large number of images and slides quickly), the CPU usage is very low and the GPU usage is very high, and the interface will still drop frames. The only way to avoid this situation is to minimize the display of a large number of pictures in a short period of time, and to display as many pictures as possible.

When the image is too large to exceed the maximum texture size of the GPU, the image needs to be preprocessed by the CPU, which will bring additional resource consumption to the CPU and GPU. For now, iPhone 4S and up, texture size is up to 4096 x 4096, more information can be found here: iosres.com. So try not to let images and views exceed this size.

Blending of views

When multiple views (or Calayers) are displayed on top of each other, the GPU blends them together first. If the view structure is too complex, the mixing process can also consume a lot of GPU resources. To reduce GPU consumption in this situation, applications should minimize the number and level of views and indicate opaque attributes in opaque views to avoid useless Alpha channel composition. Of course, this can also be done by pre-rendering multiple views as a single image.

Graph generation.

CALayer’s border, rounded corners, shadows and masks, CASharpLayer’s vector graphics display, usually trigger offscreen rendering, which usually happens on the GPU. When a list view shows a large number of calayers with rounded corners and a quick swipe, you can observe that the GPU resource is full and the CPU resource consumption is low. The interface still slides normally, but the average number of frames drops to a very low level. ShouldRasterize to avoid this, try turning on the calayer.shouldrasterize property, but this will shift the off-screen rendering onto the CPU. For situations where only rounded corners are needed, you can also simulate the same visual effect by overlaying the original view with an already drawn rounded corner image. The most radical solution is to draw the graphics that need to be displayed as images in background threads, avoiding rounded corners, shadows, masks, and other attributes.

Rendering frameworks in iOS

The iOS rendering framework still conforms to the basic architecture of the rendering pipeline, with the technical stack shown above. On the basis of hardware, iOS has Core Graphics, Core Animation, Core Image, OpenGL and other software frameworks to draw content, and carries out higher-level encapsulation between CPU and GPU.

GPU Driver: The above software frameworks also depend on each other, but all frameworks eventually connect to the GPU Driver through OpenGL. The GPU Driver is a code block that communicates directly with the GPU.

OpenGL: is an API that provides 2D and 3D graphics rendering. It can work closely with the GPU to make the most of GPU capabilities and achieve hardware-accelerated rendering. An efficient implementation of OpenGL (using graphic-acceleration hardware) is typically provided by the display device vendor and is heavily dependent on that vendor’s hardware. Many extensions to OpenGL, such as Core Graphics, eventually rely on OpenGL, and in some cases for greater efficiency, such as game programs, even call OpenGL interfaces directly.

Core Graphics: Core Graphics is a powerful 2d image rendering engine. It is the Core Graphics library of iOS. Commonly used Graphics such as CGRect are defined under this framework.

Core Image: Core Image is a high-performance Image processing and analysis framework. It has a series of ready-made Image filters to efficiently process existing images.

MetalMetal is similar to OpenGL ES. It is a third-party standard implemented by Apple. Core Animation, Core Image, SceneKit, SpriteKit and other rendering frameworks are all built on top of Metal.

Core AnimationIt is essentially a composite engine that renders, builds, and implements animations.

Core Animation is usually used to implement animations efficiently and easily, but in fact its predecessor is called Layer Kit and Animation implementation is only part of its functionality. For iOS apps, no matter whether Core Animation is directly used or not, it is deeply involved in app construction at the bottom level. For OS X app, some functions can be easily realized by using Core Animation.

Core Animation is the perfect low-level support for AppKit and UIKit. It is also integrated into the workflow of Cocoa and Cocoa Touch. It is the most basic architecture for app interface rendering and construction. Core Animation’s job is to combine the different visual content on the screen as quickly as possible. This content is broken down into separate layers (specifically CALayer in iOS) and stored as a tree hierarchy. This tree also forms the basis of UIKit and everything you see on screen in iOS apps. Basically, everything on the screen that users can see is managed by CALayer. So how exactly does CALayer manage? In addition, in iOS development, the most commonly used view control is actually UIView rather than CALayer, so what is the relationship between the two?

CALayer is the basis of the display: it stores bitmaps

Simply put, CALayer is the foundation of on-screen display. So how did CALayer pull it off? In calayer. h, CALayer has this property contents:

/** Layer content properties and methods. **/ /* An object providing the contents of the layer, typically a CGImageRef, * but may be something else. (For example, NSImage objects are * supported on Mac OS X 10.6 and later.) Default value is nil. strong) id contents;Copy the code

Contents provides the contents of layer and is a pointer type, which in iOS is CGImageRef (or NSImage in OS X). As we further check, Apple defines CGImageRef as A bitmap image or image mask.

In fact, the contents property in CALayer holds the bitmap rendered by the device rendering pipeline (also known as the backing Store), and when the device screen is refreshed, the generated bitmap is read from the CALayer and rendered to the screen.

So, if we set the contents property of CALayer in our code like this:

// typedef struct CGImage CGImageRef; // typedef struct CGImage CGImageRef; layer.contents = (__bridge id)image.CGImage;Copy the code

Then, when the code is copied, the operating system will call the underlying interface and render the image through the RENDERING pipeline of CPU+GPU to obtain the corresponding bitmap, which will be stored in calayer.contents. When the device screen is refreshed, the bitmap will be read and presented on the screen. Because the content to be rendered is stored statically each time, the Core Animation triggers the drawRect: method to use the stored bitmap for a new display.

Relationship between CALayer and UIView

UIView, the most commonly used view control, is also closely related to CALayer. So what is the relationship between UIView and CALayer? What are the differences? Of course, there are obvious differences, such as the ability to respond to click events. But in order to understand these problems fundamentally and thoroughly, we must first understand the responsibilities of both.

UIView is the basic structure in your app, and it defines some uniform specifications. It takes care of rendering the content and handling the interaction events. Specifically, it is responsible for the following three categories:

  • Drawing and animation
  • Layout and subview Management: Layout and subview management
  • Event Handling: Click Event handling

CALayer’s main role is to manage internal visual content, which is consistent with what we talked about earlier. When we create a UIView, the UIView automatically creates a CALayer, provides itself with a place to store bitmaps, and sets itself up as the backing store of the CALayer.

From here, we can roughly summarize the following two core relationships:

  • CALayer is one of the UIView properties responsible for rendering and animating and providing visual presentation of content.
  • UIView provides an encapsulation of some of the functions of CALayer and is also responsible for handling interactive events.

With these two key underlying relationships, it’s easy to explain the obvious similarities and differences that often crop up in interview answers. A few examples:

  • The same hierarchy: We are familiar with the hierarchy of UIView, since each UIView corresponds to a CALayer that is responsible for drawing pages, so CALayer also has a corresponding hierarchy.

  • Partial effect Settings: Because UIView only encapsulates part of CALayer’s functions, other effects such as rounded corners, shadows, borders, etc. need to be set by calling layer properties.

  • Whether to respond to click events: CALayer is not responsible for click events, so it does not respond to click events, whereas UIView does.

  • Different inheritance relationships: CALayer inherits from NSObject, UIView inherits from UIResponder because it’s responsible for interaction events.

Of course, the last question is why CALayer should be separated from UIView? Why not have one unified object for everything? The main reason for this design is to separate responsibility, split function, convenient code reuse. The Core Animation framework is responsible for rendering visual content, which can be rendered using Core Animation on both iOS and OS X. At the same time, the two systems can further encapsulate unified controls according to the different interaction rules. For example, iOS has UIKit and UIView, while OS X has AppKit and NSView.

Offscreen Rendering

Off-screen rendering process

If we want to display content on the screen, we need at least one frame buffer with the same amount of pixel data on the screen as the pixel data storage area, and this is where the GPU stores the rendering results. If the rendering result cannot be written directly to the frame buffer due to some limitations, but is temporarily stored in another area of memory before writing to the frame buffer, this process is called off-screen rendering.

The efficiency of off-screen rendering

GPU operation is highly pipelined. All of the computational work was being methodically output to the frame Buffer, when suddenly we received an instruction that we needed to output to another piece of memory, and everything that was going on in the pipeline had to be discarded and switched to serving only our current “rounding” operation. Wait until you are done, then clear again, and return to the normal flow of output to the Frame Buffer.

In addition, the Offscreen Buffer itself needs extra space, and a large number of off-screen rendering may cause too much pressure on memory. At the same time, the total size of the Offscreen Buffer is also limited and cannot exceed 2.5 times the total pixels of the screen. The overhead of visible off-screen rendering is very high. Once too much content needs to be rendered off-screen, it is easy to cause the problem of dropping frames. So for the most part, we should try to avoid off-screen rendering.

Why use off-screen rendering

So why use an off-screen rendering? Mainly because of the following two reasons:

  • 1. For the purpose of efficiency, the content can be rendered in advance and stored in Offscreen Buffer. For example, some layer data frequently used later can be cached in off-screen Buffer and reused directly when used
  • 2. There are some special effects that cannot be completed by normal process and must be rendered off-screen, such as rounded corners, shadows and masks, Gaussian blur, translucent layer blending, etc

What circumstances trigger an off-screen rendering

  • Layer with mask (layer.mask)
  • Layer (layer.maskstobounds/view.clipstobounds)
  • Set up a group to YES, transparency and transparency of 1 layer (layer. AllowsGroupOpacity/layer. Opacity)
  • Added shadow layer (layer.shadow*)
  • ShouldRasterize (layer. ShouldRasterize)
  • Layer with Text drawn (UILabel, CATextLayer, Core Text, etc.)

How to optimize

  • The AsyncDisplayKit(Texture) can be used as the main rendering framework, and the asynchronous rendering of text and images is handled by the framework.
  • For fillet corners of the picture, the “precomposite” strategy is adopted uniformly, that is, the fillet corners of the picture are cut using CoreGraphics in advance without cutting through the container
  • As for the rounded corners of the video, since real-time cutting consumes performance, four white arc-shaped layers can be created to cover the four corners and create the rounded corners visually
  • If you don’t have a backgroundColor, use a cornerRadius
  • For all shadows, use shadowPath to avoid off-screen rendering
  • For special shaped views, use layer Mask and shouldRasterize to cache the render results
  • For the blur effect, instead of using the UIVisualEffect provided by the system, CIGaussianBlur is implemented separately and the rendering result is managed manually

Reference: blog.ibireme.com/2015/11/12/… Juejin. Cn/post / 684490… zhuanlan.zhihu.com/p/72653360