This article explores the process of rendering an image from file to screen in order to understand the rendering process of shaders.
Graphic image rendering process
Display display image principle
- The CPU calculates the frame of the image -> image decode -> draw the texture image and hand it to the GPU for rendering.
- The GPU waits for the vSYNC signal V-sync and performs shader rendering after receiving the VSYNC signal, that is, the vertex data is processed into pixel slices by coordinate transformation of the vertex shader… raster, and the texture is mixed after the pixel shader processing, and finally put into the off-screen buffer.
- The pointer of the video controller points to the screen buffer and is sent to the display. After a frame of image is displayed, the two buffers are switched, and the pointer of the video controller points to the new buffer content, and at this time, the GPU will receive a V-sync to continue rendering the new frame content.
It can be seen from the above process that the image display is completed through the cooperation of CPU and GPU. One problem occurs in this process, that is, when GPU receives v-sync signal, CPU and GPU start to work, and neither CPU nor GPU finishes their work when the next V-sync signal appears. As a result, there is no frame data in the off-screen buffer at this time, and the stacken phenomenon will occur. For a more detailed explanation, see Ibireme’s tips for keeping the interface smooth on iOS.
Image loading workflow
Before you start exploring the process of loading images, be clear about two concepts.
- Bitmap: A bitmap is an array of pixels, each element in the array represents a pixel of the bitmap. Jpeg and PNG are bitmaps, just compressed bitmaps. Jpeg is lossy compression (0~1), PNG is lossless compression.
- Pixel: Is a basic element of an image, an image is made up of many pixels, a pixel has one and only one color, pixel is made up of 4 different vectors namely RGBA, red, green, blue, alpha (how much light passes through the current image).
So what does a CPU do?
When you load an image using several methods of UIImage, the image doesn’t get decoded immediately, it goes through a series of steps
- 1. Read images from disk into buffer: as we use
imageWithContentsOfFileLoad an image from disk. Instead of decompressing the image, read the binary data of the image (JPEG or PNG, etc.) into the kernel buffer, i.eData Buffer. - 2. Copy the kernel buffer to user memory space
- 3, the
imageSet it toimageViewThat is, when the image is to be displayed, the binary data of the image is placedunzipRestore to original bitmap data,Core AnimationLayer renders the UIImageView layer using bitmap dataVery CPU time consuming. - implicit
CATransactionCapture the changes to the UIImageView layer - 5. Next time on the main thread
RunloopArrival,Core AnimationCommit implicitlyCATransactionStart rendering the image.- 5.1 If the data is not byte aligned, Core Animation copies another data for byte alignment.
- 6, GPU processing bitmap data start rendering (this step is GPU processing)
- GPU gets texture coordinates
- The texture coordinates are converted to normalized coordinates by vertex shaders
- The converted data is assembled into primitives
- The pixel is rasterized into pixel slice
- To be colored by the chip shader. GPU processing is actually the process of rendering to the screen through the shader, see the shader rendering process below
The rendering process of the shader
- Vertex data -> Vertex shader The vertex shader processes each vertex, such as translation, scaling, rotation, various model transformations, visual transformations, projection transformations, etc., to transform it into standardized coordinates.
- Vertex shaders -> Subdivision shaders -> Geometry shaders describe the shape of the object, process the model geometry (make it smoother, drop the flanges), and generate the final state. In OpenGL ES, developers can only work with vertex shaders and chip shaders, so there’s not too much entanglement here.
- Geometry Shader -> Primitives Set the processed data to primitives
- Pixel Settings -> Cut Cut the pixel according to the viewport, clipping out what is not visible from the viewport.
- Clip-> rasterization is actually the processing of primiprimions, parsing primiprimions into mathematical descriptions, and turning them into coordinates and pixel tiles that can be displayed on the screen.
- Rasterize -> Pixel shaders color pixel tiles.
This is the rendering process of the shader.
As you can see from this, the image decoding process is very time-consuming, which is fine for an App when it is still, but when it is sliding down the list, the performance impact is very serious. We usually measure the performance of a mobile phone by the number of Frames Per Second. 60FPS (60 Frames Per Second) is the standard to measure whether the phone will be stuck. The processing time of a frame should not exceed 0.1666s or 16.7ms. So let’s see what we can do about UIImage and YYImage.
UIImage image processing
UIImage has multiple ways to get an image, so what’s the difference between these different ways? A, imageNamed
ImageNamed, when it first renders to the screen, it decodes the image in the main thread, and the bitmap data is cached, and then when you access the image, you get it from the cache, and it says on the web that when the phone sends a memory warning, it clears the UIImage cache. There are two problems: 1) the first image decoding operation is still done in the main thread; 2) Bitmap data is too large to be stored in the cache.
ImageWithContentsOfFile and imageWithData
ImageWithContentsOfFile and imageWithData will also be decoded in the main thread when rendering to the screen for the first time. If the UIImage object is released, it will still be decoded in the main thread when accessing it again. This small icon is ok, but there is a problem with the large image
Therefore, the common practice in the market is to force the image decoding ahead of time by redrawing it in a child thread using CGBitmapContextCreate before the CPU decompresses it.
YYImage image processing
Find YYKit Demo directly used, YYImage is called as follows
YYImage *image = [YYImage imageNamed:name];
[self addImage:image size:CGSizeZero text:text];
Copy the codeYYImage is integrated from UIImage, and imageNamed method is rewritten to avoid global cache of the system
+ (YYImage *)imageNamed:(NSString *)name {// if there is no extension, you need to find it. @[ext] : @[@""The @"png"The @"jpeg"The @"jpg"The @"gif"The @"webp"The @"apng"]; // Scales are an array similar to @[@1,@2,@3]. Because different models have different physical or logical resolutions, the corresponding query priorities are different. NSArray *scales = _NSBundlePreferredScales();for (int s = 0; s < scales.count; s++) {
scale = ((NSNumber *)scales[s]).floatValue;
NSString *scaledName = _NSStringByAppendingNameScale(res, scale);
for (NSString *e in exts) {
path = [[NSBundle mainBundle] pathForResource:scaledName ofType:e];
if (path) break;
}
if (path) break;
}
NSData *data = [NSData dataWithContentsOfFile:path];
return [[self alloc] initWithData:data scale:scale];
}
Copy the codeThis is basically the process of getting the binary data of the image from disk. Scales are an array similar to @[@1,@2,@3]. Because different models have different physical or logical resolutions, the corresponding query priorities are also different. All of YYImage’s methods call initWithData, and then initWithData
- (instancetype)initWithData:(NSData *)data scale:(CGFloat)scale {
if (data.length == 0) return nil;
if(scale <= 0) scale = [UIScreen mainScreen].scale; _preloadedLock = dispatch_semaphoRE_create (1); // creates a large number of temporary variables, YYImageDecoder *decoder = [YYImageDecoder decoderWithData:data scale:scale]; DecodeForDisplay :YES [decoder frameAtIndex:0 decodeForDisplay:YES]; // UIImage *image = frame.image;if(! image)return nil;
self = [self initWithCGImage:image.CGImage scale:decoder.scale orientation:image.imageOrientation];
if(! self)return nil;
_animatedImageType = decoder.type;
if (decoder.frameCount > 1) {
_decoder = decoder;
_bytesPerFrame = CGImageGetBytesPerRow(image.CGImage) * CGImageGetHeight(image.CGImage);
_animatedImageMemorySize = _bytesPerFrame * decoder.frameCount;
}
self.yy_isDecodedForDisplay = YES;
}
return self;
}
Copy the codePreloadedLock: This is a thread lock, controlled by semaphore, for preloaded data read security. YYImageDecoder: Get some basic information about the image, such as width and height, frame number and image type
Principle of decompression
YYImage uses Image I/O to decode images. There are two ways: first, CGImageGetDataProvider(imageRef)
CGColorSpaceRef space = CGImageGetColorSpace(imageRef);
size_t bitsPerComponent = CGImageGetBitsPerComponent(imageRef);
size_t bitsPerPixel = CGImageGetBitsPerPixel(imageRef);
size_t bytesPerRow = CGImageGetBytesPerRow(imageRef);
CGBitmapInfo bitmapInfo = CGImageGetBitmapInfo(imageRef);
if (bytesPerRow == 0 || width == 0 || height == 0) return NULL;
CGDataProviderRef dataProvider = CGImageGetDataProvider(imageRef);
if(! dataProvider)return NULL;
CFDataRef data = CGDataProviderCopyData(dataProvider); // decode
if(! data)return NULL;
CGDataProviderRef newProvider = CGDataProviderCreateWithCFData(data);
CFRelease(data);
if(! newProvider)return NULL;
CGImageRef newImage = CGImageCreate(width, height, bitsPerComponent, bitsPerPixel, bytesPerRow, space, bitmapInfo, newProvider, NULL, false, kCGRenderingIntentDefault);
CFRelease(newProvider);
return newImage;
Copy the codeCGImageSourceRef: use CGImageSourceCreateWithData function, According to get pictures of binary data from disk to create imageSource CGImageRef: use CGImageSourceCreateImageAtIndex function creates a CGImage CGDataProviderRef: bitmap data provider, CGImageGetDataProvider function, anyhow is to take from this bitmap data CFDataRef: CGDataProviderCopyData, obtain CFDataRef data from the DataProvider. CGImageRef:CGImageCreate creates a CGImage based on the bitmap data.
CGBitmapContextCreate
CGImageRef YYCGImageCreateDecodedCopy (CGImageRef imageRef, BOOL decodeForDisplay) {/ / whether or not to unpackif (decodeForDisplay) { //decode with redraw (may lose some precision)
CGImageAlphaInfo alphaInfo = CGImageGetAlphaInfo(imageRef) & kCGBitmapAlphaInfoMask;
BOOL hasAlpha = NO;
if(alphaInfo == kCGImageAlphaPremultipliedLast || alphaInfo == kCGImageAlphaPremultipliedFirst || alphaInfo == kCGImageAlphaLast || alphaInfo == kCGImageAlphaFirst) { hasAlpha = YES; } // BGRA8888 (premultiplied) or BGRX8888 // same as UIGraphicsBeginImageContext() and -[UIView drawRect:] CGBitmapInfo bitmapInfo = kCGBitmapByteOrder32Host; / / alpha is the least significant bit RGBA, for example, or in the most significant bit like ARGB / / when the image does not contain alpha use kCGImageAlphaNoneSkipFirst, Otherwise, use kCGImageAlphaPremultipliedFirst. bitmapInfo |= hasAlpha ? kCGImageAlphaPremultipliedFirst : kCGImageAlphaNoneSkipFirst; CGContextRef context = CGBitmapContextCreate(NULL, width, height, 8, 0, YYCGColorSpaceGetDeviceRGB(), bitmapInfo);if(! context)return NULL;
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef); // decode
CGImageRef newImage = CGBitmapContextCreateImage(context);
NSLog(@"currentThread = %@", [NSThread currentThread]);
CFRelease(context);
returnnewImage; }}Copy the codeThe decompression of CGBitmapContextCreate is used
CG_EXTERN CGContextRef __nullable CGBitmapContextCreate(void * __nullable data, size_t width, size_t height, size_t bitsPerComponent, size_t bytesPerRow, CGColorSpaceRef cg_nullable space, Uint32_t bitmapInfo) CG_AVAILABLE_STARTING (10.0, 2.0);Copy the codeYou can see that this function takes the following arguments
data: The value is usually specified as NULL to allow the system to automatically allocate and release memorywidth: Pixel width of the bitmapheight: Pixel height of the bitmapbitsPerComponent: The number of bytes occupied by each color vector of a pixel, usually 8 bytes, RGBA has 4 vectors, so a total of 32 bytes.bytesPerRow: Number of bytes occupied by a bitmap line. Generally, 0/NULL is writtenwidth * bitsPerPixelAnd the system will optimize it for us.space: Color space, fill in RGBbitmapInfo: Layout information, in the case of alphakCGImageAlphaPremultipliedFirst, in the absence of alphakCGImageAlphaNoneSkipFirst.
By printing the thread on which this function is executed, you can see that the operation is performed asynchronously, which we will see later is exactly asynchronous serial.
YYAnimatedImageView parsing
YYAnimatedImageView integrates UIImageView and rewrites many methods, starting with initialization
- (instancetype)initWithImage:(UIImage *)image { self = [super init]; _runloopMode = NSRunLoopCommonModes; // Set the mode of runloop to common. _autoPlayAnimatedImage = YES; self.frame = (CGRect) {CGPointZero, image.size }; self.image = image; // Override setter methodsreturn self;
}
Copy the codeDuring initialization, a runloopMode is defined as NSRunLoopCommonMode, which means that the GIF will also be played during the runloop switch.
- (void)setImage:(UIImage *)image {
if (self.image == image) return;
[self setImage:image withType:YYAnimatedImageTypeImage];
}
- (void)setImage:(id)image withType:(YYAnimatedImageType)type {
[self stopAnimating];
if (_link) [self resetAnimated];
_curFrame = nil;
switch (type) {
case YYAnimatedImageTypeNone: break;
case YYAnimatedImageTypeImage: super.image = image; break;
case YYAnimatedImageTypeHighlightedImage: super.highlightedImage = image; break;
case YYAnimatedImageTypeImages: super.animationImages = image; break;
case YYAnimatedImageTypeHighlightedImages: super.highlightedAnimationImages = image; break;
}
[self imageChanged];
}
Copy the code_link: CADisplayLink is a timer for playing animations. The first time you enter setImage, the imageChanged method is executed to determine the image and container size, and the tag timer waits for the next runloop to start playing the animation.
- (void)imageChanged {// Get image type YYAnimatedImageType newType = [self currentImageType]; id newVisibleImage = [self imageForType:newType]; NSUInteger newImageFrameCount = 0;if(newImageFrameCount > 1) { [self resetAnimated]; _curAnimatedImage = newVisibleImage; _curFrame = newVisibleImage; _totalLoop = _curAnimatedImage.animatedImageLoopCount; _totalFrameCount = _curAnimatedImage.animatedImageFrameCount; [self calcMaxBufferCount]; } // mark RunLoop, timer (CAD) [selfsetNeedsDisplay];
[self didMoved];
}
Copy the codeInitialization takes place in the resetAnimated method, including the _lock thread lock, the _buffer cache container initialization, the _requestQueue thread queue initialization, the timer _link initialization, and so on.
- (void)resetAnimated {
if(! _link) { _lock = dispatch_semaphore_create(1); _buffer = [NSMutableDictionary new]; _requestQueue = [[NSOperationQueue alloc] init]; / / asynchronous serial queue _requestQueue. MaxConcurrentOperationCount = 1; _link = [CADisplayLink displayLinkWithTarget:[_YYImageWeakProxy proxyWithTarget:self] selector:@selector(step:)];if (_runloopMode) {
[_link addToRunLoop:[NSRunLoop mainRunLoop] forMode:_runloopMode];
}
_link.paused = YES;
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(didReceiveMemoryWarning:) name:UIApplicationDidReceiveMemoryWarningNotification object:nil];
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(didEnterBackground:) name:UIApplicationDidEnterBackgroundNotification object:nil];
}
[_requestQueue cancelAllOperations];
LOCK(
if (_buffer.count) {
NSMutableDictionary *holder = _buffer;
_buffer = [NSMutableDictionary new];
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0), ^{
// Capture the dictionary to global queue,
// release these images inbackground to avoid blocking UI thread. [holder class]; }); }); _link.paused = YES; _time = 0;if(_curIndex ! = 0) { [self willChangeValueForKey:@"currentAnimatedImageIndex"];
_curIndex = 0;
[self didChangeValueForKey:@"currentAnimatedImageIndex"];
}
_curAnimatedImage = nil;
_curFrame = nil;
_curLoop = 0;
_totalLoop = 0;
_totalFrameCount = 1;
_loopEnd = NO;
_bufferMiss = NO;
_incrBufferCount = 0;
}
Copy the codeNSOperationQueue: the initialized thread queue is usedNSOperationQueue, cut setmaxConcurrentOperationCountThe maximum number of concurrent requests is one, indicating that this is oneA serial asynchronous queue, but it does not block the current thread and opens another thread to perform the task._YYImageWeakProxy: To prevent circular references, a class derived from NSProxy is used for message forwarding._buffer: cache pool, which avoids UIImage’s global cache and registers two notifications to clear the cache when a memory warning is received and it goes into the background. Initialization of some other parameters.
conclusion
1. Picture rendering to screen process: read file from disk -> compute Frame-> picture decoding -> submit to GPU rendering via data bus -> Vertex shader -> Raster processing -> Slice shader coloring -> Render to Frame buffer -> Video controller pointing to Frame buffer -> display. 2. YYImage avoids global caching and asynchronously forces image decompression before images are displayed, which greatly improves performance. In fact, this library has many advantages, so I will take time to look at it later.