Session 719: Core Image: Performance, Prototyping, and Python

It is believed that most iOS developers are familiar with Core Image. As an unusually powerful Image processing library equipped with the system, Core Image can meet the Image processing requirements of App in most scenarios. In addition, Core Image now supports custom filter on iOS, catching up with GPUImage (GPUImage is the de facto benchmark for Image processing on iOS). In addition to iOS12, apple plans to deprecate OpenGL and OpenGL ES to promote Metal. Core Image, which is closely associated with Metal, has a better chance of winning.

The content of this Session consists of three parts:

  1. Core Image’s new performance API;
  2. Quickly build filter prototype on Core Image system;
  3. Applying machine learning on Core Image system;

1. New performance API for Core Image

Among the cache

So before we talk about intermediate caching, we need to review Core Image. In Core Image, we can execute Filter of Image chain, as shown below:

Through the combination of Filter, we can achieve some complex image processing effects. To establish a Filter chain, refer to the following code (excerpt from Core Image Programming Guide) :

func applyFilterChain(to image: CIImage) -> CIImage {
    // The CIPhotoEffectInstant filter takes only an input image
    let colorFilter = CIFilter(name: "CIPhotoEffectProcess", withInputParameters:
        [kCIInputImageKey: image])!
    
    // Pass the result of the color filter into the Bloom filter
    // and set its parameters for a glowy effect.
    letbloomImage = colorFilter.outputImage! .applyingFilter("CIBloom",
                                                             withInputParameters: [
                                                                kCIInputRadiusKey: 10.0,
                                                                kCIInputIntensityKey: 1.0
        ])
    
    // imageByCroppingToRect is a convenience method for
    // creating the CICrop filter and accessing its outputImage.
    let cropRect = CGRect(x: 350, y: 350, width: 150, height: 150)
    let croppedImage = bloomImage.cropping(to: cropRect)
    
    return croppedImage
}
Copy the code

The whole process is straightforward, we feed the image to the first Filter, get the outputImage of the first Filter, and then feed the object to the second Filter… The Filter chain is established by analogy.

The Core Image of the Lazy

It is worth noting that when the above code is executed, Image processing does not take place, but some relations are established inside the Core Image. Only when the Image needs to be rendered, the Image processing process of each Filter will be actually executed.

Due to its Lazy nature, the most important optimization on Core Image is “Filter Concatenation “, because the final Image processing process takes place after all the filters are chained. Therefore, Core Image can merge multiple linked filters into one to save unnecessary overhead. As shown below:

Among the cache

Now go back to this scenario:

Three filters. The calculation of the first Filter is time-consuming, while the parameter of the third Filter can be manually adjusted by the user. This means that the three filters need to be recalculated after each user adjustment. In fact, the parameters of the first two filters remain unchanged, that is to say, the calculation process and results of the first two filters do not change as users adjust the parameters of the third Filter. Is it possible to optimize the repeated calculations here?

It is easy to imagine that we would just cache the results of the first two operations, as shown in the following figure:

However, as mentioned above, Core Image will automatically merge the Filter chains into a single Filter. How do we access the intermediate results?

In Core Image of iOS12, apple added an insert intermediate attribute to CIImage to solve this problem, as shown in the figure below:

We want to save the result of the second Filter by calling insertingIntermediate() in the outputImage of the second Filter to generate a new CIImage for the subsequent flow. In this way, the parameter adjustment of the third Filter will not lead to the double calculation of the first two filters.

How do you do that? Basically, the auto-merge logic is adjusted according to insert Intermediate. As shown below:

CIContext of Core Image can set whether to enable cacheIntermediate, but insertingIntermediate has a higher priority. For some specific suggestions, please refer to the following figure:

New features of the Kernal language

Two modes

IOS supports custom Filter, which is developed in Kernal (a scripting language similar to GLSL). There are currently two modes of developing CIKernel programs:

The first is to write in traditional CIKernal development language and then compile into Metal or GLSL, the second is to directly use Metal Shading language for development and then generate binary library during build. Load is directly converted to GPU instructions after the execution phase.

At present, because Apple is promoting Metal Performance Shader (MPS), method 1 has been marked as deprecated.

According to the group, speaking, reading and writing

Advantages of using Metal to develop CIKernel:

  1. Supports semi-precision floating point numbers;
  2. Support Group read & Group write;

Semi-precision floating-point is an optimization of pure operational properties. It is faster on the A11 chip and has more room for optimization because of the small registers used.

Next, focus on group reading and writing.

Suppose we perform a 3×3 convolution operation on the pixels in the red box on the left, and the result is stored in the green box on the right. Obviously, for each new pixel, you need to read the input image nine times.

However, if you read and write by group, see the following figure. We read 16 pixels at a time to calculate and write the four pixels on the right, so we wrote 4 times and read 16 times in the whole process. The average number of readings per new pixel is 4, significantly lower than the 9 readings per pixel described above.

The principle of group read and write is very simple. If we have a KerNAL developed using CIKernal Language, we can modify it to use high-speed optimization of group read and write.

Suppose our kerNAL looks like this:

Step 1, convert to metal:

The second step is to change the mode to read and write by group. The core is to use S. gaatherx to implement.

With group read and write and the classic optimization of half-floating point, you can almost get a two-fold performance improvement.

2. Quickly build filter prototype on Core Image system

Generally speaking, the typical research and development process of a filter is to carry out rapid prototyping test on the computer, and then transfer to the production environment, where there are a lot of tools (OpenCV, SciPy, Numpy, Tensorflow, etc.) for rapid prototyping, while the technology stack in the production environment is Core Image. VImage, Metal, and other completely different technical architecture stacks, this often leads to a problem: a prototype that works well on the computer, only to crash on the phone.

To solve this problem, apple released PyCoreImage.

Does it feel strange to see the term for the first time? But it’s pretty obvious that this is the interface that you can call Core Image from in Python.

We used Python + PyCoreImage in Prototype, which mimicked the real world as much as possible, and basically ported to the phone as well. And the most crucial is, want to learn a frame only good, much of learn not to finish ！！！！

PyObjc, released in OS X 10.5, implements code that calls Objective-C in Python. The main conversion rule is colon to underscore. See the example in the figure for details.

PyCoreImage interacts with the macOS CoreImage via PyObjC and outputs the result back to NumPy.

The code in the figure below first imports an image, then applies a Gaussian blur Filter to it, and then outputs the result to the variable bar.

For more information on PyCoreImage usage, please refer to the Session PPT.

3. Apply machine learning on Core Image system

Image processing and computer development up to now, has been largely through the use of machine learning and deep learning to improve the effectiveness of algorithms. Core Image also provides great support for machine learning.

CoreML Filter

Core Image can now apply images directly to a CoreML model, as opposed to deep learning capabilities for Core Image filters.

Core Image in iOS12 provides the CICoreMLModelFilter class to encapsulate CoreML’s Model into a Filter format that Core Image can recognize.

Below is an example of a classic application of the ML domain (style transfer)

But CICoreMLModelFilter is still completely unavailable on the Internet, (fog

Data to fill in

For machine learning, the completeness and coverage of the training set can largely determine the accuracy of the final model. However, the reality is that we often do not have so many training sets. In this case, the academic circle generally adopts the corresponding changes to the existing training sets to expand the data set. These technologies are known as Data Augmentation

Core Image naturally supports this type of task well, with support for the following types of changes:

  1. Image appearance;
  2. Image noise;
  3. Geometric transformation;

Here are some examples of using different filters of Core Image to convert one Image into many training images:

summary

Generally speaking, the content brought by this Session is exciting. Although some students may think it is small and does not have that kind of subversive innovation, for students engaged in the field of image, there is no doubt that these jobs all give a feeling of mind opener. It really reflects apple’s advanced thinking in these two areas of multimedia and user experience.