Core Image is a powerful and efficient Image processing framework. You can use the built-in filters provided by the framework to create beautiful effects, as well as create custom filters and image handlers. You can adjust colors, geometry, and perform complex convolution.

Making beautiful filters is an art, and one of the greatest artists was Leonardo da Vinci. In this tutorial, you will add some interesting elements to Leonardo Da Vinci’s famous paintings.

In the process, you will:

  • Learn about Core Image’s classes and built-in filters.
  • Create filters using built-in filters.
  • Convert the color of the image using a custom color kernel.
  • Transform image geometry using custom distortion kernel.
  • Learn how to debug Core Image problems.

Note: Due to an Apple error, this tutorial does not work with Xcode 13 and iOS 15. You must now use Xcode 12.

Get your brushes ready, oops, I mean your Xcode is ready. It’s time to dive into the wonderful world of Core Image!

An introduction to

You will see four of Da Vinci’s most famous works. Clicking on a painting opens a piece of paper, but the output of the image is empty.

In this tutorial, you will create filters for these images and then see the results of applying the filters to the output.

Slide down to close the worksheet. Next, click on the list of filters in the upper right corner.

This button should display a list of available built-in filters. But wait, it’s currently empty. You will solve this problem next. :]

Introduces the core image class

Before you can populate the filter list, you need to understand the basic classes of the Core Image framework.

  • CIImage: Represents an Image ready for processing or generated by the Core Image filter. A CIImage object has data for all the images in it, but not the actual image. It’s like a recipe, it contains all the ingredients to make a dish, but not the dish itself.

    You will see how to render the image to display later in this tutorial.

  • CIFilter: Obtain one or more images, process each image by applying transform and generate aCIImage as its output. You can link multiple filters and create interesting effects. Object CIFilters are mutable and not thread-safe.

  • CIContext: Displays the result of the filter processing. For example, CIContext helps create a Quartz 2D image from a CIImage object.

To learn more about these classes, see the Core Image tutorial: Getting Started.

Now that you’re familiar with the Core Image class, it’s time to populate the filter list.

Gets a list of built-in filters

Open RayVinci and select FilterListView.swift. Replace filterList in FilterListView with:

FilterList = cifilter.filterNames (inCategory: nil)Copy the code

Here, you get a list of all the available built-in filters provided by Core Image by passing filterNames(inCategory:) and nil as categories. You can view the developer documentation for CIFilter in the list of available categories.

Open the FilterDetailView. Swift. Replace Text(“Filter Details”) with body:

// 1 if let ciFilter = CIFilter (name: The filter) {/ / 2 ScrollView {Text (ciFilter. Attributes. The description)}} else {/ / 3 Text (" unknown filter!" )}Copy the code

Here you are:

  1. ciFilterInitializes the filter with the filter name. Because the name is a string and can be misspelled, the initializer returns an optional. Therefore, you need to check whether the filter exists.
  2. You can use it to check various attributes of the filterattributes. If a filter exists, you will create it hereScrollViewAnd populateTextProperty description in view.
  3. If the filter does not exist or is unknown, you will display oneTextA view explaining the situation.

Build and run. Click on the filter list. Wow, there are so many filters!

Click on any filter to view its properties.

Amazing, isn’t it? You’re just getting started! In the next section, you will use one of the built-in filters to shine sunlight on the Mona Lisa. :]

Use built-in filters

Now that you’ve seen a list of available filters, you’ll use one of them to create an interesting effect.

Open the ImageProcessor. Swift. At the top, before the class declaration, add:

enum  ProcessEffect  {  case builtIn case colorKernel case warpKernel case blendKernel }
Copy the code

Here, you declare ProcessEffect as enum. It contains all the filter cases that you will use throughout this tutorial.

Add the following to the ImageProcessor:

// 1 private func applyBuiltInEffect (input: CIImage) {// 2 let noir = CIImage ( "CIPhotoEffectNoir", parameter: [" input image ": input])? .outputimage // 3 let sunGenerate = CIFilter (name: "CISunbeamsGenerator", parameter: ["inputStriationStrength" : 1 , "inputSunRadius" : 300 , "inputCenter" : CIVector ( x: input.extent.width - input.extent.width / 5 , y: input.extent.height - input.extent.height / 10 ) ]) ? Let outputImage / / 4 CompositeImage = input. ApplyingFilter (" CIBlendWithMask parameters: [kCIInputBackgroundImageKey: noir as Any , kCIInputMaskImageKey: sunGenerate as Any ]) }Copy the code

Here you are:

  1. Declare that a will be aCIImagePrivate methods that serve as input and apply built-in filters.
  2. You can get started by creating a dark, moodyblackThe use effect ofCIPhotoEffectNoir.CIFilterTake strings as names and arguments in the form of dictionaries. You get the generated filter image fromoutputImage.
  3. Next, you useCISunbeamsGeneratorThis will create a sun mask. In parameters, you set:
    • inputStriationStrength: indicates the intensity of sunlight.
    • inputSunRadius: indicates the radius of the sun.
    • inputCenter: x and y positions in the center of the sun. In this case, you set the location in the upper right corner of the image.
  4. Here, you can useCIBlendWithMask. You caninputBy setting the resultCIPhotoEffectNoirIs the background image andsunGenerateMask the map image to apply the filter. The result of this combination isCIImage.

ImageProcessor has Output, a published property, which is UIImage. You need to convert the composite result to aUIImage to display it.

In ImageProcessor, add @published var output = UIImage() :

Let context = CIContext ()Copy the code

Here, you create an instance that all CIContext filters will use.

Add the following to the ImageProcessor:

ivate func renderAsUIImage ( _ image : CIImage ) -> UIImage ? {if let cgImage = context.createcgImage (image, from: image.extent) {return UIImage (cgImage: cgImage)} returns zero}Copy the code

Here, you use the context to create an instance of CGImagefrom, CIImage.

Use cgImage, and then create a UIImage. The user will see this image.

Displays the output of the built-in filter

Add the following to the end applyBuiltInEffect(input:) :

OutputImage = renderAsUIImage(compositeImage) {output = outputImage}Copy the code

This converts the compositeImagea CIImage to UIImageusing renderAsUIImage(_:). The result is then saved to output.

Add the following new methods to the ImageProcessor:

// 1 func process ( Painting : Painting , effect : ProcessEffect ) { // 2 guard let paintImage = UIImage (named: Painting.image), Let input = CIImage (image:paintImage) else {print ("Invalid input image") return} switch effect {// 3 case.builtin: ApplyBuiltInEffect (input: input) Default: Print (" unsupported effect ")}}Copy the code

Here you are:

  1. Create a method as the entry pointImageProcessor. It requires an instancePaintingAnd aeffectTo apply.
  2. Check for valid images.
  3. If the effect type is.builtInThe callapplyBuiltInEffect(input:)To apply the filter.

Open the PaintWall. Swift. Insert Button in the action closure of selectedPainting = paintings[index] :

var effect = ProcessEffect .builtIn if let Painting = selectedPainting { switch index { case 0 : Effect =.builtin Default: effect =.builtin} imageProcessor.shared. process (draw: draw, effect: effect)}Copy the code

Here, you will effect to.builtin for the first drawing. You also set it to default. Then, apply the filter on the ImageProcessor by calling Process (painting:, Effect :).

Build and run. Click * “Mona Lisa” *. You will see a built-in filter applied to the output!

The great job of keeping the Mona Lisa in the sun. No wonder she’s smiling! Now it’s time to create a filter using CIKernel.

Know CIKernel

With CIKernel, you can place custom code called kernel to manipulate images pixel by pixel. The GPU processes those pixels. You write kernels using Metal Shading Language, which has the following advantages over older Core Image Kernel languages, deprecated since iOS 12:

  • Supports all the powerful features of the Core Image kernel, such as wiring and tiling.
  • Precompile at build time, with error diagnostics. That way, you don’t have to wait for runtime errors.
  • Provides syntax highlighting and syntax checking.

There are different types of kernels:

  • CIColorKernel: Changes the color of a pixel without knowing its position.
  • CIWarpKernel: Changes the position of the pixel without knowing the color of the pixel.
  • CIBlendKernel: Blends two images in an optimized manner.

To create and apply a kernel, you need:

  1. First, add custom build rules to the project.
  2. Then, add the Metal source file.
  3. Load the kernel.
  4. Finally, the kernel is initialized and applied.

Next, you will implement each of these steps. Get ready for a fun ride!

Creating a build rule

You need to compile Core Image Metal code and link it to a special flag.

Select the RayVinci target in the project navigator. Then, select the Build Rules TAB. Click *+* to add a new build rule.

Then, set the first new build rule:

  1. willprocessSet toSource files with matching names:. Then set **.ci.metal* to the value.
  2. uncheckRun once per architecture.
  3. Add the following script:
xcrun metal -c -I $MTL_HEADER_SEARCH_PATHS -fcikernel "${INPUT_FILE_PATH}" \ -o "${SCRIPT_OUTPUT_FILE_0}"
Copy the code
This invokes the Metal compiler with the required *-fcikernel* flag.Copy the code
  1. inOutput fileAdd the following:
$(DERIVED_FILE_DIR)/${INPUT_FILE_BASE}.air
Copy the code
This produces an output binary ending in *.ci.air*.Copy the code

Next, click *+* again to add another new build rule.

For the second new build rule, follow these steps:

  1. willprocessSet toSource files with matching names:. Then set **.ci.air* to the value.
  2. uncheckRun once per architecture.
  3. Add the following script:
xcrun metallib -cikernel "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"
Copy the code
This invokes the Metal linker with the required *-cikernel* flag.Copy the code
  1. inOutput fileAdd the following:
$(METAL_LIBRARY_OUTPUT_DIR)/$(INPUT_FILE_BASE).metallib
Copy the code
This will generate a file in the application package ending with *.ci.metallib*.Copy the code

Next, it’s time to add the Metal source.

Add metal source

First, you will create a source file for the color kernel. In the Project Navigator, highlight the RayVinci Project under the RayVinci authority.

Right-click and select New Group. Name this new group Filters. Then, highlight the group and add a named ColorFilterKernel. Ci. Metal new metal file.

Open the file and add:

// 2 extern "C" {// 3 float4 colorFilterKernel (sample_t s) {// 4 float4 Swap colors; Switch colors. R = sg; G = sb; B = sr; swappedColor.a = sa; Return exchange color; }}}Copy the code

Here’s the code breakdown:

  1. Including the Core Image header gives you access to classes provided by the framework. This will automatically include the Core Image Metal kernel libraryCIKernelMetalLib.h.
  2. The kernel needs to be locatedextern "C"Inside the shell so that it can be accessed by name at run time. Next, you specify the namespacecoreimage. You are incoreimageDeclare all extensions in the namespace to avoid collisions with Metal.
  3. Here, you declarecolorFilterKernel, which accepts input of typesample_t.sample_tRepresents a single color sample from the input image.colorFilterKernelreturnfloat4Representing the RGBA value of the pixel.
  4. Then, you declare a new onefloat4,swappedColorAnd swap RGBA values in the input samples. It then returns a sample with swapped values.

Next, you’ll write code to load and apply the kernel.

Loading kernel code

To load and apply the kernel, you first create the CIFilter.

Create a new Swift file in the filter group. Name it colorfilter.swift and add:

CoreImage class ColorFilter: CIFilter {// 2 var inputImage: CIImage? // 3 static var kernel: CIKernel = { () -> CIColorKernel in guard let url = Bundle .main.url( forResource: "ColorFilterKernel.ci" , withExtension: "metallib" ), let data = try? Else {fatalError (" Unable to load metallib ")} Guard lets kernel = try? CIColorKernel ("colorFilterKernel", fromMetalLibraryData: Data) else {fatalError (" cannot create color kernel ")} return kernel}() // 4 override var outputImage: CIImage? {guard let inputImage = inputImage else {return nil} return colorfilter.kernel. apply(range: Inputimage. extent, roiCallback: {_, RECT return rectangle}, parameter: [inputImage])}}Copy the code

Here you are:

  1. First, import the Core Image framework.

  2. Subclassing CIFilter involves two main steps:

    • Specify input parameters. Here, you useinputImage.
    • coveroutputImage.

  3. Then, you declare a static attribute kernel, used to load ColorFilterKernel. Ci. Metallib content. In this way, the library is loaded only once. Then use CIColorKernel ColorFilterKernel. Ci. The content of the metallib create an instance.

  4. Next, you override outputImage. Here, you use the apply (among other: roiCallback: the arguments:). This extent determines how much of the input image is passed to the kernel.

    You passed the entire image, so the filter will be applied to the entire image. RoiCallback determines the outputImage of the input image required by Rect to render rectin. Here, rectofinputImage and outputImage do not change, so you return the same value and pass in the inputImage parameter array to the kernel.

Now that you have created the color kernel filter, you will apply it to the image.

Apply a color kernel filter

Open the ImageProcessor. Swift. Add the following methods to the ImageProcessor:

private func applyColorKernel ( input : CIImage ) { let filter = ColorFilter () filter.inputImage = input if let outputImage = filter.outputImage, Let renderImage = renderAsUIImage(outputImage) {output = renderImage}}Copy the code

Here, you declare applyColorKernel(input:). This takes aCIImage as input. You can create custom filter ColorFilter through the created instance.

The filter outputImage applies a color kernel. Then create a UIImageusing instance, renderAsUIImage(_:), and set it to output.

ColorKernel in process(painting:effect:) is shown below. Add the new case default above:

ColorKernel: applyColorKernel (input: input)Copy the code

Here, you call applyColorKernel(input:) to apply the custom color kernel filter.

Finally, open PaintingWall. Swift. Add the following to the statement directly below the switch in the’s closure: Case 0 ‘Button’ action

Case 1: Effect =.colorkernelCopy the code

This sets the effect of the second drawing of.colorkernel to.

Build and run. Now click on the second picture * “Last Supper” *. You will see the color kernel filter applied and the RGBA values exchanged in the image.

Very good! Next, you’ll create a cool twist on Da Vinci’s mysterious “Savior.”

Creating a twisted kernel

Like the color kernel, you’ll start by adding the Metal source file. The filter in the group to create a named WarpFilterKernel. Ci. Metal new metal file. Open the file and add:

# include < coreImage.h > //1 extern "C" {//2 float2 warpFilter(destination Dest) {float y = dest.coord().y + tan(dest.coord().y / 10 ) * 20 ; Float x = ded.coord ().x + tan(ded.coord ().x/ 10) * 20; Returns the float2 (x, y); }}}Copy the code

Here’s what you added:

  1. Just as in the color kernel Metal source, you include the Core Image header and include the method in an extern “C” shell. Then specify the CoreImage namespace.

  2. Next, you warpFilter(_:) declare destination with the input parameter of type, allowing access to the location of the pixel you are currently calculating. It returns the position in the input image coordinates, which you can then use as the source.

    You can use coord() to access the x and y coordinates of the target pixel. You then apply simple math to transform coordinates and return them as source pixel coordinates to create interesting tiling effects.

    Note: Try replacing tan with in and you’ll get an interesting distortion effect! :] sin“warpFilter(_:)

Loading the twisted kernel

Similar to the filters you created for the color kernel, you will create a custom filter to load and initialize the distortion kernel.

Create a new Swift file in the filter group. Name it warpfilter.swift and add:

WarpFilter: CIFilter {var inputImage: CIImage? // 2 static var kernel: CIWarpKernel = { () -> CIWarpKernel in guard let url = Bundle .main.url( forResource: "WarpFilterKernel.ci" , withExtension: "metallib" ), let data = try? Else {fatalError (" Unable to load metallib ")} Guard lets kernel = try? Function name: "warpFilter", fromMetalLibraryData: Data) else {fatalError (" cannot create distorted kernel ")} return kernel}() // 3 override var outputImage: CIImage? {guard make inputImage = inputImage else {return. none} return warpfilter.kernel. apply(range: inputImage. Extent, roiCallback: {_, RECT return rectangle}, image: input image, parameter: [])}}Copy the code

Here you are:

  1. createWarpFilterforCIFilterwithinputImageAs a subclass of the input parameter.
  2. Next, you declare static propertieskernelIn order to loadWarpFilterKernel.ci.metallibThe content of the. And then create aCIWarpKernelAn instance of the content used.metallib.
  3. Finally, you pass the overrideoutputImage. In theoverride, you apply the kernel toinputImageusingapply(extent:roiCallback:arguments:)And returns the result.

Apply distorted kernel filters

Open the ImageProcessor. Swift. Add the following to the ImageProcessor:

private func applyWarpKernel ( input : CIImage ) { let filter = WarpFilter () filter.inputImage = input if let outputImage = filter.outputImage, Let renderImage = renderAsUIImage(outputImage) {output = renderImage}}Copy the code

Here, you declare applyColorKernel(input:), which takes CIImage as input. Then create an instance of WarpFilter and set, inputImage.

The filter outputImage applies to the distorted kernel. Then create a UIImageusing instance, renderAsUIImage(_:), and save it to the output.

Next, add the following case to process(painting:effect:), following case.colorkernel:

WarpKernel: applyWarpKernel (input: input)Copy the code

Here, you deal with.warpkernel and apply the distorted kernel filter by calling applyWarpKernel(input:).

Finally, open PaintingWall. Swift. Add the following action in case 1 below the right of switch:

Case 2: Effect =.warpkernelCopy the code

WarpKernel sets the effect for the third painting.

Build and run. Click on Salvator Mundi’s painting. You will see an interesting application of tile effects based on distortion.

A: congratulations! You apply your style to your masterpiece! ; ]

The challenge: Implementing a hybrid kernel

The CIBlendKernel is optimized for mixing two images. As a fun challenge, here are some tips for CIBlendKernel:

  1. To create aCIFilterAccepts two subclasses of images: input image and background image.
  2. Use the built-inavailable CIBlendKernelThe kernel. For this challenge, useBuilt-in multiplicationHybrid kernels.
  3. Create a methodImageProcessorApply the hybrid kernel filter to the image and set the result as output. You can use the one provided in the project assetmulti_colorThe image serves as the background image for the filter. In addition, processing.blendKernel.
  4. Apply this filter toPaintingWall. SwiftThe fourth picture of.

You will find the solution implemented in the final project in the downloaded materials. Good luck!

Debug core image issues

Knowing how Core Image renders an Image can help you debug when the Image doesn’t display as expected. The easiest way to do this is to use Core Image Quick Look during debugging.

Quick view using core images

Open the ImageProcessor. Swift. Put breakpoints where you set line output in applyColorKernel(input:). Build and run. Click on * last Supper *.

When you hit a breakpoint, hover over outputImage. You will see a small pop-up box that displays the address.

Click the eye symbol. A window appears showing the graph that made the image. Pretty cool, huh?

Using CI_PRINT_TREE

CI_PRINT_TREE is a debugging function based on the same infrastructure as Core Image Quick Look. It has multiple modes and operations.

Select and edit the RayVinci scheme. Select the Run TAB and add CI_PRINT_TREE as a new environment variable with a value of 7 PDF.

The CI_PRINT_TREE value takes the form graph_type output_type options.

Graph_type represents the stage of core image rendering. Here are the values you can specify:

  • 1: Displays the initial graphic of the color space.
  • 2: An optimized diagram showing how Core Image is optimized.
  • 4: A connection diagram that shows how much memory you need.
  • 7: Records detailed logs. This will print all of the above charts.

For output_type, you can specify PDF or PNG. It saves the document to a temporary directory.

Build and run. Select * The Last Supper in the simulator. Now open the temporary directory on your Mac by using terminal navigation to/TMP *.

You will see all graphics as PDF files. Open one of the files with the suffix *_initial_graph.pdf*.

Input is at the bottom, output is at the top. Red nodes represent colored cores and green nodes represent distorted cores. You’ll also see the ROI and scope for each step.