At a time when short videos and live streaming occupy the iron throne of traffic, competition between products is fierce. Since douyin short video came out, numerous imitative products have followed. If a product wants to be removed from the market, it is difficult to win only by picture quality and fluency. It not only needs to save traffic costs but also ensure the ultra HIGH-DEFINITION picture quality, and at the same time, it also needs to have rich and innovative gameplay to attract users.

With the maturity of AI technology, scenes have become rich and diverse, such as: repairing old movies, face-changing effects, 2K-liter 4K, sports events, special effects APP.

However, no matter how the product needs change, we still have to work hard on image processing. So today, we will start from simple image processing, and explore video processing from shallow to deep, and contact more audio and video algorithms. If you are interested, you can also study the principle behind these algorithms.

Take the football match of a sports event as an example. The camera ensures that the picture quality is not torn in the process of automatically tracking the football (the position moves rapidly), and at the same time guarantees the overall picture quality of the HD video under the weak network mood. There are many technical difficulties. In addition to using AI to calculate the key positions (the ball and the player who predicted the ball to fly to) and the weak net situation audio and video data transmission (packet loss, replacement), these can actually be solved (optimized). For example, in the case of weak network, we only need to ensure the picture quality of the football and several players around, and all other pixels are blurred, which can greatly relieve the bandwidth pressure.

The principle of analysis

The image is composed of more than N finite pixels (bitmap non-vector graph), each pixel has its corresponding position and color value, the number of pixels determines the size of the image resolution, the more local pixels, the clearer the local image quality.

Image processing is to adjust the color intensity and transparency of a given pixel without changing the position of the pixel.

Then the image processing steps can be disassembled as:

  • Get image data: We need to specify the color values of all pixels of the image and their corresponding transparency.
  • Change image data: We can change the color value of all (or a specified range of) pixels of the image and their corresponding transparency.

For example, Meitu’s acne removal is to adjust the color intensity of the pixel position of the acne in the picture, and change the pixels in the range to the color intensity of skin color. If you have any questions, please continue reading with your questions and solve your doubts yourself.

Image data and processing

Let’s take the Web front end as an example. On the mobile end, you just need to understand the principle, and the same applies, just need to pay attention to the platform differences.

Web front ends typically use the ImageData constructor to create an ImageData object.

Grammar:

// Create an ImageData object for the specified data
const imageData = new ImageData(array, width, height);
// Create an ImageData object for blank data
const imageData2 = new ImageData(width, height);
Copy the code

The ImageData object contains the image width (imagedata.width), the height (imagedata.height) and the ImageData (imagedata.data) and other information:

  • width:

    The width of the picture

  • height:

    The high of the picture

  • Data:

    The queue used to store the position of each pixel of the image and the color value of the pixel is a one-dimensional array of type Uint8ClampedArray. The array element is an integer ranging from 0 to 255. It contains an integer array of RGBA data. The array length is 4 * imagedata.width * imagedata.height.

ImageData object width, height may be easier to understand, data attributes we can through the following questions and answers to understand:

  • Why is the array element of imageData. data an integer between 0 and 255?

    First of all, red, green and blue, as three primary colors, can be synthesized into a variety of colors, according to the intensity (ratio) of the color value of the three primary colors, which means that the more detailed the intensity grade of the three primary colors, the more colors will be mixed. In order to ensure the color value of the premise of the rich, RGB was eventually divided into 256 levels, can mix 16 million colors, and also meet the requirements of the video chip in binary storage, exactly 2 to the eighth power.

    Similarly, transparency must be an integer between 0 and 255. However, we often use 0 to 1 for transparency in the front end, which means that when we convert it to alpha, we divide it by 255.

  • Why is the imageData. data array a one-dimensional array, exactly 4 times the resolution?

    Because the array of imagedata. data records the color intensity and transparency of the red, green and blue colors of each pixel of the picture in the order of R,G,B and A. In other words, each of the four array elements identifies the color and transparency of a pixel. Not only the structure is simple, data volume and calculation speed have been greatly improved.

    The subscript of the array element 4n represents the color intensity of red at the NTH pixel, and the four elements 4n ~ 4n + 3 distinguish the color intensity and transparency of red, green and blue at the NTH pixel.

    Red = imageData.data[4n]; 
Green = imageData.data[4n + 1]; 
Blue = imageData.data[4n + 2];
Alpha = imageData.data[4n + 3];
Copy the code

An example of understanding ImageData

Friendly note: this example is not suitable for implementation, suitable for understanding theoretical knowledge. If you must, use Chrome and zoom in 500%.

Suppose we draw an image that is 3 pixels wide (PX) and 3 pixels high (PX). The first pixel is red, the second pixel is green, the third pixel is blue, and the fifth pixel is black. Then all other pixels are white.

Understand the length of imageData.data

Get ImageData for the image:

// Canvas Canvas blur effect const canvas = document.getelementById ('myCanvas'); const ctx = canvas.getContext('2d'); const img = new Image(); Img. SRC = "./test2.png"; Img. onload = function () {// Set canvas width = img.width; canvas.height = img.height; Ctx. drawImage(img, 0, 0, img.width, img.height); Const originImageData = ctx.getimageData (0, 0, img.width, img.height); consolt.log('originImageData', originImageData); };Copy the code

So the image’s ImageData must look like this:

// originImageData
ImageData: {
	width: 3,
	height: 3,
	data: Uint8ClampedArray(36),
	...
}
Copy the code

Resolution = image width * image height = 9 ImageData. Data length = resolution * 4 = 36

Understand the data for imageData.data

From the picture, we can know that the color values of these resolutions are:

  • The first pixel isred
    • RGBA values are: Red = 255, Green = 0, Blue = 0, Alpha = 255
  • The second pixel isgreen
    • RGBA values are: Red = 0, Green = 255, Blue = 0, Alpha = 255
  • The third pixel isblue
    • RGBA values are: Red = 0, Green = 0, Blue = 255, Alpha = 255
  • The fifth pixel isblack
    • RGBA values are: Red = 0, Green = 0, Blue = 0, Alpha = 255
  • Other arewhite
    • RGBA values are: Red = 255, Green = 255, Blue = 255, Alpha = 255

So let’s take a look at the imagedata.data and see if it matches what we guessed above

From the picture, we can see that:

  • Each of the four elements describes the color value of a pixel, respectively red saturation, green saturation, blue saturation and transparent saturation, and the range of integers between 0 and 255.
  • The array subscript is a value of 4 * n, representing the starting position of the NTH pixel
    • 4 * n is the red saturation
    • 4 * n + 1 represents green saturation
    • 4 * n + 2 represents the blue saturation
    • 4 * n + 3 represents transparency saturation

I wonder if you can get the pixel of the specified position when you see this.

Draw the image with ImageData

Not only can we get the ImageData (ImageData), but we can also use the ImageData constructor to create the ImageData and finally draw the image. Let’s also take the picture above as an example. What if we drew the whole picture?

Construct the ImageData. Data

We already know that imageData. data is a one-dimensional array of type Uint8ClampedArray, and we also know the color saturation of image. data, so the first step is to generate imageData. data

const arr = new Uint8ClampedArray(36); // First pixel, red, rgba(255, 0, 0, 1) arr[0] = 255 arr[1] = 0 arr[2] = 0 arr[3] = 255 Rgba (0, 255, 0, 1) arr[4] = 0 ARr [5] = 255 arr[6] = 0 arr[7] = 255 Rgba (0, 0, 255, 1) arr[8] = 0 ARr [9] = 0 arr[10] = 255 arr[11] = 255 Rgba (255, 255, 255, 1) ARr [12] = 255 arr[13] = 255 arr[14] = 255 arr[15] = 255 Rgba (0, 0, 0, 1) arr[16] = 0 ARr [17] = 0 arr[18] = 0 arr[19] = 255 (255, 255, 255, 1) arr[20] = 255 arr[21] = 255 arr[22] = 255 arr[23] = 255 (255, 255, 255, 1) arr[24] = 255 arr[25] = 255 arr[26] = 255 arr[27] = 255 (255, 255, 255, 1) arr[28] = 255 arr[29] = 255 arr[30] = 255 arr[31] = 255 (255, 255, 255, 1) arr[32] = 255 arr[33] = 255 arr[34] = 255 arr[35] = 255Copy the code
Construct the ImageData

We can generate an instance of ImageData by new an ImageData

const imgData = new ImageData(arr, 3, 3);
Copy the code
Canvas draw

Friendly note: this example is not suitable for implementation, suitable for understanding theoretical knowledge. If you must, use Chrome and zoom in 500%.

Through CanvasRenderingContext2D putImageData method draw the picture, in the end we can download the pictures to our local.

const canvas = document.getElementById('myCanvas');
const ctx = canvas.getContext('2d');
ctx.putImageData(imgData, 0, 0);
Copy the code

The Demo experience

We said above that if we want to process the picture, we must replace the color value of part of the pixel or adjust the gray level without changing the position of the pixel. In the case of already clear picture data format, it is much easier to process the picture.

Web does not allow you to modify images directly. Usually, we use CANVS to process images and finally save the processed images locally. Therefore, the following Demo will be carried out in Canvas.

Demo mainly from simple to deep step by step guide you to understand the picture processing, you can experience the Demo in the following order, to digest and understand the above knowledge points:

  • Gets the color value of any pixel of the picture
  • Gray the picture (black and white)
  • Add a Mosaic to the image

Gets the color value of the specified pixel in the picture

The code address

Black and white photos

The code address

Black and white algorithm

Mosaic

The code address

Mosaic algorithm

More (stay tuned)

I’ll have more interesting demos in the future, including AI face-changing, Gaussian blur, shadow elimination, HDR, and more.

Write in the last

Do you have a new understanding of image processing technology? Now let’s think about what image processing technology is used in these products to achieve what functions?

  • How does meitu Xiu Xiu and other retouching software work?
  • How does video processing software such as clip and reflect realize video clip and add special effects
  • Further understanding of the audio and video industry

Next web Technology analysis: We’ll take a look at image processing techniques and principles used in mainstream applications

  • Video processing
  • The Demo experience
  • Bilibili video hypersegmentation algorithm
  • Tiktok, Inke, pepper special effects processing
  • Realization principle of AVATAR “Ant tooth Black”