Image Processing and Compositing in JavaScript (4)

Quote:

The series is now conceived as the following four parts:

Base type image processing techniques for scaling, cropping and rotation (portal);
Basic types of image processing techniques for image composition (portal);
Basic type image processing techniques for text composition (portal);
Algorithm type Image processing technology (portal);

Through this accumulation, I have encapsulated several features commonly used in projects:

Image synthesis Image cropping Like to dig in addition to

Previous articles mainly introduced cropping/rotation/composition and other basic types of image processing (text composition writing… 🐒🐒🐒), we begin to introduce algorithmic image processing techniques! ~ ~ ✈ ️ ✈ ️ ✈ ️

The focus of this type is mainly on the algorithm and performance level, in the front end due to the limitations of JS and device performance, usually not ideal. In the real online business, in order to pursue better user experience, only some relatively lightweight algorithms with good performance can be run. It is better to do this on the server side.

Tips: Since I do not have a deep understanding of algorithms, this article is mainly about the outer layers and basic principles of algorithms, not about algorithms themselves. Hope you understand ha ~🙈

We take the following two 🌰 to do a preliminary understanding:

(1) Halloween app

Halloween effect

The renderings are as follows:

This little app is a Halloween activity. The puppet makeup is really cool, but it requires complex face recognition, model comparison and makeup algorithm, which is difficult to put in the front-end performance, so it is obviously a better choice to let the server to deal with it. The border and background blur processing, this type of processing is more suitable for the front end, first of all, the performance is acceptable, and more flexible, can be in different entrances at any time to replace different border materials.

For the server-side makeup algorithm, because I did not in-depth research on the algorithm, here is not to teach a lesson, we will directly comb the part of the front end:

Send the original picture to the server and accept the effect picture after makeup treatment;
After downloading the effect picture, zoom into a suitable size for fuzzy processing, get the fuzzy result picture;
Pixel-level fusion of the resulting image/blur image/border;

Tips: This is all pixel-level algorithm fusion, which can also be achieved through composition of base types.

Algorithm performance improvement

The essential principle of image algorithm processing is actually traversing pixels and transforming the RGBA value of pixels. For the transformation algorithm itself, this article will not go into depth, but can share with you the relevant experience.

As we all know, a good algorithm, one of the most important indicators is performance, and how to improve performance? One is algorithm optimization, to improve the performance within the loop or to optimize the traversal algorithm, the performance of the algorithm will be magnified countless times due to the existence of traversal. The other is to reduce the number of pixels.

Pixel traversal is an important performance consumption point of an algorithm, and the number of cycles directly determines the performance of the algorithm. The number of pixels increases exponentially with the size of the image. Therefore, scaling the image source properly before processing it can improve performance greatly. For example, a 2000*2000 image has 4 million pixels, which means it needs to be traversed 4 million times. When the image is shrunk to 800*800, the number of loops is 640,000. Here is a test I did:

let st = new Date().getTime();
let imgData = [];
for (let i = 0; i < n * 10000; i += 4) {
    let r = getRandom(0.255),
        g = getRandom(0.255),
        b = getRandom(0.255),
        a = 1;
    if (r <= 30 && g <= 30 && b<= 30) a = 0;
    imgData[i] = r;
    imgData[i + 1] = g;
    imgData[i + 2] = b;
    imgData[i + 3] = a;
}
let et = new Date().getTime();
let t = et - st;
console.log(`${n}10000 times time:${et - st}ms`, imgData);
Copy the code

Test result is (MAC – Chrome -20 times average):

Image size	Pixel number	Time (ms)	Zoom multiples	ascension
`2000 * 2000`	4 million	168	1	0%
`1600 * 1600`	2.56 million	98	0.8	42%
`1200 * 1200`	1.44 million	64	0.6	62%
`800 * 800`	640000	32	0.4	81%
`400 * 400`	160000	10	0.2	94%

You can see that the reduction of the image has a very significant performance improvement. One of the features is that the performance gains decrease as the scaling factor increases. At 0.8 the performance gains increase by 42%, while at 0.6 the gains decrease by 20%. At the same time, zooming in and out of the image means a decrease in quality, so we need to find a balance to improve the performance as much as possible without affecting the effect of the resulting image, which needs to be determined by the image quality requirements of the algorithm.

In addition, cutting the original image is also a good way to cut out the redundant background, can also achieve the effect of reducing the number of iterations, improve performance.

Fuzzy algorithm

The fuzzy part of the small application uses the fuzzy algorithm of StackBlur. Js, and the application code is as follows:

// Zoom in;
let srcImg = scaleMid(imgData);

// Create a container for the fuzzy result graph;
let blurCvs = document.createElement('canvas'),
	blurCtx = blurCvs.getContext('2d');

// Make a copy of the original data;
let blurImg = blurCtx.createImageData(srcImg.width, srcImg.height);
let size = srcImg.width * srcImg.height * 4;
for (let i = 0; i < size; i++) {
    blurImg.data[i] = srcImg.data[i];
}

// Scale to 400*400;
blurImg = scale(blurImg, 400);

// blur;
StackBlur.imageDataRGBA(blurImg, 0.0, blurImg.width, blurImg.height, 1);

// After processing and then enlarge 800*800;
blurImg = scale(blurImg, 800);
Copy the code

Image fusion

Now we have all the materials for the final image, blur the background/makeup/border images, and the final step is to merge the three. The principle of fusion is to divide the final image into different areas and fill in the corresponding material data in each area:

// Image fusion
function mix(src, blur, mtl) {
	  // The final result is fixed 800*800; Longitudinal 800 data;
    for (let i = 0; i < 800; i++) {
        let offset1 = 800 * i;
        
        // Data from the horizontal 800;
        for (let j = 0; j < 800; j++) {
            let offset = (offset1 + j) * 4;
            
            // Fill in the material at a specific position;
            if (i <= 75 || i >= 609 || j <= 126 || j >= 676) {
                let alpha = mtl.data[offset + 3] / 255.0;
                mtl.data[offset] = alpha * mtl.data[offset] + (1 - alpha) * blur.data[offset];
                mtl.data[offset + 1] = alpha * mtl.data[offset + 1] + (1 - alpha) * blur.data[offset + 1];
                mtl.data[offset + 2] = alpha * mtl.data[offset + 2] + (1 - alpha) * blur.data[offset + 2];
                mtl.data[offset + 3] = 255;
            } else {
                let alpha = mtl.data[offset + 3] / 255.0;
                let x = i - 75;
                let y = j - 126;
                let newOffset = (x * 550 + y) * 4;
                mtl.data[offset] = alpha * mtl.data[offset] + (1 - alpha) * src.data[newOffset];
                mtl.data[offset + 1] = alpha * mtl.data[offset + 1] + (1 - alpha) * src.data[newOffset + 1];
                mtl.data[offset + 2] = alpha * mtl.data[offset + 2] + (1 - alpha) * src.data[newOffset + 2];
                mtl.data[offset + 3] = 255; }}}return mtl;
}
Copy the code

(2) Cut out the portrait

This is a service based on the server-side portrait mask layer, which cuts out the portrait in the front, so that further background fusion and switching can be done. It has been used in several online projects.

Like to dig in addition to

This is based on two renderings processed by the server:

Result graph and mask graph with background:

1. We need to process the mask map first:

// Draw a mask;
// mask_zoom: the zoom factor is used to optimize performance;
mask = document.createElement('canvas');
maskCtx = mask.getContext('2d');
mask.width = imgEl.naturalWidth * ops.mask_zoom;
mask.height = imgEl.naturalHeight * ops.mask_zoom / 2;
maskCtx.drawImage(imgEl, 0, - imgEl.naturalHeight * ops.mask_zoom / 2, imgEl.naturalWidth * ops.mask_zoom , imgEl.naturalHeight * ops.mask_zoom);
Copy the code

2. Remove the black background of the mask and turn it into a transparent color, which requires pixel operation:

// Get image data;
let maskData = maskCtx.getImageData(0.0, mask.width, mask.height);

// Change the opacity of the points near black to 0;
for (let i = 0; i < data.length; i += 4) {
	let r = data[i],
	    g = data[i + 1],
	    b = data[i + 2];
		
	if (r <= 30 && g <= 30 && b<= 30)data[i + 3] = 0;
}

// Fill the mask layer with the modified data;
maskCtx.putImageData(maskData, 0.0);
Copy the code

3, image fusion, here with the help of a magical canvas, believe everybody heard, but are not familiar with – globalCompositeOperation, this value can change integration model of the canvas, there are many integration mode you can self study, used here is the source – in;

// Create the final image container;
result = document.createElement('canvas');
resultCtx = result.getContext('2d');
result.width = imgEl.naturalWidth;
result.height = imgEl.naturalHeight;

// Draw the mask layer as the background;
resultCtx.drawImage(mask, 0.0, imgEl.naturalWidth, imgEl.naturalHeight);

// Modify the fusion mode
resultCtx.globalCompositeOperation = 'source-in';

// Draw the resulting graph with a background
resultCtx.drawImage(origin, 0.0);
Copy the code

The resulting renderings:

Finally, you can use this portrait to blend with any background or material according to your business needs.

conclusion

The algorithm part due to my lack of knowledge, I can not elaborate in depth, can only share some relatively superficial experience, hope to develop to the lower field in the future.

But hopefully it will enlighten you. If you are interested in image processing, you can discuss it with me at any time. I hope to get the guidance of the great God. I also hope js can develop faster in the field of image processing.

Thank you for reading. ~ ~ ~ 😬 😬 😬 😬.