preface

This article will introduce a relatively “hard core” knowledge, digital image processing – convolution matrix. And hand in hand to teach you a digital image processing tool.

Convolution Matrix

Introduction to the

This is a slightly mathematical concept, and most filters use convolution matrices. If you’re having fun, you can use the convolution matrix to build custom filters. What is a convolution matrix? If you don’t use mathematical tools, you probably don’t know the exact concept of convolution and how it works, but you can see it in Wikipedia. Convolution has many applications in science, engineering and mathematics. Here we mainly use it for image processing, such as image blur, sharpening, edge detection and so on.

Convolution is treated as a matrix and is generally referred to as a “kernel”. An image (in digital image processing) is a two-dimensional collection of pixels in rectangular coordinates, the kernel used depends on the image effect you want to process. The kernel is typically a 5×5 or 3×3 matrix, and only the 3×3 matrix is considered here, which is the most commonly used and sufficient for all desired effects. If all boundary values of the kernel of a 5×5 matrix are set to zero, it is treated as a 3×3 matrix.

The filter needs to process each pixel of the image in turn. For each pixel in the image set, we call it the “initial pixel”, and it checks the corresponding value by multiplying the value of that pixel with the value of the eight surrounding pixels. These results are then added up and the initial pixel is set to the final result value.

As shown in the figure.

On the left is the image matrix, with each pixel marked with a value. Inside the red border are the initial pixels. The green border is the kernel operation area, the middle is the kernel (convolution matrix), and the right is the result of the convolution process.

Specific calculation process:

The filter reads all pixels of the active area of the kernel from left to right and top to bottom. It multiplies their respective values to check the corresponding values and adds the results. The initial pixel becomes 42:

‘(40 * 0) + (42 * 1) + (46 * 0)

Plus 46 times 0 plus 50 times 0 plus 55 times 0.

Plus 52 times 0 plus 56 times 0 plus 58 times 0.

= 42 `. As a graphical result, the initial pixel is moved down one pixel.

Began to practice

This is mainly divided into several steps: firstly, pre-define several convolution cores, load images to canvas canvas, filter (convolution) the original image pixel by pixel, and finally draw the processed pixels on canvas. And let these several convolution kernels run in a loop and show their processing effect.

  1. We define a preset of convolution kernels for a pivot function.
/ / convolution kernels
const ConvolutionKernel = {
  'normal': [
    0.0.0.0.1.0.0.0.0,].'sharpen': [
    0, -1.0,
    -1.5, -1.0, -1.0,].'blur': [
    1.1.1.1.1.1.1.1.1,].'blur2': [
		0.1.0.0, -1.0.0.1.0,].'edgedetect': [
    0.1.0.1, -4.1.0.1.0,].'emboss': [-2, -1.0,
    -1.1.1.0.1.2]},// The main function processImg
export const processImg = ({ el, type, imgPath, hookMounted } = {}) = > {

  const options = mergeObj(defaultOpt, {
    el,
    type,
    imgPath,
    hookMounted
  });

  letpromise = processImageData({ ... options }).catch(e= > console.log(e));

  return Object.assign(promise, { data: options });
}
Copy the code
  1. Define the image path to be processed, and define a function to load the image.
const IMGPath = '/assets/js.png';

export const loadImg = (src) = >
	new Promise((resolve, reject) = > {
		const img = new Image()
		img.onload = () = > resolve(img);
		img.onerror = reject;

		img.crossOrigin = 'anonymous'
		img.src = src;
	})

Copy the code
  1. Create Canvas-2D and load the image for drawing.
const noop = () = > {}

const genCanvasAndImg = (el, imgPath, hookMounted = noop) = >
  new Promise((resolve, reject) = > {
    loadImg(imgPath).then((img) = > {
      let canvas = handleContainer(el, img);

      hookMounted(canvas)

      const ctx = canvas.getContext('2d')
      ctx.drawImage(img, 0.0, img.width, img.height);

      resolve(ctx);
      
    }).catch(reject);
  })

Copy the code
  1. Filter (convolution) processing per pixel

const kernelWeight = 1;
// convolution processing
const handleConvolution = (imageData, kernel) = > {

  if(! imageDatainstanceof ImageData || !Array.isArray(kernel)) return;

  const [width, height] = [imageData.width, imageData.height];
  let oldData = [...imageData.data];
  let newData = imageData.data;

  / / each pixel is multiplied 3 x3 matrix reference: https://docs.gimp.org/2.6/en/plug-in-convmatrix.html
  // For the time being, the worker thread should be optimized for performance
  for (let y = 0; y < height; y++) {
    for (let x = 0; x < width; x++) {
      // Boundary directly reuse the original pixel
      if(x == 0 || x == width - 1 || y == 0 || y == height - 1) continue;
      // Upper left pixel
      const topLeft = getPixelByCoord(oldData, x - 1, y - 1, width);
      / / high
      const topMid = getPixelByCoord(oldData, x, y - 1, width);
      / / on the right
      const topRight = getPixelByCoord(oldData, x + 1, y - 1, width);
      / / left
      const midLeft = getPixelByCoord(oldData, x - 1, y, width);
      // Middle/current
      const curPix = getPixelByCoord(oldData, x, y, width);
      / / the centre-right
      const midRight = getPixelByCoord(oldData, x + 1, y, width);
      Key levels / /
      const bottomLeft = getPixelByCoord(oldData, x - 1, y + 1, width);
      / / in
      const bottomMid = getPixelByCoord(oldData, x, y + 1, width);
      / / right
      const bottomRight = getPixelByCoord(oldData, x + 1, y + 1, width);

      // calculate acc
      const sumR = (topLeft[0] * kernel[0] + topMid[0] * kernel[1] + topRight[0] * kernel[2] +
        midLeft[0] * kernel[3] + curPix[0] * kernel[4] + midRight[0] * kernel[5] +
        bottomLeft[0] * kernel[6] + bottomMid[0] * kernel[7] + bottomRight[0] * kernel[8]) / kernelWeight;

      const sumG = (topLeft[1] * kernel[0] + topMid[1] * kernel[1] + topRight[1] * kernel[2] +
        midLeft[1] * kernel[3] + curPix[1] * kernel[4] + midRight[1] * kernel[5] +
        bottomLeft[1] * kernel[6] + bottomMid[1] * kernel[7] + bottomRight[1] * kernel[8]) / kernelWeight;

      const sumB = (topLeft[2] * kernel[0] + topMid[2] * kernel[1] + topRight[2] * kernel[2] +
        midLeft[2] * kernel[3] + curPix[2] * kernel[4] + midRight[2] * kernel[5] +
        bottomLeft[2] * kernel[6] + bottomMid[2] * kernel[7] + bottomRight[2] * kernel[8]) / kernelWeight;

      // const sumA = (topLeft[3] * kernel[0] + topMid[3] * kernel[1] + topRight[3] * kernel[2] 
      // + midLeft[3] * kernel[3] + curPix[3] * kernel[4] + midRight[3] * kernel[5]
      // + bottomLeft[3] * kernel[6] + bottomMid[3] * kernel[7] + bottomRight[3] * kernel[8]) / kernelWeight;

      const index = 4 * x + 4 * width * y;
      const alpha = curPix[3];

      newData[index] = parseInt(sumR);
      newData[index + 1] = parseInt(sumG);
      newData[index + 2] = parseInt(sumB);
      newData[index + 3] = alpha;

      // console.log(sumR, sumG, sumB, alpha)}}return imageData;
}

const getPixelByCoord = (imageData, x, y, width) = > {
  const index = 4 * x + 4 * width * y;

  let [r, g, b, a] = [imageData[index], imageData[index + 1], imageData[index + 2], imageData[index + 3]].// console.log(index, [r, g, b, a])
  return [r, g, b, a];
}

Copy the code
  1. Draw the processed pixels to the canvas

const processImageData = ({ el, type, imgPath, hookMounted }) = > 
  genImageData(el, imgPath, hookMounted)
  .then(({ ctx, data }) = > {
    console.log('ConvolutionKernel', ConvolutionKernel[type])
    const imageData = handleConvolution(data, ConvolutionKernel[type]);
    // Draw a new imageData to canvas
    ctx.putImageData(imageData, 0.0);

    return ctx;
  });

const genImageData = (el, imgPath, hookMounted) = >
  genCanvasAndImg(el, imgPath, hookMounted)
  .then(ctx= > {
    const data = ctx.getImageData(0.0, ctx.canvas.width, ctx.canvas.height);
    return {
      data,
      ctx
    }
  })
  .catch(e= > {
    console.log(e)
  });

Copy the code
  1. Loop through and display
// ["normal", "blur", "blur2", "edgedetect", "emboss", "sharpen"]

Object.keys(ConvolutionKernel).slice(0)
	.forEach((v, i) = > setTimeout(() = > {
		// 
		processImg({ el: '#app'.type: v, imgPath: IMGPath });

		console.log(v);
	}, 1000 * i));

Copy the code

The final effect is as shown.

conclusion

Here, we mainly introduces an image processing related knowledge, and by using the different convolution check the digital image processing, of course, here is a list of several common knowledge of convolution kernels, in the field of image processing, the nuclear tend to want to cooperate to use for various purposes, such as image recognition preprocessing, edge detection and so on.

Question to consider: Can the image processing techniques mentioned above be applied to image OCR recognition? Can the recognition accuracy be quantified by traditional similarity calculation? Give it a try

Learning a knowledge point and applying it in a project and getting a project can weigh the pros and cons of each technical implementation scheme or knowledge point. The latter requires a large amount of knowledge reserve and practical experience, while the former is a process of gradual accumulation and quantitative change. The cycle repeats. As Internet technology updates more and more quickly, it is necessary for every technician to constantly learn and update.