A few days ago, my senior recommended an interesting project — real-time Personal-removal. This project uses tensorflow.js and image processing in canvas to make people disappear in videos. Take this opportunity to review image processing in the Canvas base.

Based on the API

Canvas’s image processing capability processes pixel data through ImageData objects. The main apis are as follows:

  • CreateImageData () : Creates a blank ImageData object
  • GetImageData () : Get canvas pixel data, each pixel has 4 values — RGBA
  • PutImageData () : Writes pixel data to the canvas
imageData = {
  width: Number.height: Number.data: Uint8ClampedArray
}
Copy the code

Width is the width of the canvas or the number of pixels on the X-axis; Height is the height of the canvas or the number of pixels on the y axis; Data is the pixel data array of the canvas, total length w * H * 4, each 4 values (RGBA) represents a pixel.

Processing of pictures

Let’s take a look at the image processing capabilities of Canvas through a few examples.

Original image effect:

const cvs = document.getElementById("canvas");
const ctx = cvs.getContext("2d");
const img = new Image();
img.src="Image URL";
img.onload = function () {
  ctx.drawImage(img, 0.0, w, h);
}
Copy the code

Negative/negative effects

Algorithm: 255 and pixel RGB difference, as the current value.

function negative(x) {
  let y = 255 - x;
  return y;
}
Copy the code

Effect:

const imageData =  ctx.getImageData(0.0, w, h);
const { data } = imageData;
let l = data.length;
for(let i = 0; i < l; i+=4) {
  const r = data[i];
  const g = data[i + 1];
  const b = data[i + 2];
  data[i] = negative(r);
  data[i + 1] = negative(g);
  data[i + 2] = negative(b);
}
ctx.putImageData(imageData, 0.0);
Copy the code

Effect of monochromatic

The monochromatic effect is to retain one of the RGB 3 values of the current pixel and remove the other color values.

for(let i = 0; i < l; i+=4) { // remove r and g values
  data[i] = 0;
  data[i + 1] = 0;
}
Copy the code

Effect:

grayscale

Grayscale: An image with only one color value per pixel. A color value from 0 to 255. The color changes from black to white.

for(let i = 0; i < l; i+=4) {
  const r = data[i];
  const g = data[i + 1];
  const b = data[i + 2];
  const gray = grayFn(r, g, b);
  data[i] = gray;
  data[i + 1] = gray;
  data[i + 2] = gray;
}
Copy the code
  • Algorithm 1 — Average method: 
    const gray = (r + g + b) / 3;
Copy the code

Effect:

  • Algorithm 2 — human eye perception: according to the perception degree of red, green and blue: green > red > blue, given weight division 
    const gray = r * 0.3 + g * 0.59 + b * 0.11
Copy the code

Effect:

In addition, there are:

  • Take a maximum or a minimum.
const grayMax = Math.max(r, g, b); // The value is large and bright
const grayMin = Math.min(r, g, b); // The value is small and dark
Copy the code
  • Take a single channel, that is, ONE of the RGB 3 values.

Binary figure

Algorithm: determine a color value, compare the current RGB value, greater than this value show black, otherwise show white.

for(let i = 0; i < l; i+=4) {
  const r = data[i];
  const g = data[i + 1];
  const b = data[i + 2];
  const gray = gray1(r, g, b);
  const binary = gray > 126 ? 255 : 0;
  data[i] = binary;
  data[i + 1] = binary;
  data[i + 2] = binary;
}
Copy the code

Effect:

Gaussian blur

Gaussian blur is one of the “blurring” algorithms in which the value of each pixel is a weighted average of the values of its neighboring pixels. The value of the original pixel has the largest Gaussian distribution value (with the largest weight), and the weight of adjacent pixels decreases as they get farther and farther away from the original pixel.

First-order formula:

(First-order formula is used because the algorithm for first-order formula is simpler.)

const radius = 5; // Blur the radius
const weightMatrix = generateWeightMatrix(radius); // Weight matrix
for(let y = 0; y < h; y++) {
  for(let x = 0; x < w; x++) {
    let [r, g, b] = [0.0.0];
    let sum = 0;
    let k = (y * w + x) * 4;
    for(let i = -radius; i <= radius; i++) {
      let x1 = x + i;
      if(x1 >= 0 && x1 < w) {
      let j = (y * w + x1) * 4;
      r += data[j] * weightMatrix[i + radius];
      g += data[j + 1] * weightMatrix[i + radius];
      b += data[j + 2] * weightMatrix[i + radius];
      sum += weightMatrix[i + radius];
      }
    }
    data[k] = r / sum;
    data[k + 1] = g / sum;
    data[k + 2] = b / sum; }}for(let x = 0; x < w; x++) {
  for(let y = 0; y < h; y++) {
    let [r, g, b] = [0.0.0];
    let sum = 0;
    let k = (y * w + x) * 4;
    for(let i = -radius; i <= radius; i++) {
      let y1 = y + i;
      if(y1 >= 0 && y1 < h) {
        let j = (y1 * w + x) * 4;
        r += data[j] * weightMatrix[i + radius];
        g += data[j + 1] * weightMatrix[i + radius];
        b += data[j + 2] * weightMatrix[i + radius];
        sum += weightMatrix[i + radius];
      }
    }
    data[k] = r / sum;
    data[k + 1] = g / sum;
    data[k + 2] = b / sum; }}function generateWeightMatrix(radius = 1, sigma) { // Standard deviation of sigma normal distribution
  const a = 1 / (Math.sqrt(2 * Math.PI) * sigma);
  const b = - 1 / (2 * Math.pow(sigma, 2));
  let weight, weightSum = 0, weightMatrix = [];
  for (let i = -radius; i <= radius; i++){
    weight = a * Math.exp(b * Math.pow(i, 2));
    weightMatrix.push(weight);
    weightSum += weight;
  }
  return weightMatrix.map(item= > item / weightSum); // normalize
}
Copy the code

Effect:

Other effects

Here is a brief introduction to other image effects processing, because the example is simple and repetitive, so no code and renderings.

  • Brightness adjustment: add a given value to RGB values.
  • Transparent processing: Change the A value in rGBA value.
  • Contrast Enhancement: Multiply each RGB value by 2, then subtract a given value.

conclusion

Ok, so that’s some basic image processing algorithms.

The resources

  • Gaussian blur algorithm
  • Gaussian blur