Many apps now have the ability to add special effects or filters to videos or images. In this article, I’ll show you how to do this on the Web. The processing of images and videos is basically the same, you can think of an image as a frame in a video, but there are a few things to be aware of when processing video, which I’ll cover at the end.
How do I get pixel data on an image
There are usually three basic types of image processing:
- Single pixel effects, such as saturation, brightness, contrast, etc. This type of image processing is only associated with a single pixel
- Multi-pixel effects, such as blur and sharpening. This type of image processing is associated with multiple pixels
- Whole image effects, such as cropping, tilting, stretching, etc
These three types of processing require the acquisition of the pixel data of the original image, modification of the pixel data of the original image, and creation of a new image, all of which require the use of canvas. However, we can choose whether to use CPU processing (2D Canvas) or GPU processing (WebGL), and I will introduce the two methods separately
2D canvas
The first step is to draw the picture onto the canvas
const source = document.getElementById('source-image');
const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');
canvas.width = source.naturalWidth;
canvas.height = source.naturalHeight;
context.drawImage(theOriginalImage, 0.0);
Copy the codeGets pixel data on the canvas
const imageData = context.getImageData(0.0, canvas.width, canvas.height);
const pixels = imageData.data;
Copy the codePixels is a Uint8ClampedArray array whose length is Canvas. Width * canvas. Height * 4. Every four elements in the array represent the color of a pixel. Pixels are sorted from the top left, left to right, and top to bottom.
pixels[0]: the value of the red channel of the first pixel [1]: the value of the first pixel green channel pixels[2]: the value of the blue channel of the first pixel [3]: the value of the first pixel alpha channel...Copy the codeGiven a coordinate (x,y) you can get the pixel information for that position as follows:
const index = (x + y * imageWidth) * 4;
const red = pixels[index];
const green = pixels[index + 1];
const blue = pixels[index + 2];
const alpha = pixels[index + 3];
Copy the codeOnce you have the pixel data, you can modify the pixel data (at this stage you can apply the desired effects to the image) and then draw the modified pixel data onto the canvas
context.putImageData(imageData, 0.0);
Copy the codeWebGL
WebGL is a big topic, and it’s not an easy topic to cover in an article. If you want to learn more about WebGL, you can read the article I prepared at the end of this article. In this article, WE will briefly describe what we need to do to process a single image using WebGL.
WebGL is not a 3D graphics API, it’s good for one thing, and that’s drawing triangles. In your application, you have to use triangles to describe what you really want to draw, which is pretty easy in the case of drawing a 2D image because the rectangle consists of two similar right triangles with the hypotenuse in the same position.
The basic process is:
- Create a WebGL context
- Create shader program
- Buffer the data
- Read buffer data to GPU
- Execute the shader program to draw
There are two types of shaders in the drawing process, vertex shaders and slice shaders, vertex shaders use vertex buffer data to calculate the position of the three points of each triangle drawn on the canvas. After executing the vertex shader program, the GPU knows which pixels in the canvas need to be drawn. The pixel shader then calls each pixel once and returns the color that pixel should be drawn. The pixel shader can read information from one or more textures to determine the color of the pixel. Here is a fragment shader code that takes color from a texture:
precision mediump float; // Coordinates varying VEC2 v_texture; // Texture uniform sampler2D u_sampler; Gl_FragColor = texture2D(u_sampler, v_texture); void main(){// Use coordinates to get the color from the texture. }Copy the codeShould you use 2D Canvas or WebGL for image processing
For professional, high-quality graphics in real time, you should use WebGL. Processing images on a GPU means that each pixel can be computed in its own thread, which makes processing images on a GPU an order of magnitude faster than processing images on a CPU. For any real-time image processing, processing speed is critical, and if you’re just doing one-time transformations, You can consider using a 2D canvas
Special type
In the previous section, I mentioned that there are three basic types of image processing, and I will introduce each of them
Single pixel effects
This type of effect is simpler than the other two because it takes the color value of a single pixel as input and results in another color value. For example, brightness control can be achieved by multiplying the values of the red, green, and blue channels of the pixel by a brightness value, 0 making the image completely black and 1 leaving the image unchanged. A value greater than 1 will make the image brighter.
const brightness = 1.1; // Make the image brightness 10% brighter
for (let i = 0; i < imageData.data.length; i += 4) {
imageData.data[i] = imageData.data[i] * brightness;
imageData.data[i + 1] = imageData.data[i + 1] * brightness;
imageData.data[i + 2] = imageData.data[i + 2] * brightness;
}
Copy the codeIf WebGL is used for processing, the fragment shader code is as follows:
precision mediump float; // Coordinates varying VEC2 v_texture; // Texture uniform sampler2D u_sampler; void main(){ vec4 color = texture2D(u_sampler, v_texture); Float brightness = 1.1; gl_FragColor = vec4(color.rgb * brightness, color.a); }Copy the codeSome effects may require additional information, such as the average brightness of the image, but this information only needs to be computed once and reused later in the process.
Multipixel effects
In this case, you need to read the original color of the image. In the previous 2D Canvas example, we changed the original color of the image directly. In multi-pixel effects, we can’t change the original color directly. The simplest solution here is to create a copy of imageData before the image is processed
const originalPixels = new Uint8Array(imageData.data);
Copy the codeIf you use WebGL to process images, you don’t need to create a copy of imageData because the shader doesn’t change the color value of the input texture
Gaussian blur is a common multi-pixel special effect. In the process of Gaussian blur processing, a convolution filter will be used to calculate the color value of the resulting pixel by using the colors of multiple adjacent pixels. For details, you can refer to Canvas 2D for image beautify
| 0 1 0 | | 1 4 1 | | | 0 0 1Copy the codeIf you want to calculate the output color of the pixel at coordinates (23,19), you take the color value of the pixel at coordinates (23,19) and the color value of the 8 pixels around it, and multiply the color value of each pixel by the corresponding weight.
(22, 18) x 0 (23, 18) x 1 (24, 18) x 0
(22, 19) x 1 (23, 19) x 4 (24, 19) x 1
(22, 20) x 0 (23, 20) x 1 (24, 20) x 0
Copy the codeThe results are then added together, and finally normalized. The result is the output color of the pixels with coordinates of (23,19)
The processing process of 2D Canvas is as follows:
const kernel = [[0.1.0],
[1.4.1],
[0.1.0]].for (let y = 0; y < imageHeight; y++) {
for (let x = 0; x < imageWidth; x++) {
let redTotal = 0;
let greenTotal = 0;
let blueTotal = 0;
let weightTotal = 0;
for (let i = -1; i <= 1; i++) {
for (let j = -1; j <= 1; j++) {
// Filter out pixels at the edges of the image
if (x + i > 0 && x + i < imageWidth &&
y + j > 0 && y + j < imageHeight) {
const index = (x + i + (y + j) * imageWidth) * 4;
const weight = kernel[i + 1][j + 1];
redTotal += weight * originalPixels[index];
greenTotal += weight * originalPixels[index + 1];
blueTotal += weight * originalPixels[index + 2]; weightTotal += weight; }}}const outputIndex = (x + y * imageWidth) * 4;
imageData.data[outputIndex] = redTotal / weightTotal;
imageData.data[outputIndex + 1] = greenTotal / weightTotal;
imageData.data[outputIndex + 2] = blueTotal / weightTotal; }}Copy the codeWhole image special effects
Some of these effects are very simple, such as clipping and scaling an image using a 2D Canvas that requires only an API call
// Set the canvas size
canvas.width = source.naturalWidth - 100;
canvas.height = source.naturalHeight - 100;
// Draw this area of the image onto the canvas
const sx = 50;
const sy = 50;
const sw = source.naturalWidth - 100;
const sh = source.naturalHeight - 100;
// Draw the image to this area on the canvas
const dx = 0;
const dy = 0;
const dw = canvas.width;
const dh = canvas.height;
context.drawImage(theOriginalImage,
sx, sy, sw, sh,
dx, dy, dw, dh);
Copy the codeUsing a 2D Canvas for rotation and translation, just call the API to transform the coordinate system before drawing
// Move the X-axis right
context.translate(-canvas.width / 2.0);
// flip the Y axis along the X axis
context.scale(-1.1);
// Rotate the coordinate system 90 degrees clockwise
context.rotate(Math.PI / 2);
Copy the codeYou can also write the 2D transformation as a 2 by 3 matrix and call setTransform. Here’s an example of applying rotation and translation:
const matrix = [
Math.cos(rot) * x1, (-Math.sin(rot)) * x2, tx,
Math.sin(rot) * y1, Math.cos(rot) * y2, ty,
];
context.setTransform(
matrix[0], matrix[1], matrix[2],
matrix[3], matrix[4], matrix[5]);Copy the codeMore complex effects, such as lens distortion or water ripples, involve applying the color of the offset pixel to the target pixel. For example, to create the effect of horizontal water ripples, you can offset the x coordinate of the source pixel based on the Y coordinate.
for (let y = 0; y < canvas.height; y++) {
const xOffset = 20 * Math.sin(y * Math.PI / 20);
for (let x = 0; x < canvas.width; x++) {
// Clamp the source x between 0 and width
const sx = Math.min(Math.max(0, x + xOffset), canvas.width);
const destIndex = ((y * canvas.width) + x) * 4;
const sourceIndex = ((y * canvas.width) + sx) * 4;
imageData.data[destIndex] = originalPixels.data[sourceIndex];
imageData.data[destIndex + 1] = originalPixels.data[sourceIndex + 1];
imageData.data[destIndex + 2] = originalPixels.data[sourceIndex + 2]; }}Copy the codeAdd special effects to the video
The processing of images and videos is basically the same, 2D Canvas and WebGL only draw the current frame of the video when drawing the video on the canvas
2D Canvas
context.drawImage(video, 0.0);
Copy the codeWebGL
// Use the video as a texture
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);
Copy the codeVideos are usually network resources, and it takes time to load network resources. You need to carefully consider when to start drawing. Such as: In the case of video.onloadedmetadata, it means that the metadata has been loaded. At this time, the video length and size can be obtained, but the video file itself has not been loaded properly. At this time, if webGL is used as the video texture, the shader program will report an error. If you use a 2D canvas the canvas will be completely black. Video.loadeddata will be triggered after the first frame of data is loaded. Video.canplay will be triggered when the video can be played in the terminal, and the video may not be finished loading at this time. Which event triggers when to start drawing, depending on what your requirements are
If you want to add special effects to the playing view, you need to use drawImage/texImage2D on each frame to grab a new video frame and then process the pixels of the current frame.
const draw = () = > {
requestAnimationFrame(draw);
context.drawImage(video, 0.0);
/ /... image processing goes here
}
Copy the codeSpeed becomes especially important when working with video. For still images, users may not notice a 100-millisecond delay between clicking the button and applying the effects. However, in animation, a delay of only 16 milliseconds can cause significant jitter.
Related Reading recommendations
- WebGL Fundamentals
- Kernel (image processing)
- Image Kernels
- 2D canvas transformations