Recently the team is building a WEB player using WASM + FFmpeg. We decode the video with FFmpeg by writing C, and convert WASM to run on the browser to communicate with JavaScript by compiling C. By default, the data decoded by FFmpeg is YUV, while Canvas only supports RGB rendering. At this time, we have two methods to deal with yuV. The first is to use FFmpeg exposed method to directly convert YUV into RGB and render to Canvas. The second one uses WebGL to convert YUV to RGB and render on canvas. The first advantage is that the writing method is very simple, only FFmpeg exposed method will directly convert YUV into RGB, the disadvantage is that it will consume a certain AMOUNT of CPU, the second advantage is the use of GPU for acceleration, the disadvantage is that the writing method is complicated, and need to be familiar with WEBGL. Considering that in order to reduce CPU consumption and use GPU for parallel acceleration, we adopted the second method.

Before we get to YUV, let’s take a look at how YUV is obtained:

Since we are writing players, the steps to implement a player must go through the following steps:

  1. Take the files of video like MP4, AVI, FLV, etc. Mp4, AVI, FLV is like a container that contains some information like compressed video, compressed audio, etc., and demultiplexed them, extract the compressed video and audio from the container, The compressed video is usually H265, H264 or other formats, and the compressed audio is usually AAC or MP3.
  2. The compressed video and compressed audio are decoded respectively to obtain the original video and audio. The original audio data is generally PCM, while the original video data is generally YUV or RGB.
  3. Then the audio and video synchronization. You can see that when you decode the compressed video data, you typically get yuV.

YUV

What is the YUV

For front-end developers, YUV is actually a little strange, for audio and video development will generally come into contact with this, to put it simply, YUV and we are familiar with RGB, are color coded, but their three letters represent a different meaning from RGB. YUV’s “Y” stands for Luminance, or Luma, or grayscale; The “U” and “V” stand for Chrominance or Chroma, which describes the color and saturation of an image and specifies the color of a pixel.

To give you a more visual sense of YUV, let’s see what Y, U, and V look like separately. Here we use FFmpeg command to convert a picture of Uchihiboku of Naruto into YUV420P:

ffmpeg -i frame.jpg -s 352x288 -pix_fmt yuv420p test.yuv
Copy the code

Open test.yuv on GLYUVPlay and display the original picture:

Benefits of using YUV

  1. As you’ve just seen, Y alone displays a black and white image, so YUV’s conversion from color to black and white is easy and compatible with older black and white TVS, a feature used in TV signals.
  2. YUV data size is generally smaller than RGB format, which can save transmission bandwidth. (But if you use YUV444, it’s the same as RGB24.)

YUV sampling

Common YUV samples include YUV444, YUV422, and YUV420:

Note: the black dot represents the Y component of the sampled pixel, and the hollow circle represents the UV component of the sampled pixel.

  1. YUV 4:4:4 sampling, each Y corresponds to a set of UV components.
  2. YUV 4:2:2 sampling, every two Y share a set of UV components.
  3. YUV 4:2:0 sampling, each four Y share a set of UV components.

YUV storage mode

YUV is stored in two types: Packed and planar:

  • Packed YUV format, Y,U,V of each pixel point is continuously interlaced storage.
  • Planar YUV format stores Y of all pixels successively, followed by U of all pixels, followed by V of all pixels.

For example, for planar mode, YUV is YYYYUUVV and for Packed mode, YUYVYUYV.

YUV format generally has a variety of, YUV420SP, YUV420P, YUV422P, YUV422SP, etc., let’s look at the more common format:

  • YUV420P (every four Y will share a set of UV components) :

  • YUV420SP (Packed, every four Y will share a group of UV components, different from YUV420P, YUV420SP is interleaved when U and V are stored) :

  • For YUV422P (planar, two Y share a planar UV component, so U and V add one line each to YUV420P U and V) :

  • YUV422SP (Packed, Every two Y shares a Group of UV components) :

YUV420P and YUV420SP can be divided into two formats according to the order of U and V:

  • YUV420P: YUV420P, also called I420, YUV420P: U before V after V, called YV12.

  • YUV420SP: U The name after the front V is NV12, and the name after the front V is NV21.

The data are arranged as follows:

I420: YYYYYYYY UU VV =>YUV420P

YV12: YYYYYYYY VV UU =>YUV420P

NV12: YYYYYYYY UV UV =>YUV420SP

NV21: YYYYYYYY VU VU =>YUV420SP
Copy the code

As for why there are so many formats, after a lot of search found that the reason is to adapt to different TV broadcast system and device system, for example, ios only this mode NV12, Android mode NV21, such as YUV411, YUV420 format is more common in digital camera data, the former used for NTSC system, The latter is used for PAL system. As for the introduction of TV broadcast system, we can see this article [standard] INTRODUCTION of NTSC, PAL, SECAM three systems

YUV calculation method

For example, YUV420P stores a 1080 x 1280 image. Its storage size is (1080 x 1280 x 3) >> 1) bytes. Take a look at the following chart:

W x H = 1080x1280
(W/2) * (H/2)= (W*H)/4 = (1080x1280)/4
(W*H)/4 = (1080x1280)/4
Y+U+V = (1080x1280)*3/2 

Y:0 到 1080*1280U:1080*1280To (1080*1280) *5/4V: (1080*1280) *5/4To (1080*1280) *3/2
Copy the code

WEBGL

What is WEBGL

Simply put, WebGL is a technology for drawing and rendering complex 3D graphics on web pages and allowing users to interact with them.

WEBGL composition

In the WebGL world, the only basic graphic elements that can be drawn are points, lines and triangles. Each image is composed of large and small triangles, as shown in the figure below. No matter how complex the graph is, its basic component is composed of triangles.

shader

Shaders are programs that run on the GPU and are written in OpenGL ES shader language, somewhat similar to C:

Introduction to GLSL (Opengl – Shader – Language)

To draw graphics in WEBGL, you must have two shaders:

  • Vertex shader
  • Chip shader

The main function of the vertex shader is to process vertices, and the pixel shader is to process each pixel generated by the rasterization stage (PS: pixel can be read as pixel), and finally calculate the color of each pixel.

WEBGL drawing process

Because the program is very silly, do not know the graph of each vertex, we need to provide their own vertex coordinates can be manually written or exported by the software:

let f = document.createDocumentFragment()
document.body.appendChild(f)

After we provide vertices, GPU will execute vertex shader program one by one according to the number of vertices we provide to generate the final coordinates of vertices and assemble the graph. It can be understood as making a kite, it is necessary to build the kite skeleton first, and pixel assembly is in this stage.

3. Rasterization this stage is like making a kite. After building the skeleton of the kite, it cannot fly at this time, because the inside is empty, you need to add cloth to the skeleton. And rasterization is at this stage, the assembled geometric graph of the pixel is transformed into a pixel (PS: the pixel can be understood as pixel).

4. Coloring and rendering

The WEBGL drawing process can be summarized as follows:

  1. Provide vertex coordinates (we need to provide)
  2. Primitive assembly (assembly of graphics by primitive type)
  3. Rasterization (assemble the graphics of pixels to generate pixels)
  4. Provide color values (can be calculated dynamically, pixel coloring)
  5. Draw on the browser using canvas.

WEBGL YUV drawing image ideas

Since the image of each video frame is not quite the same, we certainly cannot know that many vertices, so how can we draw the image of the video frame with WebGL? One technique used here is texture mapping. In simple terms, an image is pasted on the surface of a geometric figure to make the geometric figure look like a geometric figure with images, that is, texture coordinates are one-to-one corresponded with webGL system coordinates:

The texture coordinates are in the lower left corner, so the image will be drawn upside down. There are two solutions:

  • Webgl provides an API for flipping texture images on the Y-axis:
// 1 represents the Y-axis inversion of the texture image
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, 1);
Copy the code
  • Texture coordinates and WebGL coordinates are mapped invert, for example 🌰, as shown above, the original texture coordinates(0.0, 1.0)This corresponds to the WebGL coordinates(-1.0, 1.0, 0.0).(0.0, 0.0)The corresponding is(-1.0, -1.0, 0.0)So let’s reverse this,(0.0, 1.0)The corresponding is(-1.0, -1.0, 0.0)And the(0.0, 0.0)The corresponding is(-1.0, 1.0, 0.0), so that the browser image is not inverted.

The detailed steps

  • Shader section
// the vertexShader is vertexShader
attribute lowp vec4 a_vertexPosition; // Pass the vertex coordinates through js
attribute vec2 a_texturePosition; // Pass texture coordinates through js
varying vec2 v_texCoord; // Pass texture coordinates to the slice shader
void main(){
	gl_Position=a_vertexPosition;// Set the vertex coordinates
	v_texCoord=a_texturePosition;// Set texture coordinates
}


// fragmentShader
precision lowp float;// lowp stands for computational precision
uniform sampler2D samplerY;// sampler2D is the sampler type in which the image texture is ultimately stored
uniform sampler2D samplerU;// sampler2D is the sampler type in which the image texture is ultimately stored
uniform sampler2D samplerV;// sampler2D is the sampler type in which the image texture is ultimately stored
varying vec2 v_texCoord; // Accept texture coordinates from the vertex shader
void main(){
  float r,g,b,y,u,v,fYmul;
  y = texture2D(samplerY, v_texCoord).r;
  u = texture2D(samplerU, v_texCoord).r;
  v = texture2D(samplerV, v_texCoord).r;
	
	/ / YUV420P RGB
  fYmul = y * 1.1643828125;
  r = fYmul + 1.59602734375 * v - 0.870787598;
  g = fYmul - 0.39176171875 * u - 0.81296875 * v + 0.52959375;
  b = fYmul + 2.01723046875 * u - 1.081389160375;
  gl_FragColor = vec4(r, g, b, 1.0);
}
Copy the code
  • Create and compile shaders, connect vertex shaders and fragment shaders to program, and use:
let vertexShader=this._compileShader(vertexShaderSource,gl.VERTEX_SHADER);// Create and compile a vertex shader
let fragmentShader=this._compileShader(fragmentShaderSource,gl.FRAGMENT_SHADER);// Create and compile the slice shader

let program=this._createProgram(vertexShader,fragmentShader);// Create program and connect shaders

Copy the code
  • Create a buffer to hold vertices and texture coordinates (PS: a buffer object is an area of memory in the WebGL system that can be filled with a large amount of vertex data at once and stored in it for use by vertex shaders).
let vertexBuffer = gl.createBuffer();
let vertexRectangle = new Float32Array([
	1.0.1.0.0.0.1.0.1.0.0.0.1.0.1.0.0.0.1.0.1.0.0.0
]);
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
// Write data to the buffer
gl.bufferData(gl.ARRAY_BUFFER, vertexRectangle, gl.STATIC_DRAW);
// Find the vertex position
let vertexPositionAttribute = gl.getAttribLocation(program, 'a_vertexPosition');
// Tells the graphics card to read the vertex data from the currently bound buffer
gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false.0.0);
// Connect the vertexPosition variable with the buffer object assigned to it
gl.enableVertexAttribArray(vertexPositionAttribute);

// Declare texture coordinates
let textureRectangle = new Float32Array([1.0.0.0.0.0.0.0.1.0.1.0.0.0.1.0]);
let textureBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, textureBuffer);
gl.bufferData(gl.ARRAY_BUFFER, textureRectangle, gl.STATIC_DRAW);
let textureCoord = gl.getAttribLocation(program, 'a_texturePosition');
gl.vertexAttribPointer(textureCoord, 2, gl.FLOAT, false.0.0);
gl.enableVertexAttribArray(textureCoord); 
Copy the code
  • Initialize and activate a texture unit (YUV)
// Activate the specified texture unit
gl.activeTexture(gl.TEXTURE0);
gl.y=this._createTexture(); // Create a texture
gl.uniform1i(gl.getUniformLocation(program,'samplerY'),0);// Get the location where the samplerY variable is stored, and pass the texture object to samplerY with the texture unit number 0

gl.activeTexture(gl.TEXTURE1);
gl.u=this._createTexture();
gl.uniform1i(gl.getUniformLocation(program,'samplerU'),1);// Get the location where the samplerU variable is stored, and pass the texture object to samplerU with texture unit number 1

gl.activeTexture(gl.TEXTURE2);
gl.v=this._createTexture();
gl.uniform1i(gl.getUniformLocation(program,'samplerV'),2);// Get the location where the samplerV variable is stored, and pass the texture object to samplerV with the texture unit number 2
Copy the code
  • Rendering rendering (PS: Since the data we obtained is YUV420P, the calculation method can refer to the calculation method just mentioned).
 // Set the color value when the color buffer is cleared
 gl.clearColor(0.0.0.0);
 // Clear the buffer
 gl.clear(gl.COLOR_BUFFER_BIT);

let uOffset = width * height;
let vOffset = (width >> 1) * (height >> 1);

gl.bindTexture(gl.TEXTURE_2D, gl.y);
Subarray (0, width * height); // Fill the Y texture with the width and height of Y.
gl.texImage2D(
	gl.TEXTURE_2D,
	0,
	gl.LUMINANCE,
	width,
	height,
	0,
	gl.LUMINANCE,
	gl.UNSIGNED_BYTE,
	data.subarray(0, uOffset)
);

gl.bindTexture(gl.TEXTURE_2D, gl.u);
Subarray (width * height, width/2 * height/2 + width * height) subarray(width * height, width/2 * height/2 + width * height)
gl.texImage2D(
	gl.TEXTURE_2D,
	0,
	gl.LUMINANCE,
	width >> 1,
	height >> 1.0,
	gl.LUMINANCE,
	gl.UNSIGNED_BYTE,
	data.subarray(uOffset, uOffset + vOffset)
);

gl.bindTexture(gl.TEXTURE_2D, gl.v);
Subarray (width/2 * height/2 + width * height, data.length) subarray(width/2 * height/2 + width * height, data.
gl.texImage2D(
	gl.TEXTURE_2D,
	0,
	gl.LUMINANCE,
	width >> 1,
	height >> 1.0,
	gl.LUMINANCE,
	gl.UNSIGNED_BYTE,
	data.subarray(uOffset + vOffset, data.length)
);

gl.drawArrays(gl.TRIANGLE_STRIP, 0.4); // Draw four points, namely rectangles
Copy the code

The steps above can be drawn into this diagram:

Complete code:

export default class WebglScreen {
    constructor(canvas) {
        this.canvas = canvas;
        this.gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');
        this._init();
    }

    _init() {
        let gl = this.gl;
        if(! gl) {console.log('gl not support! ');
            return;
        }
        // Image preprocessing
        gl.pixelStorei(gl.UNPACK_ALIGNMENT, 1);
        // Vertex shader code in GLSL format
        let vertexShaderSource = ` attribute lowp vec4 a_vertexPosition; attribute vec2 a_texturePosition; varying vec2 v_texCoord; void main() { gl_Position = a_vertexPosition; v_texCoord = a_texturePosition; } `;

        let fragmentShaderSource = ` precision lowp float; uniform sampler2D samplerY; uniform sampler2D samplerU; uniform sampler2D samplerV; varying vec2 v_texCoord; void main() { float r,g,b,y,u,v,fYmul; y = texture2D(samplerY, v_texCoord).r; u = texture2D(samplerU, v_texCoord).r; v = texture2D(samplerV, v_texCoord).r; FYmul = y * 1.1643828125; R = fYmul + 1.59602734375 * v - 0.870787598; G = fymul-0.39176171875 * u-0.81296875 * v + 0.52959375; B = fYmul + 2.01723046875 * u-1.081389160375; Gl_FragColor = vec4(r, g, b, 1.0); } `;

        let vertexShader = this._compileShader(vertexShaderSource, gl.VERTEX_SHADER);
        let fragmentShader = this._compileShader(fragmentShaderSource, gl.FRAGMENT_SHADER);

        let program = this._createProgram(vertexShader, fragmentShader);

        this._initVertexBuffers(program);

        // Activate the specified texture unit
        gl.activeTexture(gl.TEXTURE0);
        gl.y = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerY'), 0);

        gl.activeTexture(gl.TEXTURE1);
        gl.u = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerU'), 1);

        gl.activeTexture(gl.TEXTURE2);
        gl.v = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerV'), 2);
    }
    Buffer * @param {glProgram} program */

    _initVertexBuffers(program) {
        let gl = this.gl;
        let vertexBuffer = gl.createBuffer();
        let vertexRectangle = new Float32Array([
            1.0.1.0.0.0.1.0.1.0.0.0.1.0.1.0.0.0.1.0.1.0.0.0
        ]);
        gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
        // Write data to the buffer
        gl.bufferData(gl.ARRAY_BUFFER, vertexRectangle, gl.STATIC_DRAW);
        // Find the vertex position
        let vertexPositionAttribute = gl.getAttribLocation(program, 'a_vertexPosition');
        // Tells the graphics card to read the vertex data from the currently bound buffer
        gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false.0.0);
        // Connect the vertexPosition variable with the buffer object assigned to it
        gl.enableVertexAttribArray(vertexPositionAttribute);

        let textureRectangle = new Float32Array([1.0.0.0.0.0.0.0.1.0.1.0.0.0.1.0]);
        let textureBuffer = gl.createBuffer();
        gl.bindBuffer(gl.ARRAY_BUFFER, textureBuffer);
        gl.bufferData(gl.ARRAY_BUFFER, textureRectangle, gl.STATIC_DRAW);
        let textureCoord = gl.getAttribLocation(program, 'a_texturePosition');
        gl.vertexAttribPointer(textureCoord, 2, gl.FLOAT, false.0.0);
        gl.enableVertexAttribArray(textureCoord);
    }

    @param {string} shaderSource GLSL shader code @param {number} shaderType shaderType, VERTEX_SHADER or FRAGMENT_SHADER. * @return {glShader} shader. * /
    _compileShader(shaderSource, shaderType) {
        // Create a shader program
        let shader = this.gl.createShader(shaderType);
        // Set the shader source
        this.gl.shaderSource(shader, shaderSource);
        // Compile the shader
        this.gl.compileShader(shader);
        const success = this.gl.getShaderParameter(shader, this.gl.COMPILE_STATUS);
        if(! success) {let err = this.gl.getShaderInfoLog(shader);
            this.gl.deleteShader(shader);
            console.error('could not compile shader', err);
            return;
        }

        return shader;
    }

    /** * create a program from 2 shaders * @param {glShader} vertexShader vertexShader. * @param {glShader} fragmentShader fragmentShader. * @return {glProgram} program */
    _createProgram(vertexShader, fragmentShader) {
        const gl = this.gl;
        let program = gl.createProgram();

        // Attach shaders
        gl.attachShader(program, vertexShader);
        gl.attachShader(program, fragmentShader);

        gl.linkProgram(program);
        // Add the WebGLProgram object to the current render state
        gl.useProgram(program);
        const success = this.gl.getProgramParameter(program, this.gl.LINK_STATUS);

        if(! success) {console.err('program fail to link' + this.gl.getShaderInfoLog(program));
            return;
        }

        return program;
    }

    /** * set texture */
    _createTexture(filter = this.gl.LINEAR) {
        let gl = this.gl;
        let t = gl.createTexture();
        // Bind the given glTexture to the target (binding point)
        gl.bindTexture(gl.TEXTURE_2D, t);
        / / Texture packaging reference https://github.com/fem-d/webGL/blob/master/blog/WebGL based learning (Lesson # % 207). The article md - > Texture wrapping
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
        // Set the texture filter mode
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, filter);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, filter);
        return t;
    }

    @param {number} width @param {number} height height */
    renderImg(width, height, data) {
        let gl = this.gl;
        // Set the viewport, that is, specify x and Y affine transformations from standard devices to window coordinates
        gl.viewport(0.0, gl.canvas.width, gl.canvas.height);
        // Set the color value when the color buffer is cleared
        gl.clearColor(0.0.0.0);
        // Clear the buffer
        gl.clear(gl.COLOR_BUFFER_BIT);

        let uOffset = width * height;
        let vOffset = (width >> 1) * (height >> 1);

        gl.bindTexture(gl.TEXTURE_2D, gl.y);
        // Fill the texture
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width,
            height,
            0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(0, uOffset)
        );

        gl.bindTexture(gl.TEXTURE_2D, gl.u);
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width >> 1,
            height >> 1.0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(uOffset, uOffset + vOffset)
        );

        gl.bindTexture(gl.TEXTURE_2D, gl.v);
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width >> 1,
            height >> 1.0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(uOffset + vOffset, data.length)
        );

        gl.drawArrays(gl.TRIANGLE_STRIP, 0.4);
    }

    @param {number} width @param {number} height height @param {number} maxWidth Maximum width */
    setSize(width, height, maxWidth) {
        let canvasWidth = Math.min(maxWidth, width);
        this.canvas.width = canvasWidth;
        this.canvas.height = canvasWidth * height / width;
    }

    destroy() {
        const {
            gl
        } = this; gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT | gl.STENCIL_BUFFER_BIT); }}Copy the code

Finally, let’s take a look at the renderings:

Problems encountered

In actual development process, we test some live streams, sometimes rendering image display is normal, but the green color, the study found that the live streaming video of different anchor width is different, such as width at anchor in the pk 368, popular anchor width to 720, a small anchor width is 540, A width of 540 will appear green because WebGL preprocesses it and sets the following value to 4 by default:

// Image preprocessing
gl.pixelStorei(gl.UNPACK_ALIGNMENT, 4);
Copy the code

So the default is 4 bytes per line, 4 bytes per line, and the Y component has a width of 540, which is a multiple of 4, so the bytes are aligned, so the image will work, while the U and V components have a width of 540/2 = 270, which is not a multiple of 4, so the bytes are not aligned, so the color will be green. There are two ways to solve this problem:

  • The first is to directly make WebGL handle 1 byte per line (which affects performance) :
// Image preprocessing
gl.pixelStorei(gl.UNPACK_ALIGNMENT, 1);
Copy the code
  • The second is to make the width of the image obtained be a multiple of 8, so that YUV byte alignment can be achieved, so that the green screen will not be displayed, but it is not recommended to do so, because the CPU consumption is very high during rotation, so the first scheme is recommended.

Refer to the article

Image and video Coding and FFmpeg(2) – Introduction and Application of YUV format – Eustoma – Blog Garden

YUV pixel formats

wiki.videolan.org/YUV/

Using 8-bit YUV format video rendering | Microsoft Docs

IOS video format YUV – simple book

Webgl&three. js works in cnwander