To make a video editor, you need to obtain a template package file from the server, unzip the file, and display the contents of the file.

Unpack the ZIP package

First of all, each file has its own set of “structure rules”. For example, PSD is directly according to the PSD file specification, the data is directly stored in binary files. Office2007 + is according to the ECMA376 specification, the file is stored in multiple XML format documents. Then compress it into a PPTX, DOCX, etc file. So for ZIP, in fact, is also a set of its own algorithm zip (file format), but really to achieve this decompression algorithm or a lot of work, the key is that the community has done a good job, we do not need to repeat the wheel, directly use Stuk/jszip, Here are two official examples.

/ / compression
var zip = new JSZip();

zip.file("Hello.txt"."Hello World\n");

var img = zip.folder("images");

img.file("smile.gif", imgData, {base64: true});

zip.generateAsync({type:"blob"})

    .then(function(content) {
        // see FileSaver.js
        saveAs(content, "example.zip");
    });

Copy the code
/ /
var zip = new JSZip();

zip.loadAsync(zipDataFromXHR);

require("fs").readFile("hello.zip".function (err, data) {
  if (err) throw err;
  var zip = new JSZip();
  zip.loadAsync(data)
      .then(function (zip) {
          console.log(zip.files);
      });
}
Copy the code

After unzipping, you can get the list of files, but it is not the “files” that we normally unzipped on the client side, or just the “files” in memory.

Zip. Files is a list of files that can be converted to the corresponding format by calling file.async(). The following are async arguments:

  1. base64 : the result will be a string, the binary in a base64 form.
  2. text (or string): the result will be an unicode string.
  3. Binarystring: The result will be a string in the “binary” form, using 1 byte per char (2 bytes).
  4. array: the result will be an Array of bytes (numbers between 0 and 255).
  5. uint8array : the result will be a Uint8Array. This requires a compatible browser.
  6. arraybuffer : the result will be a ArrayBuffer. This requires a compatible browser.
  7. blob : the result will be a Blob. This requires a compatible browser.
  8. nodebuffer : the result will be a nodejs Buffer. This requires nodejs.

If it’s just plain text or images, you can use text or base64 at this point

A blob object

One of the formats in the jszip file is bloB. Let’s start by looking at the description of BLOB in MDN

A Blob object represents an immutable, raw data-like file object. Blobs don’t necessarily represent data in JavaScript’s native format. The File interface is based on Blob, inheriting the functionality of Blob and extending it to support files on the user’s system.

So what does it do to get the blob?

  1. Transfer the File object
const file = new File(\[blob\], fileName, options);

// file
Copy the code
  1. Turn the blob url. –
const url = URL.crateObjectURL(blob);  // 'blob:httpx://xxxx.xxxx.xxxxxxxxxxxxxxxx'

img.src = url;


// <img src='blob:httpx://xxxx.xxxx.xxxxxxxxxxxxxxxx'/>
Copy the code
  1. Turn arrayBuffer
blob.arrayBuffer().then(console.log);
// Outputs an arrayBuffer object
Copy the code
  1. Use FileReader read
const fr = new FileReader();

fr.onload = e= > {
     const result = e.target.result;
     / / output base64
};

fr.readAsDataURL(blob);
Copy the code

FileReader.readAsArrayBuffer() FileReader.readAsBinaryString() FileReader.readAsDataURL() FileReader.readAsText()

Above 4 methods, from the function name can easily see the final read content format, here will not be repeated.

By the way, the four formats here correspond to the formats extracted from JsZip.

Arraybuffer, Type array object, DataView

The ArrayBuffer object is used to represent a generic, fixed-length buffer of raw binary data. An ArrayBuffer does not operate directly, but through a type array object or DataView object, which represents the data in the buffer in specific formats and reads and writes the contents of the buffer in those formats.

Arraybuffer has a constructor of the same name that creates a zeroed Arraybuffer object of a specified length. The constructor takes an argument specifying the length of the object to be created. Such as:

const ab = new ArrayBuffer(8);
// Create an 8-byte ArrayBuffer
Copy the code

Since we can’t operate on an Arraybuffer directly, we need to work with other objects. So there’s TypedArray and DataView.

  1. TypedArray, TypedArray is a class of objects, in fact, JS does not have a TypedArray object or constructor. So you can’t use TypedArray directly. Here are 9 TypedArray objects/constructors
Int8Array(a);Uint8Array(a);Uint8ClampedArray(a);Int16Array(a);Uint16Array(a);Int32Array(a);Uint32Array(a);Float32Array(a);Float64Array(a);Copy the code

For details, see MDN: TypedArray

TypedArray is not really an array, but it has almost the same API as an array, and we can manipulate TypedArray just like an array, so with TypedArray we can convert an ArrayBuffer to TypedArray, and then in the read and write operations, to achieve the purpose of operating binary, the following is an example


    const arrayBuffer = new ArrayBuffer(8);

    console.log(arrayBuffer[0]);  // undefined

    const uint8Array = new Uint8Array(arrayBuffer);

    console.log(uint8Array);  // [0, 0, 0, 0, 0, 0, 0]

    uint8Array[0] = 1;
    console.log(uint8Array[0]); / / 1

    console.log(uint8Array);  // [1, 0, 0, 0, 0, 0, 0]

Copy the code

As you can see, it is not possible to get the contents of an arrayBuffer object directly using arrayBuffer[0], whereas TypedArray does.

[[Int8Array]] [[Int16Array]] [[Int32Array]] [[Uint8Array]] 4 TypedArray Data, but this is supposed to be converted by the browser to make it easier for developers to view the data, rather than ArrayBuffer actually owning the data. After all, the object names don’t look so formal (contained in [[]]).

Assigning a value to a subscript of an arrayBuffer object with arrayBuffer[0] = 1 will not fail, and you can later retrieve the value console.log(arrayBuffer[0]) // 1 using the same path. But that doesn’t mean you’re manipulating the data in an ArrayBuffer, just like you do with an array.

  1. DataView, DataView provides functions similar to TypedArray. Unlike TypedArray, DataView is a real object that provides various methods to manipulate data of different types.
let arrayBuffer = new ArrayBuffer(8);

    const dataView = new DataView(arrayBuffer);

    console.log(dataView.getUint8(1)); / / 0

    dataView.setUint8(1.2);
    console.log(dataView.getUint8(1)); / / 2
    console.log(dataView.getUint16(1)); / / 512

    dataView.setUint16(1.255);
    console.log(dataView.getUint16(1)); / / 255
    console.log(dataView.getUint8(1)); / / 0

Copy the code

As you can see, we can call different methods to read/write different types (lengths) of data on the same data, but most of the time it’s hard to get the desired results. Like the output above, it doesn’t look that normal because a 16-bit binary, read in 8-bit format, can be read as two 8-bit binaries. Take a chestnut


// 16 bits of 1

0000 0000 0000 0001

// Use 8 bits to read

0000 0000  / / 0

0000 0001  / / 1

Copy the code

Because the first eight bits are all zeros, it doesn’t look any different except for multiple zeros, right? When the numbers are big


// Should be 256? I'm not very good at this either

0000 0001 0000 0001

// Use 8 bits to read

0000 0001  / / 1

0000 0001  / / 1

Copy the code

Anyway, let’s go back and look at what DataView offers

/ / read the DataView. Prototype. GetInt8 () DataView. Prototype. GetUint8 () DataView. Prototype. GetInt16 () DataView.prototype.getUint16() DataView.prototype.getInt32() DataView.prototype.getUint32() DataView. Prototype. GetFloat32 () DataView. Prototype. GetFloat64 () / / write DataView. Prototype. SetInt8 () DataView.prototype.setUint8() DataView.prototype.setInt16() DataView.prototype.setUint16() DataView.prototype.setInt32()  DataView.prototype.setUint32() DataView.prototype.setFloat32() DataView.prototype.setFloat64()Copy the code

Uint8ClampedArray() : DataView

What is the use of this “array”?

  1. Canvan’s ImageData can directly receive Uint8ClampedArray type data.

  2. new Blob(arrayBuffer);

  3. new File([arrayBuffer], filename, options);

base64

When it comes to base64, my first reaction is to assign it to img.src. But where did base64 come from? What format can it convert to?

source

  1. FileReader.readAdDataURL()

  2. canvas.toDataURL();

  3. window.btoa();

  4. Other older Base64 algorithms;

where

  1. window.atob();

  2. Assign to a DOM element;

  3. Other older Base64 algorithms;

The File object

In addition to the blob and ArrayBuffer that can be converted to objects, strings and binaryStrings can also be converted to File objects.

const file = new File(["aaa"].'1111.txt', {type: 'plain/text'});

// File
Copy the code

The File object constructed in this way is the same as the local File selected by calling DOM . It can be directly used as a File and uploaded to the server or for other purposes. Since a File is inherited from a Blob object, you can also call url.createObjecturl () to convert it into a blob-URL directly assigned to the DOM element.

Image conversion

Front-end development is almost inseparable from the picture, especially we do a variety of editors, inevitably have to deal with the picture, a lot of times, need to further processing of the picture.

In order to convert images to pixels, it is probably the first choice to draw pictures on canvas and then do further processing. So first let’s see what other elements canvas can draw besides DOM such as tag.

Here is the object (CanvasImageSource) that can be used as the first argument to Canvas.drawImage ()

  1. CSSImageValue
  2. HTMLImageElement
  3. SVGImageElement
  4. HTMLVideoElement
  5. HTMLCanvasElement
  6. ImageBitmap
  7. OffscreenCanvas

We can through the canvas. Context. GlobalCompositeOperation to set the blend mode, do some simple processing of image.

  1. source-over
  2. source-in
  3. source-out
  4. source-atop
  5. destination-over
  6. destination-in
  7. destination-out
  8. destination-atop
  9. lighter
  10. copy
  11. xor
  12. multiply
  13. screen
  14. overlay
  15. darken
  16. lighten
  17. color-dodge
  18. color-burn
  19. hard-light
  20. soft-light
  21. difference
  22. exclusion
  23. hue
  24. saturation
  25. color
  26. luminosity

See the globalCompositeOperation on the MDN for details of the blending effect

Of course, for complex cases, you may not be able to use the above blending modes, so you may need to manipulate the pixel data yourself.

Canvas provides a set of interfaces to manipulate ImageData, allowing you to define your own algorithms to manipulate pixel data. Here is the code I used to calculate the mask video effect of a Web video template

    const imageData = maskCtx.getImageData(0.0, width, height);
    const data = imageData.data;
    const len = data.length;

    for(let i = 0; i < len; i += 4) {
        data[i + 3] = 255 - (data[i] + data[i + 1] + data[i + 2) /3;
    }

    maskCtx.putImageData(imageData, 0.0);
Copy the code

Here, the ImageData is first extracted from the canvas, which is the pixel data, and then the pixel data is traversed, a simple calculation is done (turning the color photo into black and white), and then the image is put back to the canvas to achieve the purpose of modifying the picture.

The ImageData doesn’t have to come from the canvas to get getImageData. You can also construct one by using the ImageData constructor

const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const arr = new Uint8ClampedArray(40000);

// Iterate through every pixel
for (let i = 0; i < arr.length; i += 4) {
  arr[i + 0] = 0;    // R value
  arr[i + 1] = 190;  // G value
  arr[i + 2] = 0;    // B value
  arr[i + 3] = 255;  // A value
}

// Initialize a new ImageData object
let imageData = new ImageData(arr, 200);

// Draw image data to the canvas
ctx.putImageData(imageData, 20.20);

Copy the code

Remember when TypedArray was mentioned above, canvan’s ImageData can directly receive Uint8ClampedArray data?

. I tried, through above FileReader readAsArrayBuffer then turn Uint8ClampedArray data, is not by calling the ImageData constructor, getting the right image, the results of Guess the reason was that the ImageData need is exact pixel data, and FileReader. ReadAsArrayBuffer read after compression algorithm of binary files, need to go through the corresponding decoding, can obtain the pixel data.

Image export

After processing the image, our final goal is of course to use the image, in addition to directly display on canvas canvas, Canvas also supports image export.

Export into base64

const base64URL = canvas.toDataURL();
Copy the code

Export a blob

canvas.toBlob(blob= > {
    console.log(blob);
});
Copy the code

The toBlob method takes three parameters, the first is the callback, which is mandatory, the second is the type, and the third is the image quality (JPG only)

Such as:

canvas.toBlob(blob= >{... },"image/jpeg".0.95);
Copy the code

Video processing.

One of the objects that canvas.drawImage() can draw is HTMLVideoElement, which is our

const video = document.querySelector('#video');
ctx.drawImage(video, 0.0);
Copy the code

As we know, the video is actually composed of many pictures, so which picture should canvas. DrawImage draw? It can be simply understood as the picture drawn on canvas, which is the picture played at the current time of the video.

What’s the use of drawing a video on canvas?

  1. Video screenshot
  2. Process the playback effect of the video
  3. Image compression

It should be noted that mp4, WebM and OGG are the only formats supported by mainstream browsers. The other two formats are still far from universal.

So it can be said that the common video formats of video only support H264 encoding MP4 format of video.

Finally, I attached a diagram that I combed out

Images cannot be viewed by clicking here github.com/noahlam/art…