Front-end Video Frame Extraction FFMPEG + Webassembly(2)

In the previous article “Front-end Video Frame extraction FFMEPG + WASM”, we implemented the function of video frame extraction through WASM, and optimized the compilation of FFMPEG and wASM loading.

However, the solution in the previous article also has the following problems:

1. The scheme in the previous article first reads the video file as an ArrayBuffer and copies it into the memory. Then wASM reads the memory and calls it. In addition to extracting the memory required by the video frame, it also needs to occupy at least one memory size of the video file. This problem is more obvious when the video file is larger

2. The memory usage exceeds the limit, causing a parsing failure. In the V8 engine of Chrome, the current memory limit for wasm is 4GB for 64-bit browsers and 2GB for 32-bit browsers. Because of the technical solution in the last article, the whole video file needs to be saved in memory, which will cause parsing failure due to insufficient memory when we extract some hd video or long video frames.

In order to solve the original problem of large memory footprint, we need to change the technical solution. Fortunately, WebAssembly provides File System modules and operations, and Emscripten provides a wrapper around the File System API.

I. Technical scheme design

After checking the documents, it can be found that Emscripten mainly provides four File System API schemes. The following are the comparison and analysis one by one

1. MEMFS, where all files are kept in memory, is inconsistent with our goal of reducing memory footprint
2. NODEFS, which can only run in a NodeJS environment, cannot run in a browser
3. IDBFS, based onindexDBImplement file persistence
4. WORKERFS, running onWeb WorkerFor woker internalFile 和 BolbRead-only access to objects without having to copy the entire data into memory fits our needs

Therefore, we can design our new technical solution based on WORKERFS:

Use the Web Worker to load and run wASM. After input selects the file and transfers the file to the corresponding Web Worker, WASM accesses the video file through the FS API and extracts the corresponding video frame. The image data is then transferred back to the main thread and drawn on the canvas

Second, related code modification

After designing the technical scheme, the ORIGINAL C code of reading video stream from memory and then calling FFMPEG should be modified to read and parse by using File System API. The original setFile set memory first and then capture capture video frame interface call method will be changed to read directly from the file, no longer need to setFile

1. The wasm module

The following modifications are made:

1. To get rid ofsetFileMethod,captureMethod to add the file path parameter and useavformat_open_inputTo achieve the corresponding file path reading, the key code is as follows:

ImageData *capture(int ms, char *path) {

    ImageData *imageData = NULL;

    AVFormatContext *pFormatCtx = avformat_alloc_context();  

    if (avformat_open_input(&pFormatCtx, path, NULL.NULL) < 0) {
        fprintf(stderr."avformat_open_input failed\n");
        return NULL; }... }Copy the code

2. Compilation command perfect, addedFile System API 和 cwrapSuch as configuration items. becausewasmDefault callcFunction parameters can only be transmittedintType, so need to passcwrapTo help transfer string types, thus implementing the path towasm, the key codes are as follows:

-lworkerfs.js \
-s EXPORTED_RUNTIME_METHODS='["ccall", "cwrap"]' \
Copy the code

After adjusting the main two items, you also need to adjust the details of memory allocation and reclamation, which can be directly referred to the project code

2. Js module

Due to the way of using Web Worker and File System API, as well as the modification of extraction process, js module needs to be adjusted accordingly, mainly including:

1. useEmscriptenOfficially providedFSInterface for file mounting and recycling, the key code is as follows:

// Mount files to /working directory
const MOUNT_DIR = '/working';

if (!this.isMKDIR) {
    FS.mkdir(MOUNT_DIR);
    this.isMKDIR = true;
}

FS.mount(WORKERFS, { files: [file] }, MOUNT_DIR); .// Recycle the file after using it
FS.unmount(MOUNT_DIR);

Copy the code

2. usecwrapIs called bywasmMethod to achieve the transfer of the path string, the key code is as follows:

if (!this.cCapture) {
    this.cCapture = Module.cwrap('capture'.'number'['number'.'string']);
}

// Corresponds to the pass parameter (int ms, char *path) in capture code
let imgDataPtr = this.cCapture(timeStamp, `${MOUNT_DIR}/${file.name}`);

Copy the code

For detailed usage and documentation of cwrap, please refer to the official Emscripten documentation

Others include the modification of Web Worker loading and initialization related processes, which can be directly referred to the project code

Third, other optimization

In addition to the above technical solution modifications, there are also optimizations for Web Worker and WASM loading.

First, the local text can be directly used to initialize the Web Worker through url.createObjecturl. The relevant code string can be replaced by global variables during compilation, as shown in the following example

function createWorker() {
    const workerBolb = new Blob([WORKER_STRING], {
        type: 'application/javascript'
    });

    const workerURL = URL.createObjectURL(workerBolb);

    const captureWorker = new Worker(workerURL);

    return captureWorker;
}
Copy the code

Secondly, Emscripten compiled code contains glue code and wASM file. Glue code can be directly packaged into Web Worker code by merging and compiling, while wASM file cannot be packaged directly.

If you use the custom loading method in the previous article, it can solve the repeated loading caused by inconsistent response headers, but there will still be loading time and call time to deal with problems. The WASM file must be loaded before asynchronously initialization. If the wASM file is called before initialization, it may fail to respond.

Thus, wASM can be initialized by reading the WASM file into base64 format and packaging it into code, and then using Base64 to ArrayBuffer during initialization, as shown in the following example

var binary_string = self.atob(WASM_STRING);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
    bytes[i] = binary_string.charCodeAt(i);
}
Copy the code

After modifying the technical scheme and carrying out a series of optimizations, compared with the original scheme, the memory usage and extraction performance have been significantly improved, and the call method is more concise

Four,

As ffMPEG is a powerful audio and video library, extracting video frames is only a small part of its functionality, there should be more application scenarios of FFMPEG + Webassembly to explore.

Under the development trend of increasing bandwidth of Internet transmission and decreasing latency, audio and video field will still maintain a good development prospect in the foreseeable future, and relying on FFMPEG + Webassembly, more attempts and applications can be made in the web side

V. Project address

Github.com/jordiwang/w…

Six, the previous background

Front-end video frame extraction FFMPEG + Webassembly