Our business is NOW live broadcast, which is very attractive. After each live broadcast, we will save the live broadcast replay for a period of time, and each live broadcast replay is full of many highlights. However, it is not so easy to accurately identify these highlights from 2 or 3 hours of live broadcast replay. Therefore, the story starts with a demand announcement. Our product hopes to clip the highlight moment of the anchor in the playback. As the front-end, we would have been better to listen to it, after all, video cutting has been done in the background for a long time, but this time, as the front-end of IVWEB, we decided to take WASM to try.

1. Programmes from many years ago

At the Node Knockout Contest in 2013 (this year is 2020), someone came up with an idea called Video Funhouse (too old for me to find more information), which led to the VideoConverter solution on Github. Videoconverter converts ffMPEG, the Swiss Army knife of audio and video, to javascript via Emscripten (a compiler that generates ASM/WASM from C/C++ code), for easy operation of video on the browser, including video cropping/conversion. It’s demo can also run at present, address is as follows: bgrins. Making. IO/videoconver…

Webm -vf showinfo -strict 2 output.mp4 If you change the time parameter into, say, -ss 10 -t 60, you can also clip the video from 10s to get an output of 60s. It uses the Web worker to execute the JS version of FFMPEG, and reads the local input.webm to achieve transcoding/clipping experience is relatively smooth.

However, it was a pure JS video scheme six years ago, and finally stayed in a demo state. There are still many requirements for the product that cannot be met, such as:

  1. The live playback of our business is HLS, and VideoConverter cannot support HLS directly

  2. The converted JS is quite large, with ffmPEG-all-Codec.js being 26m before gzip and 6.8m after gzip

Now, six years later, EmScripten has been upgraded from 1.2.1 to 1.38.45, and we have a new solution for video manipulation, but VideoConverter provides us with an idea of how to do it.

2. Wasm rebirth

This article is not an introduction to WebAssembly and Emscripen (wASM for short). Wasm is smaller, faster, and smaller than webAssembly. Nowadays, Emscripten can easily convert C/C ++ code into ASM/WASM. The Module object of Emscripten can control the execution of WASM code, realize data interaction and function call. An article devoted to emscripten Module objects will follow.

The implementation process of the whole scheme is shown in the figure below:

Refer to VideoConverter solution idea, the core step is to compile a browser usable FFMPEG version, so the first step is to go to the official website to download ffMPEG. You cannot install FFMPEG using BREW, you will need to compile and install it yourself.

  1. Compiling FFMPEG, like installing FFMpeg locally, requires installing third-party dependencies, especially libx264 (ffmpeg encoder does not have H264), and setting the compilation parameters. In the ffmepg directory./configure --helpYou can view the complete build configuration. through--cc="emcc"Specify the compiler as EMCC and disable unnecessary FFMPEG and modules and features that do not support WASM, for example--disable-hwaccelsHard decoding is disabled. The complete configuration can be viewed in the code repository at the bottom.

After configuring the Demuxers /decoders muxers/ Encoders you need and configuring linked third-party libraries, you can compile and install the ffMPEG version you compiled. The next step is to compile the WASM and glue JS code via EMCC.

emcc \ -O3 \ -s WASM=1 \ -s ASSERTIONS=2 \ -s VERBOSE=1 \ -s ALLOW_MEMORY_GROWTH=1 \ -s TOTAL_MEMORY=33554432 \ -v ffmpeg.bc libx264.bc libvpx.bc libz.bc \ -o .. /ffmpeg.js --pre-js .. /ffmpeg_pre.js --post-js .. /ffmpeg_post.jsCopy the code

-O3 is the optimization level for compilation. The TOTAL_MEMORY and ALLOW_MEMORY_GROWTH parameters set the memory size required by WASM and allow automatic expansion when the memory size exceeds TOTAL_MEMORY. –pre-js and –post-js set custom JS files to be used as prefixes and suffixes for the final generated glue code, to execute logic in pre-.js before wASM executes, to set some necessary parameters, to execute returns, etc. These two files refer to the VideoConverter code and set ffmPEG entry function ffmpeg_run and data callback function in pre-.js.

The final file output will be ffmPEg. wasm and ffmPEg. js. The glue code size is 250K, ffmpeg.wasm size is 5m, and the videoConverter output JS size is 26m, which is much smaller than that. And FFmpeg. wasm still continues to reduce space after compilation and configuration.

  1. Use simple ffmPEG using the command line locallyffmpeg -i input.m3u8 -c copy output.mp4Command to export the HLS video to an MP4 file, if you need 5 to 8 minutes of video, useffmpeg -i input.m3u8 -ss 300 -t 180 -c copy output.mp4You can do that.

Arguments to the FFMPEG WASM version of the command line arguments can be set using the Arguments of the Emscripten Module object. Module.arguments is an array of arguments that need to be set before execution.

3. Detail implementation

  1. HLS file analysis for the playback of HLS files, the first is to load m3U8 file, M3U8 file is a specified video file fragment text, by parsing M3U8 can know the start time of each segment, such as an M3U8 file, after removing some versions, serial number specified:
#EXTM3U ... #EXT-X-PROGRAM-DATE-TIME:2019-09-21T20:24:50+08:00 #EXTINF:5, 122070284_485656995_1.ts? start=0&end=781327&type=mpegts #EXTINF:5, 122070284_485656995_1.ts? start=781328&end=1351343&type=mpegts #EXTINF:5 ...Copy the code

The first fragment is 122070284_485656995_1.ts? Start =0&end=781327&type= mPEGts, its length is 6.002, the second fragment 122070284_485656995_1.ts? Start =781328&end=1351343&type= mPEGTS, its duration is 4.005. Based on the length of each clip, we can calculate the clipping time segment we really need through the specified time segment and the time offset from this time segment after parsing M3U8, so we can clip the video we need without loading all TS files. For example, we need 8-15s video, only the second and third clips, and the start time will become 3s.

In addition, the original M3U8 file needs to be reconstructed. After saving the previous file header, the TS fragment of the file consists of the ts clipping required, and the file name can be re-specified.

  1. After reconstructing the m3U8 file, the entire entry function is called as follows:
  ffmpeg_run({
    print: console.log,
    printError: console.error,
    files: [{name: 'playlist.m3u8'
        data: new Uint8Array(buffer)
      },
      {
        name: 'list0.ts'
        data: new Uint8Array(buffer0)
      },
      {
        name: 'list1.ts'
        data: new Uint8Array(buffer1) } ... ] .arguments: ['-i'.'playlist.m3u8'.'-ss', recalculate the start time,'-t'.'180', input.m3u8', '-c', 'copy', 'output.mp4']});Copy the code

How does FFmPEg.wasm read the playback video when it has been broken down into video clips?

Emscripen provides a file system FS to implement virtual files. The input file m3u8, TS and output file output.mp4 mentioned above can be used to implement virtual files. Use the createDataFile and createFolder of FS to create the virtual file system we need.

  Module['files'].forEach(function(file) {
    FS.createDataFile('/', file.name, file.data, true.true);
  }
Copy the code

CreateDataFile passes the specified file name and ArrayBufer data to create the file. When ffmpeg-wasm parses m3U8, it can read the m3U8 file and ts file.

Emscripen also provides Fetch Api, through which file transfer can be realized through XHR, and file request steps can be handed over to C/C ++ for processing. I have not tried this solution, and interested students can try it.

4. Optimize a little

Mp4 format is composed of box data blocks one by one, in which Moov box contains all macro description information of video files, such as video size, frame rate and other information. When playing the video, it is necessary to read the information of the Moov box first to find the position of the video and audio data. If the position of the Moov box is at the end of the video, it is necessary to load the whole video before starting to play.

For those of us who use video streaming, this is unacceptable (there are ways to support SEEK by having the server seek directly to the end of the video, but that requires extra processing). Fortunately, FFmpeg provides a way to front-load mooV by adding -movFlags FastStart to the command line argument. Using MP4 Info to view our generated MP4 file, you can see that the MOOV has been placed before the video data MDAT.

5. To summarize

Wasm + FFMPEG development requires more patience, but no pain, no gain. Wasm introduces FFMPEG to web development. I believe we will also see more pure Web audio and video applications in the future.