This powerful video processing tool can be run directly in a browser with FFmpeg’s WebAssembly version. In this article, we’ll explore FFmpeg.wASM and write a simple codebase that streams data into a video element and plays it back.

FFmpeg.wasm

Generally we use FFmpeg through its command line. For example, the following command can convert AVI files to MP4 format:

$ ffmpeg -i input.avi output.mp4
Copy the code

The same can be done in a browser. Wasm is a WebAssembly port for FFmpeg that, like other JavaScript modules, can be installed via NPM and used in Node or a browser:

$ npm install @ffmpeg/ffmpeg @ffmpeg/core
Copy the code

With FFmpeg.wasm installed, the equivalent transcoding operation can be performed in a browser:

// fetch AVI files
const sourceBuffer = await fetch("input.avi").then(r= > r.arrayBuffer());

// Create an FFmpeg instance and load it
const ffmpeg = createFFmpeg({ log: true });
await ffmpeg.load();

// Write AVI to FFmpeg file system
ffmpeg.FS(
  "writeFile"."input.avi".new Uint8Array(sourceBuffer, 0, sourceBuffer.byteLength)
);

// Execute FFmpeg command line tool to convert AVI to MP4
await ffmpeg.run("-i"."input.avi"."output.mp4");

// Remove the MP4 file from the FFmpeg file system
const output = ffmpeg.FS("readFile"."output.mp4");

// Perform subsequent operations on the video file
const video = document.getElementById("video");
video.src = URL.createObjectURL(
  new Blob([output.buffer], { type: "video/mp4"}));Copy the code

There’s a lot of interesting stuff here, so let’s dive into the details.

After the FETCH API loads the AVI file, initialize FFmpeg with the following steps:

const ffmpeg = createFFmpeg({ log: true });
await ffmpeg.load();
Copy the code

Ffmpeg.wasm consists of a thin JavaScript API layer and a large (20M) WebAssembly binary. The code above loads and initializes the WebAssembly file available for use.

WebAssembly is low-level, performance-optimized bytecode that runs in a browser. It is specifically designed to be developed and compiled in multiple languages.

FFmpeg has been around for over 20 years and more than a thousand people have contributed code. Before WebAssembly, the work involved in creating an interface that JavaScript could call could be tedious.

WebAssembly will be used more widely in the future, and it has been very successful as a mechanism for bringing a large number of mature C/C++ code bases to the Web, such as Google Earth, AutoCAD, and TensorFlow.

After initialization, the next step is to write the AVI file to the file system:

ffmpeg.FS(
  "writeFile"."input.avi".new Uint8Array(sourceBuffer, 0, sourceBuffer.byteLength)
);
Copy the code

This code is a bit strange, and to understand what is going on, you need to look more deeply at how FFmPEg.wASM is compiled.

Emscripten is a tool chain that compiles C/C++ code to WebAssembly, which compiles FFmpeg.wasm to WebAssembly. But Emscripten is more than just a C++ compiler. It provides support for many C/C++ apis through web-based equivalents in order to simplify the migration of existing code bases. For example, support OpenGL by mapping function calls to WebGL. It also supports SDL, POSIX, and PTHREAD.

Emscripten maps to in-memory storage by providing a file-system API. With ffmpeg. wasm, you can expose the underlying Emscripten file system API directly through ffmpeg.FS functions. You can use this excuse to browse directories, create files, and do various other file-system-specific operations.

The next step is where it gets really interesting:

await ffmpeg.run("-i"."input.avi"."output.mp4");
Copy the code

If you look in Chrome’s developer tools, you’ll notice that it creates a number of Web workers, each of which has ffmpeg.wasm loaded:

Here with the help of Emscripten Pthread support (emscripten.org/docs/portin…

Output #0, mp4, to 'Output. Mp4 ': Metadata: encoder: Lavf58.45.100 Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 256x240, q=-1--1, 35 fps, 17920 tbn, 35 tbc Metadata: encoder : Lavc58.91.100 libx264 Side data: CPB: bitrate Max /min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A frame= 47 FPS = 0.0q =0.0 size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x frame= 76 FPS = 68 q=30.0 size= 0kB Time =00:00:00.65 bitrate= 0.6kbits/s speed=0.589x frame= 102 FPS = 62 q=30.0 size= 0kB time=00:00:01.40 bitrate= 0.3 kbits/s speed = 0.846 xCopy the code

The last step is to read the output file and feed it to the video element:

const output = ffmpeg.FS("readFile"."output.mp4");
const video = document.getElementById("video");
video.src = URL.createObjectURL(
  new Blob([output.buffer], { type: "video/mp4"}));Copy the code

Interestingly, ffmpeg. wasm, a command-line utility with virtual file systems, is a bit like Docker!

Create a stream transcoder

Encoding conversion of large video files may take a long time. We can transcode the file into slices and gradually add them to the video buffer.

You can build streaming Media using the MediaSource Extension APIs, which include MediaSource and SourceBuffer objects. Creating and loading buffers can be tricky because both objects provide life cycle events that must be handled to add a new buffer at the right time. To manage coordination of these events, I use RxJS.

The following function creates an RxJS Observable based on ffmpeg. wasm transcoding output:

const bufferStream = filename= >
  new Observable(async subscriber => {
    const ffmpeg = FFmpeg.createFFmpeg({
      corePath: "thirdparty/ffmpeg-core.js".log: false
    });

    const fileExists = file= > ffmpeg.FS("readdir"."/").includes(file);
    const readFile = file= > ffmpeg.FS("readFile", file);

    await ffmpeg.load();
    const sourceBuffer = await fetch(filename).then(r= > r.arrayBuffer());
    ffmpeg.FS(
      "writeFile"."input.mp4".new Uint8Array(sourceBuffer, 0, sourceBuffer.byteLength)
    );

    let index = 0;

    ffmpeg
      .run(
        "-i"."input.mp4".// Encode the stream
        "-segment_format_options"."movflags=frag_keyframe+empty_moov+default_base_moof".// Encode as a 5-second fragment
        "-segment_time"."5".// Write to the file system by index
        "-f"."segment"."%d.mp4"
      )
      .then(() = > {
        // Send the rest of the file
        while (fileExists(`${index}.mp4`)) {
          subscriber.next(readFile(`${index}.mp4`));
          index++;
        }
        subscriber.complete();
      });

    setInterval(() = > {
      // Periodically check whether files have been written
      if (fileExists(`${index + 1}.mp4`)) {
        subscriber.next(readFile(`${index}.mp4`)); index++; }},200);
  });
Copy the code

The code above uses the same FFmpeg.wasm setup as before to write the transcoded files to the in-memory file system. To create segmented output, ffmpeg.run is configured differently from the previous example, requiring the proper transcoder to be set up. FFmpeg takes incremental indexes (0.mp4, 1.mp4,…) at run time Writes files to the memory file system.

To achieve streaming output, you need to poll the file system for transcoded output and send the data as an event through subscriber.next. Finally, when ffmpeg.run is complete, the rest of the file content is sent out and the stream is closed.

You need to create a MediaSource object to transfer the data stream to the video element and wait for the SourceOpen event to fire. The following code uses RxJS’s combineLatest to ensure that FFmpeg output is not processed until the event is triggered:

const mediaSource = new MediaSource();
videoPlayer.src = URL.createObjectURL(mediaSource);
videoPlayer.play();

const mediaSourceOpen = fromEvent(mediaSource, "sourceopen");

const bufferStreamReady = combineLatest(
  mediaSourceOpen,
  bufferStream("4club-JTV-i63.avi")
).pipe(map(([, a]) = > a));
Copy the code

When the first video slice or buffer is received, the MediaSource needs to be added to the SourceBuffer at the correct time and the original buffer attached to the SourceBuffer. After that, it is important to note that the new buffer cannot be added to the SourceBuffer immediately until it issues the updateend event to indicate that the previous buffer has been processed.

The following code handles the first buffer with take and reads the MIME type using the mux.js library. A new observable flow is then returned from the Updateend event:

const sourceBufferUpdateEnd = bufferStreamReady.pipe(
  take(1),
  map(buffer= > {
    // Create a buffer based on the current MIME type
    const mime = `video/mp4; codecs="${muxjs.mp4.probe
      .tracks(buffer)
      .map(t => t.codec)
      .join(",")}"`;
    const sourceBuf = mediaSource.addSourceBuffer(mime);

    // Append the channel buffer
    mediaSource.duration = 5;
    sourceBuf.timestampOffset = 0;
    sourceBuf.appendBuffer(buffer);

    // Create a new event stream
    return fromEvent(sourceBuf, "updateend").pipe(map(() = > sourceBuf));
  }),
  flatMap(value= > value)
);
Copy the code

All that is left is to append the buffer when it arrives and the SourceBuffer is ready. This can be done with the RxJS zip function:

zip(sourceBufferUpdateEnd, bufferStreamReady.pipe(skip(1)))
  .pipe(
    map(([sourceBuf, buffer], index) = > {
      mediaSource.duration = 10 + index * 5;
      sourceBuf.timestampOffset = 5 + index * 5;
      sourceBuf.appendBuffer(buffer.buffer);
    })
  )
  .subscribe();
Copy the code

With some coordination of events, the video is transcoded with very little code and the results are gradually added to the video elements.

The code for the last example is on GitHub.

Welcome to pay attention to my public number: front-end pioneer