Introduction to WebAssembly technology

WebAssembly technology has become so popular in recent years that it has become a new turning point for JavaScript. JavaScript has been criticized for its performance problems since its birth in 1995. By 2008, many browsers had added just-in-time compilers, JavaScript had started to introduce JITs, and with a lot of optimization from Google and others, performance had improved by more than 10 times. As a result, JavaScript has begun to move beyond the browser, popping up in various areas, such as Node.js in the background and Electron on the desktop. \

In short, JIT technology is to temporarily store commonly used binary code blocks during the execution of JavaScript interpretation, and directly run the temporarily stored binary code blocks when the same code block is executed next time, saving the time of interpretation. How about compiling all of your JavaScript code into binary at once to make it more efficient? WebAssembly came along to answer that question.

Before WebAssembly, JavaScript was the only programming language you could run in a browser. WebAssembly technology makes it possible for browsers to run programs written in other languages. You can currently create WASM modules using C, C++, Rust, Go, Java, C# compilers (and more). The browser runs the WASM module in a private virtual machine at runtime. WebAssembly is also seen by many front-end developers as the next generation of front-end technology because it is binary and runs much more efficiently than interpreted JavaScript scripts.

The current WebAssembly compatibility is shown below:

As you can see, WebAssembly is now available in all major browsers, both PC and mobile, and it continues to be supported by major browser vendors and will soon become very common. \

WebAssembly and live streaming are different sparks

Iqiyi has been producing live streams in MP4 and FLV formats for a long time, but the NATIVE Html5 video tag only supports THE playback of MP4, how to solve the problem of FLV format in the web end is in front of everyone. Generally speaking, FLV format has the following solutions for playing on the web page: \

1. Use the Flash player plugin ************

However, due to various performance and security issues, the major browsers have gradually weakened this approach, and Chrome will stop supporting Flash Player around 2020, so there are few people using flash Player now.

2. Webpage decoding of FLV format video ************

Using Canvas to render images and audio to play sounds is like making a player on the web side, which is also feasible. However, each major browser manufacturer will optimize the hardware acceleration rendering of the native video control for different platforms. If rendering by oneself, the hardware acceleration will also need to be done by oneself, which will cost a lot of manpower and the effect is difficult to compare with the browser’s native hardware acceleration.

3. Convert FLV format to MP4 format on the web and use native player ************

This is also the most commonly used solution. In this way, you can either play the FLV live stream or leave rendering to the native player, giving full play to the optimization capabilities of the native player.

Iqiyi live uses the third way. When the FLV live stream arrives at the front end, JavaScript is used to convert the FLV into MP4 and then hand it to the native player. However, the performance of this section was always unsatisfactory due to the low efficiency of JavaScript, so we decided to introduce WebAssembly technology to see if it could make a difference. Now open any iQiyi live studio and type __enablewasm__=true to enable the WebAssembly transcoding mode, as shown below:

In terms of experience, both modes can meet the needs of smooth watching live broadcast. As you can see, the WebAssembly module is a perfect replacement for the original javascript-written transcoding module. Here’s how to access WebAssembly. \

Steps for accessing WebAssembly

Using WebAssembly is very simple, generally broken down into the following steps: \

1. Use C to write the code of FLV to MP4

First define the WebAssembly interface file called by JS: \

If you want to compile a WASM file that can be called by JavaScript, you need to use EM_PORT_API in front of functions that can be called externally, so that WebAssembly will throw this function as a method that can be called by JavaScript at later compilation time. \

Then you need to define some interfaces for WebAssembly to call JavaScript, as shown below:

It is mainly to inform JS to convert the header information of mp4 stream and the address of the cache area that has been turned into a good part of the stream, etc. The actual call code needs to use the EM_ASM_() function package, which is filled with the name of the JavaScript method called and with parameters. \

After defining the interface is the transcoding implementation part, which involves the relevant knowledge of FLV and MP4 formats (students who are not familiar with these two formats can read the relevant documents). The overall transcoding adopts the mode of FLV and MP4 double cache, and the process is shown as follows:

  1. After JavaScript obtains the live stream, it stores it in the FLV cache.
  2. JavaScript notifies WebAssembly of cache headers and progress;
  3. WebAssembly requests cached data;
  4. WebAssembly transcoding;
  5. Save the improved MP4 clip into the MP4 cache;
  6. WebAssembly notifies JavaScript transcoding progress;

Finally, JavaScript tells the native player to play the video stream directly from the MP4 cache.

2. Compile FLV2MP4.js and flv2MP4.wasm using EMCC

First, you need to install the Emscripten environment. You can refer to the Emscripten website for detailed installation and configuration steps. \

Emcc main.c -s TOTAL_MEMORY=268435456 -g -o flvtomp4.js FlvToMp4. Js and flvToMp4. Wasm.

Where flvtomp4.wASM is the actual transcoding code, flvtomp4.js is equivalent to the interface file, player can import flvtomp4.js to load the WASM file and call the binary code in the WASM file.

Flvtomp4.js automatically generates code for initializing WebAssembly, calls webAssembly.instantiate () after fetching the WebAssembly binary, and instantiates the WebAssembly. And you can try again if you fail to get or load a WASM file.

You can also see the download process by looking at the getBinaryPromise() method. \

With automatically generated code, you can basically ignore the problems of loading WASM files, which is very convenient. \

3. Connect to the compiled WASM file

Since transcoding is a high-CPU job, run it in the web_worker so it doesn’t block the main thread rendering. \

After the worker is created, bind the event to the main thread, that is, bind the methods that wASM calls JavaScript as defined above:

Above is the definition of WebAssembly’s relevant message interface for notificating JavaScript. This has completed all interface definitions for the entire transcoding process, as shown in the sequence diagram below.

It can be seen from the sequence diagram that the player will initialize the WebAssembly module when receiving the stream. After the initialization, it enters the transcoding stage, notifying the WebAssembly to transcode and store it in the cache, and then notifying the player to play the stream. \

Actual performance comparison using WebAssembly

If the experience is consistent, how much is the actual performance improvement? Again, let the numbers speak for themselves. The iQiyi live team tested the actual performance of WebAssembly by using code points and real-time data monitoring by the browser’s own performance monitoring tool. \

1. Efficiency of live broadcast code transfer

First, test the speed of actual transcoding using WebAssembly. The transcoding module developed by JavaScript and the transcoding module of WebAssembly are respectively used for conversion, and the timing is performed before and after the conversion of FLV stream packets in the actual live broadcast room. Finally, the data obtained is shown as follows: \

In the table, the second column is the transcoding time of each packet, and the third column is the packet size. At the end of the table, the size and total time of total packets are counted. The average transmission rates of the disabled and enabled WebAssembly are 35305.6 bytes /s and 46608.1 bytes /s, respectively. It can be seen that the transcoding speed after the opening of WebAssembly is very obvious.

2. Runtime browser resource consumption

What kind of improvement can WebAssembly bring to the broadcast room when it is actually used in the broadcast room? The most obvious is the decline in CPU usage. You can use Chrome’s Performance Monitor to compare resource consumption before and after using WebAssembly. \

As shown in the figure above, you can find the Performance Monitor in the developer tool More Tools. With this tool, you can get a rough idea of the CPU usage at normal times. The following two graphs show the CPU usage of WebAssembly when stable playback is disabled and WebAssembly is enabled: \

  • WebAssembly is disabled: ****

  • Start WebAssembly: ****\

As you can see from the figure, when WebAssembly is not enabled, the CPU usage basically stabilizes around 7%, but after WebAssembly is enabled, the CPU usage stabilizes around 5%, which can be estimated as a 10-20% improvement (note: The test model is macbookpro 2018 with CPU i7 2.2ghz. The performance of different machines may be different and the fluctuation is not completely consistent, but the performance of WebAssembly can be improved to different degrees on different platforms. \

More possibilities for WebAssembly’s future

The use of WebAssembly transcoding is still only a very small use of WebAssembly, iQiyi live team will use WebAssembly technology to achieve more interesting and valuable functions, such as: \

  • Porting c++ projects. Now many image-related projects and algorithms are written by c++, if you want to run in the browser, you can only use JavaScript to rewrite; Now, with WebAssembly, it can run in a browser with minimal modification.
  • Algorithm encryption. Because the wASM compiled by WebAssembly is a binary file, the cost of decompilation is very high, and some algorithms with strong security will use WebAssembly technology.
  • H.265 encoding format support. H.265 encoding method is used by more and more products because of its excellent compression ratio, but the mainstream browser does not support h. 265 hard solution. However, it is possible to use WebAssembly to convert H.265 streams into H.264 streams, which can then be played back using native players to achieve the effect of playing H.265 streams on the Web side, which can greatly reduce bandwidth costs.

Thanks to the improved performance, WebAssembly is beginning to emerge in various areas. In the future, iQiyi live team will also try to use WebAsssembly to implement more functions to optimize the live experience of iQiyi.

end

Maybe you’d like to see more

Continuous polishing and Improvement — TECHNICAL performance of PC Web Broadcast Player

** We are all dream chasers — IQiyi 8K VR live broadcasting technology solution **

Scan the qr code below, more exciting content to accompany you!

Iqiyi technical product team

Think simply, act simply