In the Alliance for Open Media (AOMedia,AOM), hardware and software developers came together to create the AV1 standard specification. Roshan Baliga, senior Product Manager at Google, will talk about AV1 hardware in terms of what it is, its hardware adoption, its development and commercial testing, its software decoding, and its importance to the AV1 ecosystem.

The text/Roshan Baliga

Organizing/LiveVideoStack

Hello and welcome to the Open Media Alliance 2021 Summit, co-hosted by the Open Media Alliance and LiveVideoStack. I’m Roshan Baliga, senior Product Manager at Google, working on open video codecs and AV1 hardware development.

The AV1 standard specification was developed by the Open Media Alliance over a three-year period from 2015 to 2018. The Open Media Alliance was originally made up of seven companies, but its membership has since grown rapidly, with many companies contributing to THE development of AV1. Today, the Open Media Alliance has more than 47 corporate members worldwide.

1. Devices with AV1 hardware

While other presentations at the summit will focus on AV1’s software development efforts, I want to talk about AV1’s hardware. It’s been out for three years. We’ve had great success bringing AV1 hardware to market, and that success is due to the work we’ve been doing since the beginning of the development of the AV1.

2. Development of AV1

In the Open Media Consortium, hardware and software developers came together to create the AV1 standard specification. Oems and streaming companies work together to provide input. This allowed us to do two things: first, we could ensure that the tools in the codec would not be too difficult to implement on hardware; Second, this means that we can create consistent code streams during specification development that hardware designers can use to validate the application of AV1 decoders.

3. Film Grain HW example

Film Grain is a modified tool based on feedback from a hardware design engineer. The video output pipeline usually includes scaling, color correction, and denoising, so it is not always reasonable to require that the film particles must be reconstructed when the decoder outputs. Although Film Grain is a standard tool for AV1, the specification allows the re-addition of Film Film particle noise to the display path.

4. Commercial test flow

In the development of AV1, Argon Design, an Open Media Alliance member, worked to create consistent codestreams that are now on the market. AllegroDVT and Vicue Soft, two other open Media Alliance members, also offer commercial conformance streams, and the availability of these streams helps chip designers release AV1-compatible decoders quickly after AV1 is released.

5. Schedule

Above is our expected timeline for the launch of the AV1 codec. If you look at the hardware and equipment division you actually just beat expectations. In December 2019, OPPO released Reno3 5G mobile phone using Mediatek’s Breguet 1000 SoC chip, which is the first smartphone to support AV1 decoding in hardware.

6. Codec schedule

Our desire for hardware support also affected the timeline. Above is a timeline of open codecs versus MPEG /ITU. The Open Media Alliance set the AV1 release date for 2018 to ensure we had extensive hardware support before the VVC launch. We knew we could work on two more years for even greater compression, but 2018 was the perfect time to launch AV1. Why do we need a hardware decoder? Where can hardware decoders be used?

7. Software decoding 4K

Dav1d is an open source AV1 decoder developed by the VideoLAN and FFmpeg communities sponsored by the Open Media Alliance. It is the most widely used AV1 software decoder, and Google uses Dav1d for AV1 support in Google Chrome. On high-end desktop computers, such as Intel and AMD, the Dav1d can decode 4K AV1 video at over 40 frames per second. But the Dav1d isn’t as fast on low-end desktops, and sometimes it doesn’t reach 30 frames per second. Laptops are generally slower than desktops, so they may only be able to decode 1080p. And even if a computer can decode 4K in software, it may not meet the DIGITAL rights management requirements necessary to receive a high-resolution stream of code, and paid content is usually only accessible when using a hardware decoder.

8. Battery life

Another concern is battery life, an important consideration as users watch more and more video on mobile devices such as smartphones and laptops. Software decoders use more power per video stream than equivalent hardware decoders, and even if SoC chips can decode 720P or 1080P with software, hardware decoders are critical for mobile devices.

9. Google Duo adds AV1 support

The coding difference is even greater, as AV1 was added to Duo a year ago to support video calling at 320×240 resolution. The reason for the low resolution is that until the hardware AV1 encoder is applied to the mobile phone, the mobile phone can only achieve low resolution real-time software coding. This is not to say that the AV1 has not been a success for the Duo. On the contrary, these low-resolution calls, as low as 15kbps, bring video calls to people in the developing world who can only make audio calls. We need better hardware coding so that all parts of the world can enjoy better high-resolution video calls.

10. Libaom coding speed

Another challenge in coding software is the sheer volume of video content hosted by sites like YouTube and Tiktok. For these types of sites, users are constantly uploading new videos, which must be instantly transcoded and streamed out, and these videos are much larger than those on paid video sites like Netflix and IQiyi. Over the past two years, we have improved the performance of the Open Media Alliance reference AV1 encoder, and Libaom now excels in speed. It encodes about 150 times faster than when the final version was released, and roughly as fast as VP9. Even so, this approach is not cost-effective for large video sites that need to be efficient. Software coding is great for the “head” of a video site, but the “long tail” needs more CPU.

11. ASIC hardware coding

In the coming weeks, Google and YouTube will present a paper describing their use of hardware encoders at the ASPLOS2021 conference. It will describe the importance of hardware encoders for large video sites to adopt new codecs. AV1 hardware encoder IP has been available for over a year and this year we will see the launch of AV1 commercial hardware encoders. Last month, Open Media Alliance member NETINT announced their AV1 hardware encoder chip for live or streaming media.

12. Brands that support AV1 decoders

There’s little point in encoding AV1 if the video doesn’t play back smoothly, so hardware decoders on consumer devices are very important. Over the past two years, there have been many devices on the market with AV1 hardware decoders, including TVS, smartphones, laptops, desktops, and graphics processors. The images above are some of the brands that currently support AV1 hardware decoders. Since AV1 has commercial conformance streams, these products will successfully decode all compliant AV1 streams, and some products can decode 8K/60 AV1 streams.

The future of hardware

We expect more manufacturers to support AV1 in their product portfolios, especially at the lower end of the market. As AV1 is pushed towards cheaper and more affordable products, such as mid-range smartphones and cheaper televisions. The increased install base will spur streaming companies to adopt AV1, and we expect to soon launch AV1-equipped set-top boxes that will last for seven years and more. I hope you found this talk useful and learned about the evolution and adoption of AV1 hardware and its importance to the AV1 ecosystem. Stay tuned for more av1-enabled devices and consumer electronics releases in the near future, and thank you for listening.