Translation | Jiang Jinyuan

Edit | Alex

Technology review | hao, hai-tao Yang

This article is from OTTVerse by Jan Ozer.

Video exploration #006# — VVC

Talk about VVC

Now that 18 months have passed since VVC was first released as an international standard in July 2020, let’s take a look at VVC’s progress to date (licensing, performance, chip development and testing, etc.).

VVC patent owner

It’s all very well chasing video quality, but the world runs on money, so let’s start there. It is important to know that video compression standards such as VVC will have unique patents related to the new features, but will also contain patents derived from previous standards such as HEVC and H.264. Table 1 is derived from the list of HEVC and VVC patent owners in an article [1] published in March 2021.

The authors then updated the information on vVP-related patent owners on iPlytics[2], and we noticed that some significant vVVP-only patent owners were not listed in Table 1. However, the synthesis of HEVC and VVC patent contribution and the percentage of patent contribution in Table 1 will be helpful for rough analysis and discussion, but not enough to give an authoritative judgment.

I would like to point out that Although Apple is not listed in Table 1, it owns about 350 patents related to VVC [3], plus other patents Apple may have acquired [4], indicating that it has invested heavily in VVC.

I know that Apple has joined the AOM alliance, becoming one of its “founding” members 27 months after it was founded, and it is clearly not blindly following other companies. As detailed in CanIUse[5], Apple has so far fallen significantly behind other companies in TERMS of AV1 compatibility because Safari/Mac, Safari iOS, and Apple TV do not support AV1. Meanwhile, Apple is the only major platform that provides browser-based access to HEVC (which directly competes with AV1 codec standard) [6].

Table 1 also includes other AOM member companies, including Tencent, Intel, Netflix and Samsung. Therefore, at least as far as these companies are concerned, AOM membership should not be interpreted as anti-VVC.

Table 1 Ownership percentage of HEVC and VVC patents

As you can already see from Table 1, I would like to add a few points. First, notice the diversity and size of the businesses in the table. Clearly, each of these companies has invested heavily in VVC and has a strong incentive to ensure its success. For example, Table 2 shows the global smartphone chip market share from q2 2020 to Q3 2021. As you can see, most of the companies listed in Table 2 have very high patent percentages in Table 1, especially Qualcomm, Huawei and Mediatek.

Table 2 Global smartphone chip market share from q2 2020 to Q3 2021

Based on these numbers, you’d expect all of these companies to seriously consider supporting VVC in their future chips (mediatek has already done so). We will explain this in detail in the implementation section below.

Second, if you add up all the percentages (including most of the companies shown by iPlytics), they add up to more than one. I’m not sure why that is, but I believe there is a clear explanation. Also note that HEVC Advance accounts for 24.38% of the patent owners in the two patent pools listed in the table (which we discuss below), while MPEG LA is 0. If you look at another iPlytics PPT, you’ll see that several major companies with VVC standard patents are not listed in Table 1, but have joined one of the two patent pools (although the percentage of ownership not covered in Table 1 is relatively small).

Of course, none of the patents shown in Table 1 have been submitted to any patent pool, so it’s still early days. However, when you are licensed in the patent pool, the patent fee should correspond to the patented technology provided by the license. That said, the Access Advance patent pool offers more patented technologies, which also represents higher patent fees.

Next, let’s look at two patent pools.

VVC patent pool

As the name suggests, patent pooling is when groups of patent owners come together to provide a joint license for their patents. The patent pool simplifies the process of technology implementers acquiring the technology; Instead of signing a separate license with each patent owner who contributes technology to a standard, implementers can sign a license that covers all patent owners in the patent pool. A patent pool transfers only the use of patents in the pool, so if there are more than one patent pool, the implementer will have to license from all the pools and enter into agreements with patent owners who are not in the pool.

Despite efforts to persuade VVC patent owners to form a patent pool [7], two eventually emerged. The Access Advance patent Pool is the first to launch [8], and its patent fee structure [9] depends on a variety of factors, including compliance with licenses, the option to display the Access Advance trademark, and the availability of both VVC and HEVC licenses from Access Advance. Figure 3 shows the lowest rates; Please note the classification, patent fees under the classification, annual upper limit of the classification and overall upper limit.

Figure 3 Patent rates for Access Advance

The Access Advance also provides MCBA (Multi-Codec Bridging Agreement) for companies licensed both VVC and HEVC. As mentioned in news [10], companies that license VVC and HEVC and sign MCBA will enjoy a 45% patent discount on products that use both VVC and HEVC patents, the same patent fee as products that use VVC-only. You can view the list of patent owners in the Access Advance VVC patent pool [11].

MPEG LA published the relevant terms and the list of patent owners in January 2022 [12]. The relevant terms shown in Figure 4 can be viewed here [13]. As you can see, hardware has different patent fees than paid software and free software like browsers. For those unfamiliar with Latin, de minimus means less than that and no royalties are paid. You can check out the link in the notes [13] for more details on patent fee caps.

Figure 4 MPEG LA VVC clause

By contrast, many technologies, such as Wi-Fi and cellular standards, have two or more patent pools and some significant patent owners, such as Qualcomm, prefer to be directly licensors rather than joining the pool. Therefore, the advantages of forming a VVC patent license certainly outweigh the disadvantages. If there is a “downside” compared to HEVC, it is that these pools merge more quickly and the terms should be more acceptable to potential licensees. While it is true that certain patent owners who do not join the pool do create uncertainty, the fact that both pools have announced terms suggests that VVC is now open for business.

Now that you have an idea of the VVC patent costs, let’s take a look at its performance.

performance

I have evaluated Fraunhofer’s VVC codec for _Streaming Media_ magazine [14] and table 4 shows the main quality comparisons. Here, the reader should be careful to distinguish between codecs and video encoding standards. VVC is a standard, and Fraunhofer’s VVenC codec is an implementation of that standard, just as any other codec is an implementation of that standard, be it HEVC, H.264, EVC, or AV1.

When comparing these encoders using the VMAF metric, I found that Fraunhofer’s VVenC encoders were approximately 5.55% more efficient than libaom-AV1; It is only 2.36% more efficient than EVC’s Main Profile using XEVE open source encoder. Moscow State University, as you’ll see later, got very different results.

Figure 5 VMAF BD-rate implemented by multiple codecs

I tested coding time on an HP workstation with a 3.4ghz Intel I7-3770 CPU, 16GB RAM, 4 cores and 8 threads and HTT enabled. VVenC encods live playback 66 times, compared with 33 times for the X265 in very Slow gear configuration, 34 times for libaom-AV with CPU usage 3, and 7 times for x264 in Very Slow gear configuration. It can be seen that the VVenC coding speed is basically in line with expectations.

In playback tests on the same computer, Fraunhofer achieved a VVC decoder of 39fps, compared with 357fps for AV1 and 388fps for HEVC. In my opinion, this would make VVC classified as a hardware codec, which means (I predict) that most publishers won’t use software playback on current platforms (due to poor performance, battery life or both). Of course, VVC won’t play on smart TVS or similar devices without hardware support. Note that some of the codec implementation studies discussed below have shown very good playback performance, and many people may well have to rethink this conclusion.

VVC quality comparison of Moscow State University (MSU)

In this MSU report [15], the Moscow State University team studied a variety of AV1, VVC, HEVC and H.264 codecs. The table below shows the results of their comparison using YUV 4:1:1 SSIM metrics. Comparisons of other metrics are also available in the free report (there are more comparisons in the paid report). To explain the scoring mechanism, MSU sets the x265 encoder as a reference to 100% performance, and then compares other codecs to this to get a score.

FIG. 6 Baseline comparison of Moscow State University

You’ll notice that in the MSU test, VVenC was only 2% more compressed than the X265, but my tests found a 43% improvement in VVenC. Although our testing methods were completely different and I used VMAF instead of SSIM as a comparison indicator, this result was still very surprising. I asked Fraunhofer about the discrepancy during testing, but they couldn’t provide an explanation either.

MSU also found that the compression performance of Huawei’s HW266 encoder improved by 50 percent compared to the X265, alibaba’s S266_V1 VVC encoder improved by about 47 percent compared to the X265, and Tencent’s encoder improved by about 41 percent. Huawei’s VVC encoder improves compression efficiency by about 20% over the most efficient AV1 implementation (Tencent VAV1).

Although we compared the experiments in different ways, both studies showed a substantial increase in efficiency (about 43% to 59%) for VVC compared to HEVC, and a significant decrease in benefits (only about 5% to 20%) for VVC compared to AV1.

Development activities

In January 2022, Gary Sullivan from Microsoft (and the founding co-chair of the joint team of video experts developing VVC) wrote a document titled “The State of VVC Standard Deployment” detailing development activities around VVC. This is a treasure trove of information that you can download here [16], and much of what follows is taken from this document.

Hardware implementation is the project with the longest delivery time, and few companies have disclosed their hardware codec product plans to date. Here’s what we know:

  • In November 2021, Mediatek released Pentonic 2000[17], which the company claimed was the first commercial 8K TV chip to support VVC/H.266, and further stated that the chip would support Dolby Vision, Smart TVS equipped with the chip are expected to be launched in the global market in 2022.

  • In July 2021, Allegro announced that its AL-D320 video decoder semiconductor IP Core supports the latest VVC/H.266 format [18]. According to the published message, the decoder is immediately available for SoC vendor integration, so the actual chip with the decoder can only be used after this deployment.

  • The 8K Association found a VVC-enabled TCL smart TV on NAB, but details such as the decoder chip are unclear.

  • Note that Qualcomm’s Snapdragon 8 Gen 2 chip will most likely support AV1, but there is no mention of VVC.

Mediatek predicted that TVS equipped with Pentonic 2000 would be available in 2022, and TCL has realized this prediction. But even if more manufacturers are able to produce such sets, it will take years for a large number of VVC-enabled sets to attract independent publishers (such as those without VVC patents). Even if they can attract a publisher, there is a sense that VVC patent owners such as Qualcomm, Broadcom and Intel could stifle its growth by not accelerating VVC support on their chips.

The following table contains some of the other VVC implementations listed in the Sullivan document:

VVC implementation

Software decoding

If VVC does require hardware for full-frame playback on computers and full-frame, power-saving playback on mobile devices, its adoption will be delayed by 2-3 years. Therefore, testing related to software playback can be particularly interesting.

According to a study of Kuaishou [16] (researchers focus on The Android platform because 80~90% of the views on their platform come from Android phones), by testing 1280×720 videos uploaded by App users, The researchers achieved 97.49 FPS on an Android machine (single-threaded) and 249fps on an iPhone 12 (single-threaded).

Figure 7. Impressive playback performance on Android phones

Bytedance’s researchers used a mix of GPU and CPU decoding to test their decoder and achieved faster speeds than streaming 4K video in real time on M1-based Macs and iPhone 13 (see Figure 8).

What’s interesting about these results is how adoption of VVC by popular platforms like TikTok accelerates hardware support for VVC codecs, especially when publishers tie some enhancements (like higher resolution video) to VVC codec support. For example, YouTube eventually convinced Apple to support VP9 (by encoding all video streams larger than 1080p as EITHER VP9 or AV1).

While I still doubt any independent publisher would deploy VVC on a battery-powered device for pure software playback, the results are surprising. You’ll see that all of the companies included in Table 1 will be deploying VVC more aggressively, just as YouTube, Meta, and Netflix are deploying AV1.

Figure 8: Faster than live 4K 8-bit video playback on the iPhone 13

Support for browser and TV

The biggest problem with VVC software playback is probably its software support. HEVC is not supported in Chrome, Firefox, or Edge on Windows or Android platforms, so it never breaks through in browser-based playback. Publishers can resort to workarounds, such as using specific video players, but then come the cost of patents. If you have a mobile App, you can use any codec you like, but it’s difficult for companies other than header streaming service providers to make that choice. See, it’s the patent fees again.

After nearly eight years in the market, HEVC has yet to gain a foothold in browsers, but has had great success with Premium Content on TV. VVC is likely to play in this area as well, but it will take another 2-3 years. Similarly, note that VVC and LCEVC have been included in Brazil’s recent TV 3.0 project [19], and that VVC (along with AV1 and AVS3) will be included in the upcoming European TELEVISION standard DVB [20].

For the sake of completeness, I should point out that VVC also brings unique enhancements to VR and AR applications that will be converging in the coming years.

Here’s what I think:

  • VVC contributors include a number of very important companies that have invested heavily in VVC and are a significant incentive to its success. Like the members of AOM, they create a significant market for the deployment and success of VVC.

  • VVC should be about 50% more efficient than HEVC and 10-20% more efficient than AV1.

  • VVC is likely to find early adoption among interested companies such as Kuaishou, Bytedance and Tencent, as AV1 did with YouTube, Meta and Netflix.

  • In the traditional streaming media market, VVC is likely to mimic HEVC, with a lot of use in TV scenarios and little use for browser-based playback. In other words, it may take two to three years for the number of VVC-enabled TVS installed to increase in order to attract independent publishers.

  • VVC seems to be an early technology leader in VR and AR.

Finally, when we talk about adding another codec to the delivery portfolio, it feels like flipping a switch, but it’s not. The dramatic drop in bandwidth costs has greatly reduced the economics of supporting more efficient codecs. Refer to article

Computing Break even on Codec Developments[21].

For this reason, most new codecs seem to be deployed when they support new markets or dramatically changing requirements, such as HEVC, which supports 4K and HDR encoding. Will Premium Content publishers switch from HEVC to VVC to save 50% bandwidth? I doubt it for at least the next 2-3 years. On the other hand, if AR/VR really does catch on and VVC support becomes the “ticket to the game”, we will likely see massive adoption and deployment of VVC.

Figure 9 Codecs respondents plan to support in 2022, from the Bitmovin Video Developer report

Finally, I need to point out that 20% of respondents in the Bitmovin Video Developer report [22] plan to support VVC in 2022, slightly less than the 22% for AV1 and 25% for HEVC (Figure 9). Another report from Rethink Technology [23] is also very optimistic that VVC will gain support from mid-2023. For some of the numbers in Bitmovin’s report, the VVC projections may make sense for companies in the streaming ecosystem that need to do more to code, distribute, monitor, and broadcast, but they feel very aggressive to actual content publishers.

Note:

[1] www.iam-media.com/who-leading…

[2] www.iplytics.com/webinar/sli…

[3] for details see page 42, made in the presentation PPT related links: www.iplytics.com/webinar/sli…

[4] streaminglearningcenter.com/codecs/when…

[5] caniuse.com/?search=av1

[6] caniuse.com/?search=HEV…

[7] www.streamingmedia.com/Articles/Ne…

[8] accessadvance.com/2021/07/01/…

[9] accessadvance.com/vvc-advance…

[10] accessadvance.com/2021/07/01/…

[11] accessadvance.com/vvc-advance…

[12] www.mpegla.com/programs/vv…

[13] www.streamingmedia.com/Articles/Re…

[14] www.streamingmedia.com/Articles/Re…

[15] www.compression.ru/video/codec…

[16] jvet-experts.org/doc\_end\_u… (If you cannot open the link, you can go to the original website to download)

[17] www.prnewswire.com/news-releas…

[18] www.allegrodvt.com/allegro-dvt…

[19] forumsbtvd.org.br/tv3\_0/

[20] dvb.org/wp-content/…

[21] ottverse.com/computing-b…

[22] bitmovin.com/top-video-t…

[23] www.streamingmedia.com/Articles/Re…

Thank you:

This article has been authorized by author Jan Ozer. Thank you.

Original link: ottverse.com/vvc-versati…