In June this year, Watermelon Video announced full support for HDR video function, becoming the first VIDEO platform with HDR technology covering all forms (mobile phone, PC and tablet) and all links (shooting, editing and consumption).
HDR (High Dynamic Range) video, compared with ordinary SDR video, has higher color depth, wider Dynamic Range and stronger color expression, which can significantly improve video quality. When Watermelon users shoot, edit, preview, upload and watch HDR videos, the “end-to-end HDR video solution” of Bytedance’s Volcano Engine Multimedia Lab plays an important role, and the solution has been gradually opened to Volcano Engine’s enterprise customers.
In recent years, many manufacturers have provided HDR technology on different terminal devices, including the ability to support filming and playback. However, THE operation of HDR technology is not a single step, but an end-to-end technology from content creation to delivery to device display. In order to efficiently transcode HDR video content and render HDR images on terminal devices, Volcano Engine Multimedia Lab has created a complete solution that supports HDR video content in various formats from upload, transcode, distribution to terminal adaptation.
In order to improve the user experience, the solution adopts the following unique technologies in different aspects of end-to-end:
1. In order to enrich HDR video sources, unique Inverse Tone Mapping algorithm is designed to maximize the HDR “restoration” and upward compatibility of existing SDR programs;
2. In order to improve user viewing experience on ordinary SDR devices, a unique tone mapping algorithm is designed to obtain the playback effect as close as possible to THAT of HDR on SDR devices;
3. The solution of high color restoration rendering and low power consumption at the player end can support up to 4K resolution and 60fpsHDR video playback smoothly;
Iv. It provides the first universal HDR brightness management scheme for Android platform in China to ensure that different Android models can have relatively stable HDR viewing experience.
The flow chart of volcano Engine Multimedia Lab’s “end-to-end HDR Video Solution” is as follows:
The technical characteristics of the scheme in HDR transcoding and playback will be described in detail below.
HDR transcoding
Transcoding support for HDR sources in different formats
After years of development, HDR standards have been differentiated into a variety of standards, such as HDR10, HDR10+, Dolby Vision and HLG. HDR videos uploaded by users of watermelons and other products also involve different encoding bit depths (8bit, 10bit, 12bit) and a variety of video encoder standards. The current transcoding workflow accurately recognizes and is compatible with all known HDR video formats.
In order to better process high-dynamic video like HDR, Bytedance’s self-developed encoder BVC has also made a lot of targeted optimization, including but not limited to SIMD implementation and coding optimization algorithm, so as to achieve a high balance between computational complexity and coding efficiency.
Algorithm processing of HDR video generated by SDR
HDR content was first produced by film festivals and broadcast networks. The first 4K HDR movies were released by Fox in 2015, the same year Sony showed off their HLG video at the SMPTE conference, and HDR rendering was used in video games even earlier. It is only in recent years that consumer-side HDR playback has begun to take off. Until then, HDR content had to be converted to SDR for transmission in order to be compatible with mainstream playback devices.
Therefore, there are a large number of SDR contents originally in HDR format on the Internet, such as movie clips, game scenes, professional photography works, etc. For this part, the technical team reworked it by using algorithms and inverse Tone Mapping technology to restore these SDR videos to the original HDR format, giving a new life to users’ HDR devices.
Compatible support for non-HDR devices
When HDR video is played on SDR device, there will be many compatibility problems such as black screen (not supported by decoder) or abnormal color (not supported by player). In order to ensure the user experience of using SDR equipment, the server must carry out backward compatible transcoding processing. For HDR source video, the technical team uses tone Mapping algorithm to convert the corresponding SDR version and send it to these users.
The conversion of HDR to SDR video has undergone the compression of brightness dynamic range and color space (BT.2020 to BT.709) and the change of photoelectric conversion function EOTF. Meanwhile, the signal bit depth has also been reduced from 10bit to 8bit. The number of available chromatic levels for video signals has been reduced by 75% (1024 to 256). In order to retain the details of dark and light parts in HDR video to the maximum extent and ensure the color not distortion, the technical team dynamically adjusted the conversion parameters in different scenes by analyzing the image characteristics such as brightness and gamut distribution of video images, and fully utilized 256 color levels of 8bit signal to represent the original HDR content. Ensure that the images in HDR source video can be accurately and well presented to users of SDR devices.
Consider that the traditional SDR standard is based on a maximum brightness of 100nit, and today’s smartphone screens are generally brighter than that. In order to take full advantage of the brightness range of the phone’s screen and render the best display on the user’s device, it is necessary to assign more color levels to the dark details of the screen, as well as higher overall contrast of the screen. Therefore, in the conversion process, the technical team redesigned the parameters of tone Mapping algorithm based on the playing environment of consumer end, so that the transformed video can present the playing effect as close as possible to that of HDR on ordinary SDR equipment.
** HDR playback **
Compatibility and high performance optimization
Watermelon Video and other products use bytedance’s self-developed player, which supports HDR10/HLG and other HDR formats on both Android and iOS. Devices with HDR playback capability can be detected through model data, distributed and played, and stable HDR picture quality performance can be achieved on different models.
It is worth mentioning that this player also has the rendering capability of HDR rollback SDR on Android. On Android models that do not support true HDR, Tone Mapping can be used to transfer HDR to SDR rendering to ensure the optimal display of the playback quality of the user terminal. On iOS, asynchronous initialization technology is used to greatly shorten the start time of HDR videos and further improve the video playing experience.
The byte player supports smooth playback up to 4K resolution and 60fps while maintaining extreme color experience and low power consumption.
HDR brightness adaptive scheme
In order to fully display HDR content to users, Watermelon Video needs to make full use of the high dynamic range of the device screen when playing videos on the mobile terminal. According to the definition of dynamic range, the dynamic range of the screen is determined by the ratio of the maximum brightness to the lowest brightness (black bits). When people watch HDR video on an OLED screen, the dynamic range of the video is entirely determined by the maximum brightness of the screen, since the black bit is a fixed value. Therefore, in order for the user to experience a higher dynamic range when viewing HDR content than SDR video, the screen brightness needs to be raised to achieve this.
Currently, major Android device manufacturers lack a unified brightness management plan. Therefore, volcanic engine multimedia laboratory proposed the first domestic AnZhuoDuan general HDR brightness management scheme, in the laboratory to determine the best HDR watch brightness curves under different initial brightness, and integrate the watermelon video AnZhuoDuan APP, under different models and different initial brightness when playing HDR video adaptively adjust the screen brightness, This ensures that different Android models can have a relatively stable HDR viewing environment, thus improving the user’s viewing experience.
About volcano Engine Multimedia Lab
The laboratory is committed to researching and exploring cutting-edge technologies in the field of multimedia, participating in the standardization of multimedia at home and abroad, and providing software and hardware solutions in the fields of multimedia content analysis, processing, compression, transmission, innovation and interaction. At present, many innovative algorithms provided by multimedia Lab have been widely used in douyin, watermelon video and other products of voD, live, real-time communication, pictures and other multimedia services, and provide technical services to volcano Engine enterprise customers.