This audio and video master advanced will be a number of chapters to share (see how much I write), so that xiaomai can understand, due to the length of the reason, can not be fully integrated into an article, so several times published, if you need a complete PDF document, can go to my home profile contact me to get the PDF version

FFmpeg compilation process

Environment to prepare

Ffmpeg.org/releases/ff…

Dl.google.com/android/rep…

1 Preparing Documents

1. Prepare the environment (Linux, MAC). It is not recommended to compile on Windows, which is very troublesome. The following options are compiled on a MAC system. 2. Download ffMPEG source code, I compiled ffMPEG 4.2.1 version

2. Download the NDK. You are advised to download the LATEST NDK version r20C used in this document

2 Configuring the NDK environment

Open the terminal and use the command vim /etc/profile to open the environment variable configuration, which should not be a problem at this point.

3 Write a script file

The purpose of the script is to package the FFMPEG source code into the.so file we need for our Android project.

Create a new android_build.sh file in the ffmpeg folder.

#! /bin/bash export NDK=/root/ndk/android-ndk-r20 # Current system export HOST_TAG=linux-x86_64 # Supported Android CUP architecture # export ARCH= aARCH64 # export CPU= armV8-a export ARCH=armv7a export CPU= armV7-a # Minimum Supported Android OS export MIN=21 export ANDROID_NDK_PLATFORM=android-21 export PREFIX=$(pwd)/android/$CPU export MIN_PLATFORM=$NDK/platforms/android-$MIN export  SYSROOT=$NDK/sysroot export TOOLCHAIN=$NDK/toolchains/llvm/prebuilt/$HOST_TAG export AR=$TOOLCHAIN/bin/arm-linux-androideabi-ar export AS=$TOOLCHAIN/bin/arm-linux-androideabi-as export CC=$TOOLCHAIN/bin/$ARCH-linux-androideabi$MIN-clang echo "-----------------------------" echo $CC export CXX=$TOOLCHAIN/bin/$ARCH-linux-androideabi$MIN-clang++ export LD=$TOOLCHAIN/bin/arm-linux-androideabi-ld export NM=$TOOLCHAIN/bin/arm-linux-androideabi-nm export RANLIB=$TOOLCHAIN/bin/arm-linux-androideabi-ranlib export STRIP=$TOOLCHAIN/bin/arm-linux-androideabi-strip OPTIMIZE_CFLAGS="-DANDROID -I$NDK/sysroot/usr/include/arm-linux-androideabi/" ADDI_LDFLAGS="-Wl,-rpath-link=$MIN_PLATFORM/arch-arm/usr/lib -L$MIN_PLATFORM/arch-arm/usr/lib -nostdlib" sed -i "" "s/SLIBNAME_WITH_MAJOR='$(SLIBNAME).$(LIBMAJOR)'/SLIBNAME_WITH_MAJOR='$(SLIBPREF)$(FULLNAME)-$(LIBMAJOR)$(SLIBSUF)'/" configure sed -i "" "s/LIB_INSTALL_EXTRA_CMD='$$(RANLIB) "$(LIBDIR)\/$(LIBNAME)"'/LIB_INSTALL_EXTRA_CMD='$$(RANLIB) "$(LIBDIR)\/$(LIBNAME)"'/" configure sed -i "" "s/SLIB_INSTALL_NAME='$(SLIBNAME_WITH_VERSION)'/SLIB_INSTALL_NAME='$(SLIBNAME_WITH_MAJOR)'/" configure sed -i "" "s/SLIB_INSTALL_LINKS='$(SLIBNAME_WITH_MAJOR) $(SLIBNAME)'/SLIB_INSTALL_LINKS='$(SLIBNAME)'/" configure sed -i -e 's/#define getenv(x) NULL//*#define getenv(x) NULL*//g' config.h # sed -i "" "s/SHFLAGS='-shared -Wl,-soname,$(SLIBNAME)'/SHFLAGS='-shared -soname $(SLIBNAME)'/" configure # sed -i "" "s/-Wl//g" configure ./configure \ --prefix=$PREFIX \ --ar=$AR \ --as=$AS \ --cc=$CC \ --cxx=$CXX \ --nm=$NM \ --ranlib=$RANLIB \ --strip=$STRIP \ --arch=$ARCH \ --target-os=android \ --enable-cross-compile \ --disable-asm \ --enable-shared \ --disable-static \ --disable-ffprobe \ --disable-ffplay \ --disable-ffmpeg \ --disable-debug \ --disable-symver \ --disable-stripping \ --extra-cflags="-Os -fpic $OPTIMIZE_CFLAGS"\ --extra-ldflags="$ADDI_LDFLAGS" sed -i "" "s/#define HAVE_TRUNC 0/#define HAVE_TRUNC 1/" config.h sed -i "" "s/#define HAVE_TRUNCF 0/#define HAVE_TRUNCF 1/" config.h sed -i "" "s/#define HAVE_RINT 0/#define HAVE_RINT 1/" config.h sed -i "" "s/#define HAVE_LRINT 0/#define HAVE_LRINT 1/" config.h sed -i "" "s/#define HAVE_LRINTF 0/#define HAVE_LRINTF 1/" config.h sed -i "" "s/#define HAVE_ROUND 0/#define HAVE_ROUND 1/" config.h sed -i "" "s/#define HAVE_ROUNDF 0/#define HAVE_ROUNDF 1/" config.h sed -i "" "s/#define HAVE_CBRT 0/#define HAVE_CBRT 1/" config.h sed -i "" "s/#define HAVE_CBRTF 0/#define HAVE_CBRTF 1/" config.h sed -i "" "s/#define HAVE_COPYSIGN 0/#define HAVE_COPYSIGN 1/" config.h sed -i "" "s/#define HAVE_ERF 0/#define HAVE_ERF 1/" config.h sed -i "" "s/#define HAVE_HYPOT 0/#define HAVE_HYPOT 1/" config.h sed -i "" "s/#define HAVE_ISNAN 0/#define HAVE_ISNAN 1/" config.h sed -i "" "s/#define HAVE_ISFINITE 0/#define HAVE_ISFINITE 1/" config.h sed -i "" "s/#define HAVE_INET_ATON 0/#define HAVE_INET_ATON 1/" config.h sed -i "" "s/#define getenv(x) NULL/\/\/ #define getenv(x) NULL/" config.hCopy the code

Copy the above script code to our new file, open the terminal, to the FFmpeg file using the command can be compiled

sh build.sh

1. Create a project

1. Create a template project and check Include C++ Support to get a template project based on CMake.
2. Copy the header file and dynamic library generated by FFmpeg compilation to the app/ SRC /main/jniLibs app/ SRC /main/ CPP /include directory respectively
3. To configure the build. Gradle

Add it in the build.gradle file in the app directory under the Android node under the defaultConfig node

defaultConfig { applicationId "com.project.ffmpegplayer" minSdkVersion 15 targetSdkVersion 29 versionCode 1 versionName "1.0" testInstrumentationRunner "androidx. Test. Runner. AndroidJUnitRunner" externalNativeBuild {cmake {cppFlags "- frtti  -fexceptions" abiFilters 'armeabi-v7a' } } sourceSets { main { jniLibs.srcDirs = ['src/main/jniLibs'] } } }Copy the code
4. Configure the cmakelists. TXT file
Cmake_minimum_required (VERSION 3.4.1) include_directories(include) add_library(native-lib SHARED native-lib.cpp) Add_library (avcodec SHARED IMPORTED) set_target_properties(AVCOdec PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libavcodec.so) set_target_properties( avdevice PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libavdevice.so) add_library( avfilter SHARED IMPORTED) set_target_properties( avfilter PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libavfilter.so) add_library( avformat SHARED IMPORTED) set_target_properties( avformat PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libavformat.so) add_library( avutil SHARED IMPORTED) set_target_properties( avutil PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libavutil.so) add_library( swresample SHARED IMPORTED) set_target_properties( swresample PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libswresample.so) add_library( swscale SHARED IMPORTED) set_target_properties( swscale PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/.. /.. /main/jniLibs/${ANDROID_ABI}/libswscale.so) find_library( log-lib log ) target_link_libraries( native-lib avcodec avdevice avfilter avformat avutil swresample swscale ${log-lib} )Copy the code
5 write test code (header files must be extern)
#include <jni.h>
#include <string>
extern  "C"
{
#include "include/libavcodec/avcodec.h"
#include "include/libavformat/avformat.h"
}
extern "C" JNIEXPORT jstring JNICALL
Java_com_project_ffmpegplayer_MainActivity_stringFromJNI(
        JNIEnv* env,
        jobject /* this */) {
    std::string hello = "Hello from C++";
    avformat_version();
    return env->NewStringUTF(avcodec_configuration());
}
Copy the code

Video Decoding basics 01- Package format

The video files we play are usually wrapped in a package format. What is the function of the package format? Usually a video file has not only video but also audio, and the purpose of the package format is to package the video and audio together. Therefore, we first need to unpack the format to see which video streams and audio streams exist. At this time, the audio stream and video stream are still compressed data, which can not be directly used for display, which needs to be decoded. Here is a flowchart for playing a video file.

2. Video file packaging format

Package format (also called container), is already encoded and compressed video and audio tracks in accordance with a certain format into a file, that is to say, it is just a shell, or we can use it as a folder to put video and audio tracks. In layman’s terms, a video track is the equivalent of a meal, while an audio track is the equivalent of a dish, and the package format is a bowl, or a pot, used to hold the meal. The following are some common video file suffixes and their corresponding encapsulation formats.

Video file format Video package format
.avi AVI (Audio Video Interleaved)
, WMV, asf WMV (Windows Media Video)
.mpg、.mpeg、.vob、.dat、.3gp、.mp4 MPEG (Moving Picture Experts Group)
.mkv Matroska
Rm, RMVB Real Video
.mov QuickTime File Format
.flv Flash Video

Iii. Introduction of audio and video coding methods

1. Video coding method
  • Functions of video coding: compress video pixel data (RGB, YUV, etc.) into video code stream, thus reducing the amount of video data.
The name of the Launch institutions Launch time Current application field
HEVC (h. 265) MPEG/ITU-T 2013 In the research and development
H.264 MPEG/ITU-T 2003 All fields
MPEG4 MPEG 2001 The tepid
MPEG2 MPEG 1994 Digital TV
VP9 Google 2013 In the research and development
VP8 Google 2008 Don’t spread
VC-1 Microsoft Inc. 2006 Microsoft platform

Fourth, audio coding mode

Function of audio coding: compress audio sampling data (PCM, etc.) into audio code stream, thus reducing the amount of audio data. The common audio encoding methods are as follows:

The name of the Launch institutions Launch time Current application field
AAC MPEG 1997 All fields (New)
MP3 MPEG 1993 Fields (old)
WMV Microsoft Inc. 1999 Microsoft platform
AC-3 Dolby Inc. 1992 The movie
1 MP3

MP3, mPEG-1 or MPEG-2 Audio Layer III, is a once-popular digital Audio encoding and lossy compression format designed to dramatically reduce the amount of Audio data. It was invented and standardized in 1991 by a group of engineers at the Research organization Fraunhofer-Gesellschaft in Erlangen, Germany. The popularity of MP3 has caused great impact and influence on the music industry.

2 AAC

Advanced Audio Coding (AAC) is a mPEG-2-based Audio Coding technology developed by Fraunhofer IIS, Dolby LABS, AT&T, and Sony in 1997. In 2000, after the emergence of the MPEG-4 standard, AAC reintegrated its features, adding SBR technology and PS technology, in order to distinguish the traditional MPEG-2 AAC, also known as MPEG-4 AAC. AAC has a higher compression ratio than MP3, and the same size of audio files, AAC sound quality is higher.

3 WMA

WMA, Windows Media Audio, is a digital Audio compression format developed by Microsoft. It includes lossy and lossless compression formats.

H264 Coding Basics 06 – The Battle of Coding Standards

The popularity of a video standard is actually the result of the game of commercial interests.

If you want to predict the future, look back. Let’s take a look at h.265’s predecessor and the process of h.264 from standardization to popularization.

The development of h. 264

Twenty years ago, there were two mainstream organizations that formulated video codec technology in the world. One was THE International Telecommunication Union (ITU-T), which formulated h.261, H.263, H.263+ and H.264 standards. The other is the “International Organization for Standardization (ISO)” it develops standards mPEG-1, MPEG-2, MPEG-4 and so on.

1990 The world’s first video compression standard is H.261 (ITU),

1993MEPG-1(ISO)

1995 H.263 (ITU),

1996 MPEG-2(ISO)

It was initiated by VCEG in 1998. H.264 was originally known as H.26L. (ITU)

The first draft h.26L was accepted in October 1999. (ITU)

In December 2001, the evaluation found that H.26L coding technology basically met the requirements of MPEG standards, so members of MPEG and VCEG formed a new group, called Joint Video Team(JVT), to drive and manage the final standardization development of H.26L.

H.264 was released in May 2003.

In 2005, versions of H.264’s more advanced application standards MVC (Multiple View Coding) and SVC (Scalable Video Coding) were developed.

By 2017, the H265 aka HEVC was released.

One of the reasons for the success of H.264/MPEG-4 AVC is the result of cooperation or compromise between the two standardization organizations.

The popularity of a video standard depends on its rivals

H.264 and his friends

1 VP8

2 AV1

3 AVS (Audio Video Coding Standard) by China.

4 VP9

2.1 WMV

WMV(Windows Media Video) is a streaming Media Format launched by Microsoft. It is an extension of the Advanced Stream Format (ASF) upgraded from the same door. With the same video quality, WMV files can be downloaded and played at the same time, so it is very suitable for playing and transferring over the Internet.

WMV(Windows Media Video) is not just based on Microsoft’s own technology. Starting with version 7 (WMV1), Microsoft began to use its own non-standard MPEG-4Part2. However, since WMV version 9 is a separate standard for SMPTE (421M, also known as VC-1), WMV has evolved as a proprietary codec technology, unlike MPEG-4. There are 16 vC-1 patent sharing companies (as of April 2006), and Microsoft is one of the mPEG-4AVC /H.264 patent sharing companies.

Microsoft’s WMV is still very influential. However, due to Microsoft’s own limitations, the application of WMV has not been successful.

1) WMV is a product of Microsoft, which must depend on Windows. Windows means that the decoding part should also have PC, at least the motherboard of PC. This increases the cost of set-top boxes, thus affecting the popularity of video broadcast on demand.

2) THE video transmission delay of WMV technology is large.

Wmv-hd, based on the WMV9 standard, is a version of the video compression technology series developed by Microsoft. Generally, HDTV files with the suffix of. WMV are compressed using WMV-HD. In terms of compression ratio, VC-1 has a higher compression ratio of H.264, and the entropy coding of VC-1 adopts adaptive variable length coding.

2.2 VP8

VP8 comes from On2’s WebM, VPX(VP6,VP7,VP8,VP9), which is designed for Web video.

In February 2010, Google completed its acquisition of On2 Technologies for $124.6 million. At the Google LEI /O developer conference held three months after the acquisition, Google made its VP8 video coding technology available to all developers for free.

Google currently uses VPX in its Android, Chrome browser, YouTube and other services.

WebRTC (Web Real Time Communication). In May 2010, Google acquired VoIP software developer Global IP Solutions for about $68.2 million, giving it access to the company’s WebRTC technology.

WebRTC integrates VP8 and VP9.

2.3 AV1

AOMedia Video 1 (AV1) is an open, patent-free Video encoding format designed for delivering Video over the Internet. It is currently being developed by The Open Media Alliance (AOMedia), a consortium of semiconductor companies, video-on-demand providers and web browser developers founded in 2015. It is a major competitor to NetVC, the Internet Engineering Task Force’s (IETF) video standardization working group. The goal is to replace predecessor VP9 and compete with MPEG’s HEVC/H.265. AV1 can be used with HTML5 web video and future versions of WebRTC’s WebM containerized audio format, Opus.

Alliance for Open Media (AOMedia)

The main goal of the alliance is to create and deliver the next generation of technology open video compression formats and codecs that are optimized for streaming media over the Internet, including commercial and non-commercial content, including user-generated content. The alliance is currently developing a new video format called AV1. Alliance members from the chip industry (AMD, ARM, Intel, Nvidia) will ensure hardware friendly designs. AOMedia plans to complete the first version of its format (AV1) by the end of 2017. Work on the Bitstream specification will continue in 2018. AV1 is designed specifically for real-time applications and is more problem solving than the typical usage scenarios of the currently generated video format (H.264).

The consortium is designed to provide the next generation of video formats, including the following features.

0.l Interoperable and open; (interoperable and open)

0.l Optimized for the Internet; (Optimized for the Internet)

0.l Scalable to any modern device at any bandwidth; (Scalable to any modern device with any bandwidth)

0.l Designed with a low computational footprint and optimized for hardware; (Low computation, optimized for hardware)

0.l Capable of consistent, highest-quality, real-time video delivery; (Can achieve continuous high quality real-time video transmission)

0. L Flexible for both commercial and non-commercial content, including user-generated content.

Union including

Founding members: Amazon, Apple, ARM, Cisco, Facebook, Google, IBM, Intel, Microsoft, Mozilla, Netflix Vidia

General Members: Argon Design, Ateme, BBC Research & Development, Bitmovin,Broadcom, Chips&Media, Hulu, Ittiam, NGCodec, Polycom, Realtek, Sigma Designs, Socionext, VeriSilicon, VideoLAN, Vidyo, Xilinx

Google, Microsoft, Cisco and Mozilla have VPx, WMV/MPEG/VC-1, Thor, Daala and other coding algorithms respectively, which provide direct technical support for AV1.

Google, Firefox, Microsoft and Apple control 95% of the browser market, so AV1 has to be able to play directly in the browser.

AMD, Intel, NVIDIA, ARM, XILINX: PC and mobile hardware chip manufacturers gather to make AV1 hardware acceleration possible, from X86 to GPU to ARM to FPGA.

Video content providers include Google’s YouTube, Netflix, Amazon’s Prime and Ateme, which also provide OTT boxes, as well as the world’s largest IPTV operator, as well as Vidy and Polycom, members of audio and video applications.

HEVC/H265 is a bit of a outrage this time because of the royalties, the formation of AOMedia is the signal, everyone originally developed their own codec against HEVC/H265? Then join forces.

Of course, the organizations that drive HEVC will not sit idly by in this process. They’ve already started threatening patent lawsuits against AV1, and the Open Media Alliance AOMedia will spare no effort to make sure it doesn’t happen. They are conducting an extensive legal code review of AV1 to ensure it does not infringe any patents held by MPEG LA, HEVC Advance and Velos Media, as well as Technicolor. This form of code review was very successful with VP8 and VP9, both of which survived all legal challenges. MPEG LA’s actions against VP8 and VP9 are considered to have perhaps no legal basis and to be purely anti-competitive. The DOJ is investigating MPEG LA’s actions until they agree to drop the lawsuit and allow Google to allow licensing of MPEG LA’s patent pool to any user of VP8 or VP9. While we’re likely to see similar attempts to block AV1, Google has vastly expanded the patent pool and the number of companies supporting CODEC has dramatically increased (thanks to the Open Media Alliance), and both companies should be making sure they get processed for a long time.

2.6 AVS

AVS is the second generation of source coding standard with independent intellectual property rights in China. It is short for “Advanced Audio and Video Coding of Information Technology” series standards, including four major technical standards, including system, video, audio and digital rights management, and supporting standards such as compliance testing.

AVS is not a simple marketization behavior since its birth, which determines that administrative power is the main driving force of AVS. In June 2002, AVS Working Group was established with the approval of the Department of Science and Technology of the Ministry of Information Industry, affiliated to the Institute of Computing Technology, Chinese Academy of Sciences. The task of the Working Group is to: To meet the needs of China’s information industry, we cooperate with domestic enterprises and scientific research institutions to formulate (revise) common technical standards of digital audio and video compression, decompression, processing and representation, so as to provide efficient and economic codec technology for digital audio and video equipment and systems. It serves high resolution digital broadcasting, high density laser digital storage media, wireless broadband multimedia communication, Internet broadband streaming media and other important information industry applications.

In February 2006, AVS video coding became a national standard, the approved standard number is GB/T20090, and has been implemented since March 1, 2006. Working Group on Technical Standards for Digital Audio and Video Codec

On September 16, 2019, Huawei Hisi released Hi3796CV300, the world’s first 8K/ 120FPS decoding chip based on AVS3.0 video standard, with CPU processing capacity reaching four-core 1.6ghz and 28nm process, leading radio and television into the full 8K era.

Three royalties

Unlike H.264, h.265 /HEVC has a high royalty rate, which is still high despite the current price cut.

At least four organizations or companies currently collect licensing fees for HEVC: the established patent pool MPEG LA, the newer ones HEVC Advance and Velos Media, and Technicolor. Velos Media and Technicolor haven’t announced their pricing policies yet.

MPEG LA is the world’s leading one-stop licensing provider for standards and other technology platforms, having pioneered the modern patent pool since the 1990s and licensing the most widely used standard in the last generation, H.264. So the licensing policy for HEVC is generous, but also expensive.

L 0-100,000 sets/year, free of patent license fee

L $2 patent license fee will be charged for each part over 100,000 sets

L cap costs $25 million

L includes right of manufacture, right of use and right of sale

L The patent license fee shall be calculated from May 1, 2013

L No longer charges a separate patent license fee for HEVC content

L Semiconductor chip suppliers or other product suppliers that include HEVC encoders/decoders may pay patent licensing fees on behalf of their licensed customers

Four a video standard can be popular because it should solve a wide range of scenes

Application scenarios or design objectives for H.264.

1) video radio Broadcast, cable, satellites, and DSL.

2) Video Storage:DVD, HD DVD,and Blu-ray.

3) Video Conferencing:over different networks.

4) Multimedia Streaming: Live and on-demand.

5) Short video service Multimedia Messaging Services.

The so-called popularization is just a sub-scenario of H.264/5 in video application, the storage and playback of video content.

So let’s get back to the question.

What is limiting the development of H265 right now?

1) Business interests

For example, copyright patent fees, etc., the major factories stand up and divide accounts (dirty), etc. In fact, open source encoders also have commercial interests behind.

2) Opponents of HEVC/H.265

Encoder VP9, etc. In 2017 and up, HEVC/H.265’s biggest rival is actually H.264.

3) Matching of HEVC/H.265

H.265 is aimed at large-size video 2K, 4K, 8K, multi-channel video, ultra-high precision video, 3D video, etc. If the corresponding acquisition, playback and computing equipment are not ready, H.265 is useless.

According to the analysis of CISCO, it is predicted that by 2021, data and video will still be dominated by HD from 2016 to 2021, and ultra-HD accounts for about 30%. H.264 is for HD and H.265 is for ULTRA HD.

When will HEVC/H.265 be available for general use?

H.265 based on h.264 development, projected 2022 to 2025

Outlook (Next generation video coding format)

2.4H265 (one of the next generation video coding formats)

2.4.1 Better compression

HEVC offers significant improvements in compression over the H.264 codec. In fact, HEVC compresses video twice as efficiently as H.264. With HEVC, a video of the same visual quality takes up half the space. Alternatively, videos with the same file size and bit rate can present better quality.

Part of this improvement can be attributed to the increased macroblock size. A macroblock is defined as an image region used for compression computation. Larger macroblocks can effectively compress high resolution video. H.264 only allows 16-by-16-pixel macroblocks, which are too small to play video above 1080p efficiently. HEVC offers 64 x 64 pixel macroblocks (now called code tree units or CTU) for higher coding efficiency at higher resolutions.

2.4.2 Improved interframe motion prediction

A major factor in video compression is the prediction of motion (or lack thereof) between frames. While the pixel remains stationary (solid background image), a smart video codec can save space by referencing it rather than reappearing it. With improved motion prediction, HEVC can provide smaller file sizes and higher compression quality.

2.4.3 Improved inter-frame prediction

Video compression actually benefits from analyzing the “movement” within a single frame, making it more efficient to compress a single frame of video. This can be done by using mathematical functions rather than actual pixel values to describe the pixel layout. This feature takes up less space than pixel data, reducing the file size. However, the codec must support sufficiently advanced mathematical functions for the technology to really work. HEVC’s interframe prediction feature is more detailed than H.264’s, with the former supporting motion prediction in 33 directions compared to just nine.

2.4.4 Parallel processing

HEVC uses unit and fragment layers that can be decoded independently of the rest of the frame. This means that the decoding process can be split across multiple parallel processing threads and take advantage of more efficient decoding opportunities available on existing standard multicore processors. As video becomes more and more high-resolution, this increased efficiency requires decoding video at a viewable speed on low-end hardware.

2.5 VP9(one of the next generation video coding formats)

VP9 is an Open and copyright free Video coding standard developed by Google. It was named as Next Gen Open Video in the early stage of development. VP9 is also regarded as the Next generation Video coding standard of VP8. VP9 is widely used for audio and video calls. Mainstream browsers such as Firefox and Chrome and video players such as FFmpeg support VP9 encoding. One advantage is that video encoded with VP9 can save half the bandwidth of video encoded with VP8 or H.264 in the past. Applying to be a partner in VP9 code is very simple, there are not many complicated licensing issues, and the coding technology is available free of charge. But any hardware or software vendor that used H.264 in the past paid a licensing fee to MPEG LA, which was then divided up by the patent owner.

2.5.1 PERFORMANCE comparison of H.265 and VP9
Coding quality

△ Code quality test

The figure above shows the comparative test of H.265 and VP9 coding quality. The smaller the value is, the better the coding quality is. From the comparison, we found that there was no significant difference between H.265 and VP9, the overall average score was only 0.001 different, and there was almost no difference in practical application.

2.5.2 Coding time

△ Coding time test

In the comparison of encoding time, VP9 beats H.265 completely, and the encoding time of VP9 is much shorter than that of H.265 no matter it is 4K video or video with 1920 or 1280 resolution.

2.5.3 CPU consumption

△ CPU consumption test

Above, h.265 and VP9 play a CPU consumption test on a Dell Precision 390 workstation. VP9 decodes CPU more efficiently on Firefox than Chrome. H.265 is far more efficient on DivX than VLC Player. After comparing the data of the two coding standards, H.265 decoding efficiency was slightly higher than VP9.

The future belongs to H.265 or VP9

H.265 inherits the video coding standard system of H.264, and is more widely used in commercial applications, such as security, military and political, enterprise and other scenarios. However, due to too many patent holders, its commercial cost is too high, so it faces great resistance in promotion.

VP9 was developed by Google and is free to use. In actual promotion, Microsoft, Apple and other companies do not want to see VP9 dominant, and other Internet manufacturers do not want the mainstream video encoding format to be monopolized, so at present, it is mainly supported in Google’s own products, and there are not many other large manufacturers using VP9.

At present, H.265 is widely used in enterprises and security, while VP9 is widely used in Internet application scenarios due to its simple and practical solution and free development. It’s unclear which of the two will be the dominant video coding standard in the future. Of course, with the diversification of video application scenarios, it is possible that the two will go together and complement each other in the future.

More convenient multimedia cloud processing

Both H.265 and VP9 have their advantages, and users can choose the format to use according to the actual application scenario.

At present, youpaiyun media processing has supported h.265 and VP9 video coding standards. When performing audio and video processing, you only need to set the video encoding format to libx265 or libvpx-VP9 to encode the video to H.265 or VP9.