1, the preface
Take the wechat video as an example. We recorded a 540p MP4 file. For Android, it basically follows the following process:
This article will mainly analyze two common problems in the coding of video stream:
- Choice of video encoder: hard and soft encoder;
- How to quickly preprocess YUV frame output by camera: mirror, zoom and rotate.
2. Selection of video encoder
Generally speaking, there are two options:
- MediaCodec;
- FFMpeg + x264 / openh264.
3, MediaCodec
3.1 Basic Introduction
The entire API design model consists of two queues at both the input and output ends:
Taking synchronous calls as an example, the general call looks something like this (from the official example) :
EncodeDecodeTest.java
3.2 Color format problem
3.3 Encoder support features are quite limited
3.416 bit alignment requirements
4, FFMpeg + x264 / openh264
Due to space limitation, h264 parameter configuration will not be elaborated here. If you are interested, you can see the two articles on the tuning of X264 encoding parameters:
www.nmm-hd.org/d/index.php…
www.cnblogs.com/wainiwann/p…
Github.com/cisco/openh…
However, compared with X264, Openh264 has poor support for advanced features of H264:
- The value of Profile can only be baseline, Level 5.2.
- Multi-thread encoding only supports slice based, not frame based multi-thread encoding.
- In terms of coding efficiency, Openh264 isn’t any faster than X264, but the best part is that it’s free to use.
5. Contrast between hard and soft knitting
6. Pretreatment of YUV frames
6.1 zoom
The most common method is to use ffMPEG sws_scale function for direct scaling, the effect/performance is better generally choose SWS_FAST_BILINEAR algorithm:
After comparing several commonly used calculations in the industry, we finally consider implementing an algorithm that uses this fast scaling:
We choose a local mean algorithm, which calculates the four pixels of the final image by four adjacent points in the two rows before and after. For each pixel of the source image, we can use Neon to directly achieve it, taking scaling Y component as an example:
6.2 rotate
Developer.apple.com/library/con…
Using FFMPEG, we can easily assign this rotation Angle to the synthesized MP4 files:
6.3 mirror
This algorithm is fairly simple to implement with Neon:
7. Reference materials
1) column by leixiaohua1020:
Blog.csdn.net/leixiaohua1…
2) Android MediaCodec Stuff:
bigflake.com/mediacodec/
3) Coding for NEON:
Community.arm.com/processors/…
4) libyuv:
Chromium.googlesource.com/libyuv/liby…
www.qcloud.com/community/a…
Appendix 1: Summary of basic audio and video data
Instant messaging audio and video development (I) : a theoretical overview of video codec
Instant messaging audio and video development (ii) : digital video introduction to video codec
Instant messaging audio and video development (iii) : coding basis of video codec
Instant communication audio and video development (iv) : introduction to video codec predictive technology
Instant messaging audio and video development (5) : Understanding the mainstream video coding technology H.264
Instant messaging audio and Video development (vi) : How to start learning audio codec technology
Instant messaging audio and Video development (7) : Introduction to audio fundamentals and coding principles
Instant messaging audio and Video development (8) : common real-time voice communication coding standards
Instant messaging audio and Video development (ix) : Overview of echo and echo cancellation for real-time voice communications
Instant messaging audio and Video development (10) : real-time voice communication echo cancellation technology details
Instant communication audio and video development (11) : Real-time voice communication packet loss compensation technology details
Instant messaging audio and video development (12) : Multi-party real-time audio and video chat architecture discussion
Instant communication audio and Video development (13) : features and advantages of real-time video coding H.264
Instant communication audio and Video development (xiv) : Introduction of real-time audio and video data transmission protocols
Instant messaging audio and video development (15) : Talk about P2P and real-time audio and video applications
Instant Messaging audio and Video development (16) : Some suggestions for mobile real-time audio and video development
Instant communication audio and video development (17) : video coding H.264, VP8 past life
Appendix 2: Summary of articles related to wechat and QQ
[1] Technical articles on QQ and wechat:
Wechat team share: wechat Android version of the small video coding filled those pits
Wechat mobile terminal local data full text retrieval optimization road
Enterprise wechat client in the organization structure data synchronization update program optimization combat
Wechat team revealed: wechat interface stuck super bug “15…” The ins and outs of
QQ 18 years: Decrypt 800 million months of QQ background service interface isolation technology
How does the super IM wechat with a monthly life of 889 million carry out Android compatibility test
Discussion on “Light Application” in MOBILE IM by Taking MOBILE QQ as an Example
Get wechat open source mobile database component WCDB everything!
Technical interview with wechat Client Team leader: How to start client performance Monitoring and optimization
Design practice of time-sequence-based cold and hot classification architecture for massive data in wechat background
Wechat team original sharing: Android version of wechat bloated trapped and modular practice road
Wechat background team: sharing the optimization and upgrade practice of wechat background asynchronous message queue
Wechat team original share: wechat client SQLite database damage repair practice
Tencent original share (a) : how to greatly improve the mobile QQ picture transmission speed and success rate under the mobile network
(2) How to greatly reduce the Traffic consumption of APP in mobile Network (Part 2)
How to greatly reduce APP traffic Consumption in mobile Network (Part 1)
Wechat Mars: network layer encapsulation library being used inside wechat, which will be open source
As promised: Mars, a cross-platform component library for wechat’s mobile IM network layer, has been officially open source
Open source Libco library: the foundation of background framework supporting 800 million users of wechat
Wechat new generation communication security solution: MMTLS based on TLS1.3
Wechat team original sharing: Android version of wechat background to keep alive combat sharing (process to keep alive)
Wechat team original sharing: Android version of wechat background to live combat sharing (network to live)
The evolution of wechat for Android from 300KB to 30MB
Wechat’s technical evolution from 300KB to 30MB for Android
Speech by technical Director of wechat on Architecture: The Way of wechat — Avenue to Simplicity
Wechat Technical Director’s Talk on Architecture: The Way of wechat — Avenue to Simplicity (PPT presentation)
How to interpret “wechat Technical Director on Architecture: The Way of wechat — The Road to Simplicity”
Background system storage architecture behind massive wechat users (Video +PPT)
Wechat asynchronous transformation practice: background solution behind 800 million monthly live, single machine 10 million connections
Wechat Moments of mass technology PPT [Attachment download]
Technical Experiment and Analysis of wechat’s Influence on Network
A summary note of wechat background technology architecture
The way of architecture: 3 programmers have achieved 1 billion daily releases in wechat Moments
Fast Fission: Witness the evolution of wechat’s powerful background architecture from 0 to 1 (I)
Fast Fission: Witness the evolution of wechat’s powerful background architecture from 0 to 1 (II)
Android memory leak monitoring and optimization tips summary
A comprehensive summary of iOS version wechat upgrade iOS9 encountered a variety of “pit”
Wechat team original resource confusion tool: reduce your APK by 1M immediately
Android source obfuscation tool: AndResGuard
Android version of wechat installation package “weight loss” combat record
IOS version of wechat installation package “weight loss” combat record
Mobile IM practice: iOS version of wechat interface lag monitoring scheme
Technical difficulties behind wechat’s “red envelope photos”
Mobile IM practice: iOS version oF wechat small video function technical scheme record
Mobile IM Practice: How to greatly improve the Interactive performance of wechat on Android (I)
Mobile IM Practice: How to greatly improve the Interactive Performance of wechat on Android (II)
Mobile IM practice: To realize the intelligent heartbeat mechanism of wechat on Android
Mobile IM practice: Analysis of heartbeat strategy of WhatsApp, Line and wechat
Mobile IM Practices: A Study of Google’s Push Messaging Service (GCM) (from wechat)
Mobile IM practice: Discussion on multi-device font adaptation scheme of iOS version wechat
Walk through the APNS pit on iOS10 together
Tencent carrier pigeon technology sharing: ten billion real time message push experience
More of the same…
[2] Technical stories about QQ and wechat:
How much money did Tencent spend developing wechat? Is the technology really that difficult? Where is difficult?
Technology past: Tencent in the early stage of business – 16 years ago in the winter, who moved ma Huateng’s code
Technical past: the most complete QQ icon change process in history, tracing the evolution history of IM giants
Technology past: how do “QQ group” and “wechat red envelope” come from?
Development past: the depth of 2010 to 2015, wechat all the way behind the wind and rain
Development of the past: wechat millennium unchanged that flash screen picture origin
Development past: Record the story behind wechat 3.0 (9 months after the release of wechat 1.0)
A wechat intern: The wechat development team in my eyes
First revealed: QQ real-time video chat behind the mysterious organization
More of the same…