### Audio and video basics

#### Video playback mechanism

Let’s start with the principle of a simple video player. The following is a simple video playing process (excluding video encryption and other processes) :

This is a basic flow chart for video playback, from which we can see the main steps of video processing in general. We will introduce some basic concepts mentioned here in detail later.

###### Note: programming with FFmpeg is almost based on this flowchart. For example, when we’re programming, we’ll get the decoder, decode it, read the data, draw it on the screen and maybe convert YUV data to RGB and so on.

Our common video packaging formats are: FLV (separate audio and video), MP4, AVI and so on. We’ll talk more about that later.

#### Why does video need to be packaged?

Because the images captured by the camera and the audio captured by the microphone are compressed, the video files would otherwise be large.

In other words:

  1. Video recording, recording, in essence is a compressed collection of images or sound process. This process is the process of video coding compression.
  2. Playing video and audio files is essentially the process of decompression, which is also called decoding.

#### Video encapsulation format

The encapsulation format is used to store video streams and audio streams in a file in a certain format.

Encapsulation Format analysis tool: Elecard Format Analyzer

Why store audio and video separately? Because of the variety of audio and video coding formats, and coding is bound to cause confusion.

Common video encapsulation formats are:

Take two formats as examples to introduce the principles:

  1. Mpeg2-ts format is composed of a TS-packet with a fixed data size, so it can support fast forward.
  2. The FLY format consists of FLV headers and tags of varying sizes. Because FLV can be played directly in Flash (browser), it is often used in the live video neighborhood. When we do the RTMP stream, we need to send the header from the beginning. Because the data unit size is not fixed, native video players do not support fast forwarding of FLV video (some players are processed to fast forward).

#### Common video codec formats

There are many compression algorithms for video, so there will be many kinds of encoding formats. Here are some common encoding formats:

##### Video codec format:

  1. Common video encoding formats include H.264, MPEG2, VP8, etc. (The WebRTC video call acquired by Google uses VP8).
  2. Pixel data obtained from video decoding YUV and RGB. YUV format, Y represents brightness, UV represents chroma, human eyes are more sensitive to brightness, the ratio of the two is 4:1, related to biological theory.

Principle analysis:

Take H264 as an example. H264 consists of nALUs of varying sizes. NALU is essentially a data structure. H264 has a lot of sub-compression algorithm, the principle is more complex, including entropy coding, loop filtering, frame detection, frame detection and other knowledge. The coding principle of H264 is complex, so the compression efficiency of H264 is hundreds to thousands of times.

###### we need to learn FFmpeg, because this library encapsulates H264 and other formats of processing.

Video decoding (camera capture) results in video pixel data, which holds the pixel value of each pixel point on the screen. Common pixel data formats include RGB24, RGB32, YUV420P, YUV422P, YUV444P, etc. Pixel data in YUV format is generally used in compression coding, and the most common format is YUV420P.

YUV video format is uncompressed, big. Early in the television used more, such as the old black and white, color TV. In the early days of color TV, video in black and white was essentially Y (brightness), because video in black and white only had Y data.

There are also many kinds of RGB, such as RGB24, different RGB coding color richness is different.

##### Audio and video codec format:

  1. Common audio encoding formats are: AAC, MP3.
  2. Audio decoding results in audio sampling data before the speaker can play. The common format is PCM, essentially sampling values one by one. Data on vibration per unit time, including amplitude and frequency. The usual sampling rate is 44100, the highest sampling rate that the human ear can feel.

###### when doing live video: audio is often encoded by AAC and processed by FAAC library; The video is encoded in H264.

Audio sampling data PCM: saves the value of each sampling point in audio, audio sampling data volume is very large, generally need to be compressed, we usually say “lossless” is essentially no loss of compression meaning.

##### Related playback (editing) tools

  1. YUV: YUV Player
  2. PCM: Adobe Audition
  3. Check out the video message: MediaInfo
  4. Video encoded data: Elecard Format Analyzer
  5. Video coding analysis tool: Elecard Stream Eye

If you are interested, you can download it.

# # # FFmpeg is introduced

FFmpeg is an open source C/C++ audio and video processing library that is so good that many large companies use it. Almost all major video players use FFmpeg.

####FFmpeg eight function library basic introduction

As shown below:

### FFmpeg project configuration under Visual Studio

# # # # introduction

We usually write the code in VS and put it in Android, so it is necessary to build the VS development environment.

#### Obtain FFmpeg resources

First of all, we need to go to FFmpeg’s official website to get the source code, because the steps to get more trouble, solid next note down, we open ffmpeg.org/:

Click the big Download button on the official website to jump to the following interface:

Select the corresponding system and we will introduce Windows Builds first. Click on Windows Builds below to jump to the following screen:

We recommend using the old version of FFmpeg library, because if you use the new version, it is difficult to go to Baidu except for problems. I have my own 64-bit computer, so CLICK on All 64-bit Downloads. We will then jump to the following warehouse page:

Among them, FFmpeg version we need to download is 2.8 series, we recommend using 2.8 or below version. Dev is a development version library, shared is a dynamic link library, static is a compiled exe (Windows version) executable. We need to download all three.

If you don’t mind, I’ll give you the download address directly below:

https://ffmpeg.zeranoe.com/builds/win64/dev/2015/ffmpeg-20151105-git-c878082-win64-dev.7z

https://ffmpeg.zeranoe.com/builds/win64/shared/2015/ffmpeg-20151105-git-c878082-win64-shared.7z

https://ffmpeg.zeranoe.com/builds/win64/static/2015/ffmpeg-20151105-git-c878082-win64-static.7z

The following and decompression effect is as follows:

Note: dev, static, shared represent these folders.

#### Play with the static executable at the command line

We open the static folder, which has a bin directory with three exe files. Here’s what we’re going to play:

To simplify things, add the bin directory to the environment variable path.

Then we prepare a test video. For example, I prepared a test.flv video file.

Open the command line and type:

ffmpeg -i test.flv test.avi
Copy the code

This then completes a simple video format conversion. I believe that careful you will also find that FFmpeg’s official website has such a picture:

This is the simplest example.

From the command line, enter the following statement:

ffmpeg -ss 0 -t 11 -i test.flv -s 1366x768 -b:v 1500k test.gif
Copy the code

Convert test. FLV to test. GIF. Specify the time range, resolution, and bit rate for conversion.

###### Bit rate (bit rate) : the size per frame and audio per unit time. Also called bit per second. The number of bits per unit time played of continuous media such as compressed audio or video. The higher the bit rate, the clearer the audio and video.

###### it makes sense to transfer the video to GIF. For example, when we do not open the small video in wechat, it actually plays a GIF file (probably using the libgif NDK library). When users open the video, they will download the real video file from the server. This greatly reduces the strain on the server.

Finally, let’s look at an example of a player. Typing the following command will open a player, so if you want to study an Android player, you need to study the ffPlay code:

ffplay test.flv
Copy the code

####FFmpeg VS project configuration

Create an empty project, then copy the dev directories include and lib to the source directory in the root path of the project (the default is the same as the project name).

Then configure the additional library directory in the project properties:

Then configure the additional library directory:

Then configure which additional libraries (additional dependencies) are available, as shown below:

avcodec.lib
avdevice.lib
avfilter.lib
avformat.lib
avutil.lib
postproc.lib
swresample.lib
swscale.lib
Copy the code

Finally, we create a test CPP file:

#include <stdlib.h> #include <stdio.h> #include <iostream> using namespace std; Extern "C"{#include "libavcodec/avcodec.h"} void main(){// print FFmpeg configuration information; Check that cout << avcodec_configuration() << endl; system("pause"); }Copy the code

However, since the FFmpeg library we are using is 64-bit, we need to change our platform to 64-bit:

Then the compiler passes, and the output is as follows:

##### digress — basic explanation of the extern keyword

Extern can be placed in front of a variable or function to indicate that the definition of the variable or function is in another file, prompting the compiler to look for its definition in another module when it encounters the variable or function. Extern can also be used for link designation. That is, extern has two functions:

  1. Extern “C” void fun(int a, int b); extern “C” void fun(int a, int b) The compiler is told to translate the name of fun using C rules instead of C++. C++ rules will change the name of fun beyond recognition. It could be fun@aBc_int_int#%$or something else. This depends on the compiler’s “attitude” (different compilers use different methods), why do this, because C++ supports function overloading, I will not discuss this problem too much, if you are interested in the Internet search, I believe you can get satisfactory explanation!
  2. Second, when extern is not used with “C” to modify a variable or function, as in a header file: extern int g_Int; Its function is to declare the scope of the function or global variable keyword, its declared function and variable can be used in this module or other modules, remember that it is a declaration not definition! That is to say, if B module (translation units) reference module (translation units) defined in A global variable or function, it can be as long as the header file contains A module, in the compile phase, although module B can’t find this function or variable, but it won’t be an error, it will be in the connection from module A generated object code found in this function.

If you feel that my words are helpful to you, welcome to pay attention to my public number:

My group welcomes everyone to come in and discuss all kinds of technical and non-technical topics. If you are interested, please add my wechat huannan88 and I will take you into our group.