In Android audio and video development system, FFmpeg library is the most critical open source library. However, the tutorials on how to use FFmpeg for Android developers on the Internet are fragmented and outdated, so I wanted to discuss them in this series. This series of articles is based on FFmpeg 4.X version to analysis, due to the limitations of the author’s personal knowledge, there are any mistakes, but also hope that readers timely pointed out, hope not hesitate to comment.

Before learning FFmpeg, it is desirable that the reader should have some relevant knowledge: C/C++ language, JNI syntax, Linux programming techniques. For reference materials, the following books are recommended: C++ Primer Plus, JNI programming guide, Linux programming techniques in detail.

Let’s start with FFmpeg:

FFmpeg is an open source computer program that can record, convert, and stream digital audio and video. It provides a complete solution for recording, converting and streaming audio and video.

FFmpeg mainly contains the following modules:

  1. libavutil

Libavutil is a utility library for assisting multimedia programming. This library contains secure portable string functions, random number generators, data structures, additional mathematical functions, encryption, and multimedia related features such as enumerations of pixel and sample formats. Libavcodec and libavFormat do not depend on this library.

  1. libswscale

The libswscale library is used to perform highly optimized image scaling, color space, and pixel format conversion operations. Specifically, the library performs the following transformations:

  • Zoom: An operation that changes the size of a video. There are several rescaling options and algorithms available. This is usually a lossy operation.
  • Pixel Format conversion: Is the operation of converting an image format and gamut to an image, such as from flat YUV420P to RGB24 compression. It also handles packaging transformations, that is, from a packaged layout (all pixels belonging to different planes interlaced in the same buffer) to a flat layout (all samples belonging to the same plane stored in a dedicated buffer or “plane”).

If the source and target gamut are different, this is usually a lossy process.

  1. libswresample

The libswresample library is used to perform highly optimized audio resampling, rematrix, and sampling format conversion operations. Specifically, the library performs the following transformations:

  • Resampling: is the operation of changing the audio rate, for example from a high sampling rate of 44100Hz to 8000Hz. Audio conversion from high sampling rate to low sampling rate is a lossy process. Several resampling options and algorithms are available.

  • Format conversion: The operation of converting a sample type, such as from a 16-bit signed sample to an unsigned 8-bit or floating-point sample. It also handles packaging transformations when passing from a packaged layout (all samples belonging to different channels interlaced in the same buffer) to a flat layout (all samples belonging to the same channel stored in a dedicated buffer or “flat”).

  • Remodulation: The operation of changing the Channel Layout, e.g. from stereo to mono. When the input channel cannot be mapped to the output stream, the process is lossy because it involves different gain factors and mixing.

  1. libavcodec

The libavCodec library provides a general-purpose encoding/decoding framework containing multiple decoders and encoders for audio, video, and subtitle streams, as well as multiple bit-flow filters.

  1. libavformat

The libavFormat library provides a common framework for multiplexing and demultiplexing audio, video, and subtitle streams. It contains multiple multiplexers and demultiplexers for multimedia container formats.

It also supports multiple input and output protocols to access media resources.

  1. libavdevice

The libavDevice library provides a generic framework for fetching and rendering from many common multimedia input/output devices, and supports a variety of input and output devices, including Video4Linux2, VfW, DShow, and ALSA.

  1. libavfilter

The libavfilter library is a general audio/video filter framework, which contains multiple filters, sources and sinks.

In order to make things easier to understand, here are some of the more unfamiliar concepts involved in the above content:

multiplexing

In a data communication system or computer network system, the bandwidth or capacity of transmission media is often greater than the need to transmit a single signal. In order to effectively use the communication line, it is expected that a channel can transmit multiple signals simultaneously. This is called Multiplexing technology. Using multiplexing technology, multiple signals can be combined to transmit in a physical channel, which can greatly save the cost of cable installation and maintenance during long-distance transmission.

Sources and sinks

Libavfilter can handle audio and video with multiple filters, each with input and output ports. The output of one filter can be connected to the input of another filter, where the first filter is called source and it has only output port, and the last filter is called sink and it has only input port.

FFmpeg for the initial understanding of our first temporarily stop here, below we will take FFmpeg in the Android platform development of the first step ————FFmpeg cross-compilation.

  1. Compile environment

Operating system: Ubuntu 20.04.1

NDK : NDK_r23b

The above is for reference only, other versions of the OPERATING system and NDK can be used as long as they are not too old.

  1. Download FFmpeg source code

Git Clone FFmpeg: github.com/FFmpeg/FFmp

  1. Write cross-compiled scripts

Clone from FFmpeg source directory open command terminal, type the following command line, create a compilation script file:

touch build_script.sh
sudo chmod 751 build_script.sh
Copy the code

For those unfamiliar with the Linux command line, explain the above command line briefly.

The first line of the touch build_script.sh command is used to create a file named build_script.sh in the current directory. Of course, the touch command is not used to create a file, it is used to modify the properties of the file, and only create a new file if the file does not exist.

# sudo chmod 741 build_script.sh; # build_script.sh; # build_script.sh; # build_script.sh; # build_script.sh;

Using a text editor, open the build_script.sh file and paste in the following code:

TOOLS_DIR=/home/jie/sda3/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin
./configure --enable-shared \
--disable-static \
--disable-x86asm \
--enable-cross-compile \
--cc=$TOOLS_DIR/aarch64-linux-android30-clang \
--cxx=$TOOLS_DIR/aarch64-linux-android30-clang++ \
--strip=$TOOLS_DIR/llvm-strip \
--extra-cflags="-I$(pwd)/include" \
--extra-ldflags="-L$(pwd)/libs" \
--arch=arm64 \
--prefix=$(pwd)/install/aarch64 \
--target-os=android \
Copy the code

This code is a script for cross-compiling the SO library with CPU architecture aARCH64. If you need to compile the so library with other CPU architecture platforms, you only need to replace the content in — CC and — CXX.

I’ll take a closer look at what each line of the code above means and what to be aware of.

TOOLS_DIR=/home/jie/sda3/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin

Define the NDK tool directory. Replace the words in bold according to the actual installation path of the NDK.

./configure

Run an executable called configure in the current directory.

--enable-shared

Generate dynamic library, in this need to pay attention to the end of the line before the backslash space, this space must not be missed.

--disable-x86asm

The yASM assembler is not installed on the system, and an error will be reported during compilation, so this line is added to disable assembly.

--enable-cross-compile

Enable cross compilation

--cc=$TOOLS_DIR/aarch64-linux-android30-clang

Specify the path of the C compiler used during compilation. The specified compiler must correspond to the target CPU. If you need to compile ARMV7A or x86 cpus, replace the compiler name by yourself.

--cxx=$TOOLS_DIR/aarch64-linux-android30-clang++

Specifies the C++ compiler path to use at compile time, again with the compiler corresponding to the target CPU.

--strip=$TOOLS_DIR/llvm-strip

Specifies the strip tool used at compile time to optimize dynamic libraries.

--extra-cflags="-I$(pwd)/include"

$(PWD) specifies the directory where the third-party header files will be stored for compilation. This requires creating a new folder called include under the current directory. This line of code is not needed when there are no third-party libraries that need to be integrated and packaged into FFMPEG.

--extra-ldflags="-L$(pwd)/libs"

$(PWD) specifies the directory where the third-party lib files will be stored for compilation. This requires a new directory named libs. This line of code is not needed when there are no third-party libraries that need to be integrated and packaged into FFMPEG.

--arch=arm64

Specifies the architecture of the target CPU for cross-compilation, which the reader can modify as needed.

--prefix=$(pwd)/install/aarch64

When you execute the make install command, compile the installation path of the generated file.

--target-os=android

Specify the target OS platform as Android.

  1. Changing the Version number

The default FFmpeg version naming convention is not suitable for Android. In this step, we also need to change the FFmpeg version naming convention.

Let’s open the configure file and search for the major keyword to find the following code snippet:

Replace the contents of the above code with something like this:

  1. Start cross-compiling

Enter the following command line in the terminal to start compiling.

./build_install.sh
make -j4
make install
Copy the code

If an error occurs while executing./build_install.sh, open the ffbuild/config.log file to check for errors in writing scripts.

If all goes well, we’ll eventually find the generated headers and so files in the /install/aarch64 directory.

As for how to integrate FFmpeg into the Android project, it may be covered in a future article, but it is not strictly FFmpeg knowledge, so I will not make a special introduction here, please have a need for readers to explore.

Next up:

Introduction to FFmpeg command line