One, foreword

FFMPEG is a particularly powerful open source library dedicated to audio and video processing. You can either use its API for audio and video processing, or use its tools such as FFmPEG, FFPlay, and FFProbe to edit your audio and video files.

This article will give you a brief introduction to the basic directory structure of the FFMPEG library and its functions, and then explain in detail how we use the tools provided by FFMPEG to work with audio and video files in our daily work.

FFMPEG directory and function

Libavcodec: provides a series of encoder implementations.

Libavformat: implements the stream protocol, container format and its own IO access.

Libavutil: includes hash, decoder, and utility functions.

Libavfilter: Provides a variety of audio and video filters.

Libavdevice: Provides an interface to access capture and playback devices.

Libswresample: Implements mixing and resampling.

Libswscale: Enables color conversion and scaling.

3. Basic concepts of FFMPEG

Before we get into the FFMPEG command, let’s introduce some basic concepts of audio and video formats.

(1) Audio/video streaming

In the field of audio and video, we call all the way sound/video all the way stream. For example, when we used to watch Hong Kong movies on VCD when we were young, we could choose cantonese or Mandarin sound. In fact, there are two audio streams stored in the CD video file, and users can choose one to play.

(2) containers

We generally call MP4, FLV, MOV and other file formats containers. That is, in these common format files, you can store multiple audio and video files. An MP4, for example, can store one stream of video, multiple streams of audio, and multiple streams of subtitles.

(3) channel

A channel is an audio concept called a channel. In one audio stream, there can be mono, two – channel or stereo.

Four, FFMPEG function classification

FFMPEG commands can be divided into the following categories by purpose:

  • Commands for querying basic information
  • The recording
  • Decomposition/reuse
  • Processing raw data
  • A filter
  • Cut and merge
  • Diagram/view mutual rotation
  • Live related

With the exception of FFMPEG’s basic information query commands, all commands process audio and video as shown below.

The encoded packet is then passed to the decoder (unless a stream copy is selected for the data stream, see further description). Decoder generates uncompressed frames (raw video/PCM audio /…) And can be further processed by filtering (see the next section). After filtering, the frame is passed to the encoder, which outputs encoded packets. Finally, these are passed to the multiplexer, which writes the encoded packets to the output file.

By default, FFMPEG contains only one stream of each type (video, audio, subtitles) in the input file and adds it to each output file. It picks the “best” of each based on the following criteria: for video, it is the stream with the highest resolution, for audio, it is the stream with the most channels, and for subtitles, it is the first subtitle stream. In the case that several streams of the same type are equal, the stream with the lowest index is selected.

You can disable some default Settings by using the -vn / -an / -sn / -dn options. For full manual control, use the -map option, which disables the default Settings just described.

5. Basic information query commands

MPEG can use the following parameters for basic information queries. For example, if you want to find out which filters are supported by the current FFMPEG, you can use FFMPEG-filters. The parameters are described as follows:

Parameters that

-version Displays the version.

– Formats show available formats (including devices).

-demuxers Displays available demuxers.

-muxers Displays available muxers.

– Devices Displays available devices.

-codecs Displays all codecs known to libavCodec.

-decoders Displays available decoders.

-encoders displays all available encoders.

– BSFS Displays the available bitstream filter.

-protocols Displays available protocols.

-filters displays available libavfilter.

-pix_fmts Displays available pixel formats.

-sample_fmts Displays available sampling formats.

– Layouts Displays the channel name and standard channel layout.

-colors Displays the recognized color name.

Next, the command formats and parameters used by FFMPEG for audio and video processing

6. Basic command formats and parameters

Here is the basic FFMPEG command format:

ffmpeg [global_options] {[input_file_options] -i input_url} ...

                         {[output_file_options] output_url} ...
Copy the code

Ffmpeg uses the -I option to read any number of input “files” (regular files, pipes, network streams, crawl devices, etc.) and write any number of output “files”.

In principle, each input/output “file” can contain any number of different types of video streams (video/audio/subtitles/attachments/data). The number and/or type of streams is limited by the container format. Selecting which input to enter from which output will be done automatically or using the -map option.

To reference the input files in the options, you must use their indexes (starting from 0). For example. The first input file is 0, the second is 1, and so on. Similarly, streams within a file are referenced by their indexes. For example. 2:3 is the fourth stream in the third input file.

Here are some common commands for FFMPEG audio and video processing:

(1) Main parameters

-f FMT (input/output)

Enforces input or output file formats. The format is usually automatically detected for the input file and guessed from the file extension for the output file, so this option is not required in most cases.

-i URL (input)

Enter the url of the file

-y (global parameter)

Overwrite the output file without asking.

-n (global parameter)

Do not overwrite the output file, exit immediately if the specified output file already exists.

-c [: stream_specifier] codec (input/output, per stream)

Select an encoder (when used before the output file) or decoder (when used before the input file) for one or more streams. Codec is the name of the decoder/encoder or copy (output only) to indicate that the stream is not recoded. For example, ffmpeg -i input-map 0 -c:v libx264 -c:a copy OUTPUT

-codec [: stream_specifier] codec (input/output, per stream)

With the -c

-t duration (input/output)

When used as an input option (before -i), limits the duration of data read from the input file. When used as an output option (before the OUTPUT URL), output stops after the duration reaches the duration.

– SS position (input/output)

When used as an input option (before -i), find the location in the input file. Note that in most formats it is impossible to search precisely, so FFMPEG will look for the nearest search point before the location. When transcoding and -accurate_seek are enabled (the default), this additional segment between the search point and location will be decoded and discarded. It will be retained when streaming copy or when using -noaccurate_seek. When used as an output option (before the OUTPUT URL), decode but discard the input until the timestamp reaches position.

-frames [: stream_specifier] framecount (output, per-stream)

Stops writing to the stream after a frame count frame.

-filter [: stream_specifier] filtergraph (output, per-stream)

Create the filtergraph specified by the filtergraph and use it to filter the stream. A Filtergraph is a description of a Filtergraph applied to a stream, and must have a single input and a single output of the same type of stream. In filter graphics, inputs are associated with labels within labels, and outputs within labels are associated with labels. For more information about the Filtergraph syntax, see the FFMPEG-Filters manual.

(2) Video parameters

-vframes num (output)

Sets the number of video frames to output. For -frames: v, this is an outdated alias and you should use it.

-r [: stream_specifier] FPS (input/output, per stream)

Set frame rate (Hz value, fraction or abbreviation). As an input option, any timestamps stored in the file are ignored and new timestamps are generated based on the rate. This is different from the -framerate option (which was used the same way in older versions of FFmpeg). If in doubt, use -framerate instead of entering the -r option. As an output option, copy or discard the input frame to achieve a constant output FPS.

-s [: stream_specifier] size (input/output, per stream)

Set the window size. As an input option, this is a shortcut to the video_size specific option, recognized by some framers, whose frame size is not stored in the file. As an output option, this inserts the zoom video filter at the end of the corresponding filter graph. Use the scale filter directly to insert it at the beginning or elsewhere. The format is’ WXH ‘(default – same as source).

– Aspect [: stream_specifier] aspect ratio (output, per stream)

Sets the aspect ratio of the video display specified in the aspect. Aspect can be a string of floating-point numbers or a string of the form num: den, where num and den are the numerator and denominator of the aspect ratio. For example, “4:3”, “16:9”, “1.3333” and “1.7777” are valid parameter values. If used with a copy of -vCOdec, this affects the aspect ratio stored at the container level, but does not affect the aspect ratio stored in encoded frames (if present).

-vn (output)

Disable video recording.

– vCOdec codec (output)

Set the video codec. This is another name for -codec: v.

-vf Filtergraph (output)

Create the filtergraph specified by the filtergraph and use it to filter the stream.

(3) Audio parameters

-aframes (output)

Sets the number of audio frames to output. This is an outdated alias for -frames: a.

-ar [: stream_specifier] freq (input/output, per stream)

Set the audio sampling frequency. For output streams, it defaults to the frequency of the corresponding input stream. For input streams, this option applies only to the audio capture device and the original splitter, and maps to the corresponding splitter option.

-AC [: stream_specifier] channel (input/output, per stream)

Set the number of audio channels. For output streams, it defaults to the number of input audio channels. For input streams, this option applies only to the audio capture device and the original splitter, and maps to the corresponding splitter option.

-an (output)

Disable recording.

– ACOdec codec (input/output)

Set up the audio codec. This is another name for -codec: A.

-sample_fmt [: stream_specifier] sample_fmt (output, per stream)

Sets the audio sampling format. Use -sample_FMts to get a list of supported sample formats.

-AF Filtergraph (output)

Create the filtergraph specified by the filtergraph and use it to filter the stream.

Vii. Function Description

(1) recording
ffmpeg -f avfoundation -list_devices true -i ""
Copy the code
(2) record the screen
ffmpeg -f avfoundation -i 1 -r 30 out.yuv
Copy the code
  • -f Collects data using AVFoundation.
  • -i specifies where to collect data. It is a file index number. On my computer, 1 represents the desktop (the device index number can be queried using the command above).
  • -r Specifies the frame rate. -r and -framerate have the same effect, but they are different when tested. -framerate is used to limit the input, while -r is used to limit the output.

Note that desktop input has no framerate requirements, so there is no need to limit the desktop framerate. In fact, limits are useless.

(3) Screen recording + sound
Ffmpeg-f avFoundation -i 1:0 -r 29.97 -c:v libx264 -crf 0 -c:a libfdk_aac-profile :a aac_he_v2 -b:a 32k out.flvCopy the code
  • -i 1:0 The “1” before the colon indicates the screen index number. The “0” after the colon represents the sound phase number.
  • -c:v, like -vcodec, indicates the video encoder. C stands for COdec and V stands for video.
  • – CRF is an X264 parameter. 0 table lossless compression.
  • -c:a Indicates an audio encoder, the same as -acodec.
  • – Profile is the parameter of FDK_AAC. Aac_he_v2 Uses AAC_HE v2 to compress data in table format.
  • -b:a Specifies the audio bit rate. B is short for bitrate and A is short for audio.
(4) record the video
ffmpeg -framerate 30 -f avfoundation -i 0 out.mp4
Copy the code
  • -framerate limits the video capture framerate. This must be set as prompted, otherwise an error will be reported.
  • -f Collects data using AVFoundation.
  • -i Specifies the index number of the video device.
(5) Video + audio
ffmpeg -framerate 30 -f avfoundation -i 0:0 out.mp4
Copy the code
(6) the recording
ffmpeg -f avfoundation -i :0 out.wav
Copy the code
(7) Recording audio raw data
ffmpeg  -f avfoundation -i :0 -ar 44100 -f s16le out.pcm
Copy the code
(8) Decomposition and reuse

Stream copy is a mode for selecting a stream by supplying the copy parameter to the -codec option. It causes FFMPEG to omit the decoding and encoding steps of the specified stream, so it can only multiplex and multiplex. This is useful for changing container formats or modifying container-level metadata. In this case, the figure above will simplify to:

Because there is no decoding or encoding, the speed is very fast and there is no loss of quality. However, it may not work properly in some cases due to a number of factors. Applying filters is also obviously impossible, since filters deal with uncompressed data.

(9) Extract audio stream
ffmpeg -i input.mp4 -acodec copy -vn out.aac
Copy the code
  • Acodec: specifies the audio encoder. Copy indicates that only the audio is copied without codec.
  • Vn: V stands for video, n stands for no, which means no video.
(10) Convert to MP3 format
ffmpeg -i input.mp4 -acodec libmp3lame  out.mp3
Copy the code
(11) Extract video stream
ffmpeg -i input.mp4 -vcodec copy -an out.h264
Copy the code
  • Vcodec: specifies the video encoder. Copy indicates that the video is copied without codec.
  • An: A stands for video and N stands for no, which means no audio.
(12) format
ffmpeg -i out.mp4 -vcodec copy -acodec copy out.flv
Copy the code

The above command table audio and video are copied directly, but the MP4 package format is converted to FLV.

(13) Audio and video merging
ffmpeg -i out.h264 -i out.aac -vcodec copy -acodec copy out.mp4
Copy the code
(14) Extracting YUV data
Mp4-an -C :v rawVideo -pixel_format yuv420p out.yuv # ffplay -s WXH out.yuvCopy the code
  • -c:v rawVideo Converts videos to raw data
  • -pixel_format yuv420p Specifies the conversion format to YUv420p
Turn the H264 YUV (15)
ffmpeg -f rawvideo -pix_fmt yuv420p -s 320x240 -r 30 -i out.yuv -c:v libx264 -f rawvideo out.h264
Copy the code
(16) Extract PCM data
Ffmpeg-i out.mp4-vn-ar 44100-AC 2-f s16le out.pcm # ffplay-ar 44100-AC 2-F s16le out.pcm # ffplay-ar 44100-AC 2-F s16le out.pcmCopy the code
(17) PCM WAV
ffmpeg -f s16be -ar 8000 -ac 2 -acodec pcm_s16be -i input.raw output.wav
Copy the code
(18) Simple filter

Before coding, FFMPEG can process raw audio and video frames using filters in the libavFilter library. Several chain filters form a filter pattern. Ffmpeg distinguishes between two types of filter graphics: simple and complex.

Simple filter graphs are those that have only one input and one output, both of the same type. In the figure above, they can be represented by inserting an additional step between decoding and encoding:

The simple FilterGraphs are configured with the per-stream-filter option (aliases -vf and -af for video and audio respectively). An example of a simple video Filtergraph could look like this:

Note that some filters change the frame properties, but not the frame contents. For example. The FPS filter in the example above changes the number of frames but does not touch the contents of the frame. Another example is the SetPTS filter, which only sets the timestamp and otherwise does not change the frame.

(19) Complex filters

Complex filter diagrams are those that cannot be simply described as a linear processing chain applied to a stream. This is the case, for example, when the graph has multiple inputs and/or outputs, or when the output stream type is different from the input. They can be represented as follows:

Complex filter diagrams are configured using the -filter_complex option. Note that this option is global, because complex filter graphs by nature cannot be explicitly associated with a single stream or file.

The -lavfi option is equivalent to -filter_complex.

A simple example of a complex filter diagram is the overlay filter, which has two video inputs and one video output and contains one video superimposed on top of the other. Its audio counterpart is the AMIX filter.

(20) Add watermark
ffmpeg -i out.mp4 -vf "movie=logo.png,scale=64:48[watermask]; [in][watermask] overlay=30:10 [out]" water.mp4Copy the code
  • – Movie in VF specifies the logo position. Scale specifies the logo size. Overlay Specifies the logo placement position.
(21) Delete watermark

First find the location to delete the LOGO through ffplay

ffplay -i test.flv -vf delogo=x=806:y=20:w=70:h=80:show=1
Copy the code

Delete the LOGO using the Delogo filter

ffmpeg -i test.flv -vf delogo=x=806:y=20:w=70:h=80 output.flv
Copy the code
(22) Video size is doubled
fmpeg -i out.mp4 -vf scale=iw/2:-1 scale.mp4
Copy the code
  • -vf scale specifies the simple filter scale. Iw in IW /2:-1 specifies the width of the video as an integer. -1 indicates that the height varies with the width.
(23) Video clipping
ffmpeg -i VR.mov  -vf crop=in_w-200:in_h-200 -c:v libx264 -c:a copy -video_size 1280x720 vr_new.mp4
Copy the code

Crop format: crop=out_w:out_h:x:y

  • Out_w: output width. You can use the in_W table to enter the width of the video.
  • Out_h: output height. You can use the in_H table to enter the height of the video.
  • X: the x coordinate
  • Y: the y coordinate

If x and y are set to 0, clipping starts at the upper-left corner. If you don’t write it is clipped from the center.

(24) Double speed playback
Ffmpeg-i out. Mp4-filter_complex "[0:v]setpts=0.5*PTS[v]; [0: a] atempo = 2.0 [a] "- map" [v] "- map" [a] "speed2.0. Mp4Copy the code

-filter_complex Complex filter, [0:v] represents the video of the first file (file index number is 0) as input. Setpts =0.5*PTS means that the PTS timestamp of each video frame is multiplied by 0.5, i.e. the difference is reduced by half. [v] indicates the alias of the output. I’m not going to go into the audio as well.

Map can be used to handle complex outputs, such as the ability to output a specified multiple stream to one output file, or to specify the output to multiple files. [v]” Alias for complex filter output as a stream of output files. The use of map above is to output video and audio output from a complex filter to a specified file.

(25) Symmetric video
ffmpeg -i out.mp4 -filter_complex "[0:v]pad=w=2*iw[a]; [0:v]hflip[b]; [a][b]overlay=x=w" duicheng.mp4Copy the code
  • Hflip flips horizontally

If you want to change to vertical flip, you can use vFlip.

Picture in picture (26)
ffmpeg -i out.mp4 -i out1.mp4 -filter_complex "[1:v]scale=w=176:h=144:force_original_aspect_ratio=decrease[ckout]; [0:v][ckout]overlay=x=W-w-10:y=0[out]" -map "[out]" -movflags faststart new.mp4Copy the code
(27) Record picture in picture
ffmpeg -f avfoundation -i "1" -framerate 30 -f avfoundation -i "0:0" -r 30 -c:v libx264 -preset ultrafast -c:a libfdk_aac -profile:a aac_he_v2 -ar 44100 -ac 2 -filter_complex "[1:v]scale=w=176:h=144:force_original_aspect_ratio=decrease[a]; [0:v][a]overlay=x=W-w-10:y=0[out]" -map "[out]" -movflags faststart -map 1:a b.mp4Copy the code
(28) Multi-channel video stitching
ffmpeg -f avfoundation -i "1" -framerate 30 -f avfoundation -i "0:0" -r 30 -c:v libx264 -preset ultrafast -c:a libfdk_aac -profile:a aac_he_v2 -ar 44100 -ac 2 -filter_complex "[0:v]scale=320:240[a]; [a]pad=640:240[b]; [b][1:v]overlay=320:0[out]" -map "[out]" -movflags faststart -map 1:a c.mp4Copy the code
(29) cutting
ffmpeg -i out.mp4 -ss 00:00:00 -t 10 out1.mp4
Copy the code
  • -ss Specifies the start time of clipping, accurate to seconds
  • -t Specifies the duration after clipping.
Merge (30)

Create an elsions. TXT file with the following contents:

file ‘1.flv’

file ‘2.flv’

file ‘3.flv’

Then execute the following command:

ffmpeg -f concat -i inputs.txt -c copy output.flv

(31) HLS slice
ffmpeg -i out.mp4 -c:v libx264 -c:a libfdk_aac -strict -2 -f hls  out.m3u8
Copy the code
  • – strict-2 Indicates that the audio is AAC.
  • -f HLS converts to the M3U8 format.
(32) Video to JPEG
ffmpeg -i test.flv -r 1 -f image2 image-%3d.jpeg
Copy the code
(33) Video to GIF
ffmpeg -i test.flv -r 1 -f image2 image-%3d.jpeg
Copy the code
(34) Picture to video
ffmpeg  -f image2 -i image-%3d.jpeg images.mp4
Copy the code
Push flow (35)
ffmpeg -re -i out.mp4 -c copy -f flv rtmp://server/live/streamName
Copy the code
(36) Pull flow preservation
ffmpeg -i rtmp://server/live/streamName -c copy dump.flv
Copy the code
Turn flow (37)
ffmpeg -i rtmp://server/live/originalStream -c:a copy -c:v copy -f flv rtmp://server/live/h264Stream
Copy the code
(38) Real-time push flow
ffmpeg -framerate 15 -f avfoundation -i "1" -s 1280x720 -c:v libx264  -f  flv rtmp://localhost:1935/live/room
Copy the code
(39) Play YUV data
ffplay -pix_fmt nv12 -s 192x144 1.yuv
Copy the code
(40) Play Y plane in YUV
ffplay -pix_fmt nv21 -s 640x480 -vf extractplanes='y' 1.yuv
Copy the code