FFmpeg has multiple timestamps internally, based on different time benchmarks. Understanding these time concepts helps in audio and video development through FFmpeg.

In my opinion, FFmpeg has two time bases: AV_TIME_BASE and AVStream->time_base.

Internal time base: AV_TIME_BASE

AV_TIME_BASE indicates how long 1S corresponds to, and AV_TIME_BASE_Q is the corresponding fraction, as shown below:

/** * Internal time base represented as integer * */
#define AV_TIME_BASE            1000000

/** * Internal time base represented as fractional value */
#define AV_TIME_BASE_Q          (AVRational){1, AV_TIME_BASE}
Copy the code

Most of the parameters in AVFormatContext are based on AV_TIME_BASE, as shown below:

/** * Position of the first frame of the component, in * AV_TIME_BASE fractional seconds
int64_t start_time;
/** * Duration of the stream, in AV_TIME_BASE fractional * seconds
int64_t duration;
Copy the code

For example, using AVFormatContext to calculate the duration of audio and video files:

/** * Convert an AVRational to a `double`. */
static inline double av_q2d(AVRational a){
    return a.num / (double) a.den;
}

/ / unit of seconds
AVFormatContext->duration * av_q2d(AV_TIME_BASE_Q)
Copy the code

So, timestamp conversions based on internal time benchmarks are simple:

// The timestamp is converted to seconds
time_in_seconds = timestamp * av_q2d(AV_TIME_BASE_Q)
// Convert seconds to timestamp
timestamp = time_in_seconds * AV_TIME_BASE
Copy the code

Stream time baseline: AVStream->time_base

AVStream represents a specific stream structure in AVFormatContext, such as audio stream, video stream, subtitle stream, etc. AVStream->time_base is also a fractional structure AVRational, only instead of fixed values, it depends on the concrete stream. The timestamps in AVStream, AVPacket and AVFrame are mostly based on the corresponding stream time baseline.

For example: frame start timestamp and stream duration in AVStream:

/** * This is the fundamental unit of time (in seconds) in terms * of which frame timestamps are represented. * How long does one second take */
AVRational time_base;
/** * Decoding: pts of the first frame of the stream in presentation order, in stream time base. */
int64_t start_time;
/** * Decoding: duration of the stream, in stream time base. */
int64_t duration;
Copy the code

To calculate the flow duration information (in seconds), look like this:

// Calculate how many seconds AVStream takesDuration (seconds) = AVStream->duration * av_q2d(AVStream->time_base);Copy the code

For example, the time_base of a video stream is 1/90000, that is, 90000 indicates 1 second. So 2700000 is actually 30 seconds.

Look again at the timestamp in AVFrame:

/** * frame timestamp estimated using various heuristics, in stream time base * */
int64_t best_effort_timestamp;
/** * Presentation timestamp in time_base units (time when frame should be shown to user). */
int64_t pts;
Copy the code

In general, both can represent PTS timestamps and are based on a stream time baseline.

// Indicates PTS in secondsTimestamp (seconds) = AV_frame_get_best_effort_timestamp (AVFrame) * av_q2d(AVStream->time_base);Copy the code

Conversion between different time bases

To convert between different time benchmarks, FFmpeg provides the av_rescale_q function for conversion

/** * Rescale a 64-bit integer by 2 rational numbers. * The operation is mathematically equivalent to `a * bq / cq`. * A represents the original value, bq represents the original time baseline; Cq represents the time base to which you want to convert */
int64_t av_rescale_q(int64_t a, AVRational bq, AVRational cq) av_const;
Copy the code

For example, to convert a stream timestamp to an internal timestamp:

// Convert the PTS of a video frame to an internal time baseline
av_rescale_q(AVFrame->pts, AVStream->time_base, AV_TIME_BASE_Q)
Copy the code

When seek is performed in FFMPEG (AV_seek_frame), the timestamp must be based on the stream time baseline, such as seek to the 5th second.

// First calculate the timestamp based on the video stream time baseline
int64_t timestamp_in_stream_time_base = av_rescale_q(5 * AV_TIME_BASE, AV_TIME_BASE_Q, video_stream_->time_base);
/ / then the seek
av_seek_frame(av_format_context, video_stream_index, timestamp_in_stream_time_base, AVSEEK_FLAG_BACKWARD);
Copy the code