This paper makes an overall analysis of the HLS protocol case itself, and analyzes the overall process from the perspective of the client
What does the client need to do
The client is responsible for selecting the appropriate requested resource, downloading the resource, and then decoding the display (integrating the player functionality). The client starts by getting the index file, usually using the given URL to identify information about the stream. This index file generally gives the location of available media files, decryption keys, and other optional streams. After the client selects the stream, it begins to download each available media file sequentially. Each file contains successive fragments of a particular stream. Once the client has downloaded enough data, it is ready to decode the data and display it. If required, the client is responsible for reading all decryption keys, authentication, or providing the user with an interface for authentication or decryption. The client can continue this process until it encounters the *# ext-x-endList * tag in the index file; If the label does not exist, the index file is a live broadcast source. The client needs to periodically update the index file and repeat the preceding procedure.
In the common HLS system, the audio input is encoded as AAC and the video input is encoded as H264 using a hard encoder, and the two are reused into MPEG-TS, which is then divided into a series of small TS files using a sharding tool. These files will be available on the Web server. The sharding tool also creates and maintains an index file (called *.m3u8 * in HLS) that contains a list of available media files. The URL of the index file is published on the Web server. The client can read the index file and then request the listed media files in order, and the fragments can be played seamlessly.
hls.c
The first two chapters talk about avformat_open_INPUT. We will deduce the corresponding AVInputFormat according to the file path, and then read the header and read_packet. To review, the general process is as follows.
Avformat_open_input (http.xxx.m3u8) init_input(s, filename, & TMP)) &score))) io_open(s, &s->pb, filename, AVIO_FLAG_READ | s->avio_flags, Options)) < 0) io_open_default ffIO_open_whitelist ffurl_alloc // Probe is HTTP protocol URLProtocol FF_http_protocol url_find_protocol(filename); Ffurl_connect // Sends the HTTP header, Download http.xxx.m3u8 file // read m3u8 file probe demultiplexing is AVInputFormat * iFormat =" HLS,applehttp" * FMT = av_probe_input_buffer2(s->pb, &s->iformat, filename, s, 0, s->format_probesize); s->iformat->read_header(s); //iformat iformat hls.c-->hls_read_headerCopy the codehls.c/read_header
static int hls_read_header(AVFormatContext *s)
{...// Parse the playList, which is the m3U8 file
if ((ret = parse_playlist(c, s->url, NULL, s->pb)) < 0)
goto fail;
if (c->n_variants == 0) {
av_log(s, AV_LOG_WARNING, "Empty playlist\n");
ret = AVERROR_EOF;
goto fail;
}
/* If the playlist only contained playlists (Master Playlist), * parse each individual playlist. */
// If Master Playlist is parsed, parse again for each
if (c->n_playlists > 1 || c->playlists[0]->n_segments == 0) {
for (i = 0; i < c->n_playlists; i++) {
struct playlist *pls = c->playlists[i];
pls->m3u8_hold_counters = 0;
if ((ret = parse_playlist(c, pls->url, pls, NULL))"0) {
av_log(s, AV_LOG_WARNING, "parse_playlist error %s [%s]\n", av_err2str(ret), pls->url);
pls->broken = 1;
if (c->n_playlists > 1)
continue;
gotofail; }}}for (i = 0; i < c->n_variants; i++) {
if (c->variants[i]->playlists[0]->n_segments == 0) {
av_log(s, AV_LOG_WARNING, "Empty segment [%s]\n", c->variants[i]->playlists[0]->url);
c->variants[i]->playlists[0]->broken = 1; }}/* If this isn't a live stream, calculate the total duration of the * stream. */
// If the finished flag is displayed, this is not a live stream, then calculate the entire duration duration
if (c->variants[0]->playlists[0]->finished) {
int64_t duration = 0;
for (i = 0; i < c->variants[0]->playlists[0]->n_segments; i++)
duration += c->variants[0]->playlists[0]->segments[i]->duration;
s->duration = duration;
}
/* Associate renditions with variants */
for (i = 0; i < c->n_variants; i++) {
struct variant *var = c->variants[i];
if (var->audio_group[0])
add_renditions_to_variant(c, var, AVMEDIA_TYPE_AUDIO, var->audio_group);
if (var->video_group[0])
add_renditions_to_variant(c, var, AVMEDIA_TYPE_VIDEO, var->video_group);
if (var->subtitles_group[0])
add_renditions_to_variant(c, var, AVMEDIA_TYPE_SUBTITLE, var->subtitles_group);
}
/* Create a program for each variant */
for (i = 0; i < c->n_variants; i++) {
struct variant *v = c->variants[i];
AVProgram *program;
program = av_new_program(s, i);
if(! program)goto fail;
av_dict_set_int(&program->metadata, "variant_bitrate", v->bandwidth, 0);
}
/* Select the starting segments */
for (i = 0; i < c->n_playlists; i++) {
struct playlist *pls = c->playlists[i];
if (pls->n_segments == 0)
continue; pls->cur_seq_no = select_cur_seq_no(c, pls); highest_cur_seq_no = FFMAX(highest_cur_seq_no, pls->cur_seq_no); }}Copy the codeFirst look at a part of the code, mainly through the parse_playlist method, parse out the relevant program information, and assign values to the HLSContext
parse_playlist
static int parse_playlist(HLSContext *c, const char *url,
struct playlist *pls, AVIOContext *in)
{
// Parse the playerList file from the m3U8 file
int ret = 0, is_segment = 0, is_variant = 0;
int64_t duration = 0;
enum KeyType key_type = KEY_NONE;
uint8_t iv[16] = "";
int has_iv = 0;
char key[MAX_URL_SIZE] = "";
char line[MAX_URL_SIZE];
const char *ptr;
int close_in = 0;
int64_t seg_offset = 0;
int64_t seg_size = - 1;
uint8_t *new_url = NULL;
struct variant_info variant_info;
char tmp_str[MAX_URL_SIZE];
struct segment *cur_init_section = NULL;
int is_http = av_strstart(url, "http".NULL);
struct segment支那prev_segments = NULL;
int prev_n_segments = 0;
int64_t prev_start_seq_no = - 1;
if(is_http && ! in && c->http_persistent && c->playlist_pb) { in = c->playlist_pb; ret = open_url_keepalive(c->ctx, &c->playlist_pb, url,NULL);
if (ret == AVERROR_EXIT) {
return ret;
} else if (ret < 0) {
if(ret ! = AVERROR_EOF) av_log(c->ctx, AV_LOG_WARNING,"keepalive request failed for '%s' with error: '%s' when parsing playlist\n",
url, av_err2str(ret));
in = NULL; }}if(! in) { AVDictionary *opts =NULL;
av_dict_copy(&opts, c->avio_opts, 0);
if (c->http_persistent)
av_dict_set(&opts, "multiple_requests"."1".0);
ret = c->ctx->io_open(c->ctx, &in, url, AVIO_FLAG_READ, &opts);
av_dict_free(&opts);
if (ret < 0)
return ret;
if (is_http && c->http_persistent)
c->playlist_pb = in;
else
close_in = 1;
}
if (av_opt_get(in, "location", AV_OPT_SEARCH_CHILDREN, &new_url) >= 0)
url = new_url;
ff_get_chomp_line(in, line, sizeof(line));
if (strcmp(line, "#EXTM3U")) {
ret = AVERROR_INVALIDDATA;
goto fail;
}
if (pls) {
prev_start_seq_no = pls->start_seq_no;
prev_segments = pls->segments;
prev_n_segments = pls->n_segments;
pls->segments = NULL;
pls->n_segments = 0;
pls->finished = 0;
pls->type = PLS_TYPE_UNSPECIFIED;
}
while(! avio_feof(in)) { ff_get_chomp_line(in, line,sizeof(line));
if (av_strstart(line, "#EXT-X-STREAM-INF:", &ptr)) {
// represents the Master Playlist, which describes many information of the Variant, such as download bandwidth, audio and video coding information, video resolution, and so on
is_variant = 1;
memset(&variant_info, 0.sizeof(variant_info));
ff_parse_key_value(ptr, (ff_parse_key_val_cb) handle_variant_args,
&variant_info);
} else if (av_strstart(line, "#EXT-X-KEY:", &ptr)) {
// This field indicates an encrypted PlayList
struct key_info info = {{0}};
ff_parse_key_value(ptr, (ff_parse_key_val_cb) handle_key_args,
&info);
key_type = KEY_NONE;
has_iv = 0;
if (!strcmp(info.method, "AES-128"))
key_type = KEY_AES_128;
if (!strcmp(info.method, "SAMPLE-AES"))
key_type = KEY_SAMPLE_AES;
if (!strncmp(info.iv, "0x".2) || !strncmp(info.iv, "0X".2)) {
ff_hex_to_data(iv, info.iv + 2);
has_iv = 1;
}
av_strlcpy(key, info.uri, sizeof(key));
} else if (av_strstart(line, "#EXT-X-MEDIA:", &ptr)) {
// Alternate Media provides an alternate audio, video, and subtitle format for HLS, which can be used without changing the generated HLS fragment information.
// Provide new optional media information to the client. To do this, the AUIOD and VIDEO properties are added to stream-INF to implement the association
//#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",AUTOSELECT=YES,DEFAULT=YES
struct rendition_info info = {{0}};
ff_parse_key_value(ptr, (ff_parse_key_val_cb) handle_rendition_args,
&info);
new_rendition(c, &info, url);
} else if (av_strstart(line, "#EXT-X-TARGETDURATION:", &ptr)) {
//EXT -x-targetduration Slice duration
ret = ensure_playlist(c, &pls, url);
if (ret < 0)
goto fail;
pls->target_duration = strtoll(ptr, NULL.10) * AV_TIME_BASE;
} else if (av_strstart(line, "#EXT-X-MEDIA-SEQUENCE:", &ptr)) {
/ / serial number
uint64_t seq_no;
ret = ensure_playlist(c, &pls, url);
if (ret < 0)
goto fail;
seq_no = strtoull(ptr, NULL.10);
if (seq_no > INT64_MAX) {
av_log(c->ctx, AV_LOG_DEBUG, "MEDIA-SEQUENCE higher than "
"INT64_MAX, mask out the highest bit\n");
seq_no &= INT64_MAX;
}
pls->start_seq_no = seq_no;
} else if (av_strstart(line, "#EXT-X-PLAYLIST-TYPE:", &ptr)) {
// PlayerList type vod: on-demand EVENT: EVENT Playlist
ret = ensure_playlist(c, &pls, url);
if (ret < 0)
goto fail;
if (!strcmp(ptr, "EVENT"))
pls->type = PLS_TYPE_EVENT;
else if (!strcmp(ptr, "VOD"))
pls->type = PLS_TYPE_VOD;
} else if (av_strstart(line, "#EXT-X-MAP:", &ptr)) {
// For '# ext-x-map ', store the header of a full video in a separate file and the rest in a separate video file. If you catch one of the videos, you won't be able to play it.
// This is a way to prevent chain theft, "putting all eggs in different baskets"
struct init_section_info info = {{0}};
ret = ensure_playlist(c, &pls, url);
if (ret < 0)
goto fail;
ff_parse_key_value(ptr, (ff_parse_key_val_cb) handle_init_section_args,
&info);
cur_init_section = new_init_section(pls, &info, url);
cur_init_section->key_type = key_type;
if (has_iv) {
memcpy(cur_init_section->iv, iv, sizeof(iv));
} else {
int64_t seq = pls->start_seq_no + pls->n_segments;
memset(cur_init_section->iv, 0.sizeof(cur_init_section->iv));
AV_WB64(cur_init_section->iv + 8, seq);
}
if(key_type ! = KEY_NONE) { ff_make_absolute_url(tmp_str,sizeof(tmp_str), url, key);
if(! tmp_str[0]) {
av_free(cur_init_section);
ret = AVERROR_INVALIDDATA;
goto fail;
}
cur_init_section->key = av_strdup(tmp_str);
if(! cur_init_section->key) { av_free(cur_init_section); ret = AVERROR(ENOMEM);gotofail; }}else {
cur_init_section->key = NULL; }}else if (av_strstart(line, "#EXT-X-ENDLIST", &ptr)) {
// If there is an end flag, it is not a live playerList
if (pls)
pls->finished = 1;
} else if (av_strstart(line, "#EXTINF:", &ptr)) {
//EXTINF ts Duration
is_segment = 1;
duration = atof(ptr) * AV_TIME_BASE;
} else if (av_strstart(line, "#EXT-X-BYTERANGE:", &ptr)) {
// Using this field, you can not store a large number of small files on the server, but only through a file
// to support sharding by offset and length. The playList equivalent to the M3U8 in the on-demand PlayList is as follows:
//#EXT-X-BYTERANGE:75232@0
seg_size = strtoll(ptr, NULL.10);
ptr = strchr(ptr, The '@');
if (ptr)
seg_offset = strtoll(ptr+1.NULL.10);
} else if (av_strstart(line, "#".NULL)) {
av_log(c->ctx, AV_LOG_INFO, "Skip ('%s')\n", line);
continue;
} else if (line[0]) {
// If it contains the master PlayerList, record it and concatenate the URL
if (is_variant) {
if(! new_variant(c, &variant_info, line, url)) { ret = AVERROR(ENOMEM);goto fail;
}
is_variant = 0;
}
/ / ts segment of the url
if (is_segment) {
struct segment *seg;
ret = ensure_playlist(c, &pls, url);
if (ret < 0)
goto fail;
seg = av_malloc(sizeof(struct segment));
if(! seg) { ret = AVERROR(ENOMEM);goto fail;
}
if (has_iv) {
memcpy(seg->iv, iv, sizeof(iv));
} else {
int64_t seq = pls->start_seq_no + pls->n_segments;
memset(seg->iv, 0.sizeof(seg->iv));
AV_WB64(seg->iv + 8, seq);
}
if(key_type ! = KEY_NONE) { ff_make_absolute_url(tmp_str,sizeof(tmp_str), url, key);
if(! tmp_str[0]) {
ret = AVERROR_INVALIDDATA;
av_free(seg);
goto fail;
}
seg->key = av_strdup(tmp_str);
if(! seg->key) { av_free(seg); ret = AVERROR(ENOMEM);gotofail; }}else {
seg->key = NULL;
}
ff_make_absolute_url(tmp_str, sizeof(tmp_str), url, line);
if(! tmp_str[0]) {
ret = AVERROR_INVALIDDATA;
if (seg->key)
av_free(seg->key);
av_free(seg);
goto fail;
}
seg->url = av_strdup(tmp_str);
if(! seg->url) { av_free(seg->key); av_free(seg); ret = AVERROR(ENOMEM);goto fail;
}
if (duration < 0.001 * AV_TIME_BASE) {
av_log(c->ctx, AV_LOG_WARNING, "Cannot get correct #EXTINF value of segment %s,"
" set to default value to 1ms.\n", seg->url);
duration = 0.001 * AV_TIME_BASE;
}
seg->duration = duration;
seg->key_type = key_type;
dynarray_add(&pls->segments, &pls->n_segments, seg);
is_segment = 0;
seg->size = seg_size;
if (seg_size >= 0) {
seg->url_offset = seg_offset;
seg_offset += seg_size;
seg_size = - 1;
} else {
seg->url_offset = 0;
seg_offset = 0; } seg->init_section = cur_init_section; }}}...if (pls)
pls->last_load_time = av_gettime_relative();
fail:
av_free(new_url);
if (close_in)
ff_format_io_close(c->ctx, &in);
c->ctx->ctx_flags = c->ctx->ctx_flags & ~(unsigned)AVFMTCTX_UNSEEKABLE;
if(! c->n_variants || ! c->variants[0]->n_playlists || ! (c->variants[0]->playlists[0]->finished ||
c->variants[0]->playlists[0]->type == PLS_TYPE_EVENT))
c->ctx->ctx_flags |= AVFMTCTX_UNSEEKABLE;
return ret;
}
Copy the codeIf you don’t know much about PlayList, you may be confused. Here is a brief introduction to several commonly used PlayList. After understanding the field parsing in the code, it will be clearer and clearer.
- On demand(
VOD, Video On Demand) Playlist - live
playlist - The secret key
playlist master playlist
On demandplaylist
The playList on demand in HLS is a static file, which is generally not allowed to be modified after being generated. The server side can generate slice files in advance. The format is as follows:
#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXTINF:9.009,
http://media.example.com/first.ts
#EXTINF:9.009,
http://media.example.com/second.ts
#EXTINF:3.003,
http://media.example.com/third.ts
#EXT-X-ENDLIST
Copy the codeTypical features of this type of playlist are
playlistContained in the#EXT-X-ENDLISTField, indicating the end flagEXT-X-PLAYLIST-TYPEField isVOD(optional)
liveplaylist
For live PlayList, the simplest feature is that there is no Ext-x-endList tag, and there is no Ext-x-playlist-type tag. Live PlayList is a typical sliding window where the server transcodes the source format in real time (with a bit of delay). Periodically clear the published fragments. A typical sliding window size is 3-5 fragments. The m3U8 format is as follows
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:10,
fileSequence1.ts
#EXTINF:10,
fileSequence2.ts
#EXTINF:10,
fileSequence3.ts
Copy the codeThe secret keyplaylist
The KEY here is specified by the ext-x-key field, which gives the encryption method and the path where the KEY is stored.
#EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:7794 #EXT-X-TARGETDURATION:15 # EXT - X - KEY: METHOD = AES - 128, URI = "https://priv.example.com/key.php?r=52" # EXTINF: 2.833, http://media.example.com/fileSequence52-A.ts # EXTINF: 15.0, http://media.example.com/fileSequence52-B.ts # EXTINF: 13.333, http://media.example.com/fileSequence52-C.ts #EXT-X-KEY:METHOD=AES-128,URI="https://priv.example.com/key.php?r=53" # EXTINF: 15.0, http://media.example.com/fileSequence53-A.tsCopy the codemaster playlist
The Master Playlist does not contain information about sharding, but describes different encoding formats of the same source, which is called variant in HLS. The Master PlayList describes a lot of information on the Variant, such as download bandwidth, audio and video coding information, video resolution, and so on. A typical example is as follows:
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=150000,RESOLUTION=416x234, \ CODECS = "avc1.42 e00a mp4a. 40.2" http://example.com/low/index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=240000,RESOLUTION=416x234, \ CODECS = "avc1.42 e00a mp4a. 40.2" http://example.com/lo_mid/index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=640000,RESOLUTION=640x360, \ CODECS = "avc1.42 e00a mp4a. 40.2" http://example.com/high/index.m3u8 # EXT - X - STREAM - INF: PROGRAM - ID = 1, BANDWIDTH = 64000, CODECS = "mp4a. 40.5" http://example.com/audio/index.m3u8Copy the codeOf course, there are some types of Playlist will not be described here one by one, interested can be their own extra learning.
ffio_init_context
/* Open the demuxer for each playlist */
for (i = 0; i < c->n_playlists; i++) {
struct playlist *pls = c->playlists[i];
char *url;
ff_const59 AVInputFormat *in_fmt = NULL;
if(! (pls->ctx = avformat_alloc_context())) { ret = AVERROR(ENOMEM);goto fail;
}
if (pls->n_segments == 0)
continue;
pls->index = i;
pls->needed = 1;
pls->parent = s;
/* * If this is a live stream and this playlist looks like it is one segment * behind, try to sync it up so that every substream starts at the same * time position (so e.g. avformat_find_stream_info() will see packets from * all active streams within the first few seconds). This is not very generic, * though, as the sequence numbers are technically independent. */
if(! pls->finished && pls->cur_seq_no == highest_cur_seq_no -1 &&
highest_cur_seq_no < pls->start_seq_no + pls->n_segments) {
pls->cur_seq_no = highest_cur_seq_no;
}
pls->read_buffer = av_malloc(INITIAL_BUFFER_SIZE);
if(! pls->read_buffer){ ret = AVERROR(ENOMEM); avformat_free_context(pls->ctx); pls->ctx =NULL;
goto fail;
}
ffio_init_context(&pls->pb, pls->read_buffer, INITIAL_BUFFER_SIZE, 0, pls,
read_data, NULL.NULL);
pls->ctx->probesize = s->probesize > 0 ? s->probesize : 1024 * 4;
pls->ctx->max_analyze_duration = s->max_analyze_duration > 0 ? s->max_analyze_duration : 4 * AV_TIME_BASE;
pls->ctx->interrupt_callback = s->interrupt_callback;
url = av_strdup(pls->segments[0]->url);
ret = av_probe_input_buffer(&pls->pb, &in_fmt, url, NULL.0.0);
if (ret < 0) {
/* Free the ctx - it isn't initialized properly at this point, * so avformat_close_input shouldn't be called. If * avformat_open_input fails below, it frees and zeros the * context, so it doesn't need any special treatment like this. */
av_log(s, AV_LOG_ERROR, "Error when loading first segment '%s'\n", url);
avformat_free_context(pls->ctx);
pls->ctx = NULL;
av_free(url);
goto fail;
}
av_free(url);
pls->ctx->pb = &pls->pb;
pls->ctx->io_open = nested_io_open;
pls->ctx->flags |= s->flags & ~AVFMT_FLAG_CUSTOM_IO;
if ((ret = ff_copy_whiteblacklists(pls->ctx, s)) < 0)
goto fail;
// Open the 0th ts slice
ret = avformat_open_input(&pls->ctx, pls->segments[0]->url, in_fmt, NULL);
if (ret < 0)
goto fail;
if (pls->id3_deferred_extra && pls->ctx->nb_streams == 1) {
ff_id3v2_parse_apic(pls->ctx, pls->id3_deferred_extra);
avformat_queue_attached_pictures(pls->ctx);
ff_id3v2_parse_priv(pls->ctx, pls->id3_deferred_extra);
ff_id3v2_free_extra_meta(&pls->id3_deferred_extra);
}
if (pls->is_id3_timestamped == - 1)
av_log(s, AV_LOG_WARNING, "No expected HTTP requests have been made\n");
/* * For ID3 timestamped raw audio streams we need to detect the packet * durations to calculate timestamps in fill_timing_for_id3_timestamped_stream(), * but for other streams we can rely on our user calling avformat_find_stream_info() * on us if they want to. */
if (pls->is_id3_timestamped || (pls->n_renditions > 0 && pls->renditions[0]->type == AVMEDIA_TYPE_AUDIO)) {
ret = avformat_find_stream_info(pls->ctx, NULL);
if (ret < 0)
gotofail; } pls->has_noheader_flag = !! (pls->ctx->ctx_flags & AVFMTCTX_NOHEADER);/* Create new AVStreams for each stream in this playlist */
ret = update_streams_from_subdemuxer(s, pls);
if (ret < 0)
goto fail;
/* * Copy any metadata from playlist to main streams, but do not set * event flags. */
if (pls->n_main_streams)
av_dict_copy(&pls->main_streams[0]->metadata, pls->ctx->metadata, 0);
add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_AUDIO);
add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_VIDEO);
add_metadata_from_renditions(s, pls, AVMEDIA_TYPE_SUBTITLE);
}
update_noheader_flag(s);
return 0;
Copy the codeAfter we get the program information, we will unprotocol for each playlist, ffio_init_context, initialize the IOContext information, and notice the read_data that is passed in. This function argument is for reading network data, which is what the real IO layer does
The following method should be familiar with av_probe_input_buffer, to infer the AVInputformat based on the file name (if not familiar, see the previous chapters), here is the TS file, so the final conclusion is ff_mPEGTS_demuxer, in mpets.c
AVInputFormat ff_mpegts_demuxer = {
.name = "mpegts",
.long_name = NULL_IF_CONFIG_SMALL("MPEG-TS (MPEG-2 Transport Stream)"),
.priv_data_size = sizeofMpegTSContext,.read_probe = MPEGTS_probe,.read_header = MPEGTS_read_header, .read_packet = mpegts_read_packet, .read_close = mpegts_read_close, .read_timestamp = mpegts_get_dts, .flags = AVFMT_SHOW_IDS | AVFMT_TS_DISCONT, .priv_class = &mpegts_class, };Copy the coderead_packet
static int hls_read_packet(AVFormatContext *s, AVPacket *pkt)
{
HLSContext *c = s->priv_data;
int ret, i, minplaylist = - 1;
recheck_discard_flags(s, c->first_packet);
c->first_packet = 0;
for (i = 0; i < c->n_playlists; i++) {
struct playlist *pls = c->playlists[i];
/* Make sure we've got one buffered packet from each open playlist * stream */
/* Read an AVPacket */ from each open playlist
if(pls->needed && ! pls->pkt->data) {while (1) {
int64_t ts_diff;
AVRational tb;
ret = av_read_frame(pls->ctx, pls->pkt);
if (ret < 0) {
if(! avio_feof(&pls->pb) && ret ! = AVERROR_EOF)return ret;
break;
} else {
/* stream_index check prevents matching picture attachments etc. */
if (pls->is_id3_timestamped && pls->pkt->stream_index == 0) {
/* audio elementary streams are id3 timestamped */
fill_timing_for_id3_timestamped_stream(pls);
}
if(c->first_timestamp == AV_NOPTS_VALUE && pls->pkt->dts ! = AV_NOPTS_VALUE) c->first_timestamp = av_rescale_q(pls->pkt->dts, get_timebase(pls), AV_TIME_BASE_Q); }if (pls->seek_timestamp == AV_NOPTS_VALUE)
break;
// The following is all about the operation with seek
if (pls->seek_stream_index < 0 ||
pls->seek_stream_index == pls->pkt->stream_index) {
if (pls->pkt->dts == AV_NOPTS_VALUE) {
pls->seek_timestamp = AV_NOPTS_VALUE;
break;
}
tb = get_timebase(pls);
ts_diff = av_rescale_rnd(pls->pkt->dts, AV_TIME_BASE,
tb.den, AV_ROUND_DOWN) -
pls->seek_timestamp;
if (ts_diff >= 0 && (pls->seek_flags & AVSEEK_FLAG_ANY ||
pls->pkt->flags & AV_PKT_FLAG_KEY)) {
pls->seek_timestamp = AV_NOPTS_VALUE;
break; }}/* if the AVPacket is discarded, read */ againav_packet_unref(pls->pkt); }}/* Check if this stream has the packet with the lowest dts */
// Find the smallest DTS packet and record its index value
if (pls->pkt->data) {
struct playlist *minpls = minplaylist < 0 ?
NULL : c->playlists[minplaylist];
if (minplaylist < 0) {
minplaylist = i;
} else {
int64_t dts = pls->pkt->dts;
int64_t mindts = minpls->pkt->dts;
if(dts == AV_NOPTS_VALUE || (mindts ! = AV_NOPTS_VALUE && compare_ts_with_wrapdetect(dts, pls, mindts, minpls) <0)) minplaylist = i; }}}/* If we got a packet, return it */
/* Read AVPacket successfully, need to return to the upper layer caller */
if (minplaylist >= 0) {
struct playlist *pls = c->playlists[minplaylist];
AVStream *ist;
AVStream *st;
ret = update_streams_from_subdemuxer(s, pls);
if (ret < 0) {
av_packet_unref(pls->pkt);
return ret;
}
// If sub-demuxer reports updated metadata, copy it to the first stream
// and set its AVSTREAM_EVENT_FLAG_METADATA_UPDATED flag.
if (pls->ctx->event_flags & AVFMT_EVENT_FLAG_METADATA_UPDATED) {
if (pls->n_main_streams) {
st = pls->main_streams[0];
av_dict_copy(&st->metadata, pls->ctx->metadata, 0);
st->event_flags |= AVSTREAM_EVENT_FLAG_METADATA_UPDATED;
}
pls->ctx->event_flags &= ~AVFMT_EVENT_FLAG_METADATA_UPDATED;
}
/* check if noheader flag has been cleared by the subdemuxer */
if(pls->has_noheader_flag && ! (pls->ctx->ctx_flags & AVFMTCTX_NOHEADER)) { pls->has_noheader_flag =0;
update_noheader_flag(s);
}
if (pls->pkt->stream_index >= pls->n_main_streams) {
av_log(s, AV_LOG_ERROR, "stream index inconsistency: index %d, %d main streams, %d subdemuxer streams\n",
pls->pkt->stream_index, pls->n_main_streams, pls->ctx->nb_streams);
av_packet_unref(pls->pkt);
return AVERROR_BUG;
}
ist = pls->ctx->streams[pls->pkt->stream_index];
st = pls->main_streams[pls->pkt->stream_index];
av_packet_move_ref(pkt, pls->pkt);
pkt->stream_index = st->index;
if(pkt->dts ! = AV_NOPTS_VALUE) c->cur_timestamp = av_rescale_q(pkt->dts, ist->time_base, AV_TIME_BASE_Q);/* There may be more situations where this would be useful, but this at least * handles newly probed codecs properly (i.e. request_probe by mpegts). */
if(ist->codecpar->codec_id ! = st->codecpar->codec_id) { ret = set_stream_info_from_input_stream(st, pls, ist);if (ret < 0) {
returnret; }}return 0;
}
return AVERROR_EOF;
}
Copy the codeThis code calls the av_read_frame method, but as you can see, the whole function does some parsing and judgment work, and there is no I/O layer process. The real processing is in av_read_frame, and the final call is to the read_data method. That’s the function callback that we passed in when we initialized IOContext.
read_data
static int read_data(void *opaque, uint8_t *buf, int buf_size)
{
struct playlist *v = opaque;
HLSContext *c = v->parent->priv_data;
int ret;
int just_opened = 0;
int reload_count = 0;
struct segment *seg;
// The go statement is used
restart:
if(! v->needed)return AVERROR_EOF;
if(! v->input || (c->http_persistent && v->input_read_done)) {int64_t reload_interval;
/* Check that the playlist is still needed before opening a new * segment. */
// Verify that the playList is valid
v->needed = playlist_needed(v);
if(! v->needed) { av_log(v->parent, AV_LOG_INFO,"No longer receiving playlist %d ('%s')\n",
v->index, v->url);
return AVERROR_EOF;
}
/* If this is a live stream and the reload interval has elapsed since * the last playlist reload, reload the playlists now. */
// In case of a live stream, refresh the duration again.
reload_interval = default_reload_interval(v);
reload:
reload_count++;
if (reload_count > c->max_reload)
return AVERROR_EOF;
// Refresh playList again to get the ts list if the live stream has reached reload time
if(! v->finished && av_gettime_relative() - v->last_load_time >= reload_interval) {if ((ret = parse_playlist(c, v->url, v, NULL))"0) {
if(ret ! = AVERROR_EXIT) av_log(v->parent, AV_LOG_WARNING,"Failed to reload playlist %d\n",
v->index);
return ret;
}
/* If we need to reload the playlist again below (if * there's still no more segments), switch to a reload * interval of half the target duration. */
// If you need to request a playlist, the request interval can be set to half of the segment length
reload_interval = v->target_duration / 2;
}
// If the current seqnumer is smaller than the starting seq, skip the middle segment and refresh to the latest data.
if (v->cur_seq_no < v->start_seq_no) {
av_log(v->parent, AV_LOG_WARNING,
"skipping %"PRId64" segments ahead, expired from playlists\n",
v->start_seq_no - v->cur_seq_no);
v->cur_seq_no = v->start_seq_no;
}
if (v->cur_seq_no > v->last_seq_no) {
v->last_seq_no = v->cur_seq_no;
v->m3u8_hold_counters = 0;
} else if (v->last_seq_no == v->cur_seq_no) {
v->m3u8_hold_counters++;
if (v->m3u8_hold_counters >= c->m3u8_hold_counters) {
returnAVERROR_EOF; }}else {
av_log(v->parent, AV_LOG_WARNING, "maybe the m3u8 list sequence have been wraped.\n");
}
// If current serial number > start serial number + total number of TS slices.
if (v->cur_seq_no >= v->start_seq_no + v->n_segments) {
// If the video is on demand, the whole video has reached the end and is over
if (v->finished)
return AVERROR_EOF;
// If it is live, then sleep until the reload time is reached
while (av_gettime_relative() - v->last_load_time < reload_interval) {
if (ff_check_interrupt(c->interrupt_callback))
return AVERROR_EXIT;
av_usleep(100*1000);
}
/* Enough time has elapsed since the last reload */
// Refresh playList once the reload time is reached
goto reload;
}
/* Start reading the segment */
v->input_read_done = 0;
seg = current_segment(v);
/* load/update Media Initialization Section, if any */
ret = update_init_section(v, seg);
if (ret)
return ret;
if (c->http_multiple == 1 && v->input_next_requested) {
FFSWAP(AVIOContext *, v->input, v->input_next);
v->cur_seg_offset = 0;
v->input_next_requested = 0;
ret = 0;
} else {
/* The HTTP request is made here */
ret = open_input(c, v, seg, &v->input);
}
if (ret < 0) {
if (ff_check_interrupt(c->interrupt_callback))
return AVERROR_EXIT;
// Seq +1: seq+1: seq+1: seq+1
av_log(v->parent, AV_LOG_WARNING, "Failed to open segment %"PRId64" of playlist %d\n",
v->cur_seq_no,
v->index);
v->cur_seq_no += 1;
goto reload;
}
just_opened = 1;
}
if (c->http_multiple == - 1) {
uint8_t *http_version_opt = NULL;
int r = av_opt_get(v->input, "http_version", AV_OPT_SEARCH_CHILDREN, &http_version_opt);
if (r >= 0) {
c->http_multiple = (!strncmp((const char *)http_version_opt, "1.1".3) || !strncmp((const char *)http_version_opt, "2.0".3));
av_freep(&http_version_opt);
}
}
seg = next_segment(v);
if (c->http_multiple == 1 && !v->input_next_requested &&
seg && seg->key_type == KEY_NONE && av_strstart(seg->url, "http".NULL)) {
ret = open_input(c, v, seg, &v->input_next);
if (ret < 0) {
if (ff_check_interrupt(c->interrupt_callback))
return AVERROR_EXIT;
av_log(v->parent, AV_LOG_WARNING, "Failed to open segment %"PRId64" of playlist %d\n",
v->cur_seq_no + 1,
v->index);
} else {
v->input_next_requested = 1; }}if (v->init_sec_buf_read_offset < v->init_sec_data_len) {
/* Push init section out first before first actual segment */
int copy_size = FFMIN(v->init_sec_data_len - v->init_sec_buf_read_offset, buf_size);
memcpy(buf, v->init_sec_buf, copy_size);
v->init_sec_buf_read_offset += copy_size;
return copy_size;
}
seg = current_segment(v);
/* This is the actual buF_size data read over HTTP, */ from the server
ret = read_from_url(v, seg, buf, buf_size);
if (ret > 0) {
if(just_opened && v->is_id3_timestamped ! =0) {
/* Intercept ID3 tags here, elementary audio streams are required * to convey timestamps using them in the beginning of each segment. */
intercept_id3(v, buf, buf_size, &ret);
}
return ret;
}
/* Close HTTP resource */
if (c->http_persistent &&
seg->key_type == KEY_NONE && av_strstart(seg->url, "http".NULL)) {
v->input_read_done = 1;
} else {
ff_format_io_close(v->parent, &v->input);
}
// Segment number ++, restart
v->cur_seq_no++;
c->cur_seq_no = v->cur_seq_no;
goto restart;
}
Copy the codeHere you can see that read_data is what the actual IO layer is doing. It reads data from the server and puts it into the buffer. The actual data parsing is done by Demuxer.
read_seek
Finally, seek operation, the basic idea is to find the corresponding stream read position according to the given time point (timestamp), and then continue to read data. So before we do seek, we need to clear the cached data
static int hls_read_seek(AVFormatContext *s, int stream_index,
int64_t timestamp, int flags)
{
HLSContext *c = s->priv_data;
struct playlist *seek_pls = NULL;
int i, j;
int stream_subdemuxer_index;
int64_t first_timestamp, seek_timestamp, duration;
int64_t seq_no;
if ((flags & AVSEEK_FLAG_BYTE) || (c->ctx->ctx_flags & AVFMTCTX_UNSEEKABLE))
return AVERROR(ENOSYS);
first_timestamp = c->first_timestamp == AV_NOPTS_VALUE ?
0 : c->first_timestamp;
// Convert the final seek timestamp based on the timestamp and index passed in
seek_timestamp = av_rescale_rnd(timestamp, AV_TIME_BASE,
s->streams[stream_index]->time_base.den,
flags & AVSEEK_FLAG_BACKWARD ?
AV_ROUND_DOWN : AV_ROUND_UP);
duration = s->duration == AV_NOPTS_VALUE ?
0 : s->duration;
/* Check the validity of the seek position */
if (0 < duration && duration < seek_timestamp - first_timestamp)
return AVERROR(EIO);
/* find the playlist with the specified stream */
/* Find playlist */ for stream_index
for (i = 0; i < c->n_playlists; i++) {
struct playlist *pls = c->playlists[i];
for (j = 0; j < pls->n_main_streams; j++) {
if (pls->main_streams[j] == s->streams[stream_index]) {
seek_pls = pls;
stream_subdemuxer_index = j;
break; }}}/* check if the timestamp is valid for the playlist with the * specified stream index */
/* Check whether the given seek timestamp is valid for the specified stream */
if(! seek_pls || ! find_timestamp_in_playlist(c, seek_pls, seek_timestamp, &seq_no))return AVERROR(EIO);
/* set segment now so we do not need to search again below */
seek_pls->cur_seq_no = seq_no;
seek_pls->seek_stream_index = stream_subdemuxer_index;
/* Below is the processing of the stream being read */
for (i = 0; i < c->n_playlists; i++) {
/* Reset reading */
struct playlist *pls = c->playlists[i];
ff_format_io_close(pls->parent, &pls->input);
pls->input_read_done = 0;
ff_format_io_close(pls->parent, &pls->input_next);
pls->input_next_requested = 0;
av_packet_unref(pls->pkt);
pls->pb.eof_reached = 0;
/* Clear any buffered data */
// Clear the buffer data
pls->pb.buf_end = pls->pb.buf_ptr = pls->pb.buffer;
/* Reset the pos, to let the mpegts demuxer know we've seeked. */
pls->pb.pos = 0;
/* Flush the packet queue of the subdemuxer. */
ff_read_frame_flush(pls->ctx);
pls->seek_timestamp = seek_timestamp;
pls->seek_flags = flags;
/* Handle playlists that are not seek */
if(pls ! = seek_pls) {/* set closest segment seq_no for playlists not handled above */
find_timestamp_in_playlist(c, pls, seek_timestamp, &pls->cur_seq_no);
/* seek the playlist to the given position without taking * keyframes into account since this playlist does not have the * specified stream where we should look for the keyframes */
pls->seek_stream_index = - 1;
pls->seek_flags |= AVSEEK_FLAG_ANY;
}
}
c->cur_timestamp = seek_timestamp;
return 0;
}
Copy the codeNote that the fourth parameter, flag, indicates how to seek to a specific frame.
#deflagfine AVSEEK_FLAG_BACKWARD 1 #define AVSEEK_FLAG_BYTE 2 #define AVSEEK_FLAG_BYTE 2 #define #define AVSEEK_FLAG_FRAME #define AVSEEK_FLAG_FRAME #define AVSEEK_FLAG_FRAME #define AVSEEK_FLAG_FRAME #define AVSEEK_FLAG_FRAME #Copy the codesummary
In this paper, the main reference FFmpeg/libavformat/ HLS. C, the code logic to do a simple collection and collation. Overall, this article summarizes the ffMPEG hlS_demxuer implementation logic, hope to help readers, thanks for viewing ~~ ~