Problem description

Recently the same group of colleagues encountered a problem: FFmpeg spliced TS files generated mp4 (demux -> mux, no codec) can play on Android platform, but not on MAC (QuickTime Player) and iOS. Where FFmpeg version is 3.3, TS stream contains audio track LC AAC and video track HEVC, as shown below:

// Stream #0 [0x101]: Video: Hevc (Main) ([36][0][0][0] [0][0] / 0x0024), YUV420p (TV), 720x1280, 250 TBR, 90K TBN, 90k TBC Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 100 kb/sCopy the code

The basic information of mp4 is shown below. If you play it in QuickTime Player on MAC, there will be a message indicating that the format is incompatible, but you can play it with QQ audio and VLC. It is also available on Android via MediaPlayer and custom player.

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'leon.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: Isomiso2mp41 encoder: Lavf57.71.100 Duration: 00:02:04.42, Start: 99.935011, bitrate: 173 KB /s Stream #0:0 Video: Hevc (Main) (HEv1/0x31766568), YUV420p (TV), 720x1280, 159 KB /s, SAR 1:1 DAR 9:16, 1.93 FPS, 250 TBR, 90K TBN, 90k tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 12 kb/s (default) Metadata: handler_name : SoundHandlerCopy the code

Preliminary problem resolution

What’s the problem? One of the world’s largest gay dating sites. Hev1 or HvC1 are the two codec tags that represent the different packaging methods for heVC streams in mp4 containers. Quicktime Player and iOS no longer support HEv1 tag MP4. Hev1 or HVC1 are just two different ways of packaging HEVC in MP4. The encoding format itself is HEVC, so you can switch between these two tags without re-encoding.

Try converting MP4’s Codec tag with the following command:

ffmpeg -i leon.mp4 -c:v copy -tag:v hvc1 -c:a copy leon-hvc1.mp4
Copy the code

Error: FFmpeg3.3 or earlier is found:

<mux.c init_muxer 376> Tag hvc1 incompatible with output codec id '174' ([35][0][0][0])
Copy the code

In versions 3.4 and later, it works fine.

Leon. mp4 is in HEv1 tag mode. You can convert it to HvC1 tag mode by using -tag:v hvc1 and play it in Quicktime Player.

Similarly, in FFmpeg3.4 and later versions, you can convert ts to mp4 by using the following command, and control the codec tag mode of mp4 with -tag:v.

// Convert ts (AAC and HEVC) to MP4. By default, leon. MP4's codec tag is HEv1. Ts -c:v copy -c:a copy leon.mp4 // Convert ts (aAC and heVC) to MP4, force hvc1, So the codec tag of leon-hvc1 is Hvc1, and the Quicktime Player can play ffmpeG-i leon.ts -c:v copy-tag :v hVc1 -c:a copy leon-hvc1. Mp4Copy the code

-tag:v hvc1; -tag:v hev1; -tag:v hvc1; -tag:v hev1;

<mux.c init_muxer 376> Tag hev1 incompatible with output codec id '174' ([35][0][0][0])
Copy the code

This means FFmpeg3.3 and earlier versions do not support -tag:v hvc1 and -tag:v hev1, but FFmpeg3.4 and later versions do.

The root cause

Differences between FFmpeg versions can only be addressed in the source code. Fortunately, the error message already specifies the file name, function name, and line number: libavformat/mux.c file, init_muxer function, line 376.

The call stack is: avformat_write_header -> avformat_init_output -> init_muxer, triggered when the header information of the wrapper container is written.

The core code snippet is as follows:

// par is the encoding parameter for heVC stream: AVStream-> codecPAR
// st is the heVC stream: AVStream
/ / s is AVFormatContext
Mp4: AVFormatContext->oformat;
if (par->codec_tag) { // If AVStream-> coDECPAR -> coDEC_tag is not 0
    // Check if AVStream-> coDECPAR -> coDEC_tag is in the list of coDEC_tags supported by the wrapper container, that is: AVOutputFormat-> coDEC_tag list
    if (!validate_codec_tag(s, st)) {
        // Select the codec_tag from AVOutputFormat->codec_tag
        const uint32_t otag = av_codec_get_tag(s->oformat->codec_tag, par->codec_id);
        Codec_tag = otag; // Codec_id = otag
        av_log(s, AV_LOG_ERROR, "Tag %s incompatible with output codec id '%d' (%s)\n".av_fourcc2str(par->codec_tag), par->codec_id, av_fourcc2str(otag));
        ret = AVERROR_INVALIDDATA;
        gotofail; }}else {
    // If AVStream-> coDECPAR -> coDEC_tag is 0 (the default is 0), FFmpeg will find a coDEC_tag from the coDEC_tag list of the encapsulated format based on the encoding ID (AVCodecID).
    par->codec_tag = av_codec_get_tag(of->codec_tag, par->codec_id);
}
    
Copy the code

Avformat_write_header Checks codec_tag when writing to the wrapper’s header: If AVStream-> coDECPAR -> coDEC_tag has a value, then AVStream-> coDECPAR -> coDEC_tag is checked to see if it is in the list of coDEC_tags supported by the encapsulation format. If it is not, an error message is printed. If AVStream-> coDECPAR -> coDEC_tag is 0, a matching CODEC_tag will be found from the coDEC_tag list of the encapsulated format based on AVCodecID.

Up to this point, FFmpeg3.3 and later versions have the same logic, so the hVC1 tag difference is probably due to the different COdec_tag lists supported by MP4 in different FFmpeg versions.

Let’s look at FFmpeg3.3 error message:

// the current hvc1 coDEC_tag set for heVC stream is not supported. 174 is the AVCodecID of heVC stream. Codec_tag incompatible with output COdec ID '174' <mux.c init_muxer 376> Tag Hvc1 incompatible with output codec ID '174' ([35] [0] [0] [0])Copy the code

OK, the cause of the error is clear: in FFmpeg3.3, the HVC1 tag we set for HEVC is not supported by MP4 package format. Let’s take a look at the list of COdec_tag supported by MP4.

Libavformat /movenc.c is the Muxer class for mp4 and MOV, where AVOutputFormat->codec_tag specifies the list of codec_tags supported by a package format. Mp4 | codec_tag | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type | ff_mp4_obj_type |

{ AV_CODEC_ID_H264        , 0x21 },
{ AV_CODEC_ID_HEVC        , 0x23 },
{ AV_CODEC_ID_AAC         , 0x40 },
Copy the code

The codec_tag corresponding to heVC is 0x23, which is base 10 35, exactly the same as the error message above. That is, the only codec_tag supported for HEVC encoding in MP4 is 0x23, or hev1, so we failed to specify HVC1 explicitly.

Take a look at the latest trunk version of FFmpeg, the list of codec_tag supported by MP4.

{ AV_CODEC_ID_H264        , MKTAG('a'.'v'.'c'.'1') },
{ AV_CODEC_ID_H264        , MKTAG('a'.'v'.'c'.'3') },
{ AV_CODEC_ID_HEVC        , MKTAG('h'.'e'.'v'.'1') },
{ AV_CODEC_ID_HEVC        , MKTAG('h'.'v'.'c'.'1') },
{ AV_CODEC_ID_AAC         , MKTAG('m'.'p'.'4'.'a')},Copy the code

The codec_tag corresponding to heVC is hev1 and hvc1, and since hev1 comes first in the list, hev1 is used by default, that is: If we do not specify -tag:v, then the codec_tag of hevc is hev1. Quicktime Player cannot play the tag, so we must specify codec_tag by tag:v hvc1.

If FFmpeg3.3 supports hev1 tag, then why does it fail to specify v hev1 tag? That’s because FFmpeg3.3 and the latest version of FFmpeg3.3 represent codec_tag differently (also hev1 tag, FFmpeg3.3 is OX23, and the latest version is MKTAG(‘ H ‘, ‘e’, ‘v’, ‘1’)), although both are HEv1, the value has been changed. The two versions are not compatible, if you specify -tag:v 35, FFmpeg3.3 will not be wrong.

So far, all questions have been answered, so to summarize:

  • For FFmpeg3.3 and earlier versions, the MP4 container only supports the HEV1 tag of HEVC, and the value is 0x23.
  • For FFmpeg3.4 and later, the mp4 container supports both hev1 and hvc1 tags for hevc, and the hev1 tag is used by default.

If you want to use THE HEVC HVC1 tag in FFmpeg3.3 or earlier, what should you do? Mov container can be used instead, analysis of the code found: MOV package format in different versions of FFmpeg, T support both HEV1 and HVC1 tag of HEVC, and the default use of HEV1 tag.

Set the codec_tag for the heVC stream before setting the avformat_write_header:

AVStream->codecpar->codec_tag = MKTAG('h'.'v'.'c'.'1');
Copy the code

What is the difference between the two tags in Mp4? Different tags correspond to different Box types.

  • Tag: STSD -> HEV1 -> hvcC