demand

The encoded video stream, which can come from a network stream or a file, is decoded into raw video data, which can then be rendered to the screen.

Realize the principle of

As we know, the encoded data is only for transmission and cannot be directly rendered to the screen, so here FFmpeg is used to parse the encoded video stream in the file and decode the compressed video data (H264 / H265) into the video raw data in the specified format (YUV,RGB) to render to the screen.

Note: this example is mainly for decoding, need to use FFmpeg to build modules, video parsing module, rendering module, these modules are linked in the following premise can be accessed directly.

Read the premise

  • Fundamentals of Audio and Video
  • Set up the iOS FFmpeg environment
  • FFmpeg parses video data
  • OpenGL renders video data

Code Address:Video Decoder

Address of nuggets:Video Decoder

Letter Address:Video Decoder

Blog Address:Video Decoder

The overall architecture

Simple and easy process

FFmpeg parse process

  • Create the format context:avformat_alloc_context
  • Open file stream:avformat_open_input
  • Looking for streaming information:avformat_find_stream_info
  • Get the index value of audio and video stream:formatContext->streams[i]->codecpar->codec_type == (isVideoStream ? AVMEDIA_TYPE_VIDEO : AVMEDIA_TYPE_AUDIO)
  • Get audio and video streams:m_formatContext->streams[m_audioStreamIndex]
  • Parsing audio and video data frames:av_read_frame
  • Get extra data:av_bitstream_filter_filter

FFmpeg decode process

  • Determine the decoder type:enum AVHWDeviceType av_hwdevice_find_type_by_name(const char *name)
  • Create a video stream:int av_find_best_stream(AVFormatContext *ic,enum FfmpegaVMediaType type,int wanted_stream_nb,int related_stream,AVCodec **decoder_ret,int flags);
  • Initialize the decoder:AVCodecContext *avcodec_alloc_context3(const AVCodec *codec)
  • Populate the decoder context:int avcodec_parameters_to_context(AVCodecContext *codec, const AVCodecParameters *par);
  • Open the specified type of device:int av_hwdevice_ctx_create(AVBufferRef **device_ctx, enum AVHWDeviceType type, const char *device, AVDictionary *opts, int flags)
  • Initialize the encoder context object:int avcodec_open2(AVCodecContext *avctx, const AVCodec *codec, AVDictionary **options)
  • Initialize a video frame:AVFrame *av_frame_alloc(void)
  • Find the first I frame and start decoding:packet.flags == 1
  • Send compressed data from parse to the decoder:int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt)
  • Received decoded data:int avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame)
  • Construct timestamp
  • Save the decoded data toCVPixelBufferRefAnd turn it intoCMSampleBufferRef, decoding completed

File structure

Quick to use

  • Initialize the preview
- (void)viewDidLoad {
    [super viewDidLoad];
    [self setupUI];
}

- (void)setupUI {
    self.previewView = [[XDXPreviewView alloc] initWithFrame:self.view.frame];
    [self.view addSubview:self.previewView];
    [self.view bringSubviewToFront:self.startBtn];
}
Copy the code
  • Parse and decode the video data in the file
- (void)startDecodeByFFmpegWithIsH265Data:(BOOL)isH265 { NSString *path = [[NSBundle mainBundle] pathForResource:isH265 @?"testh265" : @"testh264" ofType:@"MOV"];
    XDXAVParseHandler *parseHandler = [[XDXAVParseHandler alloc] initWithPath:path];
    XDXFFmpegVideoDecoder *decoder = [[XDXFFmpegVideoDecoder alloc] initWithFormatContext:[parseHandler getFormatContext] videoStreamIndex:[parseHandler getVideoStreamIndex]];
    decoder.delegate = self;
    [parseHandler startParseGetAVPackeWithCompletionHandler:^(BOOL isVideoFrame, BOOL isFinish, AVPacket packet) {
        if (isFinish) {
            [decoder stopDecoder];
            return;
        }
        
        if(isVideoFrame) { [decoder startDecodeVideoDataWithAVPacket:packet]; }}]; }Copy the code
  • Render the decoded data onto the screen
-(void)getDecodeVideoDataByFFmpeg:(CMSampleBufferRef)sampleBuffer {
    CVPixelBufferRef pix = CMSampleBufferGetImageBuffer(sampleBuffer);
    [self.previewView displayPixelBuffer:pix];
}
Copy the code

The specific implementation

1. Initialize the instance object

Because the video data source in this example is a file, and the Format Context context is actually initialized by the Parse module, you simply pass it into the decoder here.

- (instancetype)initWithFormatContext:(AVFormatContext *)formatContext videoStreamIndex:(int)videoStreamIndex {
    if (self = [super init]) {
        m_formatContext     = formatContext;
        m_videoStreamIndex  = videoStreamIndex;
        
        m_isFindIDR = NO;
        m_base_time = 0;
        
        [self initDecoder];
    }
    return self;
}

Copy the code

2. Initialize the decoder

Void initDecoder {// AVStream *videoStream = m_formatContext->streams[m_videoStreamIndex]; / / create the decoder context object m_videoCodecContext = [self createVideoEncderWithFormatContext: m_formatContext stream: videoStream videoStreamIndex:m_videoStreamIndex];if(! m_videoCodecContext) {log4cplus_error(kModuleName, "%s: create video codec failed",__func__);
        return; } // create video frame m_videoFrame = av_frame_alloc();if(! m_videoFrame) {log4cplus_error(kModuleName, "%s: alloc video frame failed",__func__); avcodec_close(m_videoCodecContext); }}Copy the code
2.1. Create a decoder context object
- (AVCodecContext *)createVideoEncderWithFormatContext:(AVFormatContext *)formatContext stream:(AVStream *)stream videoStreamIndex:(int)videoStreamIndex { AVCodecContext *codecContext = NULL; AVCodec *codec = NULL; Const char *codecName = AV_hwDevice_get_type_name (AV_HWDEVICE_TYPE_VIDEOTOOLBOX); // Convert the decoder name to the corresponding enum type, enum AVHWDeviceTypetype = av_hwdevice_find_type_by_name(codecName);
    if (type! = AV_HWDEVICE_TYPE_VIDEOTOOLBOX) {log4cplus_error(kModuleName, "%s: Not find hardware codec.",__func__);
        returnNULL; Int ret = av_find_best_STREAM (formatContext, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);if (ret < 0) {
        log4cplus_error(kModuleName, "av_find_best_stream faliture");
        returnNULL; } // Allocate memory for the decoder context object codecContext = AVCODEC_alloc_context3 (codec);if(! codecContext){log4cplus_error(kModuleName, "avcodec_alloc_context3 faliture");
        returnNULL; Ret = AVCODEC_parameters_to_context (codecContext, formatContext->streams[videoStreamIndex]->codecpar);if (ret < 0){
        log4cplus_error(kModuleName, "avcodec_parameters_to_context faliture");
        returnNULL; } ret = InitHardwareDecoder(codecContext,type);
    if (ret < 0){
        log4cplus_error(kModuleName, "hw_decoder_init faliture");
        returnNULL; } // Initialize the decoder context object ret = AVCoDEC_open2 (codecContext, codec, NULL);if (ret < 0) {
        log4cplus_error(kModuleName, "avcodec_open2 faliture");
        return NULL;
    }
    
    return codecContext;
}

#pragma mark - C Function
AVBufferRef *hw_device_ctx = NULL;
static int InitHardwareDecoder(AVCodecContext *ctx, const enum AVHWDeviceType type) {
    int err = av_hwdevice_ctx_create(&hw_device_ctx, type, NULL, NULL, 0);
    if (err < 0) {
        log4cplus_error("XDXParseParse"."Failed to create specified HW device.\n");
        return err;
    }
    ctx->hw_device_ctx = av_buffer_ref(hw_device_ctx);
    return err;
}
Copy the code

  • Av_find_best_stream: Find the best stream information in the file.

    • IC: media file
    • type: video, audio, subtitles…
    • Wanted_stream_nb: stream number requested by the user. -1 indicates automatic selection
    • Related_stream: Tries to find a related stream, if there is none to fill -1
    • Decoder_ret: Returns a decoder reference non-null
    • Flags: Reserved field

  • Avcodec_parameters_to_context: Populates the decoder context based on the values in the supplied decoder parameters

Any assigned field PAR with the corresponding field in the decoder is simply released and replaced with a copy of the corresponding field in the PAR. Does not involve fields in the decoder that do not have corresponding entries in par.

  • Av_hwdevice_ctx_create: Opens a device of the specified type and creates AVHWDeviceContext for it.
  • Avcodec_open2: Initializes the AVCodecContext with the given AVCodec. Before using this function, memory must be allocated with avcodec_alloc_context3 ().
int av_find_best_stream(AVFormatContext *ic,
                        enum FfmpegaVMediaType type,
                        int wanted_stream_nb,
                        int related_stream,
                        AVCodec **decoder_ret,
                        int flags);
Copy the code
2.2. Create video frames

AVFrame acts as a container for the original audio and video data after decoding. Avframes are typically assigned once and repeated multiple times (for example, a single AVFrame to hold frames received from the decoder). In this case, av_frame_unref () will release any references held by the framework and reset them to their original clean state before reuse.

    // Get video frame
    m_videoFrame = av_frame_alloc();
    if(! m_videoFrame) {log4cplus_error(kModuleName, "%s: alloc video frame failed",__func__);
        avcodec_close(m_videoCodecContext);
    }
Copy the code

3. Start decoding

The first I frame in the encoded data stream is found, and then avcodec_send_packet is called to send the compressed data to the decoder. Finally, the video data decoded by AVCODEC_receive_frame is received in a loop. Construct the timestamp and populate CVPixelBufferRef with the decoded data and convert it to CMSampleBufferRef.

- (void)startDecodeVideoDataWithAVPacket:(AVPacket)packet {
    if (packet.flags == 1 && m_isFindIDR == NO) {
        m_isFindIDR = YES;
        m_base_time =  m_videoFrame->pts;
    }
    
    if (m_isFindIDR == YES) {
        [self startDecodeVideoDataWithAVPacket:packet
                             videoCodecContext:m_videoCodecContext
                                    videoFrame:m_videoFrame
                                      baseTime:m_base_time
                              videoStreamIndex:m_videoStreamIndex];
    }
}

- (void)startDecodeVideoDataWithAVPacket:(AVPacket)packet videoCodecContext:(AVCodecContext *)videoCodecContext videoFrame:(AVFrame *)videoFrame baseTime:(int64_t)baseTime videoStreamIndex:(int)videoStreamIndex {
    Float64 current_timestamp = [self getCurrentTimestamp];
    AVStream *videoStream = m_formatContext->streams[videoStreamIndex];
    int fps = DecodeGetAVStreamFPSTimeBase(videoStream);
    
    
    avcodec_send_packet(videoCodecContext, &packet);
    while (0 == avcodec_receive_frame(videoCodecContext, videoFrame))
    {
        CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)videoFrame->data[3];
        CMTime presentationTimeStamp = kCMTimeInvalid;
        int64_t originPTS = videoFrame->pts;
        int64_t newPTS    = originPTS - baseTime;
        presentationTimeStamp = CMTimeMakeWithSeconds(current_timestamp + newPTS * av_q2d(videoStream->time_base) , fps);
        CMSampleBufferRef sampleBufferRef = [self convertCVImageBufferRefToCMSampleBufferRef:(CVPixelBufferRef)pixelBuffer
                                                                   withPresentationTimeStamp:presentationTimeStamp];
        
        if (sampleBufferRef) {
            if([self.delegate respondsToSelector:@selector(getDecodeVideoDataByFFmpeg:)]) { [self.delegate getDecodeVideoDataByFFmpeg:sampleBufferRef]; } CFRelease(sampleBufferRef); }}}Copy the code

  • Avcodec_send_packet: Sends compressed video frame data to the decoder

    • AVERROR(EAGAIN): In the current state, the user must passavcodec_receive_frame()Read the output buffer. (Once all output has been read,packet should be re-sent without the call failing.)
    • AVERROR_EOF: The decoder has been flushed and no new packets can be sent to it.
    • AVERROR(EINVAL): The decoder is not open
    • AVERROR(ENOMEM): Failed to add Packet to the internal queue.

  • Avcodec_receive_frame: receives decoded data from the decoder

    • AVERROR(EAGAIN): The output is not available and the user must try to send a new input
    • AVERROR_EOF: The decoder is completely refreshed, there are no more output frames
    • AVERROR(EINVAL): The decoder is not open.
    • Other negative numbers: decoding error.

4. Stop decoding

Releasing related Resources

- (void)stopDecoder {
    [self freeAllResources];
}

- (void)freeAllResources {
    if (m_videoCodecContext) {
        avcodec_send_packet(m_videoCodecContext, NULL);
        avcodec_flush_buffers(m_videoCodecContext);
        
        if (m_videoCodecContext->hw_device_ctx) {
            av_buffer_unref(&m_videoCodecContext->hw_device_ctx);
            m_videoCodecContext->hw_device_ctx = NULL;
        }
        avcodec_close(m_videoCodecContext);
        m_videoCodecContext = NULL;
    }
    
    if(m_videoFrame) { av_free(m_videoFrame); m_videoFrame = NULL; }}Copy the code