This article was first published on the wechat official account Byteflow

FFmpeg development series serial:

FFmpeg Development (01) : FFmpeg compilation and integration

FFmpeg development (02) : FFmpeg + ANativeWindow video decoding playback

FFmpeg development (03) : FFmpeg + OpenSLES audio decoding playback

FFmpeg development (04) : FFmpeg + OpenGLES audio visual playback

FFmpeg development (05) : FFmpeg + OpenGLES video decoding playback and video filter

FFmpeg development (06) : FFmpeg player to achieve audio and video synchronization in three ways

FFmpeg development (07) : FFmpeg + OpenGLES implementation of 3D panorama player

FFmpeg Development (08) : FFmpeg player video rendering optimization

FFmpeg development (09) : FFmpeg, X264 and FDK-AAC compiler integration

FFmpeg Development (10) : FFmpeg video recording – Video adding filters and coding

FFmpeg development (11) : FFmpeg + Android AudioRecorder audio recording encoding

FFmpeg development (12) : Android FFmpeg with filter micro channel video recording function

FFmpeg development (13) : Android FFmpeg streaming media playback while recording function

In the previous FFmpeg series of articles, audio and video playback has been realized, recording has added filters and other functions, this article will use FFmpeg + MediaCodec to do a player, video hard decoding and audio and video synchronization.

MediaCodec introduction

MediaCodec is a class provided by Android for audio and video codec. It accesses the underlying CODEC to realize the codec function. It is a part of the Android Media basic framework. Usually used with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface and AudioTrack.

Details see the official document: developer.android.com/reference/a…

AMediaCodec is a native interface for MediaCodec, which is provided by Google from Android 5.0. Native code should be compiled with mediandk library. Github.com/android/ndk…

FFmpeg + ANativeCodec

Before Android opened up the ModecCodec interface in the Native layer, FFmpeg’s implementation of hard decoding required copying video and audio data to the Java layer, where MediaCodec (Java object methods called through JNI) was called.

This article will implement FFmpeg and AMediaCodec in combination, FFmpeg is responsible for demultiplexing and audio decoding, MediaCodec is responsible for video decoding and output to Surface (AtiveWindow) object, Among them, demultiplexing, audio decoding and video decoding are carried out in a subthread respectively, and audio and video packets are managed by queue.

Note: The video processed in this demo is encoded in H.264. Since AVPacket Data is not a standard NALU, Replace the first 4 bytes of AVPacket with 0x00000001 with av_bitSTREAM_filter_filter to get standard NALU data so that MediaCodec decoded correctly.

Configure the AMediaCodec object to decode only the video stream:

        m_MediaExtractor = AMediaExtractor_new(a);media_status_t err = AMediaExtractor_setDataSourceFd(m_MediaExtractor, fd,static_cast<off64_t>(outStart),static_cast<off64_t>(outLen));
        close(fd);
        if(err ! = AMEDIA_OK) { result =- 1;
            LOGCATE("HWCodecPlayer::InitDecoder AMediaExtractor_setDataSourceFd fail. err=%d", err);
            break;
        }

        int numTracks = AMediaExtractor_getTrackCount(m_MediaExtractor);

        LOGCATE("HWCodecPlayer::InitDecoder AMediaExtractor_getTrackCount %d tracks", numTracks);
        for (int i = 0; i < numTracks; i++) {
            AMediaFormat *format = AMediaExtractor_getTrackFormat(m_MediaExtractor, i);
            const char *s = AMediaFormat_toString(format);
            LOGCATE("HWCodecPlayer::InitDecoder track %d format: %s", i, s);
            const char *mime;
            if (!AMediaFormat_getString(format, AMEDIAFORMAT_KEY_MIME, &mime)) {
                LOGCATE("HWCodecPlayer::InitDecoder no mime type");
                result = - 1;
                break;
            } else if (!strncmp(mime, "video/".6)) {
                // Omitting most error handling for clarity.
                // Production code should check for errors.
                AMediaExtractor_selectTrack(m_MediaExtractor, i);
                m_MediaCodec = AMediaCodec_createDecoderByType(mime);
                AMediaCodec_configure(m_MediaCodec, format, m_ANativeWindow, NULL.0);
                AMediaCodec_start(m_MediaCodec);
            }
            AMediaFormat_delete(format);
        }
Copy the code

FFmpeg demultiplexes the audio and video encoded packets into two queues in one thread.

int HWCodecPlayer::DoMuxLoop(a) {
    LOGCATE("HWCodecPlayer::DoMuxLoop start");

    int result = 0;
    AVPacket avPacket = {0};
    for(;;) {
        double passTimes = 0; .if(m_SeekPosition >= 0) { / / the seek operations
            //seek to frame
            LOGCATE("HWCodecPlayer::DoMuxLoop seeking m_SeekPosition=%f", m_SeekPosition); . } result =av_read_frame(m_AVFormatContext, &avPacket);
        if(result >= 0) {
            double bufferDuration = m_VideoPacketQueue->GetDuration(*)av_q2d(m_VideoTimeBase);
            LOGCATE("HWCodecPlayer::DoMuxLoop bufferDuration=%lfs", bufferDuration);
            // Prevent too many buffered packets
            while (BUFF_MAX_VIDEO_DURATION < bufferDuration && m_PlayerState == PLAYER_STATE_PLAYING && m_SeekPosition < 0) {
                bufferDuration = m_VideoPacketQueue->GetDuration(*)av_q2d(m_VideoTimeBase);
                usleep(10 * 1000);
            }

			// Audio and video encoded packets are placed in two queues respectively.
            if(avPacket.stream_index == m_VideoStreamIdx) {
                m_VideoPacketQueue->PushPacket(&avPacket);
            } else if(avPacket.stream_index == m_AudioStreamIdx) {
                m_AudioPacketQueue->PushPacket(&avPacket);
            } else {
                av_packet_unref(&avPacket); }}else {
            // Pause the decoder after demultiplexing
            std::unique_lock<std::mutex> lock(m_Mutex); m_PlayerState = PLAYER_STATE_PAUSE; }}LOGCATE("HWCodecPlayer::DoMuxLoop end");
    return 0;
}
Copy the code

In the video decoding thread, Native uses AMediaCodec to decode the video and takes the packet from the AVPacket queue of the video for decoding.

void HWCodecPlayer::VideoDecodeThreadProc(HWCodecPlayer *player) {
    LOGCATE("HWCodecPlayer::VideoDecodeThreadProc start");
    AVPacketQueue* videoPacketQueue = player->m_VideoPacketQueue;
    AMediaCodec* videoCodec = player->m_MediaCodec;
    AVPacket *packet = av_packet_alloc(a);for(;;) {...ssize_t bufIdx = - 1;
        bufIdx = AMediaCodec_dequeueInputBuffer(videoCodec, 0);
        if (bufIdx >= 0) {
            size_t bufSize;
            auto buf = AMediaCodec_getInputBuffer(videoCodec, bufIdx, &bufSize);
            av_bitstream_filter_filter(player->m_Bsfc, player->m_VideoCodecCtx, NULL, &packet->data, &packet->size, packet->data, packet->size,
                                       packet->flags & AV_PKT_FLAG_KEY);
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc 0x%02X 0x%02X 0x%02X 0x%02X \n",packet->data[0],packet->data[1],packet->data[2],packet->data[3]);
            memcpy(buf, packet->data, packet->size);
            AMediaCodec_queueInputBuffer(videoCodec, bufIdx, 0, packet->size, packet->pts, 0);
        }
        av_packet_unref(packet);
        AMediaCodecBufferInfo info;
        auto status = AMediaCodec_dequeueOutputBuffer(videoCodec, &info, 1000);
        LOGCATI("HWCodecPlayer::VideoDecodeThreadProc status: %d\n", status);
        uint8_t* buffer;
        if (status >= 0) {
            SyncClock* videoClock = &player->m_VideoClock;
            double presentationNano = info.presentationTimeUs * av_q2d(player->m_VideoTimeBase) * 1000;
            videoClock->SetClock(presentationNano, GetSysCurrentTime());
            player->AVSync(a);// Audio and video synchronization

            size_t size;
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc sync video curPts = %lf", presentationNano);
            buffer = AMediaCodec_getOutputBuffer(videoCodec, status, &size);
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc buffer: %p, buffer size: %d", buffer, size);
            AMediaCodec_releaseOutputBuffer(videoCodec, status, info.size ! =0);
        } else if (status == AMEDIACODEC_INFO_OUTPUT_BUFFERS_CHANGED) {
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc output buffers changed");
        } else if (status == AMEDIACODEC_INFO_OUTPUT_FORMAT_CHANGED) {
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc output format changed");
        } else if (status == AMEDIACODEC_INFO_TRY_AGAIN_LATER) {
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc no output buffer right now");
        } else {
            LOGCATI("HWCodecPlayer::VideoDecodeThreadProc unexpected info code: %zd", status);
        }
        if(isLocked) lock.unlock(a); }if(packet ! =nullptr) {
        av_packet_free(&packet);
        packet = nullptr;
    }
    LOGCATE("HWCodecPlayer::VideoDecodeThreadProc end");
}
Copy the code

The delay time is fine-tuned according to the time difference between the actual frame rate and the target frame rate.

void HWCodecPlayer::AVSync(a) {
    LOGCATE("HWCodecPlayer::AVSync");
    double delay = m_VideoClock.curPts - m_VideoClock.lastPts;
    int tickFrame = 1000 * m_FrameRate.den / m_FrameRate.num;
    LOGCATE("HWCodecPlayer::AVSync tickFrame=%dms", tickFrame);
    if(delay <= 0 || delay > VIDEO_FRAME_MAX_DELAY) {
        delay = tickFrame;
    }
    double refClock = m_AudioClock.GetClock(a);// Synchronize video to audio
    double avDiff = m_VideoClock.lastPts - refClock;
    m_VideoClock.lastPts = m_VideoClock.curPts;
    double syncThreshold = FFMAX(AV_SYNC_THRESHOLD_MIN, FFMIN(AV_SYNC_THRESHOLD_MAX, delay));
    LOGCATE("HWCodecPlayer::AVSync refClock=%lf, delay=%lf, avDiff=%lf, syncThreshold=%lf", refClock, delay, avDiff, syncThreshold);
    if(avDiff <= -syncThreshold) { // Video is slower than audio
        delay = FFMAX(0,  delay + avDiff);
    }
    else if(avDiff >= syncThreshold && delay > AV_SYNC_FRAMEDUP_THRESHOLD) { // Video is much faster than audio
        delay = delay + avDiff;
    }
    else if(avDiff >= syncThreshold)
        delay = 2 * delay;

    LOGCATE("HWCodecPlayer::AVSync avDiff=%lf, delay=%lf", avDiff, delay);

    double tickCur = GetSysCurrentTime(a);double tickDiff =  tickCur - m_VideoClock.frameTimer;// The actual interval between two frames
    m_VideoClock.frameTimer = tickCur;

    if(tickDiff - tickFrame >  5) delay-=5;// Adjust delay time
    if(tickDiff - tickFrame < - 5) delay+=5;

    LOGCATE("HWCodecPlayer::AVSync delay=%lf, tickDiff=%lf", delay, tickDiff);
    if(delay > 0) {
        usleep(1000* delay); }}Copy the code

Implementation code path: github.com/githubhaoha…