After the learning of the first two and , we have learned some knowledge about audio and video collection and playback. Next, we will enter the application part, and use the knowledge learned before to complete the functions with practical significance

One, knowledge point review

After learning the first two chapters, we have a certain understanding of the collection and playing of audio and video, but we played audio and video separately before, while in actual development, we often need to play audio and video normally

First, let’s review some relevant knowledge about audio and video

1.1 audio

  • The original format of the audio isPCM, you can useAudioRecordSampling the microphone data to getPCMdata
  • forPCMAudio data can be usedMediaCodecYou hard-code it, and you getAACData in encoded format
  • In turn, forAACEncoded data can also be usedMediaCodecI hardcoded it, and I gotPCMRaw format data
  • AACEncoded data can be played directly using a regular player,PCMRaw format data can be usedAudioTrackplay

1.2 video

  • The original format of the video isYUVorRGBCan be obtained from the camera previewNV21(YUV420A kind of) data
  • forYUVVideo data can be usedMediaCodecYou hard-code it, and you getAVCData in encoded format
  • In turn, forAVCEncoded data can also be usedMediaCodecI hardcoded it, and I gotYUVRaw format data, of course, can still be usedSurfaceRender the decoded data directly

In addition, the usage of MediaExtractor and MediaMuxer are also introduced

1.3 MediaExtractor

  • throughMediaExtractorCan be separatedMp4Audio and video tracks in a package format
  • After separation, it can be aligned for reading frame by frame
  • Read data, generally used for incomingMediaCodecDecode operation

1.4 MediaMuxer

  • throughMediaMuxerCan be synthesizedMp4, the audio and video encoding format is written to a file
  • The mode of operation is also throughTrack id
  • After getting the encoded data, pass in the correspondingTrack id, writeMediaMuxerIn the can

Above, the overall review of the and introduction of the content, we will make a simple player according to the knowledge learned

1.5 Playback/Recording Flow chart

play

From the top, the whole thing is playing

  • decapsulation
  • decoding
  • Apply colours to a drawing

The recording

From the bottom up, the whole process is recording

  • collect
  • coding
  • synthetic

Second, video player

In the previous decoding of audio and video, we used a thread for operation, so now is to do a complete video playback function, it needs two threads, one processing audio, one processing video

2.1 Procedure

Audio thread

  • MediaExtractorfromMp4Parse the audio track in
  • Initialize the audioMediaCodec
  • Initialize theAudioTrack
  • fromMediaExtractorRetrieves data to be decoded and passes inMediaCodec
  • fromMediaCodecRetrieve decoded data, passed inAudioTrack
  • Stop playing and release resources

Video thread

  • MediaExtractorParse video tracks from Mp4
  • Initialize the videoMediaCodec
  • fromMediaExtractorRetrieves data to be decoded and passes inMediaCodec
  • fromMediaCodecTake the decoded data and render it directly toSurface
  • Stop playing and release resources

2.2 Specific Implementation

Audio player thread

The decoding process is described in detail in the chapter audio decoding

private static class AudioPlayThread extends Thread {
    private static final long TIMEOUT_MS = 2000L;
    private String path;
    private MediaExtractor mediaExtractor;
    private MediaCodec mediaCodec;
    private AudioTrack audioTrack;
    private MediaFormat format;
    private String mime;
    private boolean isStopPlay = false;
    public AudioPlayThread(String path) {
        this.path = path;
    }
    @Override
    public void run(a) {
        super.run();
        initMediaExtractor();
        initMediaCodec();
        initAudioTrack();
        play();
    }
    private void initMediaExtractor(a) {
        if (TextUtils.isEmpty(path)) {
            return;
        }
        try {
            mediaExtractor = new MediaExtractor();
            mediaExtractor.setDataSource(path);
            int trackCount = mediaExtractor.getTrackCount();
            for (int i = 0; i < trackCount; i++) {
                format = mediaExtractor.getTrackFormat(i);
                mime = format.getString(MediaFormat.KEY_MIME);
                if(! TextUtils.isEmpty(mime) && mime.startsWith("audio/")) {
                    mediaExtractor.selectTrack(i);
                    break; }}}catch (IOException e) {
            e.printStackTrace();
            mediaExtractor = null;
            format = null;
            mime = null; }}private void initMediaCodec(a) {
        if (format == null || TextUtils.isEmpty(mime)) {
            return;
        }
        try {
            mediaCodec = MediaCodec.createDecoderByType(mime);
            mediaCodec.configure(format, null.null.0);
        } catch (IOException e) {
            e.printStackTrace();
            mediaCodec = null; }}private void initAudioTrack(a) {
        if (format == null) {
            return;
        }
        int sampleRateInHz = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
        int channelConfig = AudioFormat.CHANNEL_OUT_STEREO;
        int audioFormat = AudioFormat.ENCODING_PCM_16BIT;
        int bufferSizeInBytes = AudioTrack.getMinBufferSize(
                sampleRateInHz,
                channelConfig, audioFormat
        );
        audioTrack = new AudioTrack(
                AudioManager.STREAM_MUSIC,
                sampleRateInHz,
                channelConfig,
                audioFormat,
                bufferSizeInBytes,
                AudioTrack.MODE_STREAM
        );
    }
    private void play(a) {
        if (mediaExtractor == null || mediaCodec == null || audioTrack == null) {
            return;
        }
        long startMs = System.currentTimeMillis();
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        mediaCodec.start();
        audioTrack.play();
        for(; ;) {if (isStopPlay) {
                release();
                break;
            }
            int inputBufferId = mediaCodec.dequeueInputBuffer(TIMEOUT_MS);
            if (inputBufferId >= 0) {
                ByteBuffer inputBuffer = mediaCodec.getInputBuffer(inputBufferId);
                int readSize = -1;
                if(inputBuffer ! =null) {
                    readSize = mediaExtractor.readSampleData(inputBuffer, 0);
                }
                if (readSize <= 0) {
                    mediaCodec.queueInputBuffer(
                            inputBufferId,
                            0.0.0,
                            MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                    isStopPlay = true;
                } else {
                    mediaCodec.queueInputBuffer(inputBufferId, 0, readSize, mediaExtractor.getSampleTime(), 0); mediaExtractor.advance(); }}int outputBufferId = mediaCodec.dequeueOutputBuffer(info, TIMEOUT_MS);
            if (outputBufferId >= 0) {
                ByteBuffer outputBuffer = mediaCodec.getOutputBuffer(outputBufferId);
                if(outputBuffer ! =null && info.size > 0) {
                    while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
                        try {
                            sleep(10);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                            break; }}byte[] data = new byte[info.size];
                    outputBuffer.get(data);
                    outputBuffer.clear();
                    audioTrack.write(data, 0, info.size);
                }
                mediaCodec.releaseOutputBuffer(outputBufferId, false); }}}void stopPlay(a) {
        isStopPlay = true;
        try {
            join(2000);
        } catch(InterruptedException e) { e.printStackTrace(); }}private void release(a) {
        if(mediaExtractor ! =null) {
            mediaExtractor.release();
            mediaExtractor = null;
        }
        if(mediaCodec ! =null) {
            mediaCodec.stop();
            mediaCodec.release();
            mediaCodec = null;
        }
        if(audioTrack ! =null) {
            audioTrack.stop();
            audioTrack.release();
            audioTrack = null; }}}Copy the code

The decoding process of the video player thread is described in detail in the chapter of video decoding

private static class VideoPlayThread extends Thread {
    private static final long TIMEOUT_MS = 2000L;
    private String path;
    private Surface surface;
    private MediaExtractor mediaExtractor;
    private MediaCodec mediaCodec;
    private MediaFormat format;
    private String mime;
    private boolean isStopPlay = false;
    public VideoPlayThread(String path, Surface surface) {
        this.path = path;
        this.surface = surface;
    }
    @Override
    public void run(a) {
        super.run();
        initMediaExtractor();
        initMediaCodec();
        play();
    }
    private void initMediaExtractor(a) {
        if (TextUtils.isEmpty(path)) {
            return;
        }
        try {
            mediaExtractor = new MediaExtractor();
            mediaExtractor.setDataSource(path);
            int trackCount = mediaExtractor.getTrackCount();
            for (int i = 0; i < trackCount; i++) {
                format = mediaExtractor.getTrackFormat(i);
                mime = format.getString(MediaFormat.KEY_MIME);
                if(! TextUtils.isEmpty(mime) && mime.startsWith("video/")) {
                    mediaExtractor.selectTrack(i);
                    break; }}}catch (IOException e) {
            e.printStackTrace();
            mediaExtractor = null;
            format = null;
            mime = null; }}private void initMediaCodec(a) {
        if (format == null || TextUtils.isEmpty(mime) || surface == null) {
            return;
        }
        try {
            mediaCodec = MediaCodec.createDecoderByType(mime);
            mediaCodec.configure(format, surface, null.0);
        } catch (IOException e) {
            e.printStackTrace();
            mediaCodec = null; }}private void play(a) {
        if (mediaExtractor == null || mediaCodec == null) {
            return;
        }
        long startMs = System.currentTimeMillis();
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        mediaCodec.start();
        for(; ;) {if (isStopPlay) {
                release();
                break;
            }
            int inputBufferId = mediaCodec.dequeueInputBuffer(TIMEOUT_MS);
            if (inputBufferId >= 0) {
                ByteBuffer inputBuffer = mediaCodec.getInputBuffer(inputBufferId);
                int readSize = -1;
                if(inputBuffer ! =null) {
                    readSize = mediaExtractor.readSampleData(inputBuffer, 0);
                }
                if (readSize <= 0) {
                    mediaCodec.queueInputBuffer(
                            inputBufferId,
                            0.0.0,
                            MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                    isStopPlay = true;
                } else {
                    mediaCodec.queueInputBuffer(inputBufferId, 0, readSize, mediaExtractor.getSampleTime(), 0mediaExtractor.advance(); }}int outputBufferId = mediaCodec.dequeueOutputBuffer(info, TIMEOUT_MS);
            if (outputBufferId >= 0) {
                ByteBuffer outputBuffer = mediaCodec.getOutputBuffer(outputBufferId);
                if(outputBuffer ! =null && info.size > 0) {
                    while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
                        try {
                            sleep(10);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                            break; }}byte[] data = new byte[info.size];
                    outputBuffer.get(data);
                    outputBuffer.clear();
                    // The resulting data is the YUV data, which can be used to do the corresponding business
                }
                mediaCodec.releaseOutputBuffer(outputBufferId, true); }}}void stopPlay(a) {
        isStopPlay = true;
        try {
            join(2000);
        } catch(InterruptedException e) { e.printStackTrace(); }}private void release(a) {
        if(mediaExtractor ! =null) {
            mediaExtractor.release();
            mediaExtractor = null;
        }
        if(mediaCodec ! =null) {
            mediaCodec.stop();
            mediaCodec.release();
            mediaCodec = null;
        }
        surface = null; }}Copy the code

Outside calls

External need to call, then need to create two thread objects, and start, in no need, remember to stop playing, and release resources

private AudioPlayThread audioPlayThread;
private VideoPlayThread videoPlayThread;
​
public void stat(String path, Surface surface) {
    stop();
    audioPlayThread = new AudioPlayThread(path);
    videoPlayThread = new VideoPlayThread(path, surface);
    audioPlayThread.start();
    videoPlayThread.start();
}
​
public void stop(a) {
    if(audioPlayThread ! =null) {
        audioPlayThread.stopPlay();
        audioPlayThread = null;
    }
    if(videoPlayThread ! =null) {
        videoPlayThread.stopPlay();
        videoPlayThread = null; }}Copy the code

Now, a simple video player is complete!!

Third, making

VideoPlayer.java

VideoPlayActivity.java