Hello everyone, I’m Kenney, a programmer. Today, I’m going to tell you how to do video decoding and rendering on Android. Video decoding has a variety of methods, today to introduce to you is to use Android with its own MediaCodec hard decoding, the so-called hard decoding is the use of hardware decoding, fast speed, and the opposite is soft decoding, slow speed, but good compatibility. MediaCodec video decoding is based on producer/consumer mode. There are some buffers in it. When a frame needs to be decoded, a buffer is taken out and filled with data, and then sent to the buffer for decoding.

Let’s take a look at how to achieve video hard decoding and rendering step by step:

1. Create a Surface

If you’re using a SurfaceView, it comes with a surface, and if you’re directly decoding it, it will automatically show up. In this article, because it involves rendering, I’m decoding it on a surface I created myself. The surface is created from the Surface Texture, which in turn is created from an oes Texture, so it ends up decoding to a texture that can then be rendered using OpenGL.

2. Initialize MediaExtractor and MediaCodec

MediaExtractor is used to extract data from the video file, the data to fill the buffer mentioned above comes from this, the initial chemical operation is mainly to set the video file path for it, and select the track, this article explains the video decoding, because only care about the video track, the sound is ignored.

mediaExtractor = MediaExtractor()
mediaExtractor.setDataSource(filePath)
val trackCount = mediaExtractor.getTrackCount()
for (i in 0 until trackCount) {
    val trackFormat = mediaExtractor.getTrackFormat(i)
    val mime = trackFormat.getString(MediaFormat.KEY_MIME)
    if (mime.contains("video")) {
        videoTrackIndex = i
        break}}if (videoTrackIndex == - 1) {
    mediaExtractor.release()
    return
}
mediaExtractor.selectTrack(videoTrackIndex)
Copy the code

Then we initialize MediaCodec. As you can see, we’ll pass it a surface. Once it’s initialized, let it work:

mediaCodec = MediaCodec.createDecoderByType(videoMime)
mediaCodec.configure(videoFormat, surface, null.0)
mediaCodec.start()
Copy the code

3. Read and decode data

This step is slightly more complicated. As mentioned earlier, MediaCodec decodes video based on producer/consumer mode. We first ask the dequeueInputBuffer for a buffer to hold the data to be decoded.

val inputBufferIndex = mediaCodec.dequeueInputBuffer(10000)
if (inputBufferIndex >= 0) {
	val buffer = mediaCodec.getInputBuffers()[inputBufferIndex]
}
Copy the code

It then reads the data from the video file. If the length of the data read is less than zero, it has read all the data. If it does not read BUFFER_FLAG_END_OF_STREAM, it will fill the buffer. QueueInputBuffer is then sent back to MediaCodec and the read location of MediaExtractor goes forward:

val sampleSize = mediaExtractor.readSampleData(buffer, 0)
if (sampleSize < 0) {
    mediaCodec.queueInputBuffer(inputBufferIndex, 0.0.0, MediaCodec.BUFFER_FLAG_END_OF_STREAM)
    eos = true
} else {
    mediaCodec.queueInputBuffer(inputBufferIndex, 0, sampleSize, mediaExtractor.sampleTime, 0)
    mediaExtractor.advance()
}
Copy the code

If the buffer has a BUFFER_FLAG_END_OF_STREAM flag, then the decoding is complete:

val outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 10000)
if((bufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM) ! =0) {
	return false
}
Copy the code

Some of the results obtained from the dequeueOutputBuffer are poorly decoded. For decoded cases, the buffer is returned via releaseOutputBuffer:

when (outputBufferIndex) {
    MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED, 
    MediaCodec.INFO_OUTPUT_FORMAT_CHANGED, 
    MediaCodec.INFO_TRY_AGAIN_LATER -> {
    }
    else -> {
        mediaCodec.releaseOutputBuffer(outputBufferIndex, true)
        return true}}Copy the code

ReleaseOutputBuffer the second parameter, if true, will render to the surface. The surface texture used to construct the surface will receive the onFrameAvailable() callback, so we know that a frame is decoded. Call the Surface Texture updateTexImage() method to update the decoded data to the texture. Once you have that texture, you can render it in OpenGL the same way you did in the previous OpenGL tutorial. Remember to stop MediaCodec and release the related resources.

See my sample code for details: github.com/kenneycode/…

Thanks for reading!