1. How MediaCodec works
MediaCodec class Android provides an interface for accessing the low-level multimedia codec. It is part of Android’s low-level multimedia architecture and is usually used in conjunction with MediaExtractor, MediaMuxer, AudioTrack, Can encode and decode such as H.264, H.265, AAC, 3GP and other common audio and video formats. Broadly speaking, MediaCodec works by processing input data to produce output data. Specifically, MediaCodec uses a set of input/output caches to process data synchronously or asynchronously during codec: First, the client writes the data to be encoded and decoded to the codec input cache and submits it to the codec. After the codec finishes processing, the data is stored in the output cache of the encoder, and the ownership of the input cache is recovered by the client. The client then reads the encoded data from the fetch to codec output cache for processing, after which the codec reclaims the client’s ownership of the output cache. The whole process is repeated until the encoder stops working or exits abnormally.
2. MediaCodec coding process
In the whole codec process, MediaCodec is configured, started, processed, Stopped and Released. The corresponding states can be Stopped, executed and Released. The Stopped state can be subdivided into Uninitialized, Configured, and Error. Subdivided into uploaded data, Running, and end-of-stream, the Executing state can also be subdivided into uploading data, Running, and end-of-stream. The overall state structure of MediaCodec is as follows:
From the above figure, when MediaCodec is created, it will enter the uninitialized state. After setting the configuration information and calling start(), MediaCodec will enter the running state and can read and write data. If an error occurs during this process, MediaCodec will go to the Stopped state, so we need to use the reset method to reset the codec, or the resources held by MediaCodec will eventually be released. Of course, if MediaCodec is working properly, we can also send EOS directives to the codec and call stop and release to terminate the codec.
2.1 Creating a codec
MediaCodec provides the createEncoderByType(String Type) and createDecoderByType(String Type) methods to create codecs, both of which require passing in a MIME type multimedia format. Common MIME types are in the following multimedia formats:
● "video/ X-vnd.on2. vp8" - VP8 video (i.e. video in.webm) ● "Video/X-vnd.on2. vp9" - VP9 video (i.e. video in.webm) ● "Video /AVC" - H.264/AVC video ● "VIDEO/MP4V-ES" - MPEG4 video ● "video/ 3GPP "- H.264 video ●" Audio / 3GPP "- AMR Narrowband Audio ● "audio/ AMR-WB "-AMR Wideband audio ● "audio/ MPEG" -MPEg1/2 Audio Layer III ● "Audio/MP4A-latm" - AAC audio (note, this is raw AAC packets, not packaged in LATM!) ● "audio/g711-mlaw" -g.711 ulaw audio ● "audio/g711-mlaw" -g.711 ulaw audio ● "g711-mlaw" -g.711 ulaw audioCopy the codeOf course, MediaCodec also provides a createByCodecName (String Name) method that allows you to create a codec using the specific name of the component. However, this method is a bit cumbersome to use, and the official recommendation is to use it in conjunction with MediaCodecList, which keeps track of all available codecs. Of course, we can also use this class to check the minmeType argument passed in to see if MediaCodec supports the mineType codec. For example, if the MIME type is video/avc, the code is as follows:
private static MediaCodecInfo selectCodec(String mimeType) {
// Get the number of supported codecs
int numCodecs = MediaCodecList.getCodecCount();
for (int i = 0; i < numCodecs; i++) {
// Codec correlation information is stored in MediaCodecInfo
MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
// Check whether it is an encoder
if(! codecInfo.isEncoder()) {continue;
}
// Get the MIME type supported by the encoder and match it
String[] types = codecInfo.getSupportedTypes();
for (int j = 0; j < types.length; j++) {
if (types[j].equalsIgnoreCase(mimeType)) {
returncodecInfo; }}}return null;
}
Copy the code2.2 Configuring and Enabling the codec
The codec configuration uses MediaCodec’s configure method, which first extracts the map of data stored in MediaFormat and then calls the local method native_configure to configure the codec. The format, surface, crypto, and flags parameters are passed into the configure method. Format is an instance of MediaFormat, which stores multimedia data format information in the form of key-value pairs. Surface is used to indicate that the data source for the decoder comes from the surface; Crypto is used to specify a MediaCrypto object for secure decryption of media data. Flags indicates that an encoder (CONFIGURE_FLAG_ENCODE) is configured.
MediaFormat mFormat = MediaFormat.createVideoFormat("video/avc".640 ,480); / / create MediaFormat
mFormat.setInteger(MediaFormat.KEY_BIT_RATE,600); // Specify the bit rate
mFormat.setInteger(MediaFormat.KEY_FRAME_RATE,30); // Specify the frame rate
mFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,mColorFormat); // Specify the encoder color format
mFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL,10); // Specify the keyframe interval
mVideoEncodec.configure(mFormat,null.null,MediaCodec.CONFIGURE_FLAG_ENCODE);
Copy the codeCreateVideoFormat (” VIDEO/AVC “, 640,480) createVideoFormat(” VIDEO/AVC “, 640,480) createVideoFormat(” AVC “, 640,480) If audio data is encoded or decoded, the createAudioFormat(String MIME,int sampleRate,int channelCount) method of MediaFormat is called. In addition to some configuration parameters such as video frame rate and audio sampling rate, the mediaformat. KEY_COLOR_FORMAT configuration attribute is used to specify the color format of the video encoder. The specific color format selected depends on the input video data source color format. For example, we all know that the image flow collected by Camera preview is usually NV21 or YV12, so the encoder needs to specify the corresponding color format, otherwise the encoded data may appear splash screen, overlapping, color distortion and other phenomena. MediaCodecInfo.CodecCapabilities. All color formats supported by the encoder are stored. Common color formats are mapped as follows:
Raw NV12 data encoder (YUV420sp) -- -- -- -- -- -- -- -- -- > COLOR_FormatYUV420PackedSemiPlanar NV21 -- -- -- -- -- -- -- -- -- -- > COLOR_FormatYUV420SemiPlanar YV12(I420) ----------> COLOR_FormatYUV420PlanarCopy the codeWhen the codec is configured, MediaCodec’s start() method can be called, which invokes the low-level native_start() method to start the encoder, The low-level method ByteBuffer[] getBuffers(input) is called to create a series of input and output buffers. The start() method is as follows:
public final void start(a) {
native_start();
synchronized(mBufferLock) {
cacheBuffers(true /* input */);
cacheBuffers(false /* input */); }}Copy the code2.3 Data Processing
MediaCodec supports two modes of codecs, synchronous and asynchronous. Synchronous mode means that the input and output of the codec data are synchronous and the codec will receive the input data again only after the output is processed. The input and output of asynchronous codec data are asynchronous, and the codec does not wait for the output data to finish processing before receiving the input data again. Here, we mainly introduce synchronous codec, because this way we use more. We know that when the codec is after the start, each codec will have a set of input and output buffering, but these buffer temporarily unable to be used, only through MediaCodec dequeueInputBuffer/dequeueOutputBuffer methods for input/output buffer authorization, These caches are manipulated by the returned ID. The following is an extended analysis of the code provided by the official:
MediaCodec codec = MediaCodec.createByCodecName(name); The codec. The configure (format,...). ; MediaFormat outputFormat = codec.getOutputFormat();// option B
codec.start();
for (;;) {
int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
if (inputBufferId >= 0) {ByteBuffer inputBuffer = codec.getinputBuffer (...) ;// fill inputBuffer with valid data... Codec. QueueInputBuffer (inputBufferId,...). ; }intOutputBufferId = codec. DequeueOutputBuffer (...). ;if (outputBufferId >= 0) {
ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
// bufferFormat is identical to outputFormat
// outputBuffer is ready to be processed or rendered.... Codec. ReleaseOutputBuffer (outputBufferId,...). ; }else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
// Can ignore if using getOutputFormat(outputBufferId)
outputFormat = codec.getOutputFormat(); // option B
}
}
codec.stop();
codec.release();
Copy the codeWhen the codec starts, it enters a for(;). Loop, which is an infinite loop to continuously fetch a cache containing data from the codec’s input cache pool, and then fetch codec output data from the output cache pool.
- Gets the input cache of the codec and writes data
First, MediaCodec’s dequeueInputBuffer(Long timeoutUs) method is called to retrieve an input buffer from the encoder’s set of input buffers and return index. If index=-1, the buffer is temporarily available. When timeoutUs=0, dequeueInputBuffer returns immediately. MediaCodec’s getInputBuffer(int index) is then called, which passes the index to the local getBuffer(true /* input */, index) and returns the ByteBuffer of that buffer. The obtained ByteBuffer and its index are stored in a BufferMap object so that the buffer can be released and returned to the codec after input. GetInputBuffer (int index)
@Nullable
public ByteBuffer getInputBuffer(int index) {
ByteBuffer newBuffer = getBuffer(true /* input */, index);
synchronized(mBufferLock) {
invalidateByteBuffer(mCachedInputBuffers, index);
// mDequeuedInputBuffers is an instance of BufferMap
mDequeuedInputBuffers.put(index, newBuffer);
}
return newBuffer;
}
Copy the codeThen, after the input buffer is obtained, data is filled into the data and submitted to the codec for processing using queueInputBuffer, while the input cache is released back to the codec. QueueInputBuffer is queueInputBuffer.
public final void queueInputBuffer(
int index,
int offset, int size, long presentationTimeUs, int flags)
throws CryptoException {
synchronized(mBufferLock) {
invalidateByteBuffer(mCachedInputBuffers, index);
// Remove the input cache
mDequeuedInputBuffers.remove(index);
}
try {
native_queueInputBuffer(
index, offset, size, presentationTimeUs, flags);
} catch (CryptoException | IllegalStateException e) {
revalidateByteBuffer(mCachedInputBuffers, index);
throwe; }}Copy the codeQueueInputBuffer is implemented by calling the low-level method native_queueInputBuffer. This method requires five parameters, index is the subscript of the input buffer, and the codec uses index to find the location of the buffer. Offset is the offset of valid data stored in the buffer. Size is the size of the valid input original data. PresentationTimeUs displays a timestamp for the buffer, usually 0; Flags is the input cache flag, usually set to BUFFER_FLAG_END_OF_STREAM.
- Gets the output cache of the codec and reads the data
First, similar to getting the input buffer through dequeueInputBuffer and getInputBuffer, MediaCodec also provides dequeueOutputBuffer and getOutputBuffer methods to help us get the output buffer of the codec. But unlike dequeueInputBuffer, dequeueOutputBuffer also needs to pass in a Mediacodec.bufferInfo object. Mediacodec.bufferinfo is an internal class of MediaCodec that records the offset and size of the codec data in the output cache.
public final static class BufferInfo {
public void set(
int newOffset, int newSize, long newTimeUs, @BufferFlag int newFlags) {
offset = newOffset;
size = newSize;
presentationTimeUs = newTimeUs;
flags = newFlags;
}
public int offset / / the offset
public int size; // Cache valid data size
public long presentationTimeUs; // Display the timestamp
public int flags; // Cache flag
@NonNull
public BufferInfo dup(a) {
BufferInfo copy = new BufferInfo();
copy.set(offset, size, presentationTimeUs, flags);
returncopy; }};Copy the codeHowever, the dequeueOutputBuffer source code shows that the output buffer is valid only if the dequeueOutputBuffer returns a value greater than =0. When a call to the local method native_dequeueOutputBuffer returns INFO_OUTPUT_BUFFERS_CHANGED, The cacheBuffers method is called to retrieve a new set of output buffers mCachedOutputBuffers(ByteBuffer[]). This explains that if we use the getOutputBuffers method (deprecated after API21, using getOutputBuffer(index) instead) to get the output buffer of the codec, we need to determine its return value by calling dequeueOutputBuffer. If the return value is mediacodec.info_output_buffers_CHANGED, you need to retrieve the output cache collection again. In addition, there are two more return values for dequeueOutputBuffer: INFO_TRY_AGAIN_LATER and mediacodec. INFO_OUTPUT_FORMAT_CHANGED: the former indicates that the codec output cache timeout is obtained, and the latter indicates that the codec data output format is changed and subsequent data output will use the new format. Therefore, we need to determine whether the returned value is INFO_OUTPUT_FORMAT_CHANGED when we call dequeueOutputBuffer, and we need to reset the MediaFormt object with MediaCodec’s getOutputFormat.
public final int dequeueOutputBuffer(
@NonNull BufferInfo info, long timeoutUs) {
int res = native_dequeueOutputBuffer(info, timeoutUs);
synchronized(mBufferLock) {
if (res == INFO_OUTPUT_BUFFERS_CHANGED) {
// The getBuffers() underlying method will be called
cacheBuffers(false /* input */);
} else if (res >= 0) {
validateOutputByteBuffer(mCachedOutputBuffers, res, info);
if(mHasSurface) { mDequeuedOutputInfos.put(res, info.dup()); }}}return res;
}
Copy the codeFinally, when the data in the output buffer has been processed, the output buffer is released by calling MediaCodec’s releaseOutputBuffer and returned to the codec. The output buffer cannot be used until the next time it is retrieved via dequeueOutputBuffer. The releaseOutputBuffer method takes two arguments: Index and render, where Index is the output buffer Index. Render: When surface is specified when encoder is configured, it should be set to true and output cache data will be passed to Surface. The source code is as follows:
public final void releaseOutputBuffer(int index, boolean render) {
BufferInfo info = null;
synchronized(mBufferLock) {
invalidateByteBuffer(mCachedOutputBuffers, index);
mDequeuedOutputBuffers.remove(index);
if (mHasSurface) {
info = mDequeuedOutputInfos.remove(index);
}
}
releaseOutputBuffer(index, render, false /* updatePTS */.0 /* dummy */);
}
Copy the code