# MediaCodec fundamentals and usage

Transfer: Basic principles and use of MeidaCodec

Author: Hong Tao is reflecting

MediaCodec class Android provides an interface for accessing the low-level multimedia codec. It is part of Android’s low-level multimedia architecture and is usually used in conjunction with MediaExtractor, MediaMuxer, AudioTrack, The ability to encode and decode common audio and video formats such as H.264, H.265, AAC, and 3GP. Broadly speaking, MediaCodec works by processing input data to produce output data. Specifically, MediaCodec uses a set of input/output caches to process data synchronously or asynchronously during codec: First, the client writes the data to be encoded and decoded to the codec input cache and submits it to the codec. After the codec finishes processing, the data is stored in the output cache of the encoder, and the ownership of the input cache is recovered by the client. The client then reads the encoded data from the fetch to codec output cache for processing, after which the codec reclaims the client’s ownership of the output cache. The whole process is repeated until the encoder stops working or exits abnormally.

# MediaCodec encoding process

In the whole codec process, MediaCodec is configured, started, processed, Stopped and Released. The corresponding states can be Stopped, executed and Released. The Stopped state can be subdivided into Uninitialized, Configured, and Error. Subdivided into uploaded data, Running, and end-of-stream, the Executing state can also be subdivided into uploading data, Running, and end-of-stream. The overall state structure of MediaCodec is as follows:

From the above figure, when MediaCodec is created, it will enter the uninitialized state. After setting the configuration information and calling start(), MediaCodec will enter the running state and can read and write data. If an error occurs during this process, MediaCodec will go to the Stopped state, so we need to use the reset method to reset the codec, or the resources held by MediaCodec will eventually be released. Of course, if MediaCodec is working properly, we can also send EOS directives to the codec and call stop and release to terminate the codec.

# MediaCodec Api:

MediaCodec can handle specific video streams in these ways

  • GetInputBuffers: Gets the queue of input streams that need to encode data, returning a ByteBuffer array
  • QueueInputBuffer: Input flows into the queue
  • DequeueInputBuffer: Encodes data from the input stream queue
  • GetOutputBuffers: Gets the codec data output stream queue, returning a ByteBuffer array
  • DequeueOutputBuffer: Fetch the data from the output queue after the encoding operation
  • ReleaseOutputBuffer: ByteBuffer data is released after processing

# steps:

Create encoder

MediaCodec provides the createEncoderByType(String Type) and createDecoderByType(String Type) methods to create codecs, both of which require passing in a MIME type multimedia format. Common MIME types are in the following multimedia formats:

  • “Video/X-vnd.on2. vp8” – VP8 Video (i.e. Video in.webm)
  • “Video/X-vnd.on2. vp9” – VP9 Video (i.e. Video in.webm)
  • “Video/avc” – h. 264 / avc video
  • “Video/mp4v – es” – MPEG4 video
  • “Video / 3 GPP” – h. 263 video
  • “Audio / 3GPP” – AMR Narrowband Audio
  • “Audio/AMR-WB” – AMR wideband audio
  • Audio/MPEG – MPEg1/2 Audio Layer III
  • “Audio/MP4A-latm” -AAC audio (Note, this is raw AAC packets, not packaged in latm!
  • “Audio/vorbis” – vorbis audio
  • “Audio/G711-alaw” – G.711 alaw Audio
  • “Audio /g711-mlaw” -G.711 ulaw Audio

Of course, MediaCodec also provides a createByCodecName (String Name) method that allows you to create a codec using the specific name of the component. However, this method is a bit cumbersome to use, and the official recommendation is to use it in conjunction with MediaCodecList, which keeps track of all available codecs. Of course, we can also use this class to check the minmeType argument passed in to see if MediaCodec supports the mineType codec. For example, if the MIME type is video/avc, the code is as follows:

private static MediaCodecInfo selectCodec(String mimeType) {
     // Get the number of supported codecs
     int numCodecs = MediaCodecList.getCodecCount();
     for (int i = 0; i < numCodecs; i++) {
        // Codec correlation information is stored in MediaCodecInfo
         MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
         // Check whether it is an encoder
         if(! codecInfo.isEncoder()) {continue;
         }
        // Get the MIME type supported by the encoder and match it
         String[] types = codecInfo.getSupportedTypes();
         for (int j = 0; j < types.length; j++) {
             if (types[j].equalsIgnoreCase(mimeType)) {
                 returncodecInfo; }}}return null;
 }
Copy the code

Configure the boot codec

The codec configuration uses MediaCodec’s Configure method, which first extracts the data map stored by MediaFormat, and then calls native configure to configure the codec. During configuration, the configure method needs to pass in format, surface, crypto, and flags parameters. Format is an instance of MediaFormat, which stores multimedia data format information in the form of key-value pairs. Surface is used to indicate that the data source for the decoder comes from the surface; Crypto is used to specify a MediaCrypto object for secure decryption of media data. Flags indicates that an encoder (CONFIGURE_FLAG_ENCODE) is configured.

MediaFormat mFormat = MediaFormat.createVideoFormat("video/avc".640 ,480);     / / create MediaFormat
mFormat.setInteger(MediaFormat.KEY_BIT_RATE,600);       // Specify the bit rate
mFormat.setInteger(MediaFormat.KEY_FRAME_RATE,30);  // Specify the frame rate
mFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,mColorFormat);  // Specify the encoder color format
mFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL,10); // Specify the keyframe interval
mVideoEncodec.configure(mFormat,null.null,MediaCodec.CONFIGURE_FLAG_ENCODE); 

Copy the code

CreateVideoFormat (” VIDEO/AVC “, 640,480) is a MediaFormat object of the “video/ AVC” type (i.e. H.264) encoder. If audio data is encoded or decoded, the createAudioFormat(String MIME,int sampleRate,int channelCount) method of MediaFormat is called. In addition to some configuration parameters such as video frame rate and audio sampling rate, the mediaformat. KEY_COLOR_FORMAT configuration attribute is used to specify the color format of the video encoder. The specific color format selected depends on the input video data source color format. For example, we all know that the image flow collected by Camera preview is usually NV21 or YV12, so the encoder needs to specify the corresponding color format, otherwise the encoded data may appear splash screen, overlapping, color distortion and other phenomena. MediaCodecInfo.CodecCapabilities. All color formats supported by the encoder are stored. Common color formats are mapped as follows:

Raw NV12 data encoder (YUV420sp) -- - > COLOR_FormatYUV420PackedSemiPlanar NV21 - - - - > COLOR_FormatYUV420SemiPlanar YV12 (I420) -- - >  COLOR_FormatYUV420PlanarCopy the code

When the codec is configured, MediaCodec’s start() method can be called, which calls the low-level native_start() method to start the encoder and the low-level ByteBuffer[] getBuffers(input) method to create a series of input and output buffers. The start() method is as follows:

public final void start(a) {
        native_start();
        synchronized(mBufferLock) {
            cacheBuffers(true /* input */);
            cacheBuffers(false /* input */); }}Copy the code

# Data processing

MediaCodec supports two modes of codecs, synchronous and asynchronous. Synchronous mode means that the input and output of the codec data are synchronous and the codec will receive the input data again only after the output is processed. The input and output of asynchronous codec data are asynchronous, and the codec does not wait for the output data to finish processing before receiving the input data again. Here, we mainly introduce synchronous codec, because this way we use more. We know that when the codec is after the start, each codec will have a set of input and output buffering, but these buffer temporarily unable to be used, only through MediaCodec dequeueInputBuffer/dequeueOutputBuffer methods for input/output buffer authorization, These caches are manipulated by the returned ID. The following is an extended analysis of the code provided by the official:

MediaCodec codec = MediaCodec.createByCodecName(name); The codec. The configure (format,...). ; MediaFormat outputFormat = codec.getOutputFormat();// option B
 codec.start();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
   if (inputBufferId >= 0) {ByteBuffer inputBuffer = codec.getinputBuffer (...) ;// fill inputBuffer with valid data... Codec. QueueInputBuffer (inputBufferId,...). ; }intOutputBufferId = codec. DequeueOutputBuffer (...). ;if (outputBufferId >= 0) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is identical to outputFormat
     // outputBuffer is ready to be processed or rendered.... Codec. ReleaseOutputBuffer (outputBufferId,...). ; }else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     outputFormat = codec.getOutputFormat(); // option B
   }
 }
 codec.stop();
 codec.release();
Copy the code

When the codec starts, it enters a for(;). Loop, which is an infinite loop to continuously fetch a cache containing data from the codec’s input cache pool, and then fetch codec output data from the output cache pool