In order to provide the ultimate video playback experience, MediaCodec provides two modes: synchronous mode and asynchronous mode.

MediaCodec Asynchronous mode

Android L and Android L and above provide asynchronous mode, The data processing is done in asynchronous threads by setting callback in codec– >configure, and then processing the decoded data in callbacks for the developer, where you can process the raw data according to your needs.

    1. To configure the MediaCodec callback, configure the MediaCodec callback interface. OnInputBufferAvailable, onOutputBufferAvailable, onOutputFormatChanged, onError, MediaCodec uses these four callback methods to automatically inform clients when the input buffer is valid, when the output buffer is valid, when the media format changes, and when the operation fails. It is also in these methods that the Client sends data to the Codec and gets the results of the processing and some other information about the Codec.
    1. In asynchronous mode, MediaCodec transitions slightly differently, going straight to Running after calling the start method.

In asynchronous processing mode, Codec immediately enters the Running sub-state after mediacodec.start () is called. The set callback method onInputBufferAvailable() will automatically receive the available (empty) input buffer, Call getInputBuffer(ID) based on the input buffer ID to get the buffer, write the data to the buffer, and finally call queueInputBuffer(ID,…). Submit the buffer to Codec for processing; Codec writes the result of each frame to an empty output buffer and notifies the Client to read the result via the onOutputBufferAvailable callback. The Client can call getOutputBuffer(ID) based on the output bufffer ID to obtain the buffer and read the result. ReleaseOutputBuffer (ID,…) Release the buffer to Codec for reuse.

MediaCodec codec = MediaCodec.createByCodecName(name); MediaFormat mOutputFormat; // member variable codec.setCallback(new MediaCodec.Callback() { @Override void onInputBufferAvailable(MediaCodec mc, int inputBufferId) { ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId); // Fill inputBuffer with valid data... Codec. QueueInputBuffer (inputBufferId,...). ; } @Override void onOutputBufferAvailable(MediaCodec MC, int outputBufferId...) { ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId); MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // Option A // bufferFormat is equivalent to mOutputFormat // outputBuffer is ready to be processed or rendered. Codec. ReleaseOutputBuffer (outputBufferId,...). ; } @Override void onOutputFormatChanged(MediaCodec mc, MediaFormat format) { // Subsequent data will conform to new format. // Can ignore if using getOutputFormat(outputBufferId) mOutputFormat = format; // Override void onError(...) {... }}); The codec. The configure (format,...). ; mOutputFormat = codec.getOutputFormat(); // option B codec.start(); // wait for processing to complete codec.stop(); codec.release();Copy the code

Codec.start () performs the core processing in asynchronous threads, so it is also called asynchronous mode

MediaCodec Synchronization mode

Synchronous mode, of course, does not start another thread to decode the data, and after codec.start() is executed, the InputBuffer and OuputBuffer are retrieved from the codec instance.

  • 1. In synchronous mode, after MediaCodec calls the start() method, the uploading state becomes Flushed. Then, after dequeueInputBuffer() is called for the first time, the uploading state becomes Running.
  • 2. In this mode, the program needs to run in an infinite loop by calling dequeueInputBuffer(…). And dequeueOutputBuffer (…). To continuously request Codec if an input buffer or output buffer is available:
  • 3. If there is an input buffer available, getInputBuffer(ID) is called to get the buffer and write data to it, and queueInputBuffer(ID,..) is called. Submit to Codec for processing
  • 4. If any output buffer is available, call getOutputBuffer(ID) to obtain the buffer id, read the processing result, and then call releaseOutputBuffer(ID,..). Release the buffer for reuse by Codec
  • 5. Buffer ids that may be marked during processing, such as mediacodec.info_output_format_CHANGED, should be treated appropriately
MediaCodec codec = MediaCodec.createByCodecName(name); The codec. The configure (format,...). ; MediaFormat outputFormat = codec.getOutputFormat(); // option B codec.start(); for (;;) { int inputBufferId = codec.dequeueInputBuffer(timeoutUs); If (inputBufferId >= 0) {ByteBuffer inputBuffer = codec.getinputBuffer (...) ; // Fill inputBuffer with valid data... Codec. QueueInputBuffer (inputBufferId,...). ; } int outputBufferId = codec. DequeueOutputBuffer (...). ; if (outputBufferId >= 0) { ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId); MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // Option A // bufferFormat is identical to outputFormat // outputBuffer is ready to be processed or rendered.... Codec. ReleaseOutputBuffer (outputBufferId,...). ; } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) { // Subsequent data will conform to new format. // Can ignore if using getOutputFormat(outputBufferId) outputFormat = codec.getOutputFormat(); // option B } } codec.stop(); codec.release();Copy the code

In order to improve the rendering performance of ExoPlayer, we have two core ideas:

  • MediaCodec switches from synchronous mode to asynchronous mode
  • The player submits the input buffer to MediaCodec in a separate thread

Application of asynchronous mode in ExoPlayer

The mediacodecrenderer. Java initializes the MediaCodec function, which creates synchronous or asynchronous codec based on the mode set.

codec = MediaCodec.createByCodecName(codecName); if (mediaCodecOperationMode == OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD && Util.SDK_INT >= 23) { codecAdapter = new AsynchronousMediaCodecAdapter(codec, getTrackType()); } else if (mediaCodecOperationMode == OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_ASYNCHRONOUS_QUEUEING && Util.SDK_INT  >= 23) { codecAdapter = new AsynchronousMediaCodecAdapter( codec, /* enableAsynchronousQueueing= */ true, getTrackType()); } else { codecAdapter = new SynchronousMediaCodecAdapter(codec); }Copy the code

AsynchronousMediaCodecAdapter. Java inheritance MediaCodec. Callback, realize MediaCodec decoding asynchronous mode, at the same time to define a special HandlerThread, used to process the decoding messages. In AsynchronousMediaCodecAdapter. At the same time defined in Java a buffer queue to deal with the codec decoding of the raw data.

  @Override
  public void onInputBufferAvailable(MediaCodec codec, int index) {
    synchronized (lock) {
      mediaCodecAsyncCallback.onInputBufferAvailable(codec, index);
    }
  }

  @Override
  public void onOutputBufferAvailable(MediaCodec codec, int index, MediaCodec.BufferInfo info) {
    synchronized (lock) {
      mediaCodecAsyncCallback.onOutputBufferAvailable(codec, index, info);
    }
  }

Copy the code

After calling onInputBufferAvailable, the free input buffer is queued, and then the dequeueInputBufferIndex obtains the corresponding input buffer position, writes the raw data to it, and begins decoding. After decoding, the onOutputBufferAvailable callback gets the decoded raw data. After reading the decoded raw data,Codec needs to release the output buffer. Free up space for subsequent decoding data writing.

According to the analysis of the process before and after the improvement, the improved process saves the time of waiting for audio parsing, and can immediately render the video stream data in the asynchronous thread, thus improving the rendering performance.