How to draw a picture on Android platform

【Android audio and video learning road (2) 】AudioRecord API detailed explanation and application

【Android audio and video learning road (3) 】AudioTrack use and detailed explanation

【Android Audio and Video learning road (4) 】 The use of Camera

【Android audio and video learning road (five) 】MediaExtractor and MediaMuxer explanation

preface

After the previous round of Basic Android audio and video related knowledge, and sorting out the related API, we should have a certain outline of the basic audio and video, the following start to contact a fairly important Android audio and video API: MediaCodec. With this API, we can do a lot of Android audio and video work. When learning this API, we aim for the following two things:

  • Learn MediaCodec API, complete audio AAC hard coding, hard solution
  • Learn MediaCodec API, complete video H.264 hard coding, hard solution

1. Introduction to MediaCodec

The MediaCodec class can be used to use some basic multimedia codecs (audio and video codec components), It is part of Android’s basic multimedia support infrastructure and is usually associated with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, Used together with AudioTrack.

A codec can process input data to produce output data. The codec uses a set of input and output buffers to process data asynchronously. You can create an empty input buffer, fill it with data and send it to the codec for processing. The codec uses the input data to convert and output to an empty output buffer. Finally you get the data from the output buffer, consume the data in it, and release it back to the codec. If there is more data to process, the codec will repeat these operations. The output process is as follows:

Supported data types:

The encoder can process raw audio data and raw video data. It is possible to process these three types of data using ByteBuffers, but you will need to provide a Surface to display the raw video data, which will also improve the performance of the codec. Surface uses a local video buffer, which is not mapped or copied to ByteBuffers. This mechanism makes the codec more efficient. Normally with Surface, you can’t access the raw video data, but you can use ImageReader to access the decoded original video frames. In the mode using ByteBuffer, you can use the Image class and getInput/OutputImage (int) to access the original video frame.

Codec life cycle:

The main life cycles are Stopped, Executing, Released.

  • In the Stopped state, there are three seed states: Uninitialized, Configured, and Error.
  • Executing three uploading states: uploading, Running, and end-stream.

Here is an illustration of the life cycle:

As can be seen in the figure:

  1. The codec was in an uninitialized state when it was created. First you need to call configure(…). Method to put it in the Configured state and then call the start() method to put it in the Executing state. In Executing state, you can use the above buffer to process the data.

  2. Executing three uploading states: uploading, Running, and end-stream. After the start() call, the codec is in the Flushed state that stores all of its buffers. Once the first input buffer is present, the codec automatically runs to the Running state. When a buffer with the end-of-stream flag is inserted, the codec enters the end-stream state, in which the codec no longer receives an input buffer but still generates an output buffer. You can then call flush() to reset the Flushed codec as it appears.

  3. The call to stop() returns the codec to its uninitialized state, which can then be reconfigured. After you have finished using the codec, you must release it by calling release().

  4. In rare cases, the codec may encounter an error and go to an error state. This is communicated using invalid return values from queued operations or sometimes through exceptions. Call reset() to make the codec available again. You can call it from any state to move the codec back to its uninitialized state. Otherwise, call Release () to move to the terminal release state.

2. Description of MediaCodec API

MediaCodec can handle specific video streams in the following ways:

  • DequeueInputBuffer (timeoutUs): Obtains the ByteBuffer subscript of the input stream queue. TimeoutUs indicates delay. 0 indicates immediate return
  • GetInputBuffers (index): Retrieves the input stream queue and returns a ByteBuffer
  • QueueInputBuffer (): Input flows into a queue
  • DequeueOutputBuffer () : Retrieves data from the output queue after the encoding operation and returns the output stream queue subscript
  • GetOutputBuffer (index) : obtain the data after decoding output stream queue and returns a ByteBuffer, need has been circulating load, until dequeueOutputBuffer () returns the index of the less than 0
  • ReleaseOutputBuffer: The processing is complete and the ByteBuffer data is released

Three, MediaCodec flow control

3.1 Basic concepts of flow control

Flow control is flow control. Why should control, because the condition is limited! TCP and video coding are involved:

For TCP, it is to control the amount of data sent per unit time. For encoding, it is to control the amount of data output per unit time.

  • TCP is limited by network bandwidth. Flow control uses network bandwidth as much as possible without causing or aggravating network congestion. If the bandwidth is enough and the network is good, we will speed up the transmission of packets. After the delay increases and the loss of packets, we will slow down the transmission of packets (because continuing to send packets at a high speed may aggravate network congestion and send packets more slowly).
  • With the development of CODEC, the decoding capacity is no longer the bottleneck, but the transmission bandwidth/file size. We want to control the amount of data, and the picture quality is as high as possible.

Generally, encoders can set a target bit rate, but the actual output bit rate of the encoder will not fully conform to the setting, because in the coding process, what can actually be controlled is not the final output bit rate, but a Quantization Parameter (QP) in the coding process, which has no fixed relationship with the bit rate. It depends on the content of the image.

Whether it’s a TCP packet to be sent or an image to be encoded, there can be “spikes,” which are large amounts of data in a short period of time. TCP can resist spikes (especially if the network is already congested), which is fine, but if the video code also resists spikes, the image quality will suffer. If you have a few very large frames, but you still have to keep the bit rate at the original level, you have to lose more information, so the image will be more distorted.

3.2 Android hardcoded flow control

There are not many interfaces related to MediaCodec flow control. One is to set the target bit rate and the bit rate control mode during configuration, and the other is to dynamically adjust the target bit rate (Android 19 or above).

Specify the target bit rate and the bit rate control mode:

mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate)
mediaFormat.setInteger(MediaFormat.KEY_BITRATE_MODE,MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR)
videoCodec.configure(mediaFormat, null, null,MediaCodec.CONFIGURE_FLAG_ENCODE)
Copy the code

There are three bit rate control modes:

  • CQ means that the code rate is not controlled at all and the image quality is guaranteed as much as possible;
  • CBR means that the encoder will try to control the output bit rate to the set value, that is, we mentioned before “unmoved”;
  • VBR means that the encoder will dynamically adjust the output bit rate according to the complexity of the image content (actually, the size of the inter-frame variation). If the image is complex, the bit rate is high; if the image is simple, the bit rate is low;

Dynamically adjust the target bit rate:

val param = Bundle();
param.putInt(MediaCodec.PARAMETER_KEY_VIDEO_BITRATE, bitrate)
mediaCodec.setParameters(param)
Copy the code

3.3 Selecting an Android Flow control Policy

  • The CQ bit rate control strategy can be selected when the quality requirement is high, the bandwidth does not care, and the decoder supports dramatic fluctuation of the bit rate.
  • VBR output bit rate will fluctuate in a certain range, for small sloshing, the block effect will be improved, but for violent sloshing is still powerless; Continuously lowering the bit rate will result in a sharp drop in the bit rate, and if this is unacceptable, then VBR is not a good choice.
  • The advantage of CBR is that it is stable and controllable, which is helpful to the real-time assurance. Therefore, CBR is generally used in WebRTC development.