An overview of the

In order to achieve H264 video coding on android platform, we can usually use libx264, FFMPEG and other third-party video coding library, but if there are certain requirements for coding speed, to achieve real-time or even super real-time high-speed video coding, we do not have many options. You can only use the MediaCodec hardcoding module provided by Android. The MediaCodec module will encounter many problems in the actual use. This article mainly discusses how to achieve the balance of speed and clarity when using MediaCodec to encode video for OpenGL rendered images.

Note that by default you will be familiar with the basic process of encoding OpenGL images using MediaCodec and SurfaceTexture

Analysis of the

Factors that affect coding speed

There are not many other factors that affect the speed of MediaCodec to encode video images outside the device hardware. We mainly found the following points after exploring the practice:

  1. Screen size Screen size is the width and height of the video frame to be encoded. It directly defines the size of the screen data to be encoded per frame, so it mainly determines the amount of work Mediacodec must do per frame. This is easy to understand.

  2. Frame rate Frame rate is the number of still frames per second of video. Since Mediacodec encodes each still frame in frames, the frame rate directly determines the number of frames that Mediacodec needs to encode when encoding a fixed length of video. For example, encoding the same 10-second video at a frame rate of 30 frames per second, regardless of screen content and other factors, would require MediaCodec to encode twice as much as at 15 frames per second, so it would take twice as long.

  3. MediaCodec works in two modes: synchronous and asynchronous. Asynchronous is only supported on Android 5.0 or higher. If you want to be compatible with systems below 5.0, you have to use synchronous mode, and many of the tutorials on MediaCodec’s coding process that have been circulated on the Internet (such as the classic BigFlake) are based on synchronous mode. With these documents and examples, everyone uses roughly the same Settings when coding with MediaCodec. In synchronous mode, we need to call MediaCodec’s dequeueOutputBuffer interface to retrieve the contents of the encoded output buffer from the queue. The second parameter to the dequeueOutputBuffer interface, timeoutUs, is a subtle time value that defines the timeout period for retrieving the contents of the output buffer from MediaCodec. Most of the current documentation tutorials and examples set this value to 10000 microseconds, which is 10 milliseconds. However, when we reduced this value to 1 millisecond or less, we saw a significant increase in the speed of the MediaCodec coding process, and based on our tests, the coding results were not significantly affected visually. Of course, the Official Android documentation does not provide specific recommendations on the appropriate range of timeoutUs, but MY conclusion here is that it was tested in the actual development of our project, and the results met our needs. If your project does not need to maintain support for systems below 5.0, I recommend that you code in asynchronous mode to avoid this synchronous wait problem directly.

  4. Encoder Profile A Profile is a description of video compression features. It defines a collection of encoding tools. The higher the Profile, the more advanced compression features are used. In general, more advanced compression takes more time to compress, meaning that the higher the profile is set, the more time it takes to encode. For example, when we are using FFMPEG encoding, if we set the encoding configuration to ultra-fast, profile will be forced to set baseline.

Factors that affect picture clarity

  1. Encoder Profile Level A Profile is a description of video compression features. It mainly defines a set of encoding tools. The higher the Profile, the more advanced the compression features, the higher the definition of the encoded video file, and the smaller the bit stream. There are three H264 coding profiles supported by android

    • Baseline Profile
    • Main Profile
    • High Profile

    Baseline is supported after Android 3.0. According to our test, the customization of Main Profile and higher Profile is supported after Android 7.0. Even with Android7.0 and later, some models (such as some OPPO and huawei models) do not support it. So if your product needs to maintain support for mainstream models, Baseline is the only reliable Profile option.

  2. Biterate is the number of bits of video data (video color, brightness, pixel) output per second. The common unit is KBPS. In general, the higher the bit rate, the higher the video definition. However, due to the limited resolution range of human eyes, it is difficult to distinguish the image sharpness caused by setting the bit rate too high by human eyes, so it is meaningless to set the bit rate too high. In general, I recommend a formula to calculate how to set the corresponding bit rate for the encoder: Biterate = Width * Height * FrameRate * Factor where Width, Height and FrameRate denote the Width, Height and FrameRate of the video, and Factor is a coefficient used to control the bit rate. Generally, in the network streaming media usage scenario, the Factor can be set to 0.1~0.2, so as to ensure that the size of the generated video file is small without serious picture loss. In the common local browsing scenario, the Factor can be set to 0.25~0.5, so as to ensure the clarity of the picture will not cause too much loss to the naked eye due to coding, so as to generate a relatively large video file. In the case of hd video processing, you can set the Factor to above 0.5. However, when the color of the video picture is richer and the picture changes faster, the video coding needs a higher bit rate. If you encounter this kind of video scene, you need to increase the bit rate appropriately to ensure the clarity. It should be noted that most Android models have a limit on Bitrate support. If you set the Bitrate above this limit, the encoder may throw an exception and the coding process may fail. So when designing the value of the Bitrate, need through MediaCodecInfo. CodecCapabilities provide relevant interface to check the system to support the maximum Bitrate.

  3. Biterate Mode Although we can set the relevant bit rate for the encoder via the interface, there are several different modes for the encoder to handle the bit rate we set during encoding.

    • BITRATE_MODE_CQ Ignores the bit rate set by the user. The encoder controls the bit rate itself, and tries to ensure the balance between the picture clarity and the bit rate.
    • BITRATE_MODE_CBR Regardless of the screen content of the video, as much as possible to follow the user set bit rate
    • BITRATE_MODE_VBR complies with the bit rate set by the user as much as possible, but dynamically adjusts the bit rate according to the motion vector between frames (commonly understood as the degree of change between frames). If the motion vector is large, the bit rate will be raised during this period, if the transformation of the picture is small, the bit rate will be reduced.

    Therefore, when we set the Bitrate, we should also pay attention to the setting of the Bitrate Mode. Different Settings have different effects on the size and clarity of the generated video files.

  4. When encoding OpenGL render content using MediaCodec, you can set the frame rate of the encoder by setting the MediaFormat.KEY_FRAME_RATE field of the MediaFormat. You can also set the time stamp of each frame by setting the presentationTimeUs value of the Mediacodec. BufferInfo object. But often overlooked, we ran the OpenGL drawing commands, you need to call before calling swapbuffer eglPresentationTimeANDROID interface to set the timestamp of the current frame. EglPresentationTimeANDROID if there is no call to set the timestamp of the current frame, only set up MediaCodec. BufferInfo object presentationTimeUs, encoding the video on time there will be no any abnormalities, So many developers often overlooked eglPresentationTimeANDROID interface call. But in fact, if you don’t call eglPresentationTimeANDROID, encoding the video with additional losses on their definition. Because MediaCodec is designed for the use of real-time video stream encoding, MediaCodec adjusts the speed of the encoding according to the speed at which the user inputs the screen to it. If we don’t through eglPresentationTimeANDROID before coding to set picture time stamp, then MediaCodec tend to we to the speed of the input image by default for the real-time speed, to adjust the encoding speed. This adjustment will result in a lower bit rate and reduced video clarity. When we pass eglPresentationTimeANDROID before coding timestamp on the image to the right Settings, MediaCodec will based on actual timestamp per frame to adjust code material. This ensures that there is no additional reduction in the bit rate and that the sharpness Settings take effect.

The solution

Combined with the above analysis, in order to achieve the balance of coding speed and picture sharpness when we use MediaCodec for video coding, we need to optimize comprehensively in several aspects.

1. Profile method

In summary, in order to ensure the perfect adaptation of apps on various Android models, there are not many options we can make in terms of Profile. The safest situation is to set the Profile Baseline.

2. The Bitrate

To set the Bitrate, I recommend using the formula Biterate = Width * Height * FrameRate * Factor to set it according to the usage scenario of the product.

3. Biterate Mode

The default setting for Biterate Mode is BITRATE_MODE_VBR and I recommend setting it to BITRATE_MODE_CQ as much as possible with system and machine support. In the case of BITRATE_MODE_CQ, the coder itself can often achieve the desired effect by adjusting the bit rate and coding speed, and the generated video files are not too many, but the picture definition is excellent. For example, if you do not support BITRATE_MODE_CQ, MediaCodec will fail to configure. In this case, you need to revert to the default Biterate Mode.

4. The time stamp is set correctly

We are in complete each frame OpenGL rendering, must call first before calling swapbuffer eglPresentationTimeANDROID method to set the right frame timestamp. EglPresentationTimeANDROID default does not directly exposed, we defined before contains header files to expand macros to load the function. It’s load eglPresentationTimeANDROID function relevant code:

#define EGL_EGLEXT_PROTOTYPES
#include <EGL/eglext.h>

eglPresentationTimeANDROID(mDisplay, mSurface, (EGLnsecsANDROID) time); // Set the current timestamp correctly before calling eglSwapBuffers
eglSwapBuffers(mDisplay, mSurface);
Copy the code