In “How to Quickly achieve mobile Terminal Short video Function? “, we mainly introduce the current hot trend of short video and the functions and services involved in developing a short video application. In this article, we’ll focus on implementing short video on Android.
Recommended reading
Video Private Cloud: Building on-demand Private Cloud Platform Based on Docker
How to Quickly Implement short Video Function of mobile Terminal
The short video service consists of two core functions: Video recording and video editing.
Video recording includes: video collection, real-time beauty, custom bit rate, camera switching, zoom, focus, exposure adjustment and filter and other functions.
Video editing includes: video cropping, video splicing, sound mixing, video animation effect, dynamic mapping and other functions.
AndroidThe technical scheme of terminal short video recording
Let’s start with video recording:
The general implementation process of video recording is shown in the figure above. First, the Camera and AudioRecord collect the most original Camera pictures and sounds, and then the collected data are processed before filter and noise reduction. After the processing is complete, hardware coding is carried out by MediaCodec. Finally use MediaMuxer to generate the final MP4 file.
The advantage of this scheme is that because GPU and hardware coding are used in the whole process, there is basically no operation on the CPU, so the CPU temporary rate is very low in the case of high frame rate (30fps), high resolution (720P) and high bit rate, and it can run well even on poor performance mobile phones. But that’s where the difficulty lies.
Those of you who have done audio and video know that under normal circumstances, what we call the processing of audio and video is mainly the operation of YUV, H264 audio PCM, AAC and other data formats. This kind of operation has the relevant RFC technology and is relatively mature. It is easier to realize, and it is easier to locate problems. That’s how it’s usually handled on devices like PCS.
However, this scheme is different for mobile phones. Although the performance of mobile phones has been greatly enhanced in recent years, many flagship phones are basically 8 core CPU, but to operate such a large amount of picture data and floating-point calculations on the CPU consumption is still very large. A high CPU temporary rate will cause hot mobile phone, which will lead to frame loss collected by Camera. Even under some manufacturers such as Xiaomi, hot mobile phone will also cause CPU underfrequency. Once the CPU underfrequency, the proportion of CPU temporarily used by APP will be higher, and high CPU temporary rate will cause fast power consumption.
Therefore, the above scheme is more suitable for short video recording on Android.
AndroidThe implementation of short video recording in terminal
Now that we have identified the technical solution, let’s look at the concrete implementation. Here are a few concepts to know:
SurfaceTexture
We know that in some simple custom Camera applications, to implement a Camera, we only need to set a SurfaceHolder to the Camera, and the Android system will automatically help us to draw the data collected by the Camera to the screen. However, in short videos, we need to pre-process the data collected by the camera, such as filters and so on, and to achieve the visible is what you get, so we need to cache the data collected by the camera first, and then draw it on the screen ourselves. This is where SurfaceTexture is needed. SurfaceTexture is an invisible canvas that receives Camera data, according to the Android documentation. This canvas exists in the GPU memory.
TextureID
The texture ID is mainly used to identify the memory address of each texture in the GPU memory, similar to the memory pointer in C language. Each GPU texture (which can be thought of as a canvas used to display images) is identified by a TextureID. The SurfaceTexture is also created with a unique texture ID.
Knowing these two concepts, we can know where the data collected by the Camera resides on the GPU. The raw data can then be preprocessed and encoded.
One thing to note here is that the Textures that the Android Camera collects on SurfaceTexture are of type GL_TEXTURE_EXTERNAL_OES, whereas many of the current filter algorithms on the market, For example, many filters in the open source GPUImage are based on GL_TEXTURE_2D texture for image processing.
Therefore, we need to convert GL_TEXTURE_EXTERNAL_OES texture into GL_TEXTURE_2D texture before we use open source filter algorithm or self-developed algorithm. Or add “#extension GL_OES_EGL_image_external: require” to the GPU Shader to indicate that the texture is OES texture and write an image processing algorithm based on OES texture.
At present, netease first converts OES into TEXTURE_2D for pre-processing, so as to facilitate the unification with iOS algorithm and better access to some open source filter algorithm.
With SurfaceTexture out of the way, MediaCodec and MediaMuxer were easier to follow. MediaCodec in Android naturally allows you to draw textures from the GPU to MediaCodec’s Surface, and then hardware-encode images directly from the Surface. The image data always flows through the GPU space, not the CPU.
The important thing to note here is that MediaCodec and MediaMuxer require API 18 and above to work. Also note the compatibility of MediaCodec on different devices.
AndroidTerminal video editing function
Finally, for the function of video editing, there is no good system API on Android, that is, hardware processing mode, mainly using FFMPEG for corresponding superposition, clip mixing and other post-processing. One point that can be optimized is that the original MP4 video is decoded first, and then the decoded YUV data is mapped to the GPU texture to adjust the brightness, saturation and other parameters, so that what you see is what you get for file editing, and then the adjusted parameters are configured as FFMPEG for editing.
Want to get more products, technology dry goods, remember to pay attention to netease Cloud letter blog.