Peeling and beautifying functions in videos have become urgent needs, so how to achieve 720P peeling and beautifying recording in Android short videos? In this article,
Netease cloud letterSenior development engineer will introduce the specific operation method to you.
Related Reading recommendations
Short Video Technology: Short Video Development Technology on Android
How to Quickly Implement short Video function on mobile Terminal?
In order to achieve a recording function on Android, it is necessary to have several aspects of knowledge: the development of custom camera, the understanding of video data format, coding knowledge and video synthesis technology, at the same time, if you need to beautification, skin grinding and other filter operations also need some knowledge of openGL. If there is a need for in-depth audio and video development of the students recommended to understand the above basic knowledge points.
Since we want to achieve 720P, 30 frames, and at the same time to record the video data filter processing, we must first determine a correct implementation scheme. If you choose the wrong solution, even if you do record, it’s not good if you don’t get to 30 frames or your phone gets hot because you’re using too much CPU.
Video coding and recording are mainly soft and hard coding two schemes:
Soft coding is the use of CPU to the camera to collect the original data after encoding and audio together into a MP4 format file.
The advantage is that the technology is relatively mature, online open source coding and synthesis library, relatively fast implementation, and good compatibility.
The disadvantages are high CPU transient rate, poor performance of the phone can not reach 720P 30 frames, at the same time reference a large number of third-party libraries, resulting in a large package.
The specific implementation scheme of soft editing is shown in the figure below, and the process is relatively clear and simple:
Hardcoding is to use the hardcoding interface provided by the mobile phone, using the hardware chip directly to encode and synthesize.
The advantages are fast speed, high efficiency, very little CPU usage, even long hd recording will not be hot, and due to the use of system API, the library is relatively small.
The disadvantage is that some strange models need to deal with compatibility issues. Meanwhile, the hard programming on Android is closely related to Surface and openGL, and there is less knowledge on the Internet, so you need to find the holes by yourself.
The main process of hard programming is shown in the figure below. It can be seen that all data are transferred in GPU from collection, coding, display and synthesis.
Combined with the two solutions analyzed above, we can see that on mobile platforms like Android, using the second hardcoding method is more appropriate. Since the local recording of short video does not require much bandwidth like live broadcast and requires dynamic adjustment of the encoder’s bit rate and frame rate, the local recording can set the encoder’s bit rate to a higher level without dynamic change of resolution. Therefore, adopting hardware coding can not only save CPU performance but also achieve efficient coding of 720P.
After determining the scheme, we will focus on the implementation of each step of the hardcoded scheme.
Custom camera development
We know that according to the Android system Camera API, we can set a SurfaceView’s SurfaceHolder to the Camera through the setPreviewDisplay interface, so that the data collected by the Camera can be displayed on the SurfaceView. The advantage of this interface is that the system helps us to process various angles collected by the camera and draw at the same time. It is suitable for simple recording, but we need to filter the data collected by the camera, so this interface is not suitable.
So we need to use another interface called setPreviewTexture:
By setting a Camera to a SurfaceTexture, we can map the data collected by the Camera to the SurfaceTexture, and then obtain the Camera data on the GPU according to the TextureID that created the Camera
Filters and local painting
SurfaceTexture binding TextureID can be used to retrieve video data collected by the Camera on the GPU. You can then send TextureID to some third-party filter library for beauty filters or write your own Shader for skin polish and whitening. Writing Shader requires opengL and image algorithm knowledge, usually requires specialized developers, here will not do the detailed expansion (of course, the simplest is access to netease cloud short video SDK, which realizes skin grinding, beauty and a number of filters).
Local drawing is mainly done by openGL. We need to create an EGLContext, EGLDisplay and EGLSurface on the Camera collection callback thread. EGLContext is openGL’s context on this thread. EGLDisplay is a virtual display area in GPU, which is mainly used to cache video data on GPU. EGLSurface is the mapping from View to openGL, which is a tool for drawing on View. When receive the Camera after the callback of a frame data, we first by SurfaceTexture. UpdateTexImage () method to map the data from the Camera to SurfaceTexture. Then, the data corresponding to TextureID is drawn to EGLDisplay according to GLSL language. It should be noted that Camera acquisition has angles, and the angles of horizontal and vertical screens are different. The Angle matrix can be obtained by using getTransformMatrix method of SurfaceTexture. And then pass the matrix to EGLDisplay for rotation. After EGLDisplay rotation drawing is complete, eglSwapBuffers can be used to copy the data from EGLDisplay to EGLSurface for display. The GLSurfaceView in Android finally displays the data to the screen we see using eglSwapBuffers.
Hardware encoding
Android hardware coding is mainly implemented by MediaCodeC API. Below is a typical data processing diagram of MediaCodeC.
As you can see from the figure, MediaCodeC’s main processes are:
- Create and configure a MediaCodec object
- Loop until complete:
- Release the MediaCodec object
MediaCodeC supports both ByteBuffers and Surface inputs. The Camera data is SurfaceTexture. So here we use Surface mode as the input source.
We mentioned in the display section above that EGLSurface is the module that actually outputs the display, as is MediaCodec. We first create a Surface through MediaCodec, and then bind the Surface to an EGLSurface. When the data collected by Camera is called back, we only need to repeat the operation of drawing the module. SwapBuffers collected by Camera on the SurfaceTexture will be used on the EGLSurface. Then loop through the MediaCodec output buffer and MediaCodec will return the encoded data to us. The advantage of this is that the display and coding are completely separated, and we can code even without UI Views, such as switching between activities, without affecting our normal coding.
Video synthesis
Video composition on Android is mainly achieved through MediaMuxer API. The MediaMuxer class is relatively simple, especially when used with MediaCodec. We just need to add the video and audio channel interface through addTrack. The AddTrack interface needs to pass in a MediaFormat object. MediaFormat is a MediaFormat class used to describe format parameters of media, such as video frame rate and audio sampling rate. So fortunately we’re using MediaCodeC, MediaCodeC is going to return MediaFormat to us, so if you’re using soft editing and then merging with MediaMuxer, there’s a big hole here, so if you create MediaFormat objects manually, Be sure to set the “CSD-0” and “CSD-1” parameters: “CSD-0” and “CSD-1” correspond to the SPS and PPS for video, and the ADTS for AAC audio. It will crash if you don’t set it. After setting this up, as soon as the encoder comes in a frame of data, we send it to MediaMuxer and it’s ready to go out MP4.
The final general flow chart is:
The above is the realization of 720P skin polishing beauty recording sharing in Android short video.
In addition, to get more product dry goods, technical dry goods, remember to pay attention to netease yunxin blog.