preface

As we all know, Android platform development is divided into Java layer and C++ layer, namely Android SDK and Android NDK. Normal product functionality only needs to be involved in the Java layer, unless there is a special need to introduce the NDK. But what about audio and video development?

Prior to MediaCodec, Android’s Java API support for audio and video remained at a very abstract API level (that is, simple parameters and methods, little behavior to control, no intermediate data, and no complex functionality to develop, let alone extend). After the launch of MediaCodec, the problem was not completely solved, for the following reasons: 1. The Android version of MediaCodec is not low, and the use of MediaCodec is not compatible with the lower version of the machine and system version; 2. Due to the open source and customization features of Android, MediaCodec is implemented differently by various manufacturers, resulting in the same code for machine A running one way, and machine B running another way. So programmers turned to the NDK, but Google didn’t provide any APIS for audio and video processing (like parsing generated files and decoding frames), so they decided to use the open source C/C++ framework. The first ones are of course the most famous ffMPEG, X264, MP3lame, faac, etc. The problem is that FFMPEG was best for x86 at the beginning, not so good for ARM or MIPS (things have improved since I researched FFMPEG 2.0). That can only use soft solution soft plait, the speed can not keep up with is what experience pro you know? For example, if you want to record 640×480 video, audio and video all use soft coding, X264 if the pure soft coding and the processing performance of the phone CPU 50 milliseconds or even 100 milliseconds a frame, in short, it is slow, but also include audio and compression coding. If you want to 25 frame rate of video recording, a frame of the coding time can’t more than 40 milliseconds, otherwise the speed to keep up with, spend time on other business function, this time at least wants to below 30 milliseconds, then use the multi-thread asynchronous coding way should optimize the reluctance to get pictures while generating video files. It is because of such inconvenience, the author just after a few months of research, found a not too perfect solution for your reference, this article will be a comprehensive introduction to the technical implementation of each link, and finally attached to the project source code. By the way, my Android development experience before this job is basically 1 (not 0 because I wrote HelloWorld before), but I have mastered C/C++, Java, and developed projects using ObjC on ios, so I think Android is not very different, the language is not the same. Platform is different, API is different, system mechanism is different, other should be the same.

What apis are available for the NDK?

Start by opening the NDK include to find out what interfaces the NDK provides. Google is humane enough to have some audio and video apis in addition to Linux system level apis.

OpenSL, which can operate audio acquisition and playback devices directly on the C++ layer to record and play sound, starting with API9.

EGL, can create OpenGL rendering environment in C++ layer, used for video image rendering, can also be used for some image processing such as cropping, stretching, rotation, and even more advanced filter effects processing. In addition, we have to say that in C++ to create their own OpenGL rendering environment than the use of Java layer GLSurfaceView flexibility, controllability, scalability directly improve several orders of magnitude. EGL is already supported in API9.

OpenGL(ES), NDK provides the Java layer OpenGL interface, and the NDK layer also provides the more native OpenGL header file, and to use GLSL that must have OpenGLES2.0+, but the NDK also has long supported. OpenGLES2.0 is supported in API5.

OpenMAXAL, one of the libraries found in the census, is a bit of a bumperer because of its interface definition, which provides functions such as playing video and opening the camera in a more abstract way. Play video is not used, behind their own decoding of their own rendering implementation, see the interface to open the camera, the heart was delighted, the result is that MY MX3 actually told me that the interface did not achieve. The result is that the camera data must be passed from the Java layer to the C++ layer. But OpenMAXIL, its cousin, is a good thing, but Google doesn’t have an open interface yet.

In this way, the image acquisition must open the Camera from the Java layer, and then get the data and pass it to the C++ layer through JNI. The View that renders the image also creates a SurfaceView from the Java layer and then passes a handle to the C++ layer to initialize the OpenGL rendering environment using EGL. The collection and playback of sound and Java has nothing to do with the bottom layer can be directly processed.

Choose an Open source framework

Ffmpeg: file parsing, image stretching, pixel format conversion, most of the decoders, the author chose version 2.7.5, there are many optimizations for ARM, decoding speed is still good.

X264: H264 encoder, the new version of the ARM also has a lot of optimization, if using multithreading encoding frame 640×480 can be as low as 3-4 ms.

The encoder for MP3lame was not used in the test project (the combination of MP4(H264+AAC) was used in the test project), but it was compiled and added to the encoder list

Faac: The encoder of AAC has not been updated for a long time. The encoding speed is slow, so there is a curve design to solve the problem of audio encoding.

Complete solution diagram

Audio encoding is slow

For x264 and FFMPEG, download the newer versions, and then turn on ASM, neon and other optimization options to compile, and the codec speed is acceptable. But FAAC is still a bit slow. The author then thought of a way, is to store temporary files, in the recording time video data directly call X264 encoding, not ffMPEG transfer (so that more flexible configuration of X264 parameters, to achieve the purpose of faster), and audio data directly written to the file. Recorded this temporary files and serious video file size difference is not big, actually will not result in a magnetic card write speed slow bottleneck problem, and can also solve the edit drag the progress bar accuracy problem, at the same time solve the problem of key frame extraction frame, because the temporary file is written, what content can control in the file. I have to say that one problem is the definition of abstract video file read and write interface Reader and Writer, and the implementation of reading and writing to MP4 files and the implementation of reading and writing to temporary files are implemented by the Reader and Writer. So in the future when you want to change to recording directly, you just need to create an MP4 when you initialize it, you just need to create another object. There is also a slow solution to the problem of multithreading asynchronous writing, the collection thread gets the data and throws it to another thread for coding write, as long as the average speed of coding write up to the frame rate can meet the demand.

Introduce OpenGL2D/3D engine

Once an OpenGL rendering environment has been created using EGL in the C++ layer, any C/C++ -based framework can be used. The author introduced Cocos2D-X to add some special effects to the video, such as sequence frames and particle effects. Cocos2d-x has its own rendering thread and OpenGL rendering environment. After you have eliminated this code, write some code to render Cocos2D-X to your own EGL environment. In addition, cocos2D-X’s object recycling mechanism is simulated objective-C reference counting and automatic recycling pool. The author also simplified the cocos2D-X recycling mechanism in the engineering source code, to tell the truth, I think its reference counting simulation is also ok, and COM is similar to the principle, unified base class can be achieved. The automatic collection pool doesn’t have to be exactly objective-C. There’s no need to push the collection pool. A global collection pool is enough. (Purely personal opinion)

Primary and secondary thread mode

GlMakeCurrent in OpenGL is thread-sensitive, as you all know. All operations related to OpenGL are thread-sensitive, i.e. text loading, GLSL script compiling links, context creation, and glDraw operations all need to be in the same thread. The Android platform does not have the same method as iOS MainOperationQueue, so THE author designed a primary and secondary thread mode (I named it myself), that is, the main thread is the Android UI thread, responsible for the UI drawing response button Action. All other operations are then handed over to the sub-thread. That is, the response function for each user operation does not do anything directly, but learns MFC’s way of Posting a message and data to a sub-thread. Subthreads must then schedule message loops and multitasks in a single-thread manner, not to mention MFC mode. Single-threaded scheduling multi-tasking is a process that takes a traditional single-threaded task and splits it into slices of time, allowing the thread to process one slice at a time, then caching the processing state until its next turn.

For example, the task interface is IMission {bool onMissionStart(); bool onMissionStep(); void onMissionStop(); } Execute onMissionStart once; if false is returned, execute onMissionStop; If the former returns true, onMissionStep is repeatedly called until false is returned, and onMissionStop is executed. The specific processing is wrapped into the implementation class of the task interface and thrown into the task list.

Imagine, under this design architecture, whether all operations are in the same thread, OpenGL calls are in the same thread, and the side effect is that mom no longer has to worry about the performance and bug problems caused by multi-threaded concurrent processing locking everywhere, don’t doubt its performance, Because even if you have multiple threads at the CPU level, you have single threads. Redis is a single – threaded, fast bar.

To summarize

  • Use OpenSL for recording and broadcasting

  • Use EGL to create an OpenGL environment at the C++ layer

  • Modify cocos2D-x to use OpenGL environment created by myself

  • Directly use X264 instead of FFMPEG transfer, according to the fastest encoding method to configure parameters, be sure to turn on x264 multi-threaded encoding.

Both X264 and FFMPEG need to be downloaded relatively new and compiled using ASM, neon, etc. (I compiled it cross-platform on Ubuntu)

If video and audio are not fast enough to be recorded, write temporary files, encode images, and save sound directly to PCM.

In addition to the Android main thread, only one secondary thread is opened for scheduling, specific small module time-consuming tasks are opened separately, there are only two threads on the main body of the framework, a master and a pair.

Complete project source code

Development using API15 can actually be as low as API9.

Source address: http://download.csdn.net/detail/yangyk125/9416064

Operate demo: http://www.tudou.com/programs/view/PvY9MMugbRw/

Position of generated video after rendering: /SD card/e4Fun /video/*.mp4

What needs to be explained is:

  • 1, com. Android. Video. Camera. The front EFCameraView class two private fields define the current chosen camera resolution is the width and height, request the current camera support this resolution.

  • 2, jni/WORKER/EFRecordWorker. CPP createRecordWorker function, define the current video recording of all kinds of basic parameters, please according to test the performance of the machine configuration of freedom.

  • 3, jni/WORKER/EFRecordWorker. CPP on_create_worker function, a set setAnimationInterval calls, set the OpenGL rendering frame rate, and the video frame rate are two different things, please take into consideration the Settings.

Thanks to someone who read this blog post and pointed out where it could be improved

  • 1. If ffMPEG is used for audio and video, then AAC should use FDK_AAC instead of FAAC which has not been updated for a long time.

  • It is inefficient to read back data. The author is trying to upgrade to GLES3.0 to see if there is any way to get the rendered result image quickly. If you know, please leave a message behind, thanks!

Do audio and video processing on Android, if you want faster codec, if it is Java layer can not escape MediaCodec, if it is C++ layer, you can look down, such as OpenMAXIL and so on.

Postscript:

After half a year of efforts, we have solved some of the efficient problems

(1) Encoding and decoding part

In the codec part, X264+FFMPEG open source scheme was used in the previous article. After further study, I found a unique implementation scheme on Android.

Version <4.4: X264 + FFmPEG or private API(libstagefright. So).

Version =4.4: JNI reverses Android.media.MediaCodec or is developed in the Java layer.

Version >4.4: NdkMediaCodec(JNI interface for Android.mediacodec).

(2) AAC is a better open source scheme

The AAC open source program FDKAAC has been updated and its efficiency has improved, while faAC has not been updated for a long time. So… You know.

AAC can also use MediaCodec or NdkMediaCodec

(3) Read back OpenGL framebuffer data

GLES version < 3.0: use glReadPixels or EGLImageKHR (eglCreateImageKHR glEGLImageTargetTexture2DOES)

GLES version =3.0: Pixel Pack Buffer + glMapBufferRange.

Android version >=4.2: Create a New SurfaceTexture and Surface, create an OpenGL Context, and render it again and again. The FrameBuffer can be rendered onto this SurfaceTexture, and the surface can also serve as input to the encoder. This can not only quickly transfer data from the render result to the encoder, but also achieve cross-thread transfer of texture data, which is provided by the Android platform itself, non-OpengL built-in ability. SurfaceTexture is 4.2 because the SurfaceTexture was not fully SurfaceTexture until 4.2, when it was unstable.

https://github.com/yangyk125/AndroidVideo

Original author: granite is sweet, the original link: https://blog.csdn.net/yangyk125/article/details/50571304