Android Face Recognition based on RTMP video Stream

I believe everyone is familiar with live broadcasting, and the technology of live broadcasting is becoming more and more mature. At present, there is such a technology that when a bullet screen falls on the face of the anchor, the bullet screen will disappear automatically and continue to be displayed outside the face range. This principle is very simple, in fact, is face recognition, face recognition within the range of all the bullets hidden. Easier said than done, this article will discuss how to implement face recognition for RTMP video streams.

• Scheme selection
• Ffmpeg access
• Ffmpeg data parsing
• OpenGL data rendering
• Face tracking and face frame drawing

First, the choice of scheme

At first, I wanted to use a player that someone else had packaged and play it by typing in the address. After access, I found that access and use are indeed very simple, and can display, but there is a very fatal problem, that is, there is no interface to obtain naked data, so there is no way to face recognition, then I switched to FFMPEG. Of course, if you just want to play the RTMP stream on the device, Bilibili’s ijkPlayer framework is absolutely no problem, easy to access and use. Here is their address.

• Github.com/Bilibili/ij…

Analysis scheme has been selected, the next is drawing and face recognition, drawing I use OpenGL. The reason for this is that we have a custom surfaceView that we can use directly. Face recognition engine I choose rainbow soft engine, there are two reasons, one is relatively simple to use, Rainbow soft demo write good, a lot of things can be directly copied over; Second, it is free. I have also used other companies’ engines, which have a trial period. I don’t like things with a period of time.

Two, ffMPEG access

1. Directory structure

Create the CPP and jniLibs directories under the SRC /main directory and put the FFMPEG library, as shown in the following figure.

2.CMakeLists

First we create two new files in SRC /main/ CPP, cmakelists. TXT and rtmpplayer-lib. CMake is used to manage and build files between libraries, and rtmpPlayer-lib is used to put our JNI code to parse data streams.

CMakeLists.txt

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.Cmake_minimum_required (VERSION 3.4.1 track)# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        rtmpplayer-lib

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        rtmpplayer-lib.cpp)
include_directories(ffmpeg)

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        rtmpplayer-lib

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib}
        android
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavcodec.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavdevice.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavfilter.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavformat.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libavutil.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libpostproc.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libswresample.so
        ${PROJECT_SOURCE_DIR}/.. /jniLibs/${CMAKE_ANDROID_ARCH_ABI}/libswscale.so
        )
Copy the code

3.build.gradle

We need to specify the location of our CMake file above, and specify the schema to build.

android{ defaultConfig { ... . externalNativeBuild { cmake { abiFilters"armeabi-v7a"
            }
        }
        ndk {
            abiFilters 'armeabi-v7a'ExternalNativeBuild {cmake {//path is the address path of CMakeLists"src/main/cpp/CMakeLists.txt"
            version "3.10.2"}}}Copy the code

4. Complete the build

After the above steps are completed, we can construct, click on the build of refresh linked prject, c + + clicking on the right side of Gradle/other/externalNativeBuildDebug, Wait for after the completion of the building can be in the build/intermediates/cmake under building a successful so libraries can see myself, if I can see libnative – lib. So congratulations, ffmpeg access is done.

Third, FFMPEG data analysis

1.JNI data flow analysis

As mentioned above, native-lib.cpp is where we will write Jni code that parses the RTMP data stream.

#include <jni.h>
#include <string>
#include <android/log.h>
#include <fstream>

#define LOGE(FORMAT, ...) __android_log_print(ANDROID_LOG_ERROR, "player", FORMAT, ##__VA_ARGS__);
#define LOGI(FORMAT, ...) __android_log_print(ANDROID_LOG_INFO, "player", FORMAT, ##__VA_ARGS__);

extern "C" {
#include "libavformat/avformat.h"
#include "libavcodec/avcodec.h"
#include "libswscale/swscale.h"
#include "libavutil/imgutils.h"
#include "libavdevice/avdevice.h"
}


static AVPacket *pPacket;
static AVFrame *pAvFrame, *pFrameNv21;
static AVCodecContext *pCodecCtx;
struct SwsContext *pImgConvertCtx;
static AVFormatContext *pFormatCtx;
uint8_t *v_out_buffer;
jobject frameCallback = NULL;
bool stop;
extern "C"JNIEXPORT jint JNICALL Java_com_example_rtmpplaydemo_RtmpPlayer_nativePrepare(JNIEnv *env, jobject, Jstring URL) {// initialize#if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(55, 28, 1)
#define av_frame_alloc avcodec_alloc_frame
#endif
    if (frameCallback == NULL) {
        return- 1; } // Apply space pAvFrame = av_frame_alloc(); pFrameNv21 = av_frame_alloc(); const char* temporary = env->GetStringUTFChars(url,NULL); char input_str[500] = {0}; strcpy(input_str,temporary); env->ReleaseStringUTFChars(url,temporary); // All available file formats and encoders in the registry avcodec_register_all(); av_register_all(); avformat_network_init(); avdevice_register_all(); pFormatCtx = avformat_alloc_context(); int openInputCode = avformat_open_input(&pFormatCtx, input_str, NULL, NULL); LOGI("openInputCode = %d", openInputCode);
    if (openInputCode < 0)
        return- 1; avformat_find_stream_info(pFormatCtx, NULL); int videoIndex = -1; // Find the first video stream and record the encoding information of this streamfor (unsigned int i = 0; i < pFormatCtx->nb_streams; i++)
    {
        if(pFormatCtx->streams[I]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {break; }}if (videoIndex == -1) {
        return- 1; } pCodecCtx = pFormatCtx->streams[videoIndex]->codec; AVCodec *pCodec = avcodec_find_decoder(pCodecCtx->codec_id); avcodec_open2(pCodecCtx, pCodec, NULL); int width = pCodecCtx->width; int height = pCodecCtx->height; LOGI("width = %d , height = %d", width, height); int numBytes = av_image_get_buffer_size(AV_PIX_FMT_NV21, width, height, 1); v_out_buffer = (uint8_t *) av_malloc(numBytes * sizeof(uint8_t)); av_image_fill_arrays(pFrameNv21->data, pFrameNv21->linesize, v_out_buffer, AV_PIX_FMT_NV21, width, height, 1); PImgConvertCtx = sws_getContext(pCodecCtx->width, pCodecCtx->height, PCodecCtx ->height, // pCodecCtx->height, // / / choose which way to proceed with the change of the size, about this parameter, you can refer to: http://www.cnblogs.com/mmix2009/p/3585524.html, NULL, NULL, NULL); pPacket = (AVPacket *) av_malloc(sizeof(AVPacket)); Jclass clazz = env->GetObjectClass(frameCallback); jmethodID onPreparedId = env->GetMethodID(clazz,"onPrepared"."(II)V");
    env->CallVoidMethod(frameCallback, onPreparedId, width, height);
    env->DeleteLocalRef(clazz);
    return videoIndex;
}

extern "C"JNIEXPORT void JNICALL Java_com_example_rtmpplaydemo_RtmpPlayer_nativeStop(JNIEnv *env, jobject) {// Stop playing stop =true;
    if (frameCallback == NULL) {
        return;
    }
    jclass clazz = env->GetObjectClass(frameCallback);
    jmethodID onPlayFinishedId = env->GetMethodID(clazz, "onPlayFinished"."()V"); Env ->CallVoidMethod(frameCallback, onPlayFinishedId); env->DeleteLocalRef(clazz); // Free the resource sws_freeContext(pImgConvertCtx); av_free(pPacket); av_free(pFrameNv21); avcodec_close(pCodecCtx); avformat_close_input(&pFormatCtx); } extern"C"JNIEXPORT void JNICALL Java_com_example_rtmpplaydemo_RtmpPlayer_nativeSetCallback(JNIEnv *env, jobject, Jobject callback) {// Set the callbackif(frameCallback ! = NULL) { env->DeleteGlobalRef(frameCallback); frameCallback = NULL; } frameCallback = (env)->NewGlobalRef(callback); } extern"C"JNIEXPORT void JNICALL Java_com_example_rtmpplaydemo_RtmpPlayer_nativeStart(JNIEnv *env, jobject) {// Start play stop =false;
    if (frameCallback == NULL) {
        return; } int count = 0;while(! stop) {if(av_read_frame(pFormatCtx, pPacket) >= 0) {int gotPicCount = 0; int decode_video2_size = avcodec_decode_video2(pCodecCtx, pAvFrame, &gotPicCount, pPacket); LOGI("decode_video2_size = %d , gotPicCount = %d", decode_video2_size, gotPicCount);
            LOGI("pAvFrame->linesize %d %d %d", pAvFrame->linesize[0], pAvFrame->linesize[1],
                 pCodecCtx->height);
            if(gotPicCount ! = 0) { count++; sws_scale( pImgConvertCtx, (const uint8_t *const *) pAvFrame->data, pAvFrame->linesize, 0, pCodecCtx->height, pFrameNv21->data, pFrameNv21->linesize); Int dataSize = pCodecCtx->height * (pAvFrame->linesize[0] + pAvFrame->linesize[1]); LOGI("pAvFrame->linesize %d %d %d %d", pAvFrame->linesize[0], pAvFrame->linesize[1], pCodecCtx->height, dataSize); jbyteArray data = env->NewByteArray(dataSize); env->SetByteArrayRegion(data, 0, dataSize, reinterpret_cast<const jbyte *>(v_out_buffer)); Jclass clazz = env->GetObjectClass(frameCallback); jmethodID onFrameAvailableId = env->GetMethodID(clazz,"onFrameAvailable"."([B)V"); env->CallVoidMethod(frameCallback, onFrameAvailableId, data); env->DeleteLocalRef(clazz); env->DeleteLocalRef(data); } } av_packet_unref(pPacket); }}Copy the code

2.Java layer data callback

After the ABOVE JNI operation is done, we have the parsed raw data, and we are done just passing the raw data to the Java layer, which we use callback to do.

Void onPrepared(int width, int height); void onPrepared(int width, int height); // Data callback void onFrameAvailable(byte[] data); Void onPlayFinished(); }Copy the code

All we need to do is pass the callback to Native and pass the parsed data to Java via JNI.

 RtmpPlayer.getInstance().nativeSetCallback(new PlayCallback() { @Override public void onPrepared(int width, Rtmpplayer.getinstance ().nativestart (); rtmpPlayer.getInstance (). } @override public void onFrameAvailable(byte[] data) {// Obtain raw data in NV21 log. I (TAG,"onFrameAvailable: " + Arrays.hashCode(data));
        surfaceView.refreshFrameNV21(data);
    }

    @Override
    public void onPlayFinished() {// End of playback callback}}); Int code = rtmpPlayer.getInstance ().prepare()"RTMP: / / 58.200.131.2:1935 / livetv hunantv");
if(code == -1) {// prepare for RTMP is prepared. toast.maketext (mainactivity.this,"prepare Error", Toast.LENGTH_LONG).show();
}
Copy the code

The data generated by onFrameAvailable is the data in NV21 format that we need. The following figure shows the data callback generated by playing hunan SATELLITE TV. From hashCode, the data callback is different each time, so it can be considered that the data is refreshed in real-time.

3. Interaction between Java layer and JNI

The RtmpPlayer singleton class has been created as a channel through which Jni interacts with the Java layer.

public class RtmpPlayer {

    private static volatile RtmpPlayer mInstance;
    private static final int PREPARE_ERROR = -1;

    private RtmpPlayerPublic static RtmpPlayer () {} // Double lock prevents multiple instances from being created by multiple threadsgetInstance() {
        if (mInstance == null) {
            synchronized (RtmpPlayer.class) {
                if(mInstance == null) { mInstance = new RtmpPlayer(); }}}returnmInstance; } public int prepare(String url) {if(nativePrepare(url) == PREPARE_ERROR){
            Log.i("rtmpPlayer"."PREPARE_ERROR ");
        }
        returnnativePrepare(url); } static {system.loadLibrary ();"rtmpplayer-lib"); } // Data preparation private native int nativePrepare(String URL); Public native void nativeStart(); Public native void nativeSetCallback(PlayCallback PlayCallback); Public native void nativeStop(); }Copy the code

Four, summary

So far we have obtained NV21 raw data, due to the limited time, the article needs to write a lot of content, so it needs to be divided into two parts. In the next part, I will talk about how to draw NV21 data through OpenGL, and how to use NV21 raw data for face recognition, and draw a face frame. That’s why we try so hard to get raw NV21 data. If there is a problem with ffMPEG access, you can see the demo I uploaded in the appendix at the end of the next article. It may be easier to use the demo.

Five, the appendix

Android Face Recognition based on RTMP video stream

RtmpPlayerDemo engineering code (including display and face rendering)