OpenCV introduction

OpenCV is a cross-platform computer vision library distributed under the BSD license that runs on Linux, Windows, Android, and Mac OS operating systems. It is lightweight and efficient — it consists of a series of C functions and a small number of C++ classes. It also provides interfaces to Python, Ruby, MATLAB and other languages and implements many common algorithms in image processing and computer vision.

Using OpenCV on mobile terminal can complete a series of image processing work.

OpenCV configuration on Android

The version of OpenCV I’m using for my project is 4.x.

Create a Library in Android Studio, import OpenCV from the official website, and call the methods of Java classes in OpenCV directly.

If you want to call C++ classes, you can also use CMake to create the environment and then put it in the specified path through the include file.

Below is the cmakelists.txt used in the project

cmake_minimum_required(VERSION 3.6.0)

include_directories(
        ${CMAKE_SOURCE_DIR}/src/main/cpp/include
)

add_library(libopencv_java4 SHARED IMPORTED)
set_target_properties(
        libopencv_java4
        PROPERTIES IMPORTED_LOCATION
        ${CMAKE_SOURCE_DIR}/src/main/jniLibs/libs/${ANDROID_ABI}/libopencv_java4.so)

add_library(libc++_shared SHARED IMPORTED)
set_target_properties(
        libc++_shared
        PROPERTIES IMPORTED_LOCATION
        ${CMAKE_SOURCE_DIR}/src/main/jniLibs/libs/${ANDROID_ABI}/libc++_shared.so)


add_library(
        detect

        SHARED

        src/main/cpp/detect-lib.cpp
        src/main/cpp/detect-phone.cpp
)


find_library(
        log-lib
        log
)

target_link_libraries(
        detect libopencv_java4 libc++_shared jnigraphics
        ${log-lib}
)
Copy the code

Among them, detect-lib. CPP and detect-phone. CPP are C++ classes I created. When you type an SO file, these two classes are included.

Three. Two examples

3.1 As a bottom-pocket scheme for TWO-DIMENSIONAL code recognition

In Android native development, two-dimensional code recognition has the old open source library such as Zxing. Why use OpenCV at all?

Because OpenCV has its own advantages, with the help of it can be positioned to the location of the TWO-DIMENSIONAL code, generally can not identify the content of the two-dimensional code is mostly because it can not find its location. If you can find the location, you can quickly identify the content of the QR code.

In this way, qr code recognition needs to take a picture, from the image to find the location of the QR code. Of course, the image can also be preprocessed to be able to better find the location of the QR code.

The following code shows that after taking photos in the application layer, the path of the picture is transmitted to the JNI layer, which is converted into the corresponding Mat object, and then converted into grayscale image, and then the location of the two-dimensional code is found. If it can be found, the content of the two-dimensional code can be identified.

extern "C"
JNIEXPORT jstring JNICALL
Java_com_xxx_sdk_utils_DetectUtils_qrDetect(JNIEnv *env, jclass jc,jstring filePath) {

    const char *file_path_str = env->GetStringUTFChars(filePath, 0);
    string path = file_path_str;
    Mat src = imread(path);

    Mat gray, qrcode_roi;
    cvtColor(src, gray, COLOR_BGR2GRAY);
    QRCodeDetector qrcode_detector;
    vector<Point> pts;
    string detect_info;
    bool det_result = qrcode_detector.detect(gray, pts);
    if (det_result) {
        detect_info = qrcode_detector.decode(gray, pts, qrcode_roi);
        return env->NewStringUTF(detect_info.c_str());
    } else {
        detect_info = "";
        return env->NewStringUTF(detect_info.c_str()); }}Copy the code

Corresponding Java code for application layer to call JNI layer’s qrDetect()

public class DetectUtils {

    static {
        System.loadLibrary("detect");
    }

    /** * Identify the qr code *@param filePath
     * @return* /
    public static native String qrDetect(String filePath); . }Copy the code

Finally, the invocation of the application layer

// Use OpenCV for QR code recognition
val result = DetectUtils.qrDetect(filePath)
L.d("Opencvs recognize qr code:$result")
Copy the code

3.2 Comparison of image differences

In our actual development, we encountered an application scenario: we need to determine whether objects are stored in our mobile phone recycling machine. (The phone collector is a touch-screen device that operates the internal hardware via Android.)

We took a picture of no object in the recycling machine as a reference image in advance, and then took another picture when we needed to judge whether there was an object. Comparing the ratio of the two pictures, if the ratio exceeds the threshold value, it is considered that there are objects in the recycling machine.

The following code shows how the application layer takes a picture, compares it with the base image, and returns the result.

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_xxx_sdk_utils_DetectUtils_checkPhoneInMTA(JNIEnv *env, jclass jc,jstring baseImgPath,jstring filePath) {

    jboolean tRet = false;
    const char *file_path_str = env->GetStringUTFChars(filePath, 0);
    string path = file_path_str;
    Mat src = imread(path);

    const char *base_img_path_str = env->GetStringUTFChars(baseImgPath, 0);
    string basePath = base_img_path_str;
    Mat baseImg = imread(basePath);

    int result = checkPhoneInBox(baseImg,src,40.0.1);

    LOGI("checkPhoneInBox result = %d",result);
    if (result == 0) {
        tRet = true;
    }

    return tRet;
}
Copy the code

The actual comparison of the two images is done in the checkPhoneInBox(). Among them, maxFilter() is used to deal with the color situation, and then gaussian filtering is used for noise reduction, and then binarization processing. Finally, it is determined whether the ratio of gray difference area to the total image column exceeds the preset threshold.

int checkPhoneInBox(cv::Mat baseImg, cv::Mat snapImg, int diffThresh, double threshRatio) {

    cv::Mat baseMaxImg, snapMaxImg,baseGausImg, snapGausImg;
    if (baseImg.empty()|| snapImg.empty())
    {
        return - 1;
    }

    try {
        maxFilter(baseImg, baseMaxImg);
        maxFilter(snapImg, snapMaxImg);
    } catch(...). {return - 1;
    }

    cv::GaussianBlur(baseMaxImg, baseGausImg, cv::Size(5.5),0);
    cv::GaussianBlur(snapMaxImg, snapGausImg, cv::Size(5.5),0);

    cv::Mat diff,diffBin;
    cv::Mat noMax;
    cv::absdiff(baseGausImg, snapGausImg, diff);
    cv::threshold(diff, diffBin, diffThresh, 255, cv::THRESH_BINARY);

    float ratio = (float)cv::countNonZero(diffBin) / (long)diffBin.total(a);LOGI("ratio = %f,%d,%ld",ratio,cv::countNonZero(diffBin),(long)diffBin.total());

    if (ratio > threshRatio)
    {
        return 0;
    }
    else
    {
        return 1; }}int maxFilter(cv::Mat baseImg, cv::Mat &maxImg)
{
    if (baseImg.channels()"3)
    {
        maxImg = baseImg.clone(a); }else
    {
        maxImg.create(baseImg.size(), CV_8UC1);
        for (int r=0; r<baseImg.rows; r++) {for (int c = 0; c < baseImg.cols; c++)
            {
                uchar maxTmp=0;
                cv::Vec3b s = baseImg.at<cv::Vec3b>(r, c);
                maxTmp = (std::max)(s[0],s[1]);
                maxTmp = (std::max)(maxTmp,s[2]); maxImg.at<uchar>(r, c) = maxTmp; }}}return 0;
}
Copy the code

Jni layer checkPhoneInMTA()

public class DetectUtils {

    static {
        System.loadLibrary("detect");
    }

    /** * Check whether there is a mobile phone in the MTA *@paramBaseImageFilePath Base image *@paramImages taken by filePath *@return* /
    public static native boolean checkPhoneInMTA(String baseImageFilePath, String filePath); . }Copy the code

Finally, the invocation of the application layer

val result = DetectUtils.checkPhoneInMTA(Constants.OPENCV_PHOTO_PATH, it.absolutePath)
Copy the code

4. To summarize

OpenCV is a powerful image processing library. But it is also bulky, and using it on mobile adds at least 10 M+ to the size of the Android Apk package (depending on how many CPU architectures the App supports). If so, consider tailoring OpenCV yourself and compiling it later.

The department I work for belongs to the Middle Taiwan Department, which mainly outputs interfaces and SDK. The use of OpenCV in the SDK is certainly a problem for the business side, and the future will look at how to reduce the size of the SDK and make the SDK modular.