An overview,

In the real world, what we see is an image. When a digital device “sees”, it records the value of each point in the image.

In the image above, the digital device sees a matrix containing the values of all the pixels. Eventually all images in the computer world can be reduced to numerical moments and matrix information. OpenCV’s core module defines how to store images in memory, and also includes the definition of matrix, vector, point and other basic operations.

2. Basic image containers

OpenCV defines the Mat class as the basic image container, and Mat can simply represent a matrix. Mat consists of two data parts: the matrix header (containing matrix size, storage method, storage address and other information) and a pointer to the matrix that stores all pixel values (the matrix can be different dimensions according to the selected storage method). The size of the matrix header is constant, but the size of the matrix itself varies from image to image. For example, for an RGB picture, the matrix of Mat object is a THREE-DIMENSIONAL matrix storing the values of R, G and B channels respectively. In the visual algorithm, it is often necessary to transfer pictures and copy pictures, and it is expensive to copy the matrix every time. Therefore, OpenCV adopts the reference counting mechanism, so that each Mat object has its own information header, but shares the same matrix. The copy constructor only copies the information header and matrix pointer, but does not copy the matrix.

2.1 create a Mat

To create a Mat, do the following:

/** * @param rows; The high * @param CLOs column corresponding to the bitmap; The width of the bitmap * @param color space & data type CvType * @param matrix data **/
Mat(int rows, int cols, int type, void* data, size_t step=AUTO_STEP);
Copy the code

In Android OpenCV basics (I, OpenCV introduction), we have seen the method of creating Mat from bitmap:

void *pixels = 0;
AndroidBitmapInfo info;
// Get bitmap information
AndroidBitmap_getInfo(env, bitmap, &info);
// Get the pixel value of the bitmap
AndroidBitmap_lockPixels(env, bitmap, &pixels);
cv::Mat rgbMat(info.height, info.width, CV_8UC4, pixels);
Copy the code

2.2 copy Mat

The copy constructor, as described in Chapter 1, only copies headers and matrix Pointers, not matrices. None of the following operations copy the matrix:

Mat B(A);                                 // Use the copy constructor
C = A;                                    // The assignment operator
Copy the code

If you do need to copy the matrix itself, you can do so in two ways:

cv::Mat tmp(info.height, info.width, CV_8UC4, pixels);
cv::Mat dst;
// Method 1: copyTo
tmp.copyTo(dst);
// Method 2: clone
dst = tmp.clone
Copy the code

2.3 CvType

In the constructor of Mat, the CvType that needs to be passed in is the built-in type of OpenCV, and the format meaning is as follows:

// CvType meaning: [number of bytes per color][with or without sign][Basic data type][number of channels per color]
CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number]
// For example, CV_8UC4: each color has 8 bits, represented by an unsigned char. Each color has 4 channels (R, G, B, A).
cv::Mat rgbMat(info.height, info.width, CV_8UC4, pixels);
Copy the code

2.3.1 Color space

Color space refers to how color elements are combined to encode a given color. Common ones are:

• RGB: use Red Red, Green Green and Blue Blue as the base color, is the most common, this is because the human eye uses a similar working mechanism, it is also used by display devices.
• RGBA: Added transparency Alpha to RGB.
• YCrCb: Widely used in the JPEG image format.
• YUV: Color space for Android cameras, “Y” for Luminance or Luma (gray scale); The “U” and “V” stand for Chrominance or Chroma, which describes the color and saturation of an image and specifies the color of a pixel.

2.3.2 Data Types

The data type refers to how each element is stored, and how it is stored determines how accurately color can be controlled in its domain. For example, in RGB space, if a single R, G, and B are stored in char, char takes up 8 bits, then RGB can represent 16 million possible colors (256 * 256 * 256). Using more type storage (such as 32-bit floats or 64-bit doubles) gives you finer color resolution and increases the memory footprint of the image.

3. Bitmap and Mat

3.1 turn Bitmap Mat

void bitmapToMat(JNIEnv *env, jobject bitmap, cv::Mat &dst) {
    AndroidBitmapInfo info;
    void *pixels = 0;
    try {
        CV_Assert(AndroidBitmap_getInfo(env, bitmap, &info) >= 0);
        CV_Assert(info.format == ANDROID_BITMAP_FORMAT_RGBA_8888 ||
                  info.format == ANDROID_BITMAP_FORMAT_RGB_565);
        CV_Assert(AndroidBitmap_lockPixels(env, bitmap, &pixels) >= 0);
        CV_Assert(pixels);
        dst.create(info.height, info.width, CV_8UC4);
        if (info.format == ANDROID_BITMAP_FORMAT_RGBA_8888) {
            cv::Mat tmp(info.height, info.width, CV_8UC4, pixels);
            tmp.copyTo(dst);
        } else {
            cv::Mat tmp(info.height, info.width, CV_8UC2, pixels);
            cvtColor(tmp, dst, CV_BGR5652RGBA);
        }
        AndroidBitmap_unlockPixels(env, bitmap);
        return;
    }catch(...). {AndroidBitmap_unlockPixels(env, bitmap);
        jclass je = env->FindClass("java/lang/Exception");
        env->ThrowNew(je, "Unknown exception in JNI code {nBitmapToMat}");
        return; }}Copy the code

3.2 turn Bitmap Mat

/** * Create a Bitmap object */
jobject createBitmap(JNIEnv *env, int width, int height, std::string config) {
    jclass bitmapConfig = env->FindClass("android/graphics/Bitmap$Config");
    jfieldID configFieldID = env->GetStaticFieldID(bitmapConfig, config.c_str(),
                                                   "Landroid/graphics/Bitmap$Config;");
    jobject rgb565Obj = env->GetStaticObjectField(bitmapConfig, configFieldID);
    jclass bitmapClass = env->FindClass("android/graphics/Bitmap");
    jmethodID createBitmapMethodID = env->GetStaticMethodID(bitmapClass,"createBitmap"."(IILandroid/graphics/Bitmap$Config;) Landroid/graphics/Bitmap;");
    jobject bitmapObj = env->CallStaticObjectMethod(bitmapClass, createBitmapMethodID,
                                                    width, height, rgb565Obj);
    env->DeleteLocalRef(bitmapConfig);
    env->DeleteLocalRef(bitmapClass);
    return bitmapObj;
}

/** * from Mat to Bitmap */
void matToBitmap(JNIEnv *env, cv::Mat &src, jobject bitmap) {
    AndroidBitmapInfo info;
    void *pixels = 0;
    try {
        if (AndroidBitmap_getInfo(env, bitmap, &info) < 0) {
            return;
        }
        if(info.format ! = ANDROID_BITMAP_FORMAT_RGBA_8888 && info.format ! = ANDROID_BITMAP_FORMAT_RGB_565) {return;
        }
        if(src.dims ! =2|| info.height ! = (uint32_t) src.rows || info.width ! = (uint32_t) src.cols) {
            return;
        }
        if (src.type() != CV_8UC1 && src.type() != CV_8UC3 && src.type() != CV_8UC4) {
            return;
        }
        if (AndroidBitmap_lockPixels(env, bitmap, &pixels) < 0) {
            return;
        }
        if (pixels == 0) {
            return;
        }
        if (info.format == ANDROID_BITMAP_FORMAT_RGBA_8888) {
            cv::Mat tmp(info.height, info.width, CV_8UC4, pixels);
            if (src.type() == CV_8UC1) {
                cvtColor(src, tmp, CV_GRAY2RGBA);
            } else if (src.type() == CV_8UC3) {
                cvtColor(src, tmp, CV_RGB2RGBA);
            } else if (src.type() == CV_8UC4) {
                src.copyTo(tmp); }}else {
            // info.format == ANDROID_BITMAP_FORMAT_RGB_565
            cv::Mat tmp(info.height, info.width, CV_8UC2, pixels);
            if (src.type() == CV_8UC1) {
                cvtColor(src, tmp, CV_GRAY2BGR565);
            } else if (src.type() == CV_8UC3) {
                cvtColor(src, tmp, CV_RGB2BGR565);
            } else if (src.type() == CV_8UC4) {
                cvtColor(src, tmp, CV_RGBA2BGR565); }}AndroidBitmap_unlockPixels(env, bitmap);
        return;
    } catch (const cv::Exception &e) {
        AndroidBitmap_unlockPixels(env, bitmap);
        jclass je = env->FindClass("org/opencv/core/CvException");
        if(! je) je = env->FindClass("java/lang/Exception");
        env->ThrowNew(je, e.what());
        return;
    } catch(...). {AndroidBitmap_unlockPixels(env, bitmap);
        jclass je = env->FindClass("java/lang/Exception");
        env->ThrowNew(je, "Unknown exception in JNI code {nMatToBitmap}");
        return; }}Copy the code

4. Picture transformation

Generally speaking, an image processing operator is a function with one or more input images and an output image. Image transformation can be divided into the following two types:

• Point operators (pixel transformations) : These operators calculate the corresponding output pixel value based only on the input pixel value (sometimes with some global information or parameters). Common operators include brightness and contrast adjustment, and color correction and transformation.
• Neighborhood (region-based) operators: These operators calculate the corresponding output pixel value according to the input pixel value and the surrounding pixel value. Common operators include kernel function, filter, etc.

4.1 Picture lighting

First declare the JNI interface as follows:

public class OpenCVSample {
    static {
        try {
            System.loadLibrary("native-lib");
        } catch(Throwable throwable) { throwable.printStackTrace(); }}public static native Bitmap lightenBitmap(Bitmap bitmap);
}
Copy the code

Then, the following point operator calculation logic is implemented in native layer:

Where I and j represent the pixels of row I and column J, and alpha and beta are parameters. In the program, alpha = 2.2 and beta=50 are used to achieve the following:

extern "C"
JNIEXPORT jobject JNICALL
Java_com_bc_sample_OpenCVSample_lightenBitmap( JNIEnv *env, jclass thiz, jobject bitmap) {
    cv::Mat rgbMat;
    bitmapToMat(env, bitmap, rgbMat);
    // Create a result Mat of the same size
    cv::Mat dst = cv::Mat(rgbMat.size(), rgbMat.type());
    New_image (I,j) = alpha*image(I,j) + beta
    float alpha = 2.2 f;
    int beta = 50;
    for (int y = 0; y < rgbMat.rows; y++) {
        for (int x = 0; x < rgbMat.cols; x++) {
            for (int c = 0; c < rgbMat.channels(a); c++) {// Vec4b because we use RGBA channel, if RGB is Vec3b
                // Implement the calculation logic
                dst.at<cv::Vec4b>(y, x)[c] = cv::saturate_cast<uchar>(
                        alpha * (rgbMat.at<cv::Vec4b>(y, x)[c]) + beta);
            }
        }
    }
    jobject sharpenBitmap = createBitmap(env, dst.cols, dst.rows, "ARGB_8888");
    matToBitmap(env, dst, sharpenBitmap);
    return sharpenBitmap;
}
Copy the code

The running result is as follows:

4.2 Image sharpening

Sharpening is a simple neighborhood operator. Similar to the principle of Sharpening implemented by OpenGL, sharpening is actually recalculating the value of each pixel in the image according to the mask matrix (also known as the core). This time we use the following matrix as the sharpened Kenal kernel function:

First declare the JNI interface as follows:

public class OpenCVSample {
    static {
        try {
            System.loadLibrary("native-lib");
        } catch(Throwable throwable) { throwable.printStackTrace(); }}public static native Bitmap sharpenBitmap(Bitmap bitmap);
}
Copy the code

Then, the following neighborhood operator calculation logic is realized in the native layer. First, align the element in the center of the matrix (the element at position (0,0) in the above example, that is, 5) to the target pixel to be calculated, and then add up the product of the neighborhood pixel value and the corresponding matrix element value. :

extern "C"
JNIEXPORT jobject JNICALL
Java_com_bc_sample_OpenCVSample_sharpenBitmap( JNIEnv *env, jclass thiz, jobject bitmap) {
    cv::Mat rgbMat;
    bitmapToMat(env, bitmap, rgbMat);
    cv::Mat dst;
    sharpen(rgbMat, dst);
    jobject sharpenBitmap = createBitmap(env, dst.cols, dst.rows, "ARGB_8888");
    matToBitmap(env, dst, sharpenBitmap);
    return sharpenBitmap;
}

/** * sharpen the kernel function to implement **/
void sharpen(const cv::Mat &myImage, cv::Mat &Result) {
    CV_Assert(myImage.depth() == CV_8U);  // Only uchar images are accepted
    Result.create(myImage.size(), myImage.type());
    const int nChannels = myImage.channels(a);for (int j = 1; j < myImage.rows - 1; ++j) {
        // The current point in the matrix
        const uchar *previous = myImage.ptr<uchar>(j - 1);
        // The point before the current point in the matrix (current column -1)
        const uchar *current = myImage.ptr<uchar>(j);
        // Next to the current point in the matrix (current column +1)
        const uchar *next = myImage.ptr<uchar>(j + 1);
        uchar *output = Result.ptr<uchar>(j);
        for (int i = nChannels; i < nChannels * (myImage.cols - 1); ++i) {
            *output++ = cv::saturate_cast<uchar>(5 * current[i]
                                                 - current[i - nChannels] - current[i + nChannels] -
                                                 previous[i] - next[i]);
              // Or use another sharpening kernel
// *output++ = cv::saturate_cast
      
       (9 * current[i]
      
// - current[i - nChannels] - current[i + nChannels]
// -previous[i] - previous[i - nChannels] - previous[i + nChannels]
// -next[i] - next[i - nChannels] - next[i + nChannels]);}}// Set the boundary points to 0 instead of using masks
    Result.row(0).setTo(cv::Scalar(0)); / / on the border
    Result.row(Result.rows - 1).setTo(cv::Scalar(0)); / / lower boundary
    Result.col(0).setTo(cv::Scalar(0)); / / the left border
    Result.col(Result.cols - 1).setTo(cv::Scalar(0));/ / right border
}
Copy the code

The running result is as follows, the left is the running result of APP, and the right is the enlarged comparison:

4.3 imgproc module

The imgProc module provides a number of image processing apis. You can call the OpenCV image processing API directly. We’ll cover the imgProc module in the next chapter. You can find the corresponding API in the imgProc module:

/ / filter
CVAPI(void) cvFilter2D( const CvArr* src, CvArr* dst, const CvMat* kernel,
                        CvPoint anchor CV_DEFAULT(cvPoint(- 1.- 1)));
Copy the code

The End

Please follow me to unlock more skills: BC’s Gold Digger homepage ~ 💐 BC’s CSDN homepage ~ 💐💐

OpenCV official website: opencv.org/releases/ OpenCV github: github.com/opencv/open… BC OpenCV LearnOpenCV learning materials OpenCV 4.5.5 official documents OpenCV 2.3.2 official documents