1. YUV storage format and sampling

1.1 YUV Storage Format

YUV is a luminance signal Y and chroma signal U, V is a separate color space, it is mainly used to optimize the transmission of color video signals, so that it is backward compatible with the old black and white TV. Where “Y” stands for Luminance or Luma, or gray scale; The “U” and “V” refer to Chrominance or Chroma, which describes the color and saturation of an image and specifies the color of a pixel. YUV formats are divided into two types: Packed and Planar.

Packed type: YUV components are stored in the same array. Y, U and V of each pixel are continuously interlaced.

Planar type: store YUV components separately in three independent arrays, and store Y of all pixels successively first, then U of all pixels, and finally V of all pixels.

1.2 YUV sampling method

The storage format of YUV code stream is closely related to the sampling method. Currently, there are three mainstream sampling methods as follows: YUV444, YUV422 and YUV420. In YUV444 sampling, each Y corresponds to a set of UV components, and each pixel (YUV) occupies 32Bits. In YUV422 sampling, every two Y shares a set of UV components, with each pixel accounting for 16bits (Y accounting for 8bits, UV component accounting for 8bits). YUV420 sampling is a set of UV components shared by each of the four Y components, with each pixel (YUV) occupying either 16bits or 12bits. The three sampling formats are shown as follows:

2. Memory analysis of common YUV420 color formats

2.1 Color format of camera output

Considering that there are many kinds of camera devices, such as mobile phone camera, USB camera, WIFI camera, etc., the YUV color format involved in the output image is also different, so I choose the mobile phone camera I often contact in the development to explain. About YUV color format of mobile phone Camera image collection, you can passCamera.Parameters.getSupportedPreviewFormats ()Method to get the color format supported by Camera. The API is described as follows:From getSupportedPreviewFormats () method, the device’s Camera main supportNV21 and YV12Two YUV color formats, in which YV12 is introduced into API12. Here we focus on the analysis of the characteristics of these two formats and memory storage characteristics:

(1) NV21:YYYYYYYY VUVU

From android/graphics/ImageFormat known NV21, NV21 is the iphone Camera preview format by default, the collection for NV21 image color format. NV21 is a semi-Planner format where the Y component occupies one plane space, the VU crossover storage occupies one plane space, and the Y components of the four pixels share one VU component. The schematic diagram is as follows:

Storage in memory: (2) YV12:YYYYYYYY VV UU From android/graphics/ImageFormat known YV12, YV12 is the color of the Camera support mobile phone format. YV12 is a kind of Planner. Y components, U components and V components occupy a plane space respectively. The Y components of 4 pixels share a VU component. The schematic diagram is as follows:Storage in memory:

2.2 Color format for encoder input

The original pictures collected by USB Camera and mobile phone Camera are usually in YUV420 format. According to YUV420 sampling method, each pixel occupies 3/2 bytes (a pixel is composed of Y, U and V components, among which Y component occupies one byte, and U and V components are 1/4 of Y component). So the size of a 1280×720 image is1280 x 720 x 1.5 bytes =1382400 bytes, about 1.3MB. Assuming that the frame rate of video transmission is 20fps, 26.36MB is needed to be transmitted in 1 second. For the current bandwidth conditions, the pressure is very large, so we need to compress and encode the original image to reduce the size of the original image while retaining the available information. In Android, MediaCodec API is provided to implement hard coding. It will call the underlying encoder hardware implementation to compress and encode the original YUV image. When using MediaCodec, we need the color format supported by the encoder. MediaCodecInfo. CodecCapabilities common color format is as follows:MediaCodecInfo. Enumeration of CodecCapabilities format is more, we will analyze the writing of several commonly used in the process of decoding, COLOR_FormatYUV420Planar, COLOR_FormatYUV420SemiPlanar, COLOR_FormatYUV420PackedPlanar.(1) COLOR_FormatYUV420Planar:YYYYYYYY UU VVCOLOR_FormatYUV420Planar is a kind of Planner, Y component, U component and V component occupy a plane space respectively, and the Y component of 4 pixels share a VU component. The schematic diagram is as follows:Storage in memory: (2) COLOR_FormatYUV420SemiPlanar:YYYYYYYY UVUVCOLOR_FormatYUV420SemiPlanar is a semi-planner with Y components occupying one plane space,UV cross storage occupying one plane space, and 4 pixels of Y components sharing one UV component. The schematic diagram is as follows:Storage in memory:

Finally, we summarize the relationship and difference between the above common YUV420 color formats: NV21, COLOR_FormatYUV420SemiPlanar YUV420SP both belong to the semi Planner, that is, Y components occupy one plane, U and V components share the same plane, the former V is located in front of U (YYYYYYYY VUVU), The latter U precedes V (YYYYYYYY UVUV); YV12, COLOR_FormatYUV420Planar(YUV420P for short) belong to the plane format (Planner), that is, Y component, U component and V component occupy a plane respectively, the former V is located in front of U (YYYYYYYY VV UU), The latter U is located in the former of V (YYYYYYYY UU VV).

Note: COLOR_FormatYUV420SemiPlanar has another name: NV12; COLOR_FormatYUV420Planar is also known as I420, in fact it is the standard YUV420 format.

Semi Planner format: NV21: YYYYYYYY VUVU COLOR_FormatYUV420SemiPlanar: YYYYYYYY UVUV: COLOR_FormatYUV420SemiPlanar YUV420SP NV12: YYYYYYYY VV UU COLOR_FormatYUV420Planar: YYYYYYYY UU VV(standard YUV format) where COLOR_FormatYUV420Planar and I420 are the same

3. YUV color format processing code implementation

Friends with strong coding development experience should be familiar with, when we use Camera to obtain data collected by the Camera, we need to specify the Preview Format, that is, the original YUV image color Format collected by the Camera device, usually the default is imageformat.nv21. It can also be set to imageformat.yv12. If we use a hard encoder (MediaCodec) to compress and encode the original image captured by the camera into H.264 data stream, we also need to specify which YUV color format data the hard encoder supports as input. The common formats are COLOR_FormatYUV420Planar, COLOR_FormatYUV420Planar, etc. Due to the inconsistency between the original image format output by the camera and the color format required by the encoder, the color display of the encoded data will be abnormal, or the phenomenon of splintered screen. Based on this, we need to convert the YUV image captured by the Camera to the color format required by the encoder before “feeding” the data to the encoder.

3. 1 NV21 and COLOR_FormatYUV420SemiPlanar conversion

NV21 storage in memory:

COLOR_FormatYUV420SemiPlanar storage in memory:

From NV21 and COLOR_FormatYUV420SemiPlanar memory storage structure, they are half planar mode, we only need to change the order of U and V components. Two implementation methods are provided here, namely Java layer implementation, C/C++ implementation, the code is as follows: (1) Java code implementation

// YYYYYYYY VUVU --> YYYYYYYY UVUV
// Convert NV21 to Yuv420sp
public static byte[] nv21ToYuv420sp(byte[] src, int width, int height) {
    int yLength = width * height;
    int uLength = yLength / 4;
    int vLength = yLength / 4;
    int frameSize = yLength + uLength + vLength;
    byte[] yuv420sp = new byte[frameSize];
    / / Y component
    System.arraycopy(src, 0, yuv420sp, 0, yLength);
    for (int i = 0; i < yLength/4; i++) {
        / / U
        yuv420sp[yLength + 2 * i] = src[yLength + 2*i+1];
        / / V
        yuv420sp[yLength + 2*i+1] = src[yLength + 2*i];
    }
    return yuv420sp;
}

// YYYYYYYY UVUV(yuv420sp)--> YYYYYYYY VUVU(nv21)
// Convert YUV420SemiPlanner to NV21
public static byte[] yuv420spToNV21(byte[] src, int width, int height) {
    int yLength = width * height;
    int uLength = yLength / 4;
    int vLength = yLength / 4;
    int frameSize = yLength + uLength + vLength;
    byte[] nv21 = new byte[frameSize];
    / / Y component
    System.arraycopy(src, 0, nv21, 0, yLength);
    for (int i = 0; i < yLength/4; i++) {
        / / U
        nv21[yLength + 2*i +1] = src[yLength+2*i];
        / / V
        nv21[yLength + 2*i] = src[yLength + 2*i + 1];
    }
    return nv21;
}
Copy the code

(2) C/C++ code

JNIEXPORT jint JNICALL
Java_com_jiangdg_natives_YuvUtils_nativeNV21ToYUV420p(JNIEnv *env, jclass type, jbyteArray jarray_, jint width, jint height) 
{
    jbyte *srcData = env->GetByteArrayElements(jarray_, NULL);
    jsize srcLen = env->GetArrayLength(jarray_);
    int yLength = width * height;
    int uLength = yLength / 4;
    // Create a temporary memory space
    char *c_tmp = (char *)malloc(srcLen);
    // Copy the Y component
    memcpy(c_tmp,srcData,yLength);
    int i = 0;
       for(i=0; i<yLength/4; i++) {
        / / U
        c_tmp[yLength + 2 * i] = srcData[yLength + 2*i+1];
        / / V
        c_tmp[yLength + 2*i+1] = srcData[yLength + 2*i];
    }
    // Overwrite the c_tmp data to jarray_
    env->SetByteArrayRegion(jarray_,0,srcLen,(jbyte *)c_tmp);
    env->ReleaseByteArrayElements(jarray_, srcData, 0);
    // Release temporary memory
    free(c_tmp);
}
Copy the code

3. 2 CONVERT NV21 with COLOR_FormatYUV420Planar

NV21 storage in memory:

COLOR_FormatYUV420Planar in memory:

From the memory storage structure of NV21 and COLOR_FormatYUV420Planar, the former belongs to half-plane mode, while the latter belongs to planar mode. During conversion, we only need to extract U and V components and arrange them as required. Two implementation methods are provided here, namely Java layer implementation, C/C++ implementation, the code is as follows: (1) Java code implementation

// YYYYYYYY UU VV --> YYYYYYYY VUVU
// Convert YUV420Planner (I420) to NV21
public static byte[] yuv420pToNV21(byte[] src, int width, int height) {
    int yLength = width * height;
    int uLength = yLength / 4;
    int vLength = yLength / 4;
    int frameSize = yLength + uLength + vLength;
    byte[] nv21 = new byte[frameSize];

    System.arraycopy(src, 0, nv21, 0, yLength); / / Y component
    for (int i = 0; i < yLength / 4; i++) {
        / / U
        nv21[yLength + 2*i + 1] = src[yLength + i];
        / / V
        nv21[yLength + 2*i] = src[yLength + uLength + i];
    }
    return nv21;
}

// YYYYYYYY VUVU ---> YYYYYYYY UU VV
// Convert nv21 to YUv420p (I420)
public static byte[] nv21ToYuv420p(byte[] src, int width, int height) {
    int yLength = width * height;
    int uLength = yLength / 4;
    int vLength = yLength / 4;
    int frameSize = yLength + uLength + vLength;
    byte[] yuv420p = new byte[frameSize];
    / / Y component
    System.arraycopy(src, 0, yuv420p, 0, yLength);
    for (int i = 0; i < yLength/4; i++) {
        / / U
        yuv420p[yLength + i] = src[yLength + 2*i + 1];
        / / V
        yuv420p[yLength + uLength + i] = src[yLength + 2*i];
    }
    return yuv420p;
}
Copy the code

(2) C/C++ code

JNIEXPORT jint JNICALL
Java_com_jiangdg_natives_YuvUtils_nativeNV21ToYUV420p(JNIEnv *env, jclass type, jbyteArray jarray_, jint width, jint height) 
{
    jbyte *srcData = env->GetByteArrayElements(jarray_, NULL);
    jsize srcLen = env->GetArrayLength(jarray_);
    int yLength = width * height;
    int uLength = yLength / 4;
    // Create a temporary memory space
    char *c_tmp = (char *)malloc(srcLen);
    // Copy the Y component
    memcpy(c_tmp,srcData,yLength);
    int i = 0;
    for(i=0; i<yLength/4; i++) {
        / / U
        c_tmp[yLength + i] = srcData[yLength + 2*i + 1];
        / / V
        c_tmp[yLength + uLength + i] = srcData[yLength + 2*i];
    }
    // Overwrite the c_tmp data to jarray_
    env->SetByteArrayRegion(jarray_,0,srcLen,(jbyte *)c_tmp);
    env->ReleaseByteArrayElements(jarray_, srcData, 0);
    // Release temporary memory
    free(c_tmp);
}
Copy the code

3.3 Converting NV21 to YV12

NV21 storage in memory:

YV12 storage in memory:

According to the memory storage structure of NV21 and YV12, the former belongs to half plane mode, while the latter belongs to plane mode. During the conversion, we only need to extract U and V components and arrange them according to requirements. Two implementation methods are provided here, namely Java layer implementation, C/C++ implementation, the code is as follows: (1) Java code implementation

// YYYYYYYY VV UU --> YYYYYYYY VUVU
// Convert YV12 to NV21
public static byte[] yv12ToNV21(byte[] src, int width, int height) {
    int yLength = width * height;
    int uLength = yLength / 4;
    int vLength = yLength / 4;
    int frameSize = yLength + uLength + vLength;
    byte[] nv21 = new byte[frameSize];

    System.arraycopy(src, 0, nv21, 0, yLength); / / Y component
    for (int i = 0; i < yLength / 4; i++) {
        / / U
        nv21[yLength + 2*i + 1] = src[yLength + vLength + i];
        / / V
        nv21[yLength + 2*i] = src[yLength + i];
    }
    return nv21;
}
Copy the code

(2) C/C++ code

JNIEXPORT jint JNICALL
Java_com_jiangdg_natives_YuvUtils_nativeYV12ToNV21
(JNIEnv *env, jclass type, jbyteArray jarray_,jint width, jint height) 
{
    jbyte *srcData = env->GetByteArrayElements(jarray_, NULL);
    jsize srcLen = env->GetArrayLength(jarray_);
    int yLength = width * height;
    int vLength = yLength / 4;
    // Create a temporary memory space
    char *c_tmp = (char *)malloc(srcLen);
    // YYYYYYYY VV UU --> YYYYYYYY VUVU
    // Copy the Y component
    memcpy(c_tmp,srcData,yLength);
    int i = 0;
    for(i=0; i<yLength/4; i++) {
        / / U
        c_tmp[yLength + 2*i + 1] = srcData[yLength + vLength + i];
        / / V
        c_tmp[yLength + 2*i] = srcData[yLength + i];
    }
    // Overwrite the c_tmp data to jarray_
    env->SetByteArrayRegion(jarray_,0,srcLen,(jbyte *)c_tmp);
    env->ReleaseByteArrayElements(jarray_, srcData, 0);
    // Release temporary memory
    free(c_tmp);
}
Copy the code

Note: Android project examples will be provided later.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Audio and Video Exploration (1) : Brief analysis of YUV color space

1. YUV storage format and sampling

1.1 YUV Storage Format

1.2 YUV sampling method

2. Memory analysis of common YUV420 color formats

2.1 Color format of camera output

2.2 Color format for encoder input

3. YUV color format processing code implementation

3. 1 NV21 and COLOR_FormatYUV420SemiPlanar conversion

3. 2 CONVERT NV21 with COLOR_FormatYUV420Planar

3.3 Converting NV21 to YV12

Audio and Video Exploration (1) : Brief analysis of YUV color space

1. YUV storage format and sampling

1.1 YUV Storage Format

1.2 YUV sampling method

2. Memory analysis of common YUV420 color formats

2.1 Color format of camera output

2.2 Color format for encoder input

3. YUV color format processing code implementation

3. 1 NV21 and COLOR_FormatYUV420SemiPlanar conversion

3. 2 CONVERT NV21 with COLOR_FormatYUV420Planar

3.3 Converting NV21 to YV12

Related Posts

In 2020, I used MVVM for a project like this

Software development: why does a high-quality software product development need an excellent product manager?

Design mode (16) combination mode