About the RGB

Those who have studied physics in middle school know the three primary colors of light: Red, Green and Blue, commonly known as Red, Green and Blue, which can represent almost all colors in nature. All colors can be presented by setting RGB components. RGB three colors are mixed to make white, which is the color of sunlight we usually see.

In computer graphics, RGB is represented by 8 bits, R is represented by 8 bits, G is represented by 8 bits, AND B is represented by 8 bits. Each color component can be represented by 256 values. In fact, colors in nature cannot be exhausted by 256 bits, but they must be represented by standardized things in computer language. Otherwise everything is a mess.

RGB is also divided into many kinds, we can simply understand, these RGB classification has specific use scenarios.

  • RGB24: This is the standard RGB mode, respectively 8-bit R, 8-bit G, 8-bit B representation.
  • RGB555: RGB555 is a 16-bit RGB format, RGB components are represented by 5 bits (the remaining 1 bit is not used), expressed as follows: X R R R R RG G G GB B B B B (X is not used, can be ignored)
  • RGB565: RGB565 uses 16 bits to represent a pixel, 5 bits for R, 6 bits for G, and 5 bits for B. R R R G G G G G G B B B B
  • RGBA: This is RGB based on the addition of A representation, A is transparency, A total of 32 bits.

About YUV

With RGB color system, it should be able to represent all the colors in pictures and videos, but YUV coding is used in our program production color coding system. Mainly is the beginning of TV is based on black and white, from black and white TV transition to color TV, YUV was born, which Y component represents gray value, UV represents color, if only Y component, no UV component, that is black and white TV.

From a storage perspective, YUV can be divided into compact and flat formats.

  • Compressed format: Compressed format is to store YUV data in an array, just like RGB: R R R R RG G G G G G GB B B B B B B
  • Plane format: plane format is to store YUV three components of data separately, first store Y component data, then U component, and then V component, the advantage of such arrangement is more convenient, plane format is currently used more widely YUV arrangement.

There are many kinds of YUV. Here are some common ones:

  • Yuv444:4:4 means complete sampling
  • Yuv422:4:2:2 represents 2:1 horizontal sampling and vertical complete sampling.
  • Yuv420:4:2:0 represents 2:1 horizontal sampling and 2:1 vertical sampling.
  • Yuv411:4:1:1 indicates 4:1 horizontal sampling and 1:1 vertical sampling.

YUV444 is the same number of Y, U and V components.

YUV422 is that the Y component is twice the UV component in the horizontal direction, and the Y component is the same as the UV component in the vertical direction.

YUV420 means that the Y component is twice the UV component in the horizontal direction, and the Y component is twice the UV component in the vertical direction.

YUV411 means that the Y component is 4 times the UV component in the horizontal direction, and the Y component is 2 times the UV component in the vertical direction.

For details, please refer to dougkerr.net/Pumpkin/art…

The box on the left above represents the Y component and the black circle represents the UV component.

Here are some of the YUV concept differences in program development.

  • The difference between YUVJ420P and YUV420P?
  • What are NV12 and NV21?

The biggest difference between YUVJ420P and YUV420P is that YUVJ420P uses the JPEG color range, that is, the normal YUV420P color range is 16 ~ 235, 16 represents black, 235 represents white. YUVJ420P indicates the full color range. 0 indicates black and 255 indicates white.

NV12 is a YUV420, but YUV420P 3-plane storage mode is different, NV12 is 2-plane storage, 3-plane is Y/U/V sub-table stored in different places, 2-plane is Y/UV sub-table stored in different places. NV12 is the Y-UV storage mode, and NV21 is the Y-VU storage mode.

Android Camera is often used in NV21 mode.

In the following figure, the original image is on the right, and the Y component, U component and V component from bottom to bottom are on the left.

YUV turn RGB

Why do we need YUV to RGB? From the above analysis, we know that YUV coding system (no matter YUV or YCbCr) is used in digital TV or analog TV. But if we want to render the video content, we still need to convert to RGB mode, so YUV to RGB is something we have to consider.

YUV types are many, RGB types are also many, we need to consider many situations in the process of conversion. Before conversion, we also introduce YUV BT.601/BT.709/ Bt.2020 three different compatibility standards.

  • BT.601
  • BT.709
  • BT.2020

In normal terms, BT.609 is for STANDARD DEFINITION video, BT.709 is for HD video, BT.2020 is for ultra HIGH definition video, BT.2020 is still relatively little, mainly the first two standards.

There are three common ways to convert YUV to RGB:

  • OpenGL shader way
  • Libyuv way
  • FFmpeg swscale way

YUV and RGB can be converted, for example, YUV420P is converted to RGBA, where the range of each component in RGBA is 0 ~ 255, the range of Y component in YUV420P is 16 ~ 235, UV component is 0 ~ 127, This requires us to map YUV420P components to RGBA, which can be used in many ways, the main is matrix calculation. In engineering development, the common approach is to follow the four approaches described above.

Specific derivation calculation you can reference: en.wikipedia.org/wiki/YUV because I think this is the conventional calculation, will not go into here. Here recommend a conversion site: licheng. Sakura. Ne. Jp/hatena6 / RGB…

The following directly share with you the specific conversion formula:

  • RGB to BT. 601 YUV
Y = 0.257r + 0.504g + 0.098b + 16 Cb = -0.148r-0.291g + 0.439b + 128 Cr = 0.439r-0.368g-0.071b + 128Copy the code
  • BT.601 YUV转 RGB
R = 1.164 (Y - 16) = 1.164 + 1.596 (Cr - 128) G (Y - 16) to 0.391 (Cb - 128) to 0.813 (Cr - 128) B = 1.164 (Y - 16) + 2.018 (Cb - 128)Copy the code

YUV here is local gamut. If it is full gamut, the transformation formula is as follows:

R = Y + 1.402(cr-128) G = y-0.344 (Cb-128) -0.714 (CR-128) B = Y + 1.772(Cb-128)Copy the code
  • RGB to BT. 709 YUV
Y = 0.183r + 0.614g + 0.062b + 16 Cb = -0.101r-0.339g + 0.439b + 128 Cr = 0.439r-0.399g-0.040b + 128Copy the code
  • BT. 709 YUV RGB
R = 1.164 (Y - 16) = 1.164 + 1.793 (Cr - 128) G (Y - 16) to 0.213 (Cb - 128) to 0.533 (Cr - 128) B = 1.164 (Y - 16) + 2.112 (Cb - 128)Copy the code

YUV here is local gamut. If it is full gamut, the transformation formula is as follows:

R = Y + 1.280(cr-128) G = y-0.215 (cb-128) -0.381 (cr-128) B = Y + 2.128(cb-128)Copy the code

OpenGL shader way

According to the matrix calculation above, the following shader can be obtained using the OpenGL shader method

Bt.601 YUV changed to RGB VARYING highP VEC2 textureCoordinate; uniform sampler2D texture_y; uniform sampler2D texture_u; uniform sampler2D texture_v; Void main() {highp float y = texture2D(texture_y, textureCoordinate).r-0.0625; Highp float u = texture2D(texture_u, textureCoordinate).r-0.5; Highp float v = texture2D(Texture_v, textureCoordinate).r-0.5; Highp float r = 1.164 * y + 1.596 * v; Highp float g = 1.164 * y - 0.391 * u - 0.813 * v; Highp float b = 1.164 * y + 2.018 * u; Gl_FragColor = vec4(r, g, b, 1.0); } # BT.601 YUV transform RGB VARYING highp VEC2 textureCoordinate; uniform sampler2D texture_y; uniform sampler2D texture_u; uniform sampler2D texture_v; void main() { highp float y = texture2D(texture_y, textureCoordinate).r; Highp float u = texture2D(texture_u, textureCoordinate).r-0.5; Highp float v = texture2D(Texture_v, textureCoordinate).r-0.5; Highp float r = y + 1.402 * v; Highp float g = y-0.344 * u-0.714 * v; Highp float b = y + 1.772 * u; Gl_FragColor = vec4(r, g, b, 1.0); } # part of the color gamut BT.709 YUV transform RGB VARYING highp VEC2 textureCoordinate; uniform sampler2D texture_y; uniform sampler2D texture_u; uniform sampler2D texture_v; Void main() {highp float y = texture2D(texture_y, textureCoordinate).r-0.0625; Highp float u = texture2D(texture_u, textureCoordinate).r-0.5; Highp float v = texture2D(Texture_v, textureCoordinate).r-0.5; Highp float r = 1.164 * y + 1.793 * v; Highp float g = 1.164 * y-0.213 * u-0.533 * v; Highp float b = 1.164 * y + 2.112 * u; Gl_FragColor = vec4(r, g, b, 1.0); } # BT.709 YUV transform RGB VARYING highp VEC2 textureCoordinate; uniform sampler2D texture_y; uniform sampler2D texture_u; uniform sampler2D texture_v; void main() { highp float y = texture2D(texture_y, textureCoordinate).r; Highp float u = texture2D(texture_u, textureCoordinate).r-0.5; Highp float v = texture2D(Texture_v, textureCoordinate).r-0.5; Highp float r = y + 1.280 * v; Highp float g = y-0.215 * u-0.381 * v; Highp float b = y + 2.128 * u; Gl_FragColor = vec4(r, g, b, 1.0); }Copy the code

Libyuv way

Libyuv: libyuv: libyuv: libyuv: libyuv: libyuv: libyuvGithub.com/lemenkov/li…The header file is under include, and the source code is under source.

If you are compiling a library for Android, follow these rules when compiling.

LOCAL_PATH:= $(call my-dir) include $(CLEAR_VARS) LOCAL_CPP_EXTENSION := .cc LOCAL_SRC_FILES := \ source/compare.cc \ source/compare_common.cc \ source/convert.cc \ source/convert_argb.cc \ source/convert_from.cc \ source/convert_from_argb.cc \ source/convert_to_argb.cc \ source/convert_to_i420.cc \ source/cpu_id.cc \ source/planar_functions.cc \ source/rotate.cc \ source/rotate_any.cc \ source/rotate_argb.cc \ source/rotate_common.cc \  source/row_any.cc \ source/row_common.cc \ source/scale.cc \ source/scale_any.cc \ source/scale_argb.cc \ source/scale_common.cc \ source/video_common.cc ifeq ($(TARGET_ARCH_ABI),armeabi-v7a) LOCAL_CFLAGS += -DLIBYUV_NEON LOCAL_SRC_FILES += \ source/compare_neon.cc.neon \ source/rotate_neon.cc.neon \ source/row_neon.cc.neon \ source/scale_neon.cc.neon endif ifeq ($(TARGET_ARCH_ABI),arm64-v8a) LOCAL_CFLAGS += -DLIBYUV_NEON LOCAL_SRC_FILES += \ source/compare_neon64.cc \ source/rotate_neon64.cc \ source/row_neon64.cc \ source/scale_neon64.cc endif ifeq ($(TARGET_ARCH_ABI),$(filter $(TARGET_ARCH_ABI), x86 x86_64)) LOCAL_SRC_FILES += \ source/compare_gcc.cc \ source/rotate_gcc.cc \ source/row_gcc.cc \ source/scale_gcc.cc  endif LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include LOCAL_C_INCLUDES += $(LOCAL_PATH)/include LOCAL_MODULE := libyuv  LOCAL_MODULE_TAGS := optional include $(BUILD_SHARED_LIBRARY)Copy the code

Remember to link to the libjpeg library, otherwise formats like full-gamet YUVJ420P will not be recognized. To convert YUV to RGB, take a look at the convert_arGB header: github.com/lemenkov/li…

// Conversion matrix for YUV to RGB
LIBYUV_API extern const struct YuvConstants kYuvI601Constants;   // BT.601
LIBYUV_API extern const struct YuvConstants kYuvJPEGConstants;   // BT.601 full
LIBYUV_API extern const struct YuvConstants kYuvH709Constants;   // BT.709
LIBYUV_API extern const struct YuvConstants kYuvF709Constants;   // BT.709 full
LIBYUV_API extern const struct YuvConstants kYuv2020Constants;   // BT.2020
LIBYUV_API extern const struct YuvConstants kYuvV2020Constants;  // BT.2020 full

// Conversion matrix for YVU to BGR
LIBYUV_API extern const struct YuvConstants kYvuI601Constants;   // BT.601
LIBYUV_API extern const struct YuvConstants kYvuJPEGConstants;   // BT.601 full
LIBYUV_API extern const struct YuvConstants kYvuH709Constants;   // BT.709
LIBYUV_API extern const struct YuvConstants kYvuF709Constants;   // BT.709 full
LIBYUV_API extern const struct YuvConstants kYvu2020Constants;   // BT.2020
LIBYUV_API extern const struct YuvConstants kYvuV2020Constants;  // BT.2020 full

int I420ToARGB(const uint8_t* src_y,
               int src_stride_y,
               const uint8_t* src_u,
               int src_stride_u,
               const uint8_t* src_v,
               int src_stride_v,
               uint8_t* dst_argb,
               int dst_stride_argb,
               int width,
               int height);
Copy the code
  • Src_y, SRC_u, and SRC_V represent the data sources of Y/U/V components. Src_stride_y, SRC_STRIde_u, and SRC_stride_V represent the width and heights of Y/U/V components.
  • Dst_argb represents the data source converted to ARGB. Dst_stride_argb is 4 * width, which is the summary of the data of the original 4 components.

FFmpeg swscale way

FFmpeg already has swscale to convert YUV to RGB, but it also needs to compile libjpeg library to achieve this. The implementation is as follows:

struct SwsContext *img_convert_ctx = NULL;
img_convert_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, AV_PIX_FMT_RGB24, SWS_FAST_BILINEAR, NULL, NULL, NULL);
AVFrame *pFrameRGB = av_frame_alloc();
sws_scale(img_convert_ctx, (const unsigned char* const*)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
Copy the code
  • First, create the SWS context
  • Initialize the SWS context, determine the target PIX_FMT format to be converted, and the conversion algorithm
  • Initialize the target data and invoke the SWs_scale transformation

PCodecCtx ->pix_fmt is the source pix_fmt, depending on the specific format of the video source you entered. AV_PIX_FMT_RGB24 is the output of PIX_FMT and SWS_FAST_BILINEAR is the transform algorithm. SWS_FAST_BILINEAR is recommended for high speed and high quality.

/* values for the flags, the stuff on the command line is different */
#define SWS_FAST_BILINEAR     1
#define SWS_BILINEAR          2
#define SWS_BICUBIC           4
#define SWS_X                 8
#define SWS_POINT          0x10
#define SWS_AREA           0x20
#define SWS_BICUBLIN       0x40
#define SWS_GAUSS          0x80
#define SWS_SINC          0x100
#define SWS_LANCZOS       0x200
#define SWS_SPLINE        0x400
Copy the code

The advantages and disadvantages of three conversion modes

Conversion and OpenGL shader libyuv FFmpeg swscale
advantages Speed is fast

Do not increase package volume		 Fast speed, only second OpenGL shader mode

Good compatibility

Fully functional		 FFmpeg native
disadvantages General compatibility Libjpeg needs to be accessed Slow speed

Libjpeg needs to be accessed