Ffmpeg Development player learning Notes – know YUV

This section is the third section of ffMPEG Development player learning Notes called “Getting to know YUV”.

To put it simply, YUV is an image and video coding method. RGB expresses various colors in the real world through three colors, while YUV expresses colors through brightness and chroma saturation. RGB is easy to understand. It’s more intuitive. From the beginning, students have learned that the pigments for painting are mixed with three colors, and the paint color of cars is also mixed with RGB three colors. The emergence of YUV has its historical significance but is also inevitable. It is based on the fact that the human eye is more sensitive to brightness than to color. Y represents brightness, which can also be understood as gray value. The lowest brightness is black, the highest brightness is white, and the middle brightness can be gray. In the transition from black and white TV to color TV, black and white TV only needs one component Y in YUV to present a black and white picture. UV components can be used in color TV sets to present color. YUV can bring a higher intra-frame compression ratio, and since the human eye is more sensitive to black and white, YUV can weaken insensitive information and reduce the sampling of UV components. RGB24 requires 3*8 bytes per pixel, what about YUV? Different YUV sampling methods have different compression ratios. TV messages use YUV while digital messages use YCrCb, which is hereinafter referred to as YUV.

✅ section 1 – Hello FFmpeg ✅ section 2 – Soft unclip video stream, render RGB24 🔔 section 3 – understand YUV 📗 section 4 – hard decoding,OpenGL render YUV Disk section 5 – Metal render YUV disk section 6 – Decode audio, play with AudioQueue 📗 section 7 – audio and video synchronization 📗 Section 8 – Perfect playback control 📗 Section 9 – double speed play 📗 section 10 – increase video filtering effect section 11 – audio changes

The Demo address of this section is github.com/czqasngit/f… The example code provides both Objective-C and Swift implementations. This article cites Objective-C code for ease of illustration, since Swift code Pointers don’t look neat.

The target

• Understand YUV sampling methods
• Understand the YUV storage mode

Understand YUV sampling methods

4:4:4 sampling

Each Y corresponds to a set of UV, which is similar to RGB sampling. This sampling method preserves all the details, brightness and chroma, so the compression rate is very low, but the benefit is that the restored image is perfect. It is most used in the bandwidth requirements are not high and the image quality characteristics are high detail requirements of the scene. For the 4:4:4 sampling method, sampling code is the same as storing and restoring.

4:2:2

The 4:2:2 sampling method means that 4 Y corresponds to two sets of UVs, that is, 2 Y corresponds to one set of UVs for sampling. When you do the reduction, you also share a set of UVs with each of the adjacent Y’s. In this sampling method, part of the chromaticity data is discarded and the chromaticity is shared during restoration. Most of the images or videos are insensitive and cannot be felt, but the original sharp edges may be blurred in the images or videos that require high clear boundaries.

4:1:1

With the original data unchanged, the 4:1:1 sample compression rate is higher, but more chrominance data is discarded, and each 4 Y shares a set of UVs.

4:2:0

4:2:0 sampling doesn’t mean Y sampling, U sampling 2, V sampling 0. It means that the first row samples two Y’s and one U in a 2:1 ratio, without sampling V. The second row samples 2 Y and 2 V, not U.

You can also say horizontal sampling and vertical sampling. The result of this sampling is that the two adjacent related lines are combined into a coding group in a 2×2 way, and each 2×2 contains 6 data, which is restored to a 4-pixel combination.

This sampling method has the same compression ratio as 4:1:1. The lost data is less sensitive than the 4:1:1 sampling method.

Understand the YUV storage mode

In addition to the sampling method and RGB are different, there are also differences between the existence method and RGB, YUV has two storage methods:packagingwithflat.

Packaging storage: YUV data is interleaved and continuously stored on a plane.

The flat storage: Y, UV or Y,U,V are stored on different planes.

Different YUV formats have different storage modes. Here, 420P in FFmpeg is used as an example. YUV data is stored on three planes. Here’s how he stored it:

The common NV12 format in CoreVideo for macOS is the following:

So that’s the basics of sampling and storage.

Once you have some understanding of YUV, you can render YUV data frames.