1. An overview of the

The Intra Prediction technology of H.266/VVC mainly includes improvements and new tools in the following aspects [1][2] :

• 67 in-frame prediction modes, including Wide Angle mode extensions
• Block size and mode dependent 4-head interpolation filter
• Position Dependent Intra Prediction Combination PDPC
• Cross Component Linear Model Intra Prediction CCLM
• Multiple Reference Line Intra Prediction MRL
• Intra Sub-Partition ISP
• Matrix Weighted Intra Prediction MIP

In the VVC standard, detailed tool-off tests for coding gain performance of each newly added coding tool are summarized in the AHG 13 report for each JVET meeting [3]. Here a simple classification is made according to the general technical category of each coding tool in the process of coding. (The latest VTM 10 test data is basically the same as the VTM 9 data provided here [3].)

The above table is collated from [3]

Let’s take a look at each of these new VVC intra – frame prediction enhancements and tools.

2. Prediction mode within 67 frames

VVC extended the number of angle-based intra-frame prediction modes from 33 in HEVC to 65. Add in the original DC and Planar models, and there are 67 prediction models.

The figure above and table are from [1]

In addition, the patterns -14~-1 and 67~80 represent the prediction patterns of large Angle (i.e. wide Angle) frames for rectangular CU (Coding Unit) blocks.

2.1 Grammar of summons

The signalling syntax for the signalling mode within a VVC frame is as follows:

The table above is from [1]

Basically, it first determines whether the prediction mode in the transmitted frame is MIP mode, then whether it is MRL, ISP mode, and then whether it is Index of one of the five candidate Modes of MPM (Most Probable Modes). Finally, if the transmitted mode is not in the five candidate MPM, MPM Remainder continues to represent the most general Intra Prediction Mode.

Intra Prediction Mode

• Judge first, if it is MipChromaDirect: So single-tree, 4:4:4, intra_chroma_pred_mode = 4, In this case, the MIP Mode will use Collocated Luma Intra Mode (the Intra Mode of the brightness block corresponding to the same location) as lumaIntraPredMode.
• Otherwise, the Intra Mode of the Luma block at the same position corresponding to the block center is used as lumaIntraPredMode, unless:
  • For MIP mode, set lumaIntraPredMode to Planar mode.
  • If it is in IBC (Intra Block Copy) or Palette mode, set lumaIntraPredMode to DC mode.
• Then, according to lumaIntraPredMode, the Chroma IntraPredMode of the current block is deduced according to the corresponding relation in the following table:

The table above is from [1]

2.2 Wide-angle mode for rectangular blocks

The usual Mode 2 (45° prediction) to Mode 67 (-135° prediction) range is mainly for square CU blocks. For rectangular blocks, if the same prediction range of 45°~-135° is still followed, the corresponding reference samples for some predicted angles near 45° or -135° will exceed the usual reference sample range of (2W + 1) or (2H + 1) (W, H is the sample number of the width and length of the block). At the same time, the prediction distance between the reference samples beyond the range and the current sample will be long, thus reducing the prediction performance.

In order to maintain the same reference samples in the length and width direction (2W + 1) or (2H + 1), and reduce the long prediction distance caused by the reference samples beyond the range, so as to improve the prediction performance, VVC introduces the wide-angle prediction mode for rectangular blocks: For those predicted angles in the vicinity of 45° or -135° corresponding to the square block, which applied to the current rectangle speed would result in exceeding the original reference sample range of (2W + 1) or (2H + 1), This pattern is remapped to the reverse predicted Angle of the reference sample on the other side (or broad or long) of the same predicted Angle extension line. In other words, the predModeIntra transmission mode remains unchanged. If it is found that the current CU is a rectangular block that meets the above conditions, predModeIntra will be adaptively converted. One of the modes -14~-1, 67~80 represents the large Angle intra-frame prediction mode for rectangular CU blocks. As shown in the figure below.

The image above is from [2]

4. Mode Dependent Intra Smoothing MDIS

In some cases, the prediction performance can be improved by filtering the reconstructed reference samples used for intra-frame prediction. This technology has been applied in HEVC, and it has been further optimized and refined in VVC.

• Smoothing filtering for luminance reference samples:

  For Mode 0 (Planar), Mode 2, 34, 66 (bottom-left, top-left, top-right), and wide-angle modes of non-fractional reference sample positions (mode-14, -12, -10, -6, 72, 76, 78, 80), and if it is not MRL or ISP mode and the total number of CU samples > 32, then the prediction reference samples within the brightness frame will be smoothed by a 3-head filter with coefficients of {1/4, 2/4, 1/4}.

• Interpolation filtering for fractional reference samples:

  Otherwise, if not MRL or ISP mode: continue to judge whether the predicted Angle is close to horizontal or vertical based on the defined MDIS condition. If it is judged to be close, a 4-head interpolation filter will be used for the reference sample of fractional position, mainly for interpolation and with little smoothing effect. Otherwise, if the Angle deviates from the horizontal and the vertical is far, the 4-head Gaussian smoothing filter will be used for the reference sample at the fractional position, which has a great smoothing effect.

  MDIS conditions are defined as follows[1]:

  The above table and related contents are from [1]

  4-head interpolation and Gaussian filter are defined as follows (in part)[1]:

The above table and related contents are from [1]

4. Cross-component Linear Model CCLM prediction

In VVC, a new CCLM model is introduced to predict chromaticity signal within the frame of brightness reconstruction information, which adopts a simple linear model to predict.

Among them,
$rec_{L}\prime(i,j)$Is the reconstructed luminance sample after the downsampling[2].

The image above is from [2]

• First of all, four adjacent samples are selected from either the upper or the left undersampling brightness reconstruction samples at positions {1/8, 3/8, 5/8, 7/8}, or two adjacent samples are selected from the upper and the left respectively at positions {1/4, 3/4}.
• Then, the 4 luminance reference samples were sorted into two smaller value groups (minY1, minY2) and two larger value groups (maxY1, maxY2), and the smaller of the reference samples after the same sorting was reconstructed by combining the chroma of the corresponding positions of their respective samples. The larger value groups (minC1, minC2) and (maxC1, maxC2) were averaged to obtain (minY, maxY) and (minC, maxC).
• The model parameters α and β can be calculated by the following simple linear model derivation formula:

There are three ways to select only 4 luminance samples from the top, or 4 luminance samples from the left, or 2 luminance samples from the top, or 2 luminance samples from the left, which are the three different modes of CCLM lM-A, LM-L, LM. The definition is as follows [2] :

The image above is from [2]

5. Intra Sub-partitions ISP

Intra – frame prediction in VVC introduces the subblock partitioning technology, that is, the blocks for intra – frame prediction can be divided into smaller subblocks and reconstructed sequentially, so that the subsequent subblocks can make more accurate intra – frame prediction by using the previously reconstructed subblocks. In VVC, each subblock is either all horizontal or all vertical, and each subblock must use the same intra-frame prediction mode. In this way, the coding performance of ISP mode is mainly improved by saving the rate of multiple sub-blocks using the same prediction mode, because the intra-frame prediction mode information only needs to be transmitted once for multiple sub-blocks.

In VVC, only the brightness blocks with more details are allowed to do ISP. An intra-frame prediction brightness CU block can be divided into two (for 4×8, 8×4 CUs) or four (for larger CU blocks) sub-blocks.

• All sub-blocks must use the same Intra Prediction Mode.
• Max ISP CU size: 64×64: This ensures that the corresponding chroma block size does not violate the VVC design requirements that Virtual Pipeline Data Unit (VPDU) of 64×64 brightness can be decoded in parallel at the decoding end.
• Min PU (Prediction Unit) The minimum Prediction block width is 4: This limit is intended to improve the worst-case throughput of on-chip memory reads and writes in the decoder implemented by the hardware to support real-time decoding more efficiently. Too small predictive block widths cause hardware to read memory too frequently, which reduces the traffic efficiency of the hardware decoder. Based on this PU width limit, the VVC ISP does not allow 1xN, 2xN PU, but does allow 1xN, 2xN TU (Transform Unit).
• Interaction with other in-frame prediction modes: In order to improve the overall encoding performance, ISP mode and other in-frame prediction modes set some mutual restrictions, such as: if MRL is used, no ISP can be used; If you use ISP, you can’t use PDPC, etc.

The image above is from [2]

Multiple Reference Line Intra Prediction MRL

The VVC extends the single top or left reference sample row used in the previous standard so that a single top or left reference row can be selected from multiple top or left reference rows for intra-frame prediction.

The image above is from [2]

• In VVC, intra-frame prediction of brightness blocks is finally allowed, with the option of two additional Reference lines (Reference lines 1 and 2 in the figure above). In this way, the same reference row storage requirement can be maintained with the brightness reference sample required by CCLM’s brightness reference sample under sampling, thus ensuring the consistency and simplicity of the design.
• If MRL is used, the prediction model is encoded in MPM and can no longer be PDPC.
• MRL is not allowed for the first line of CU blocks in a CTU, because it is required to store reconstruction reference samples outside the current CTU, which is not allowed in the overall design of VVC.

7. Basic methods and principles of intra-frame prediction

Starting with simple DC, Planar, Directional, and conventional in-frame prediction patterns, we look at the general, basic approach of in-frame prediction techniques, namely, the various weighted averages based on the reconstruction of reference samples in the left and top frames. The principle of different in-frame prediction techniques is to design different types of weighted average algorithms to generate in-frame prediction samples by discovering and utilizing different types of spatial redundancy information between reference samples and current blocks through experimental analysis. Then we move on to the two new intra – frame prediction model extensions introduced in VVC, PDPC and MIP.

7.1 DC and Planar Intra Prediction

The simplest DC model commonly used is directly the arithmetic average of each reference sample. (In VVC, the arithmetic average of all reference samples for the rectangular block, that is, the longer side.)

The generally effective Planar pattern is, in principle, the weighted average of the horizontal and vertical reference samples corresponding to the current sample position in terms of the distance from the current sample. (Here, the use of the reference samples in the lower left and upper right corner of the current block in Planar Mode is equivalent to the padding-copy operation of the reference samples in the horizontal and vertical directions respectively, as an estimation of the unknown reference samples in the same position. Then the principle of Planar Mode becomes clear.)

Directional Intra Prediction

As the main body, the directional in-frame prediction model based on prediction Angle directly predicts the current sample by using the in-frame reconstruction reference sample at a certain prediction Angle.

Read in VVC, in order to facilitate memory, improve efficiency, for may at the same time use the Angle to the left and top reference sample, pattern, taken along the main most block reference samples (or left) above, the edge of the reverse, by the other side of the corresponding reference samples of predicting reverse projection Angle, to produce the required reverse extend the extension of the reference sample.

The Angle prediction of the extended reference sample using reverse lengthening is shown in the figure below.

8. Position Dependent Intra Prediction Combination PDPC

To compensate for more than a simple frame prediction model in terms of use of the space redundancy, introducing a kind of VVC according to the current position of the sample and the Angle of the frame prediction model adaptive selection of reference sample in opposite direction Angle as a relatively new complementary information to predict the frame, and then to predict and Angle of the original frame to do it based on the weighted average of sample distance. The effect is that the original Angle prediction can make full use of the reference sample information corresponding to the positive and negative directions of the Angle, so as to improve the prediction performance.

RefL [x,y]refL[x,y]refL[x,y] refT[x,y]refT[x,y]refT[x,y] refT[x,y]refT[x,y]refT[x,y] WT [y]wT[y]wT[y] wT[y] is their respective weight based on horizontal or vertical distance. Pred [x,y]pred[x,y]pred[x,y] pred[x,y]pred[x,y] is the original intra-frame prediction sample. The left pred[x,y]pred[x,y]pred[x,y] is the predicted sample value with PDPC compensation finally included. WL [x]wL[x]wL[x], wT[y]wT[y]wT[Y], the weight of PDPC will become 0 for (x, y) beyond a certain distance.

The PDPC in VVC can only be used in conjunction with DC, Planar, lower than or equal to the predicted reference Angle pattern, and greater than or equal to the vertical reference Angle pattern.

• For DC and Planar modes:

• For horizontal or vertical Angle prediction model (take horizontal prediction as an example) :

• For the prediction model of angles below the horizontal or outside the vertical (take the prediction of angles below the horizontal as an example) :

9. Matrix Weighted Intra Prediction MIP

Compared with the conventional Angle prediction model, the newly introduced MIP model in VVC represents another kind of new intra-frame prediction model obtained through machine learning training. MIP includes a total of 60 new prediction patterns for three different block sizes. Considering the influence of matrix transpose factor (one MIP mode matrix can be transposed to define two MIP modes in total), there are a total of 30 MIP mode definition matrices, aiming at different block sizes of different classes. There are more MIP matrices defined for small blocks in the population, and fewer MIP matrices defined for large blocks.

The image above is from [2]

In MIP mode, the reference samples on the left and the top are sampled and averaged, and then the matrix is multiplied by the pattern matrix related to the block size obtained by training. Finally, a constant adjustment vector term of the pattern matrix is added. The resulting predicted sample is then upsampled to the original block size. The classification of block sizes is as follows:

The image above is from [2]

Where W, H are the width and height of the current block, and IDx is the final category Index. The idx 0 class of minimum block defines 16 MIP matrices, while idx 1,2 define 8 and 6 MIP matrices respectively. By transpose, the 30 matrices represent 60 MIP patterns.

It can be seen that MIP technology actually uses machine learning methods to conduct linear modeling of efficient in-frame prediction patterns in a statistical sense by means of matrix definition. The weighted matrix defined here for each sample position in the block is actually the most generalized intra-frame prediction method using the weighted average of all reference samples for each sample in the block, as shown below.

Nodule 10.

Compared with HEVC, VVC refines and optimizes some existing intra-frame prediction techniques in multiple directions:

• Double refinement of the prediction Angle, and a more effective prediction for rectangular blocks in the wide-angle model
• More efficient adaptive 4-head interpolation and smooth filtering
• Multiple reference lines
• Multiple subblock partitioning

In addition, VVC has introduced several new intra-frame prediction techniques:

• CCLM prediction of chromaticity signal using reference sample information in luminance frame
• The PDPC predicted within the original frame was weighted by reference samples in the direction of reverse complementary Angle
• MIP of the most generalized intra – frame prediction method for matrix linear modeling using machine learning

reference

[1] B. Bross, et al., “Versatile Video Coding Refinements for Versatile Video Coding”, JVEt-T2001

[2] Chen, J., et al., “Versatile Video Coding based on multiplex Algorithm and Its Application”, JVEt-T2002

[3] W.-J. Chien, et al., “JVET REPORTING Procedure and Testing (AHG13)”, JVEt-T0013