First take a look at the effect display:

1. VAP

VAP (Video Animation Player) is an implementation scheme of penguin esports development, which is used to play cool animated pictures.

  • Compared with Webp, Apng GIF scheme has the advantages of high compression rate (smaller material) and hardware decoding (faster decoding)

  • Can achieve more complex animation effects (such as particle effects) than Lottie

Also, VAP can incorporate custom attributes (such as user name, avatar) into animations.

2. Project background

Penguin esports is a live streaming platform that needs to display cool animations of gift giving in the live broadcast room.

The more cool the animation is, the higher the requirements for the size of the material and decoding performance. We investigated many schemes, and first give the comparison table:

  • Test parameters:

  • Phone: mi mix3

  • Footage: 736 × 576 80 frames

  • Apng: 75 mass; Webp: 75 quality; VAP: 2000 bit rate

Research plan:

1. Vector animation scheme (Lottie) : Lottie vector animation has a high compression rate, but it is not suitable because it cannot display special effects (such as particle effects); 2. GIF scheme (representing GIF, Apng, Webp) :

  • GIF: only supports 8-bit color, color loss is serious, decoding performance is low, can not meet the special effects;

  • Apng, Webp: can meet the special effects, but the file is large and the soft solution efficiency is low (on low-end mobile phones, such as the red Mi 1 of that year, the decoding process can even cause the whole broadcast room to stall), these problems are difficult to accept;

3. Video scheme (representing MP4) : using H264 encoding, compared with the GIF scheme, has a very high compression rate, the hardware decoding efficiency is very high, the shortcomings are obvious, can not support transparent background;

After investigation, it was found that either the special effects performance was not up to the requirements (Lottie, GIF), or the files were too large and soft (Webp, Apng), or they did not support transparency (MP4). These solutions did not meet our needs, and the high-performance animation component VAP was born.

3. Implementation principle

Scheme selection

Mp4 video scheme is the best in terms of effect, size and decoding performance, but H264 stores YUV data without transparent channels. VAP solution based on MP4, to solve the problem of transparency in video, so that both better compression efficiency, and better decoding performance.

Video transparency Implementation

H264 decoded every frame of data is YUV, after converting to RGB, there is no Alpha channel, and we can open an extra area in the video, store the Alpha value in RGB channel, and finally use OpenGL to synthesize these data into ARGB images (images with transparent channels).

For example, after the decoder decodes a frame of raw image, the composition principle is shown below:

Each frame does the same thing, which is to get the video with transparency. The actual video looks like this:

In the original video, the black and white areas carry Alpha data. The latest version of THE VAP has implemented variable Alpha region size. By reducing the Alpha region size, the video resolution can be reduced without affecting the final display, improving compatibility, and freeing up extra region for the fusion features of the VAP.

Animation configuration information

During the animation playback, some configuration information is required to assist the playback (such as Alpha field declaration, including the fusion animation information). The configuration is in JSON format. To make the components easier to use, all related files are merged into an MP4 file, so you only need an MP4 file to play animations.

Mp4 is organized in much the same way as JSON’s key-value organization. It is called a BOX. We create a new BOX (VAPc VAP Config). The mp4 specification defines that if the BOX is not recognized, it will be ignored automatically, which does not affect the normal playback process of MP4.

VAP fusion animation

VAP also supports incorporating custom attributes, such as user names and avatars, into animations. We call it VAP fusion animation.

The video content cannot directly insert the attributes, but can only save the country by curving. By pruning the attribute picture, the user’s eyes are deceived and it looks like it is in the video content, so as to achieve the final fusion effect (the effect is shown at the beginning of the article).

In order to realize attribute image processing, it is necessary to introduce “mask” material, and use the mask and attribute image to carry out porter-duff operation to get the desired shape

Then paste the result to the corresponding coordinate position of the video to achieve the final fusion effect.

The “mask” footage is stored in each frame of video content, and the empty area is utilized by shrinking the Aplha area.

Form a complete set of tools

For everyone more convenient use of components, and supporting material production tools

Past review:

  • Interview tool! Bytedance’s 2021 Android Programmer Interview Guide is now open source
  • Alibaba internal Jetpack treasure book accidentally leaked! The ultimate classic, the ceiling of Android’s architectural components
  • Big guy is strong, bytedance high workers stay up half a month finishing the “componentized actual combat learning manual”, all is the essence!
  • Bytedance has uploaded an “Android Interview Guide for Interviewers,” and it’s all the best without any nonsense
  • 606 pages of new Android interview questions with answers