Introduction to the Playback Process

The process of playing video must be very clear to everyone. Let’s take a look at the following simple flow chart of playing video:

The data packets obtained after unencapsulation include audio data packets and video data packets. In this case, two queues are generally prepared, one for video data packets and the other for audio data packets.

  • Av_read_frame is used to read packets in FFmpeg, and the read packets are stored in AVPacket
  • If packet is found to be audio, it is placed in the audio queue
  • If the packet is found to be a video, it is placed in the video queue

The size of the queue is limited, right? If the queue size limit is reached, the current packet reading thread should be suspended until the decoding thread decodes the packets in the queue.

if (context->video_packet_queue->nb_packets >= context->video_packet_queue->max_size) {
  pthread_mutex_lock(&context->packet_full_mutex_t);
  pthread_cond_wait(&context->packet_full_cond_t, &context->packet_full_mutex_t);
  pthread_mutex_unlock(&context->packet_full_mutex_t);
}
Copy the code

Because the video data is much larger than the audio data, the video queue is also locked and the audio queue is not considered.

Another thread is started to fetch the data from the queue, and its main job is to decode it. When the packet in the queue is taken out and decoded, the locked thread is reawakened so it can continue reading the packet. The above brief introduction is the main flow of the player.

Is the size of the data queue something to consider?

  • video packet queue
  • audio packet queue

Generally, the frame rate of video is 30fps, 30 frames per second. It is appropriate to design the size of video queue as 30 * 5, which can store 5s video packet data. Such references are fine under normal circumstances, but this article just happens to analyze a particular kind of video.

Special Video introduction

Audio packet and video packet alternate in normal video, which is normal. There will be no problem when using packet queue to control audio and video synchronization. As shown below, only audio packet or video packet will not appear.

Packet, audio, 1-5058-0.114694, - 5058, 0.114694, 2048,0.046440, 186103, 651 KD Packet, video, 0,0,0.000000, - 7200, 0.080000, 3600,0.040000, 6528103, 837, K_ Packet, audio, 1-3010-0.068254, - 3010, 0.068254, 2048,0.046440, 186110, 365 KD Packet, video, 0720 0,0.080000, - 3600, 0.040000, 3600,0.040000, 11475110, 551, __ Packet, audio, 1-962-0.021814, - 962, 0.021814, 2048,0.046440, 186122, 026, K_ Packet, video, 0360 0,0.040000, 0,0.000000,,0.040000 3600, 124122, 212, __ 6,0.024626 packet, audio, 1108, 1086,0.024626,0.046440 2048, 185122, 336, K_ Packet, video, 0144, 00,0.160000, 3600,0.040000,0.040000 3600, 19436122, 521, __ 4,0.071066 packet, audio, 1313, 3134,0.071066,0.046440 2048, 186141, 957, K_ Packet, video, 0108, 00,0.120000, 7200,0.080000,0.040000 3600, 161142, 143, __ 2,0.117506 packet, audio, 1518, 5182,0.117506,0.046440 2048, 272142, 304, K_ Packet, video, 0252, 00,0.280000, 10800,0.120000,0.040000 3600, 12728142, 576, __ Packet, video, 0180, 00,0.200000, 14400,0.160000,0.040000 3600, 163155, 304, __ Packet, audio, 1723 0,0.163946, 7230,0.163946,0.046440 2048, 250155, 467, K_ Packet, video, 0216, 00,0.240000, 18000,0.200000,0.040000 3600, 189155, 717, __ 8,0.210385 packet, audio, 1927, 9278,0.210385,0.046440 2048, 236155, 906, K_ Packet, video, 0396, 00,0.440000, 21600,0.240000,0.040000 3600, 4613156, 142, __ 26,0.256825 packet, audio, 1113, 11326,0.256825,0.046440 2048, 231160, 755, K_ Packet, video, 0324, 00,0.360000, 25200,0.280000,0.040000 3600, 887160, 986, __ Packet, audio, 1133, 74,0.303265, 13374,0.303265,0.046440 2048, 212161, 873, K_ Packet, video, 0288, 00,0.320000, 28800,0.320000,0.040000 3600, 491162, 085, __ 22,0.349705 packet, audio, 1154, 15422,0.349705,0.046440 2048, 220162, 576, K_ Packet, video, 0360, 00,0.400000, 32400,0.360000,0.040000 3600, 532162, 796, __ Packet, audio, 1174, 70,0.396145, 17470,0.396145,0.046440 2048, 237163, 328, K_Copy the code

However, there are still some very special videos. Audio packet lags seriously, which is caused by problems in packaging. The following example is that there are almost all video packets but no audio packet, so the audio packet queue will be overloaded. The queue mode and audio and video synchronization mechanism should be designed to prevent the picture from getting stuck when playing the video.

packet,video,0,2048,0.135593,768,0.050847,256,0.016949,2010,32204,__

packet,video,0,1536,0.101695,1024,0.067797,256,0.016949,286,34214,__

packet,audio,1,3072,0.069660,3072,0.069660,1024,0.023220,456,34500,K_

packet,video,0,1280,0.084746,1280,0.084746,256,0.016949,135,34956,__

packet,audio,1,4096,0.092880,4096,0.092880,1024,0.023220,465,35091,K_

packet,video,0,1792,0.118644,1536,0.101695,256,0.016949,147,35556,__

packet,audio,1,5120,0.116100,5120,0.116100,1024,0.023220,431,35703,K_

packet,video,0,3072,0.203390,1792,0.118644,256,0.016949,3726,36134,__

packet,video,0,2560,0.169492,2048,0.135593,256,0.016949,214,39860,__

packet,audio,1,6144,0.139320,6144,0.139320,1024,0.023220,404,40074,K_

packet,video,0,2304,0.152542,2304,0.152542,256,0.016949,858,40478,__

packet,video,0,2816,0.186441,2560,0.169492,256,0.016949,96,41336,__

packet,video,0,4096,0.271186,2816,0.186441,256,0.016949,3472,41432,__

packet,video,0,3584,0.237288,3072,0.203390,256,0.016949,607,44904,__

packet,video,0,3328,0.220339,3328,0.220339,256,0.016949,519,45511,__

packet,video,0,3840,0.254237,3584,0.237288,256,0.016949,104,46030,__

packet,video,0,5120,0.338983,3840,0.254237,256,0.016949,945,46134,__

packet,video,0,4608,0.305085,4096,0.271186,256,0.016949,99,47079,__

packet,video,0,4352,0.288136,4352,0.288136,256,0.016949,117,47178,__

packet,video,0,4864,0.322034,4608,0.305085,256,0.016949,46,47295,__

packet,video,0,6144,0.406780,4864,0.322034,256,0.016949,1462,47341,__

packet,video,0,5632,0.372881,5120,0.338983,256,0.016949,253,48803,__

packet,video,0,5376,0.355932,5376,0.355932,256,0.016949,321,49056,__

packet,video,0,5888,0.389831,5632,0.372881,256,0.016949,94,49377,__

packet,video,0,7168,0.474576,5888,0.389831,256,0.016949,809,49471,__

packet,video,0,6656,0.440678,6144,0.406780,256,0.016949,146,50280,__

packet,video,0,6400,0.423729,6400,0.423729,256,0.016949,56,50426,__

packet,video,0,6912,0.457627,6656,0.440678,256,0.016949,105,50482,__

packet,video,0,8192,0.542373,6912,0.457627,256,0.016949,1851,50587,__

packet,video,0,7680,0.508475,7168,0.474576,256,0.016949,145,52438,__

packet,video,0,7424,0.491525,7424,0.491525,256,0.016949,336,52583,__

packet,video,0,7936,0.525424,7680,0.508475,256,0.016949,48,52919,__

packet,video,0,9216,0.610169,7936,0.525424,256,0.016949,771,52967,__

packet,video,0,8704,0.576271,8192,0.542373,256,0.016949,118,53738,__

packet,video,0,8448,0.559322,8448,0.559322,256,0.016949,71,53856,__

packet,video,0,8960,0.593220,8704,0.576271,256,0.016949,46,53927,__

packet,video,0,9984,0.661017,8960,0.593220,256,0.016949,268,53973,__

packet,video,0,9472,0.627119,9216,0.610169,256,0.016949,71,54241,__

packet,video,0,9728,0.644068,9472,0.627119,256,0.016949,44,54312,__

packet,video,0,10240,0.677966,9728,0.644068,256,0.016949,35307,54356,__

packet,video,0,11264,0.745763,9984,0.661017,256,0.016949,3893,89663,__

packet,video,0,10752,0.711864,10240,0.677966,256,0.016949,440,93556,__

packet,video,0,10496,0.694915,10496,0.694915,256,0.016949,144,93996,__

packet,video,0,11008,0.728814,10752,0.711864,256,0.016949,359,94140,__

packet,audio,1,7168,0.162540,7168,0.162540,1024,0.023220,390,94499,K_

packet,video,0,12288,0.813559,11008,0.728814,256,0.016949,5313,94889,__

packet,video,0,11776,0.779661,11264,0.745763,256,0.016949,516,100202,__

packet,video,0,11520,0.762712,11520,0.762712,256,0.016949,207,100718,__

packet,video,0,12032,0.796610,11776,0.779661,256,0.016949,106,100925,__

packet,video,0,13312,0.881356,12032,0.796610,256,0.016949,4026,101031,__

packet,video,0,12800,0.847458,12288,0.813559,256,0.016949,328,105057,__

packet,video,0,12544,0.830508,12544,0.830508,256,0.016949,91,105385,__

packet,video,0,13056,0.864407,12800,0.847458,256,0.016949,935,105476,__

packet,video,0,14336,0.949153,13056,0.864407,256,0.016949,2153,106411,__

packet,video,0,13824,0.915254,13312,0.881356,256,0.016949,251,108564,__

packet,video,0,13568,0.898305,13568,0.898305,256,0.016949,107,108815,__

packet,video,0,14080,0.932203,13824,0.915254,256,0.016949,56,108922,__

packet,video,0,15360,1.016949,14080,0.932203,256,0.016949,4559,108978,__

packet,video,0,14848,0.983051,14336,0.949153,256,0.016949,888,113537,__

packet,video,0,14592,0.966102,14592,0.966102,256,0.016949,123,114425,__

packet,video,0,15104,1.000000,14848,0.983051,256,0.016949,915,114548,__

packet,video,0,16384,1.084746,15104,1.000000,256,0.016949,601,115463,__

packet,video,0,15872,1.050847,15360,1.016949,256,0.016949,106,116064,__

packet,video,0,15616,1.033898,15616,1.033898,256,0.016949,51,116170,__

packet,video,0,16128,1.067797,15872,1.050847,256,0.016949,50,116221,__

packet,video,0,16640,1.101695,16128,1.067797,256,0.016949,156,116271,__

packet,video,0,17152,1.135593,16384,1.084746,256,0.016949,8093,116427,__

packet,video,0,16896,1.118644,16640,1.101695,256,0.016949,2801,124520,__

packet,video,0,18176,1.203390,16896,1.118644,256,0.016949,2568,127321,__

packet,video,0,17664,1.169492,17152,1.135593,256,0.016949,1169,129889,__

packet,video,0,17408,1.152542,17408,1.152542,256,0.016949,312,131058,__

packet,video,0,17920,1.186441,17664,1.169492,256,0.016949,193,131370,__
Copy the code

The solution

Dynamically expanding the Video Packet queue

For a special video, the frame rate of the video needs to be paid attention to. For example, the frame rate of the video above is 60 FPS. Set an initial size of the Video Packet queue and set it to 5 * 60 = 300. If the video queue is full and the audio queue is empty, the video queue size is expanded. However, there is a limit to the size of the video queue, so you need to set a limit.

if (context->video_packet_queue->nb_packets >= context->video_packet_queue->max_size) { if (context->audio_packet_queue->nb_packets <= 5) { context->video_packet_queue->max_size = context->video_packet_queue->max_size + 30; if (context->video_packet_queue->max_size >= 500) { pthread_mutex_lock(&context->packet_full_mutex_t); pthread_cond_wait(&context->packet_full_cond_t, &context->packet_full_mutex_t); pthread_mutex_unlock(&context->packet_full_mutex_t); } } else { pthread_mutex_lock(&context->packet_full_mutex_t); pthread_cond_wait(&context->packet_full_cond_t, &context->packet_full_mutex_t); pthread_mutex_unlock(&context->packet_full_mutex_t); }}Copy the code

Audio and video synchronization Discarded the Audio packet

There are also problems with the above Settings. What if you really encounter very special videos with very little audio data, or all at the back of the queue? In this case, it does not matter how large the queue is set, or it may cause Out of memory because the queue is too large. In fact, such a situation does occur, indicating that there is indeed a problem in video encapsulation. However, in order not to affect the experience, it is recommended to discard some expired audio in the case of audio and video synchronization. It’s audio and video syncing using the PTS of the video.

Server-side processing

The fundamental solution to this problem is server reencapsulation, which arranges video and audio packets one after another.