A hot keyword in the Internet field in the past two years is video live broadcast. From the beginning of game live broadcast and show entertainment, now live broadcast elements have been implanted in various industries.
Netease cloud letterFor many years, I have been deeply engaged in the field of audio and video. In this article, I will talk to you about some key technologies of live video broadcasting.
Related Reading recommendations
How to Quickly Implement short Video function on mobile Terminal?
Video Private Cloud: Building on-demand Private Cloud Platform based on Docker
clarity
4K, 1080p, 720p, these concepts have been hyped by the major TV manufacturers for so many years, the world has understood. 4K is still not popular in Internet video broadcast, mainly because it requires too much network data transmission. 1080p is becoming increasingly popular in high-definition scenarios such as live games, where data transfer rates are around 4Mbps. 720p is now the mainstream definition of live streaming, with a rate of about 1Mbps. In less demanding areas, there will be a 540P or 360P.
fluency
If there is a stutter and a twirl during a live broadcast, it means it is not smooth. The channel between anchors and viewers is like a water pipe with a limited amount of flow, so better clarity may mean less fluency for viewers.
Time delay
Live video is interactive, if you chat with the sister of the show, talk for a long time and no response is slightly pit dad. However, delay is not all bad. Appropriate delay means that there can be a certain amount of video stream data cache on the audience side, and it can resist small range fluctuation when network instability occurs, making the audience unaware.
The first screen time
When the audience enters the studio, the time when the first anchor screen appears is called the first screen time. In order to ensure that the live broadcast is smooth, some data will be cached before the broadcast starts. However, this is not absolute, and will be described in detail later.
Therefore, the relationship between these indicators is summarized at last.
Next, we will describe in detail the whole process of live video broadcasting, how the video streaming data is transferred between the host transmitting end, CDN and audience playing end, and what we can do technically to ensure users’ viewing experience.
1.The first screen seconds away
Start with the moment viewers enter the studio, which is the beginning of the entire live broadcast life cycle. When entering the live broadcast room, the player requests data from the CDN. At this point, assume that the anchor has sent the video stream data to frame 100. Due to some delay of data transmission, the latest data received by the CDN end may be at frame 90. When the CDN receives a request to pull a video stream, it does a very interesting thing, which is to go back a bit of data, which is shown in the figure as going back 2 seconds, which is frame 5 of the video stream. The CDN will send a stream of data from frame 5 onwards to the player via RTMP or other live protocols. So why go back 2 seconds? This is probably one of the more characteristic technical optimizations in the current live video technology, which can be used to balance the smoothness and the first screen seconds. And we’ll see how that works.
2.Fluent playing
What happened next is a good illustration of the power of the 2 seconds back. Since CDN starts to send data from frame 5, all subsequent data are cached in THE CDN server, so data can be continuously sent to the client. The figure shows data from frame 5 to frame 50, all cached in the player memory. This part of data can be used to effectively resist the influence caused by network fluctuation. Of course, one downside of this is that the player’s latency increases by 2 seconds compared to the anchor’s. So what live video is all about is finding a good balance between delay and smoothness.
3.Network congestion
Network congestion is one of the most common situations on the Internet. The next section discusses what happens when network congestion occurs. Suppose that when the viewer reaches frame 150, the user has a problem with the downlink network. If no new data arrives in the player, the picture will inevitably freeze and start to rotate. At this time, the host end will not be aware of this matter, and the host is still pushing video stream data normally. After a lag of about 4 seconds, the network on the audience side is restored and data will flow from the CDN to the player again. By the time you see the network flowing, frame 280 has been stored in the player’s cache, and the current screen is frame 150. What kind of problem does that create? Because the player plays data at a constant speed according to the time stamp of each frame, the absence of any optimization means that the delay of the live broadcast increases by a period of time that is the same as the time it is stuck.
4.Delay after
From what we’ve just described, you already know that delay accumulation is a problem that must be solved. So what the player still needs to do is delay catch-up. The player must detect the data in the cache in real time and start a delay catch-up once it exceeds a certain threshold. You can catch up by either throwing away excess data or you can fast-forward it. Fast forward mode is relatively better user experience, will not produce obvious jump, processing should pay attention to sound not because of fast and then spike. As a final note, delay catch-up should not be too aggressive and should be kept in the cache to mitigate network congestion that may occur again later.
The basic concepts of the first screen startup, smooth playback, network congestion and delayed catch-up as well as what happens in each stage are described above. The whole live broadcast goes back and forth in the three stages of fluency, congestion and delayed catch-up. After reading this article, interested readers can try to use open source software to write their own live streaming APP, which can be used for training and entertainment. If you want to launch, there are various other weird pits.
In addition, to get more product dry goods, technical dry goods, remember to pay attention to NetEase yunxin blog.