As a person who has never been exposed to real-time streaming (live streaming), I had no concept of real-time video before. However, the project I recently participated in happened to have the demand of video monitoring. Before participating in the technical selection, I had a thorough understanding of the display of front-end real-time streaming.

An overview of

Video has a concept of streaming, so it is called streaming media. The streaming of live video is easy to understand, because the video is live and needs to have a place to output the video continuously, so the entire video can be called a stream. So can the video be exported directly to the front page? Unfortunately, if I could, there would be no article of mine. RTSP protocol is widely used in the real-time video stream of the camera now, and the front end can not directly play the video stream of RTSP.

  • RTSP (Real-time Stream Protocol) is an application layer Protocol in TCP/UDP Protocol system, which is in the same layer with HTTP. RTSP is architecturally on top of RTP and RTCP, and it uses either TCP or RTP for data transfer. RTSP real-time effect is very good, suitable for video chat, video surveillance and other directions.

Then we need an intermediate layer to flow the RTSP into a protocol that can be supported by the front end. This also leads to several directions of current real-time streaming technology:

  • RTSP -> RTMP
  • RTSP -> HLS
  • RTSP -> RTMP -> HTTP-FLV



RTMP

RTMP (Real Time Messaging Protocol) is a set of video protocols belonging to Adobe. This solution requires specialized RTMP streaming media, and if you want to play on the browser, you cannot use the HTML5 video tag. Flash player only. (You can play with the video tag by using the version below [email protected], but you still need to load Flash). Its real-time performance is the best among several solutions, but since it can only use Flash solutions, it is directly GG on mobile terminals, and it is also a thing of the past on PC terminals. Since the following two methods also require RTMP, here’s how to convert an RTSP stream to RTMP, using ffmpeg+Nginx+nginx-rtmp-module:

RTMP {server {# port listen 1935; # application test {# live on; record off; }}}
# bash to convert RTSP to RTMP And push on the port to 1935 ffmpeg -i "RTSP: / / XXX. XXX. XXX, XXX / 1" - vcodec copy - acodec copy -f FLV "RTMP: / / 127.0.0.1:1935 / live/"

So we have an RTMP stream that we can play directly with VLC or IINA.

HLS

HLS (HTTP Live Streaming) is a network transmission protocol for Streaming media based on HTTP proposed by Apple Inc. Its working principle is to divide the whole stream into small HTTP-based files for downloading, and only download some files at a time. HLS is cross-platform and supports IOS /Android/ browser with strong versatility. But its real-time performance is poor: Apple’s official recommendation is to ask for three clips before starting to play. Therefore, HLS is rarely used as the transmission protocol of Internet live broadcasting. Assuming that the list contains five TS files, each of which contains five seconds of video content, the overall latency is 25 seconds. Apple’s official recommendation for small files is 10s, so this will give you 30s (n x 10) of latency.

Here is the full link of the HLS real-time stream:



As can be seen from the figure, we need a server that acts as an encoder and stream divider, accepts the stream and outputs it into stream fragments, and then the front end accesses these stream fragments through an index file. We can also use NGINX + FFMPEG to do this.

Application HLS {live on; application HLS {live on; hls on; hls_path xxx/; # save HLS file folder hls_fragment 10s; }
ALTER TABLE TABLE ALTER TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE TABLE # as Distribution in the image above, is responsible for the shard file and the output of the index file location/HLS {# provide HLS fragments, declared type types {application/VND. Apple. Mpegurl m3u8; video/mp2t ts; } root /Users/mark/Desktop/hls; # access the folder where the slice file is saved # cache-controll no-cache; expires -1; }

Then push the stream to the HLS path again using FFMPEG:

Ffmpeg -i "RTSP: / / XXX, XXX, XXX, XXX / 1" - vcodec copy - acodec copy -f FLV RTMP: / / 127.0.0.1:1935 / HLS

At this point, you can see that there are already many streams in the folder, and they are constantly updated:



Then we can use video-.js + video-.js-contrib-hls to play the video:

<html> <head> <title>video</title> <! - the introduction of CSS - > < link href = "https://unpkg.com/video.js/dist/video-js.min.css" rel = "stylesheet" > < / head > < body > < div class="videoBox"> <video id="video" class="video-js vjs-default-skin" controls> <source src="http://localhost:8080/hls/test.m3u8" type="application/x-mpegURL"> </video> </div> </body> </html> <script src="https://unpkg.com/video.js/dist/video.min.js"></script> <script type="text/javascript" SRC = "https://cdnjs.cloudflare.com/ajax/libs/videojs-contrib-hls/5.15.0/videojs-contrib-hls.min.js" > < / script > < script > videojs.options.flash.swf = "./videojs/video-js.swf" videojs('video', {"autoplay":true}).play(); </script>

In my test, the delay of HLS is around 10-20 seconds. We can reduce the delay by adjusting the size of the slices, but due to the limitations of the architecture, the delay is a problem that cannot be ignored.

HTTP-FLV

The next big thing is HTTP-FLV, which combines the versatility of HLS and the real-time capability of RTMP to play live streams in the browser with a low latency using the HTML5 Video TAB. HTTP-FLV relies on MIME characteristics to select the corresponding program to process the corresponding Content according to the content-type of the protocol, so that the streaming media can be transmitted over HTTP. In addition, it can be flexible scheduling/load balancing via HTTP 302 jump, support HTTPS encrypted transmission, and is compatible with Android, iOS and other mobile terminals. Nginx-http-flv-module is used to stream RTMP into an HTTP stream. In Nginx, we can use nginx-http-flv-module to stream RTMP into an HTTP stream.

The FLV format is still Adobe’s. The native Video tag can’t play directly, but we have Bilibili’s FLV. js, which can reuse FLV files into ISO BMFF (MP4 fragment) fragments. Then feed the MP4 snippet to the browser through the Media Source Extensions. In the browser-enabled protocol, the deferred sort would look like this: RTMP = http-flv = websocket-flv < HLS and the performance sort would look like this: RTMP = http-flv = websocket-flv < HLS RTMP > http-flv = websocket-flv > HLS RTMP > http-flv = websocket-flv > HLS

  1. First we need a new nginx plugin: nginx-http-flv-module
  2. Some new configurations in nginx.conf:
# rtmp server application myvideo { live on; gop_cache: on; HTTP server location /live {flv_live on; }
  1. Again use FFMPEG to push the stream, using the RTMP command above
  2. The front end import flv.js, and then use it to play back
// import flv.js from 'flv.js'; // import flv.js from 'flv.js'; export function playVideo(elementId, src) { const videoElement = document.getElementById(elementId); const flvPlayer = flvJs.createPlayer({ isLive: true, type: 'flv', url: src, }); flvPlayer.attachMediaElement(videoElement); flvPlayer.load(); } playVideo('#video', 'http://localhost:8080/live? port=1985&app=myvideo&stream=streamname')

You can see that flv.js is in usevideo/x-flvThis MIME returns data.



If you need more delay, try the following:

  1. Flv.js can be configuredenableStashBufferField, which is used by flv.js to control the cache buffer. When turned off, it can achieve minimum latency, but because there is no cache, you may see video lag caused by network jitters.
  2. You can try to disable nginx in the HTTP configurationgop_cache 。gop_cacheAlso known as key frame cache, its meaning is to control whether the cache between the key frames of the video is turned on.

Here we introduce the concept of a key frame: H.264, the most widely used video compression format, uses compression schemes such as intra-frame predictive compression/inter-frame predictive compression, and finally obtains three types of BPI frames:

  • I frame: key frame, using intra-frame compression technology.
  • P frame: Forward reference frame. During compression, only the previous frames are referred to. It indicates the difference between the current frame and the previous frame (which may be frame I or frame P). Interframe compression technology is used.
  • Frame B: Bidirectional reference frame, when compressed, it refers to both the previous frame and the frame behind it. Frame B records the difference between this frame and the previous frame. Interframe compression technology is used.

A typical video sequence with I, B, and P frames. A P frame only needs to refer to the preceding I or P frame, while a B frame needs to refer to both the preceding and subsequent I or P frames. Since P/B frames are directly or indirectly dependent on I frames, the player must first decode I frames in order to decode a sequence of video frames and play them back. Assuming that the GOP is 10 seconds, that is, the key frame is only available every 10 seconds. If the user starts playing at the fifth second, he will not be able to get the current key frame. This is where the GOP_CACHE comes into play: The GOP_CACHE controls whether or not the most recent keyframe is cached. Enabling the GOP_CACHE allows the client to receive a keyframe and display the image immediately after the game starts playing. Of course, the latency is increased by increasing the cache of the previous frame. If latency is more important than first screen time/playback fluency, you can try turning off the gop_cache to achieve lower latency.

thinking

Delay with lag

The delay and lag of real-time video are the two most important indicators of video quality. However, in theory, these two indicators are a contradictory relationship — the need for a lower latency means that the buffers on both the server side and the player side must be shorter, and abnormal jitter from the network is prone to cause stalling; When the business can accept higher latency, both the server and the player can have longer buffers to deal with jitter from the network and provide a smoother experience.

What do live broadcast manufacturers do?

Now all the live broadcast platforms have basically abandoned the above traditional ways and used the CDN provided by cloud service providers, but they still cannot do without the several protocols and methods mentioned above. The following picture is the picture of Aliyun’s live streaming service. As you can see, the process is roughly divided into these steps:

  1. Collect video stream (the anchor uses RTMP to push the stream)
  2. Push stream to CDN node (Upstream)
  3. The CDN node goes to the live broadcast center, which is similar to a powerful intermediate source with computing power and can provide additional services such as drop storage (recording/recording to cloud storage/VOD), transcoding, auditing, output of multiple protocols, etc.
  4. The live broadcast is distributed to CDN nodes
  5. Broadcast (Aliyun supports RTMP, FLV and HLS three broadcast streaming protocols)



PS: If you have read this and think my writing is OK, please give me a thumbs-up, thanks! 🙏