The introduction
Network video broadcast live there for a long time, TDD as mobile has increase bandwidth and lower tariffs, live video is endowed with more entertainment and social attributes, people live and enjoy anytime, anywhere to watch, the host is not satisfied with a one-way broadcast, viewers are more eager to interact, broadcast the opening time and delay into the development important indicators affect product function. So, the question is: how to achieve low latency, second on live?
First, let’s take a look at the five key processes of live video broadcasting: recording -> encoding -> network transmission -> decoding -> playback. Each step will have varying degrees of influence on the delay of live broadcast. The focus here is on mobile devices. Limited by technology maturity and hardware environment, we have simply summarized four points of live broadcast delay optimization for mobile scenes: network, protocol, codec and mobile terminal, and will be divided into four phases to decode technical details of UCloud live broadcast cloud to realize low delay and second opening.
In the previous article about Live Broadcast, All the technical details are here (1), we described how to make live broadcast content from the anchor to the audience in the “shortest” path, with the lowest delay at the transmission level. In this article, we will introduce the selection of live broadcast application layer protocol and transport layer protocol, and analyze the impact on live broadcast experience.
Selection of live broadcast agreement
There are several common live broadcast protocols in China: RTMP, HLS, HDL (HTTP-FLV) and RTP. Let’s introduce them one by one.
RTMP protocol:
Adobe patent agreement, now most foreign CDN is not supported. It is very popular in China. There are several reasons for this:
1. Stable and complete support for open source software and open source libraries. For example, OBS software commonly used by Douyu anchors, open source Librtmp library, nginx-rtmp plug-in on the server side.
2. High installation rate of the playback end. As long as the browser supports FlashPlayer, it is very easy to play live RTMP. Compared with other protocols, the initial handshake process of RTMP is too complicated (the underlying layer is based on TCP, here we are talking about the interaction of RTMP itself). Depending on different network conditions, it will bring more than 100ms delay for the initial connection. Rtmp-based broadcast content generally has a delay of 2~5 seconds.
HTTP – FLV agreement:
That is to use HTTP protocol to transmit media content streaming. Compared to RTMP, HTTP is simpler and better known, without fear of being held hostage to Adobe’s patents. Content latency can also be 2-5 seconds, which is faster to open because HTTP itself does not have complex state interactions. So http-FLV is superior to RTMP in terms of latency.
HLS protocol:
Http Live Streaming is a media Streaming protocol based on Http proposed by Apple. HLS has one big advantage: HTML5 can play directly; This means that a live broadcast link can be forwarded and shared through wechat and other channels without the need to install any independent APP and only need to have a browser, so its popularity is very high. Social broadcast APP, HLS, can be said to be just needed, let’s analyze its principle.
HLS based live streaming URL is a m3u8 file, which contains a number of recent small video TS (a video encapsulation format, here is not extended introduction) file, such as www.ucloud.cn/helloworld.. Is a live streaming retention link, its content is as follows:
Assuming that the list contains five TS files, each of which contains five seconds of video content, the overall delay is 25 seconds. It is certainly possible to shorten the length of the list and the size of the individual TS files to reduce latency, and in extreme cases it is possible to shorten the m3u8 files with a list length of 1,1 seconds of content, but it is extremely vulnerable to network fluctuations causing lag.
Through the verification of the public network, the best effect can be achieved by using the same-city network at present, which is the result of the comprehensive fluency and content delay. So can HTML5 have a live streaming technology that opens directly with lower latency? We’ll talk about that at the end.
RTP protocol:
The Real-Time Transport Protocol is a Transport layer Protocol for multimedia data flows on the Internet.
In actual application scenarios, RTP Control Protocol (RTCP) is often used together. RTCP transmits interactive Control signaling and RTP transmits actual media data.
RTP is widely used in video surveillance, video conferencing, and IP phone, because one important user experience of video conferencing and IP phone is real-time content.
One important difference between RTP and the above three or two protocols is that RTP uses UDP to transmit data by default, while RTMP and HTTP are TCP based. Why can UDP achieve such real-time effect? The difference between TCP and UDP analysis of a large number of articles, not to repeat here, a brief summary:
UDP: single datagram, do not need to establish a connection, simple, unreliable, packet loss, will be out of order;
TCP: streaming, need to establish connections, complex, reliable, orderly.
Real-time audio and video streaming scenes do not need reliable guarantee, so there is no need to have a retransmission mechanism, real-time view of the image and sound, network jitter lost some content, blurred and blurred screen, completely unimportant. TCP causes delay and unsynchronization for retransmission. For example, when a piece of content is retransmitted and arrives one second later, the entire conversation will be delayed by one second. With network jitter, the delay will increase to two or three seconds.
To summarize: In the selection of live protocols, RTMP or HTTP-FLV means a content delay of 2-5 seconds, but in terms of open delay, HTTP-FLV is superior to RTMP. HLS has a content delay of 5-7 seconds. If RTP is selected for live broadcast, the live broadcast delay can be achieved within 1 second. However, as far as we know, major CDN manufacturers do not support live broadcast based on RTP, so the mainstream in China is still RTMP or HTTP-FLV.
Is there a lower latency alternative to HLS?
The advantages of HLS are obvious: mobile terminal can be opened and watched by browsers compatible with HTML5 without installing APP. All mainstream mobile terminal browsers basically support HTML5, which has huge advantages in the transmission and experience of live broadcast.
The only apparent drawback: high content latency (there are a lot of HLS restrictions not mentioned here, such as having to be H264+AAC encoding, which can also be considered a “drawback”). If it can be solved, it will be a very big step forward for live streaming technology. Or, to put it another way, is there a live streaming solution with lower latency that can be transmitted directly with links? Not limited to HLS itself.
For browser direct video interaction, Google has been pushing WebRTC, there are a number of products that can be opened in the browser real-time conversation, live. But take a look at the browser coverage chart below:
Unfortunately, Safari does not support WebRTC until iOS 9.3. Continuing our exploration, what about Websocket support?
WebSocket support is available in all browsers except the old and unfashionable Opera Mini. That seems like good news. To sort out the problems that HTML5 WebSocket live needs to solve:
1. Back-end compatibility
2, transfer,
3. Decode and play
#1 does not seem to be a big problem, it is a basic skill for having done RTMP to HLS and RTP. #2 It is a good option for browsers to use HTTP for transport. Jsmpeg: GitHub – phoboslab/jsmpeg: MPEG1 Video Decoder in JavaScript, there is a NodeJS server for stream-server.js.
According to the test results, the code of the project is relatively thin and has not reached the maturity of industrial level. It is estimated that a lot of holes need to be filled in for large-scale application. Interested students can learn and study.
The above is the live broadcast cloud: the selection of live broadcast application layer protocol and transport layer protocol and the analysis of their impact on live broadcast experience.
— — —
So, how to solve the contradiction between delay and caton? Sometimes you have to drop your bag? !
More technical details about access network optimization, content caching and transmission strategy optimization, terminal optimization. Stay tuned for the next analysis: “All the technical details about live streaming are here (3)”.
Stay tuned!
This article is provided by the UCloud Streaming Media Development Team.
UCloud organization number will exclusively share technical insights, industry information and anything you need to know about cloud computing.
Questions & attention are welcome (*////, ////*)
The above.