• In recent years, with the rise of live broadcasting, QQ Music has also been equipped with live broadcasting capability to support live broadcasting of concerts, anchors and stars. According to the different ways of interaction, we can be divided into interactive live broadcasting and push stream live broadcasting. I was fortunate to participate in the process of live broadcasting from nothing to nothing. I have a clear understanding of live broadcasting. This paper mainly introduces the process of live broadcasting in the Web part.

Live the status quo

The apps listed above are all involved in the live streaming industry, which shows how popular live streaming is now.

According to the flow process of the whole video content, the whole live broadcast process is divided into three stages: generation stage, transmission stage and presentation stage



These stages are described below;

Generate the phase

The generation stage includes the collection and processing of audio and video;

The collection stage of audio and video is mainly to collect the original video content, that is, the source of live broadcast content. According to the difference of application scenarios, we can divide it into five categories

  1. The live broadcast on TV mainly collects existing video sources, such as TV series and variety shows, which are pre-recorded.

  2. Live broadcast of a concert is mainly to record pictures and audio of the concert through cameras and microphones

  3. For example, we usually see the live broadcast of league of Legends, Eat Chicken and other games. The main thing is that the anchors record the live broadcast pictures of the anchors through cameras and microphones, as well as OBS and other screen recording software to record the game interface and synthesize the live broadcast pictures

  4. At present, outdoor live broadcasting is also popular, similar to the model of reality show. Anchors mainly rely on mobile phones with cameras and microphones to record content

  5. Live streaming of mobile games, such as the recently popular King of Glory, Daxian, Hi’s and other anchors are live streaming of mobile games. On Android, they are mainly recorded by emulators, while on ios, videos can be recorded by Itools Apple Screen Recording Master and then synchronized to OBS.

Audio and video processing

DIY video processing

For the video content collected, some post-processing needs to be done, mainly including two aspects

(1) The recorded video may not meet the satisfaction of anchors, and some anchors want to add some special effects, so they need to beautify, skin grinding, filter, add special effects to the content. This process involves complex operations, including the application of face recognition, video synthesis and other technologies;

(2) Some videos have copyright restrictions and need to be watermarked, which can be processed in this link, as well as in the subsequent transcoding stage

DIY Audio processing

(1) The original sound may be relatively miscellaneous, so noise reduction is required

(2) For some live scenes, the anchor has to deal with mixing and inflection, and the sound effects are also dealt with in this link;

Coding processing

Coding is actually the process of video compression

So let’s do a calculation

A 6s720p original video, 474M in size without any processing, takes about 6 minutes to transmit under 10M bandwidth, which is obviously unbearable for live broadcast. At present, the video generally adopts h.264 standard coding. After this standard processing, the size can be directly reduced to less than 1M, and the transmission time can be reduced to less than 800ms.

H.264 standard is packaged based on MPEG format. We first look at the FORMAT of MPEG, which divides the picture (i.e. frame) into I, P and B. I is the internal encoding frame, P is the forward prediction frame, and B is the bidirectional interpolation frame. To put it simply, I frame is a complete picture, while P frame and B frame record changes relative to I frame. Without I frame, P frame and B frame cannot be decoded. We only need to keep the key I frame and record some associated information for other frames.

The process of compression is the process of removing redundant information, mainly including two aspects of compression:

1, the human visual system is not sensitive to some details, visual redundancy;

1. There is strong correlation and spatial redundancy between adjacent pixels of images; 2. Contents of adjacent images of video sequences are similar and time is redundant.

Take a chestnut

For example, in a video of skiing, most of the contents of successive frames are the same. The students who are skiing move forward, but the snow scene actually moves backward. P-frames can be encoded by reference frames (I or other P-frames), and the size after coding is very small and the compression ratio is very high.

In addition, audio content also needs to be compressed. Currently, ACC coding is mainly used.

The flow of the entire generation phase is as follows

Transmission phase

The transmission stage is mainly the process of video from the collection end to the user end

  1. Push stream push stream is commonly referred to as up stream. The recorded video needs to be pushed to the server first. At present, most of the push stream protocols are RTMP, which stands for Real Time Messaging Protocol and has the following characteristics

    (1) Abode proprietary agreement. RTMP protocol is supported by computers with FlashPlayer browser and generally supported by PCS

    (2) Long link, no need to repeat the handshake delay can be controlled within 2s, which is crucial to control the delay in the live broadcast process

    (3) Good CDN support, supported by mainstream CDN vendors

  2. Transcoding At present, several transcoding formats of live streams provided by Tencent Cloud are pushed to the server. In order to ensure that users at all ends can watch them, it can basically meet all viewing scenarios: RTMP stream native to mobile terminal FLV stream used by PC terminal HLS stream used by mobile terminal H5

  1. distribution

    Distribution which we usually said downward, the service side it was a major flow to the user, this major need ops students focus on how to more quickly in complicated network environment allows users to see the quality of the live, this involves the transmission quality monitoring, quality optimization, jian huang and so on part of the work, the Internet also has a lot of special articles on the subject, this article does not do.

Render phase

After the content distribution comes to the user play stage, how to better present a video to the user

  1. Playback protocol Currently, H5 live broadcast on mobile devices mainly adopts THE HLS protocol, which was originally developed by Apple for mobile devices such as iPhone, iPod, iTouch and iPad. Android3.0 and above are also supported. The following table shows that mobile terminals are basically supported.

#EXTM3U m3U file header, must be placed in the first line # ext-x-allow-cache set whether CACHE is allowed, currently not allowed # ext-x-media-sequence next request for the first TS fragment SEQUENCE number, # ext-x-targetDuration Maximum duration of each shard TS, currently 9s, #EXTINF Shard TS information, such as duration **Copy the code

The request flow for HLS is:

1 The player makes an HTTP request to retrieve the configuration information of M3U8.

2 The server returns a playlist of M3U8, as shown in the figure above. The fragment length and the number of lists reflect the delay length. For example, if the list length is 5 and the fragment length is 9s, the total delay is 45s.

3 The player terminal parses the M3U8 playlist, obtains TS data streams in sequence, and pulls down the next one when the playback is complete.

  1. H5 also encountered a lot of problems in the process of playing

(1) Mobile terminal playback problem – pseudo full screen

In many scenarios, the video is played in full screen and the information of the interactive part needs to be displayed. If the system is in full screen and the entire playing interface is uncontrollable, pseudo full screen, namely CSS style, is used to set the video size, and the full screen is spread to achieve full screen effect.

The following Settings are invalid in ios9, there is no solution at present, ios9-ios10 webKit-playsinline, Ios10 playsinline above, the webview surroundings of the need to support this form of ios WeChat hand inside the qq music app support webviewallowsInlineMediaPlayback = YES; Android client hand Q wechat are based on x5 kernel support embedded playback, but in THE QQ browser will take over the live broadcast here if you want to change the embedded need to find a browser students add a whitelist as shown below the normal effect is the right picture, QQ browser will be as the left picture effect



(2) Adaptive full screen

Since video sizes vary, it is critical to adapt the video to the screen when the screen height and width are fixed.

Specific steps



1. Calculate the height and width of the video

2. Compare video aspect ratio with screen aspect ratio

3, the value is large to the screen width as the baseline, for proportional scaling, the value is small, to the screen height as the baseline for proportional scaling;

For example

1. Left of the picture below

2. The screen is 93169 (width/height), and the video size is 252192, 252/192>93/169. Based on the screen width, the zoom ratio is 252/93, as shown in Figure 2.

3. Center the video and store it in a container, as shown in Figure 3.

(3) Playback protocol adaptive terminal

Currently, the HLS protocol used by the live stream protocol is supported by ios terminals. The HLS protocol is supported by different models, system versions and webView environments on Android terminals. For the unsupported HLS protocol, you can use canPlayType to see the support level, and this method will return three values

“Probably” – browsers are most likely to support this audio/video type

“Maybe” – the browser may support the audio/video type

“-” (empty string) The browser does not support this audio/video type

function chooseWay(){ var h5Able = {'probably': 1, 'maybe': 1}; var videoDom = document.createElement("video"); if( videoDom.canPlayType && ( videoDom.canPlayType("application/vnd.apple.mpegURL" ) in h5Able )){ $('#js_video').attr('src','http://3260.liveplay.myqcloud.com/live/3260_17c84703049011e791eae435c87f075e.m3u8'); The console. The log (" support HLS "); }else{console.log(" HLS not supported "); }; }Copy the code

(4) Whether the playback is continuing

It is also a troublesome thing to judge whether live broadcast is in progress or not. Since Android supports canplay canplaythrough playing events to different degrees and their trigger times are different, we can use some hack techniques. For example, timeUpdate can be used to determine whether it is playing based on real-time changes in the progress

$('#js_video').on('timeupdate',function(){ playtimeupdate = new Date().getTime(); }); SetInterval (function(){if(lastPlaytimeUpdate &&lastPlaytimeUpdate == playtimeUpdate){console.log(" exception "); }else{ lastplaytimeupdate = playtimeupdate; }}, 3000).Copy the code

(5) When the PC calls the system’s full-screen custom UI, the default system will have a UI. If you want to customize THE UI, you need to find a way to hide or cover the shadow node built in the video.

When you call full screen



The browser will add z-index to the full-screen node, which is an infinite value. If we directly set the video full-screen, the level of the video and shadow nodes will be set to an infinite height. Even if the Z-index of custom UI is set, the click will be invalid, so we need to do full-screen on the parent node of the video and custom UI node.

Since the research the SDK

At present, music also launched a self-developed video SDK, lightweight, good compatibility, welcome to use

Part of the vod function of H5 has been fully developed, supporting backward/fast forward/progress drag/full screen, and the live broadcast is being connected

Part of the vod function of PC has been grayscale. On the basis of H5, / volume control/custom full-screen UI/definition switch has been added. The live broadcast is being connected

The demo page y.qq.com/m/demo/demo…

H5 based on Zepto citation y.gtimg.cn/music/qmv/q…

PC based on jquery reference y.gtimg.cn/music/qmv/q…

Var params = {title: "QMV playlist ", container: ".js_videoplayer", source:["x00248tvbgv","t0024928s1j","g00133xte6r"], quality:true, autoplay:false, Mode :0,// 0 sequence 1 random valid when the number of lists is greater than 2 useConnectionPlay: false,}; var qmv = new QMV(params);Copy the code

The resources

HLS is introduced

https://github.com/ossrs/srs/wiki/v2_CN_DeliveryHLS

The difference between intra-frame compression and inter-frame compression

https://www.zhihu.com/question/20237091

H5 live video literacy

http://km.oa.com/group/19674/articles/show/266140?kmref=search&from_page=1&no=1

Mobile HTML5 video tag in-depth practice

http://km.oa.com/articles/show/261332?kmref=search&from_page=1&no=4