Penguin tutorial H5 live on demand practice

The author of this article: Chen Junwen, from IMWeb team, republication is prohibited without consent.

preface

Penguin Tutoring is an online learning service platform launched by Tencent for primary, middle and high schools. Recently, in order to facilitate users to take classes more quickly and conveniently, Penguin Tutoring added the mobile terminal H5 class form. This paper mainly introduces the selection of H5 “live broadcast” and “on-demand”, as well as the problems and solutions encountered in the development process.

Demand background

The teaching function of Penguin Tutoring is the core function of the product, and there are only two original teaching methods: APP class /PC browser class.

• APP class: the environment requirements are not high, pick up the mobile APP to experience, the first class needs to download the APP.
• PC browser class: login website can be taught, but the class environment must be in front of the computer.

Now there is a way and carrier of class — mobile terminal H5 class. Its characteristics lie in: sharing links can enter the designated course designated section, easy to spread. No need to download the APP, enter the link to sign up for the class. It is mainly used to quickly experience the scene of class

Demand for details

H5 Class page entrance

1. In a non-APP environment, the original H5 course details page and payment success page provide access to the H5 Course page (gray scale). For students who have bought the course, they can enter the tutoring page quickly and experience the class immediately without downloading the APP.

2. Through the sharing link specified in the group, if the link contains the course ID and section ID, you can automatically enter the live class of the specified section or view the playback content of the section. By default, the system automatically locates the last live class.

H5 Class page functional requirements

1. Class live broadcast: WebRTC live broadcast on Tencent Cloud (priority plan) and HLS live broadcast (downgrade plan). 2. On-demand playback: HLS playback through Tencent Cloud TcPlayer super on-demand. 3. Discussion interaction: Based on Impush, students can send and receive copywriting + emoticons. The rest of the interactive scenes are suggested through toast. 4. Course outline: Check the content and time of each course, and directly switch to the same page for live broadcast/playback.

Other requirements Based on the business logic of PC Web class, there are the following requirements: 1. Show different covers according to different states. 2. Horizontal and vertical screen detection and playback, and switch to full-screen playback. 3. Network type detection and prompt, wifi => 4G traffic pause and prompt, etc. 4, boot jump: ① purchase check boot jump purchase class. (2) Guide the APP to better experience the class.

Function of comb

The technical implementation

Live and on-demand solutions

Selection of audio and video programs

WebRTC Live downgraded HLS live

At the same time, there are discussion modules and other interactive modules (such as hand raising, answering questions, etc.). If there is a delay in audio and video that the student user can clearly perceive, the experience will be compromised. Therefore, PlanA adopts WebRTC, a low-delay live broadcast scheme, with a delay of less than 1 second.

However, since WebRTC is not compatible enough on mobile terminals, especially on Apple models, it will be gradually supported after IOS version 11.1.2. Therefore, a degrade live broadcast scheme with wide compatibility and coverage is needed as PlanB to ensure that it can live from the bottom.

The degraded live broadcast scheme adopts the HLS protocol live broadcast mode, which has good and wide compatibility with the mobile terminal H5. But the delay is higher than 20 seconds. Therefore, when the live broadcast is downgraded, the message length in the discussion forum will be extended accordingly, and the speech function will be banned. The perception of low latency is shielded by allowing students to receive interactive messages only unilaterally, but not to send messages.

Why not use other live streams with less latency than HLS as a downgrade solution? The reason is that even though other live streaming methods have much lower latency (FLV can reach 5-7 seconds), the downgrading scheme prioritizes compatibility over latency. Even if the delay of 5-7 seconds is still perceived, it needs to be eliminated by one-way information flow such as silence. The amount of delay is not so important. Therefore, HLS with better mobile support is adopted.

On-demand playback

The vod solution uses Tencent cloud voD service, and the Encrypted HLS voD solution is used on the Web side. Due to my previous experience on PC and the use of super vod player of Tencent Cloud, this part is more stable than live broadcast. As the security needs to decrypt the encrypted voD resources, the main work is token authentication to exchange decryption keys and other authentication processing.

Player access and playback process

Online education WeBRTC-live-Player is used for live streaming access to WebrTC, which relies on Tencent Cloud WebrTC SDK, and needs to be supported above 3.4.1 (the reason is that the new IOS version already supports unifiedPlan). Live player access, relatively simple. Determine container DOM and specify related service reporting modules. Then instantiate and perform preconnect.

Const opts = {// Some reported configuration items}; this.webrtcPlayer = new WebRTCLivePlayer(opts); this.preconnect();Copy the code

Preconnect here only establishes the Websocket signaling connection of WeBRTC first, because the userSIG of WebTC may not be delivered in the first time. Creating a signaling connection in advance saves the time of signaling connection establishment.

When usersig is ready, execute init and set the width and height of the player after asynchronous creation. Finally, after connet is executed to enter webrTC’s room successfully, SDP and candidate are exchanged to establish ICE connection, and the video stream can be played after waiting.

This. WebrtcPlayer. Init (sig, roomId). Then (() = > {/ / set player high this wide. WebrtcPlayer. SetContainerStyle ({width: XXX, height: xxx, }); this.webrtcPlayer.connect(); });Copy the code

Of course, with webrTC player, there are more things to do in the actual business layer, including stream update processing, video width and height change update layout, reconnection mechanism, and report processing.

Degraded Hls live access based on TCPlayerLite, pass in m3u8 mixed address, instantiate the player. Initialization parameters refer to: cloud.tencent.com/document/pr… .

TCPlayerLite uses a combination of H5

const player = new TcPlayer('tcplayer_container', {
    "m3u8": "http://2157.liveplay.myqcloud.com/2157_358535a.m3u8", / /, for example"autoplay" : false, (Remarks:trueOnly works on most PC platforms)"width":"100%"."height":"100%"."live":true."x5_player":true
});
Copy the code

In order to better understand the steps of live broadcasting, I sorted out the process and architecture diagram of live broadcasting.

Tencent cloud TCPlayer on-demand access

On-demand resources, as the resources paid by student users, can not be played by a M3U8 address, and must be encrypted. Therefore, in addition to the fileId of voD resource, user identity verification is also required for video decryption.

Encryption HLS refer to: cloud.tencent.com/document/pr… It provides two video playback schemes. In simple terms, cookies are used to generate tokens and decryption keys DK are obtained through the DK interface results of services.

Scheme 1: Pass identity authentication token information through QueryString.

http://example.vod2.myqcloud.com/path/to/a/voddrm.token.ABC123.video.m3u8Copy the code

Scheme 2: The player will attach cookies to the URL identified by the ext-X-key tag.

However, there are pits: on PC, we have adopted scheme 2. There is no problem with cookie on the business DK interface of the same domain, but it cannot be played under Android H5. Packet capture found that under Android clearly is the same domain interface can not carry any cookie. I went back to the file and found this.

It seems that we cannot use scheme 2 as simple as PC on mobile terminal H5. However, in addition to generating token according to cookie, scheme 1 also needs to play address parameters. However, we originally only have filedId and DK interface, instead of using M3U8 address to play. After consulting the classroom colleagues, it turns out that Tcplayer provides a parameter HLStoken not included in the documentation to pass tokens.

/cgi-bin/qcloud/get_dk /cgi-bin/qcloud/get_dk /cgi-bin/qcloud

This. player = new window.TCPlaye(this.elIDId, {fileID, // Please pass in the video that you want toplay.false, {HLSToken: {token, // Token generated from cookie, this parameter is not in the document. ,}}});Copy the code

Mobile terminal H5 audio and video tap point

Because mobile terminal H5 in addition to the system and version (IOS and Android) are different, browser environment (wechat, mobile Q and its own browser) is also much more complex than PC. In addition to WebRTC’s mobile compatibility issues, there are differences in performance from ios11.1.2 to the latest 12.4. So I’m going to summarize some of the major pits I’ve stepped on.

1. IOS 12.3 requires support for unifiedPlan Cause: Vendors of the WebRTC SDP standard gradually adopt unifiedPlan, but Plan-B of Apple is unavailable. Solution: Upgrade your Tencent Cloud WebRTC SDK to 3.4.1, which modified the original default plan-B to UnifiedPlan, and added the corresponding RTCUtils API to unifiedPlan path.
2. In IOS but not Safari, WebRTC has a 2-minute flow interruption. Cause: The browser kernel of IOS non-Safari app (wechat, Mobile Q) cannot obtain the quality of PeerConnection. The key is that there are no successful or failed callbacks to get quality. So you don’t have 2s to report mass, and then you cut off the flow. Solution: Check whether the UA is an IOS browser that is not Safari and force the failed callback.
3. Video Label attributes are incompatible. Solution: In H5, use the playsinline attribute; otherwise, the system automatically playsin full screen. Attributes need to be prefixed with -webkit, x5. Autoplay may be invalid and requires user action events to trigger play. All video tag attributes are handled case by case. For partial compatibility, see docs.qq.com/sheet/DTGp4…
4. The videoEVent triggering mechanism is faulty.Cause: The implementation of H5 video playing standard is inconsistent on different platforms, and the event triggering mode and result may be different.Developer.mozilla.org/en-US/docs/….error,timeupdate,load,loadedmetadata,loadeddata,progress,fullscreen,play,playing,pause,ended,seeking,seeked,resize,volumechangeSolution: Some machines trigger loadmeatadata and loadData only after triggering playback. Adjust the first frame time to playable time.
5. IOS not unfiedplan,WebRTC update stream cannot reset the stream for video. Solution: IOS12.3 below, when drawing from something to nothing, and then from nothing, re-establish WebRTC connection.
6. X5 kernel TCplayer on demand, system full screen need to shield the download button. Solution: find tcplayer colleagues, want to x5 kernel team to provide domain name, can be blocked.
7. The video system provides the control bar.Solutions:<video controls >Set it to false to implement the DOM control bar, a video-.fraternal =true/false, and a full-screen key, as well as the RequestFullscreen API.
8. The internal implementation path of WebRTC full-screen system in X5 kernel is different from that of MSE Video. Cause: I asked my X5 kernel colleague that the internal path of weBRTC full-screen system is different from that of ordinary video full-screen system. As a result, the WebRTC system is not perfect in full screen, and clicking the back button will exit the page. The overall full screen scheme is explained below.

Horizontal and vertical screen detection and full screen solution

Concept definition: Full screen (system full screen) or web full screen (pseudo full screen)

• Full screen: A full screen is displayed on the screen. After the full screen is displayed, only the video content is displayed and the browser interface such as the address bar is not displayed. This mode requires the browser to provide interfaces.
• Full screen: a web page is displayed in a full screen area. After the page is displayed in full screen, you can still see the address bar of the browser. In general, the full screen of a web page is used to solve the problem that the browser does not support full screen, so it is also called pseudo full screen. This full-screen mode is implemented by the CSS.

As for how to achieve full screen web page, here is the simplest way: enter the full screen web page, video set to Fixed, adjust the Z-index value, the control bar is also occupied. In vertical and horizontal screen processing is slightly different, compare the width and height ratio of the screen and video, set the width or height to 100%, the other side can be calculated according to the proportion, the ultimate purpose is to achieve the filling effect of contain.

requestFullScreen API

There are two kinds of interfaces that support Fullscreen. One is called Fullscreen API. After entering Fullscreen, you can still see the player interface composed of HTML CSS. The other interface is webkitEnterFullScreen, which only applies to the video label and is usually used when the mobile terminal does not support Fullscreen API. After the interface is used in Fullscreen, the player interface is the system’s own interface.

In the actual business, for full screen priority video. WebkitEnterFullscreen (); (apple only supports the API), when you perform is not successful, then executed video. WebkitRequestFullScreen (); .

Horizontal and vertical screen judgment, orientation detection

Here I use a third-party detection code: github.com/shrekshrek/… This code can not only detect horizontal and vertical screen, but also detect gravity and level Angle, etc., and do multi-terminal compatibility processing.

It is simple to use: initialize the listener and update the layout when the dir value changes.

const _orienter = new window.Orienter(); this.props.updateOrientation(_orienter.direction); // Determine the current orienter.onOrient =function (obj) {
  if(this.dir ! == obj.dir) { this.dir = obj.dir;setTimeout (() = > {/ / 200 ms delay update layout enclosing props. The updateOrientation (obj. Dir); }, 200); }}; // Start listening _orienter.on();Copy the code

Note: there is a 200 ms delay here, because when the horizontal and vertical screens change, the screen width is still the same as before, so it will be problematic to update the layout immediately. So delay updating the layout by 200 milliseconds.

Design class status and cover control status

When you get the requirements and interactive visual draft, you will find that there are multiple covers for a class, and the covers may change back and forth in a page over time. All this is due to the complexity of class status. There are three dimensions of judgment: 1. Current time. 2. Whether the teacher has a class. 3. Whether there is video streaming. Therefore, how to control the change of the cover is mainly to maintain and control the store value of the classroom state, and then map the cover.

Cover state design: using hierarchical cover, state mapping to control. The group covers overlay the sub-covers. Main cover component: responsible for the time dimension and CGI can determine the large course state, one-way flow, not switch. Sub-cover component: responsible for the control of playback state, according to the teacher class state, flow state control, non-one-way process, may switch back and forth.

Network type and disconnection detection

In the PC browser class, generally wifi and wired Internet access, but in the mobile browser to watch live and on-demand, in addition to wifi may consume traffic, the need is to be able to detect the current network status (traffic mode /wifi mode) to judge. In this way, users are prompted to respond to avoid traffic consumption without their knowledge.

Through the navigator browser itself. The connection. The type can really judge browser, but only the chrome support mobile terminal, and almost all other browsers is unavailable.

Therefore, judging the network type still depends on the ability provided by app. Judging the network type is based on the JS API provided by the two major in-app browsers, Mobile Q and wechat. The following is the code implementation.

return new Promise((resolve, reject) => {
    if(window. WeixinJSBridge) {/ / networktype WeChat detection window. The WeixinJSBridge. Invoke ('getNetworkType', {}, (e) => { resolve({ isNetWorkFlow: e.err_msg ! = ='network_type:wifi'&& e.err_msg ! = ='network_type:fail',
          isNetworkBroken: e.err_msg === 'network_type:fail'}); }); }if(window. MQQ) {/ / hand Q test networktype window. The MQQ. The device. The getNetworkInfo ((res) = > {resolve ({isNetWorkFlow: res.type === 2 || res.type === 3 || res.type === 4, isNetworkBroken: res.type === 0, }); }); }else{ reject(); }});Copy the code

Check whether the network type is isNetWorkFlow and the current network isNetworkBroken by checking the callback every three seconds. Finally based on the result of the callback based on different prompts.

Class page links query parameter control and class section switching

The PC class page links the query parameter

In the PC Web project, each class page corresponds to only one class, and the linked parameters correspond to the information of each class and the current class status.

https://fudao.qq.com/pc/webclass.html?term_id=2000010153&course_id=113536&lesson=0&lesson_id=93561&status=2&sub_termid=0Copy the code

Course_id and TERm_id: Course ID and class ID Lesson_id: section ID Status: Class status 0 Not started, 1 Being streamed, 2 replaying, 3. Generating replaying SUB_termid: id of a small class

In PC browser, jump from the course task page to the course page, the query parameter on the link has been determined in the previous page, and you can get the status parameter immediately (calculated by the time). This has the advantage that the page load can immediately load the player and the corresponding style according to the class state, without waiting for the CGI request. The downside is that switching courses involves reloading the refresh page.

Switch to a class without refreshing the page

Since H5 class page contains “course outline”, which is equivalent to concentrating the “Course Task page” and “Class page” on the original PC on one page, the product hopes to meet the following requirements: 1. Course switching does not refresh the page. 2. If there is no lesson_id on the link, you can locate the last unstarted lesson. 3. After switching the link, the shared link can be located to the current link.

The above three points are difficult to implement, because in the original code, there are quite a lot of places to obtain the parameters from the link at one time, and because of the sharing mechanism, it is necessary to modify the link parameters without refreshing the page, obtain the link parameters again, and reload the player and subtitles component. Implement a switchLesson method as follows: