From IMWeb Community by Terrance,
The original link


First, live knowledge small popular science

  1. A typical live broadcast process: recording -> encoding -> network transmission (push stream -> server processing ->CDN distribution) -> decoding -> playback
  2. IPB: A common video compression scheme in which I frame represents the key frame, B frame represents the forward differential frame, and P frame represents the bidirectional differential frame
  3. GOP (Group of Pictures) : The longer the GOP (the larger the interval between I frames), the higher the proportion of B frames, the higher the rATE-distortion performance of coding. Although B frame compression rate is high, but decoding will be more CPU pressure.
  4. Main indicators of live audio and video broadcast quality include content delay, lag (fluency) and first frame duration
  5. Major problems to be overcome in live audio and video broadcasting: network environment, multi-party connection, main and auxiliary channels, browser compatibility, CDN support, etc
  6. MSE(Media Source Extensions) : W3C standard API for HTML5 streaming (HTML5 native only supports streaming mp4/ webM, not FLV), allowing JavaScript to be built dynamically<video> 和 <audio>Media streams. Can use MediaSource. IsTypeSupported () to judge whether to support a mime type. Not supported in ios Safari.

  1. File format/Package format/Container format: A format for hosting video, such as FLV, AVI, MPG, VOB, MOV, MP4, etc. How video is encoded and decoded depends on Codec. For example, MP4 format is divided into nMP4 and fMP4 according to the different codec. NMP4 is made up of nested Boxes, while fMP4 is made up of a series of clips, so that only the latter does not need to load the entire file for playback.
  2. Codec: Multimedia digital signal Codec, capable of audio and video compression (CO) and decompression (DEC). CODEC technology can effectively reduce the space occupied by digital storage. In computer systems, using hardware to complete CODEC can save CPU resources and improve the operating efficiency of the system.
  3. Common video encoding: MPEG, H264, RealVideo, WMV, QuickTime…
  4. Commonly used audio coding: PCM, WAV, OGG, APE, AAC, MP3, Vorbis, Opus…

Ii. Comparison of existing schemes

RTMP protocol

  1. Based on the TCP
  2. Adobe monopoly, high domestic support
  3. The browser relies on Flash for playback
  4. A delay of 2-5 seconds

RTP protocol

  1. Real-time Transpo
    rt P

    Rotocol, a standard proposed by IETF in 1996

  2. Based on UDP
  3. Strong real time
  4. Used for video surveillance, video conferences, and IP phone calls
  5. The CDN vendor and browser are not supported

HLS agreement

  1. Http Live Streaming: an Http based protocol developed by Apple
  2. HTML5 direct support (video), suitable for APP live, PC only Safari, Edge support
  3. The code must be H264+AAC
  4. Because the transmission is the cut audio and video clips, resulting in a large content delay

flv.js

  1. Bilibli is open source, parses FLV data and feeds fMP4 to video tags via MSE
  2. Coding for H264 + AAC
  3. Streaming media content using HTTP streaming IO(FETCH or Stream) or WebSocket protocols
  4. 2 to 5 seconds delay, first frame faster than RTMP

WebRTC protocols

1, Google pushed hard, has become a W3C standard

2, modern browser support trend, X5 also support (wechat, QQ)

3, based on UDP, low latency, weak network resistance, more advantages than FLV.js

plan CPU Frame rate Bit rate Time delay The first frame
flv.js 0.4 30 700kbit/s 1.5 s 2s
WebRTC 1.9 30 700kbit/s 0.7 s 1.5 s

4. Supports the Web uplink capability

5. The code is H264+OPUS

6. Provide NAT penetration technology (ICE)

In practice, when the number of users is large, the performance of the device is very high, and the complex permission management is difficult to achieve, so it is basically not feasible to adopt P2P architecture. Tencent Classroom has implemented a P2S solution for the scenario where individual users provide upstream stream and mass users only pull stream. Read jaychen’s series on WebRTC live streaming technology for further study.

Three, small program + live

Technical solution

  1. RTMP is an internal transport mechanism that uses HTTP/2, but is based on UDP.
  2. Live-pusher and Live-Player do not limit third-party cloud services
  3. Tencent cloud video live can be directly used, only need to configure the push stream URL, play URL

Push the current url:

Play the url:

The following is an audio and video applet I built according to the official website tutorial. The process of building is simple and the experience of live broadcast under the same LAN is very smooth (readers can also directly search Tencent Video cloud applet for experience) :

The front-end core code is fairly concise:

  • Live-pusher component: Set parameters such as URL and stream address (FLV, RTMP format only), and use bindStatechange to obtain the change of playing status

    <view id='video-box'>  
    <live-pusher
          id="pusher"
          mode="RTC"
          url="{{pusher.push_url}}" 
          autopush='true'
          bindstatechange="onPush">
    </live-pusher>  
 </view>
Copy the code

  • Live-player component: After setting the SRC audio and video address (only in FLV and RTMP format), use bindStatechange to obtain the change of playing status

    <view id='video-box'>  
    <live-player
        wx:for="{{player}}"
        id="player_{{index}}"
        mode="RTC"
        object-fit="fillCrop"
        src="{{item.playUrl}}" 
        autoplay='true'
        bindstatechange="onPlay">
   </live-player>
 </view>
Copy the code

Can it be used with WebRTC?

For the application scenario of Tencent class, the teacher uses THE RTMP protocol to push the stream in class. Considering that WebRTC can only be used for streaming on PC, can users directly watch the live class through small programs on mobile terminal? In my opinion, if it is technically feasible, access to small program live broadcasting will be of great help to expand the influence of the platform, share in the social circle and improve the conversion of fees. The difficulty lies in the complex permission control, multi-channel audio and video stream, multi-person connection to the mic and other problems. For example, the permission control can only be completed in the room control logic alone, and the audio and video stream itself lacks such verification. In addition, separate signaling control should be added to switch between main and secondary routes, and corresponding judgment logic should be added to small programs.

4. Refer to the article

  1. Getting started with HTTP
  2. Use flv.js for live streaming
  3. Future-oriented Live Streaming Technology -WebRTC
  4. Interpretation of “Small Program Live Broadcast” by technical director of Small Program Audio and Video Capability
  5. Small program development easy tutorial
  6. Small program audio and video interpretation