background

Recently, I met the requirements of live video broadcasting and video playback in the project, and encountered many difficulties in the implementation process. Therefore, I would like to record the solutions encountered in the research process here.

Video streaming introduction

Video streams can be roughly divided into three types: HLS,RTSP and RTMP. The following three video streams are introduced in turn

HLS

Http Live Streaming (HLS) is an HTTP-based Streaming media network transmission protocol proposed by Apple. It slices the Streaming media into segments and saves the information in the M3U list file. Versions at different rates can be sliced into corresponding slices. The player can request streaming data directly using the HTTP protocol. Advantage:

  • You can freely switch between different speed versions to achieve seamless playback
  • Save the hassle of using other protocols

Disadvantage:

  • The delay size is affected by the slice size, which is not suitable for live broadcast but for video on demand.
  • Poor real-time performance and high delay. The delay of HLS is more than 10s+
  • File fragments. Features of the double-edged sword, TS slice small, will cause a large number of small files, storage and caching have certain challenges

RTMP

RTMP (Real Time Message Protocol) is a proprietary and not fully public streaming Protocol proposed by Adobe to solve the Multiplexing and packetizing problems of multimedia data transmission streams. The RTMP protocol generally transmits FLV and F4V streams. Generally, commands and data are transmitted over one TCP channel. Advantage:

  • Low latency, high stability, support all camera formats
  • Developed specifically for streaming media, the underlying optimization is better than other protocols

Disadvantage:

  • The browser needs to load the Flash plug-in to play.
  • RTMP is an Adobe proprietary protocol that cannot be played on many devices, especially iOS, and requires a third-party decoder to play
  • TCP – based transport, non – public port, may be blocked by the firewall

RTSP

Real-time Stream Protocol (RTSP) is a streaming media Protocol jointly proposed by Real Networks and Netscape. RTSP is a common Protocol and is maintained by a specialized organization. It is an application layer protocol in the TCP/IP protocol system. RTSP protocol is generally transmitted in TS, MP4 format streams, RTSP transmission generally requires 2-3 channels, command and data channel separation. Text – based multimedia playback control protocol. RTSP defines the stream format, and the stream data is transmitted through RTP. Advantage:

  1. RTSP real-time effect is very good, suitable for video chat, video surveillance and other directions.

Disadvantage:

  1. Browsers cannot play directly, only through plug-ins or transcodes

Playback Solution

HLS and RTMP playback solutions

If we only need to play on the WebView of mobile devices, we can choose to use HTML embedded HLS to play. The following HTML can be played on all Safari or Android webViews above 3.0. This eliminates the need to use third-party players.

<html lang="en"> <head> <meta charset=utf-8/> </head> <body> <div id='player'> <video width="640" height="320" controls> The < source SRC = "http://10.10.5.119/live/livestream.m3u8" type = "application/x - mpegURL" > < / video > < / div > < / body > < / HTML >Copy the code

Disadvantages:

  1. Native H5 supports MP4, OGG, WebM and M3U8 media formats
  2. M3U8 format is not compatible with PC browsers
  3. Unable to play video from RTMP stream because native does not support FLV

Advantages:

  1. Use native video without installing third-party plug-ins

Flv.js is an open source project from Bilibli. It parses FLV files and feeds them to native HTML5 Video tags to play audio and Video data, making it possible for browsers to play FLV without Flash.

import React, { PureComponent } from 'react';
import Reflv from 'reflv';

export class HttpFlv extends PureComponent {
  render() {
    return (
      <Reflv
        url={`http://localhost:7001/live/test.flv`}
        type="flv"
        isLive
        cors
      />)}}Copy the code

Disadvantages:

  1. For recording, rely on native HTML5 Video tags and the Media Source Extensions API
  2. Livestream, which relies on the playback technology required for recording and broadcasting, also relies on HTTP FLV or a protocol in WebSocket to transmit FLV. HTTP FLV needs to fetch data through streaming IO, which supports fetch or Stream
  3. Due to the Media Source Extensions, all browsers under iOS and Android4.4.4 are not supported at present, which means that flv.js is basically not available on mobile.

Advantages:

  1. Thanks to the browser’s hardware acceleration for native Video tabs, performance is good and hd is supported
  2. It also supports recording and live broadcasting
  3. Remove the reliance on Flash

Video. Js supports web page playback in all browsers, free use of many advanced features. These two video streams can be played using video.js. Since the front end is developed using React, _video.js is required. _ When playing two different streams, you need to set different types. When playing RTMP, type is RTMP/FLV, and when playing HLS, type is Application/X-mpegURL. Both video streams are flash-based and therefore require Flash support.

class VideoPlayer extend Component {
      constructor(props) {
        super(props);
        this.VideoOptions = {
            autoplay: true.controls: false.sources: [{
                src: url,
                type: "application/x-mpegURL"}].fluid: 'true'.playbackRates: [0.75.1.1.5.2]
        }
    }
    componentDidMount() {
         this.player = videojs(this.videoNode, this.VideoOptions, function onPlayerReady() { })
    }
    componentWillUnmount() {
        if (this.player) {
            this.player.dispose()
        }
    }
  render(){
    return (
      <video
      	ref={node= > this.videoNode = node}
        className='video-js vjs-16-9'
      />
    )
  }
}
Copy the code

Html5 video tag only supports MP4, WebM and OGG formats, but does not support FLV format. FLV format is reserved for Flash. RTMP streams in FLV format must use Flash player. If not, the flash player of the browser is automatically called to play the video. Flash is disabled for web sites on Edge and Google Chrome by default. You can manually enable Flash

Disadvantages:

  1. Rely on flash
  2. RTSP streams are not supported

Advantages:

  1. It’s open source and free, and you can easily access the latest code on Github.
  2. It’s so easy to use that it only takes a few seconds to set up a video playback page.
  3. It works with almost all browsers, uses HTML5 preferentially, and automatically uses Flash for playback in unsupported browsers.
  4. The interface can be customized, pure javascript and CSS. The documentation is also very detailed

RTSP streaming playback solution

1. Install VLC plug-in for server mapping transcoding by using VLC plug-in redirect browser, get ogG format video that can be opened normally with video label, and then play

<object type='application/x-vlc-plugin' pluginspage="http://www.videolan.org/" id='vlc' events='false' width="720" height="410"> < param name = 'called simply MRL' value = "RTSP: / / admin: 12345 @192.168.10.235:554 / h264 / ch1 / main/av_stream" / > < param name = 'volume' value='50' /> <param name='autoplay' value='true' /> <param name='loop' value='false' /> <param name='fullscreen' value='false' /> <param name='controls' value='false' /> </object>Copy the code

The advantages and disadvantages

  • Advantages: RTSP can be played directly without the help of any mediation server
  • Disadvantages: Plug-ins need to be manually installed; NPAPI based, not supported by the latest Chrome and Firefox
  • If all other features of your project are compatible with Internet Explorer on the client’s computer, this is the best solution.

Html5_rtsp_player + websocket_rtSP_proxy realizes live video streaming



Implementation steps

  1. Server Install the Streamedian server
  2. The client uses the video label to play
<video id="test_video" controls autoplay></video>
<script src="Free player. 1.8.4. Js." "></script>
<script>
    if (window.Streamedian) {
        var errHandler = function(err){
            console.log('err', err.message);
        };
        var infHandler = function(inf) {
            console.log('info', inf)
        };
        var playerOptions = {
            socket: "ws://localhost:8088/ws/",
            redirectNativeMediaErrors : true,
            bufferDuration: 30,
            errorHandler: errHandler,
            infoHandler: infHandler
        };
        var html5Player  = document.getElementById("test_video");
        html5Player.src = "RTSP: / / 184.72.239.149 / vod/mp4: / / BigBuckBunny_175k. Mov." ";
        var player = Streamedian.player('test_video', playerOptions);
        
        window.onbeforeunload = function(){
            player && player.destroy();
            player = null;
            Request = null;
        }
    }
</script>
Copy the code

Note: Apply for a license key from the official website. Otherwise, the socket can only identify the advantages and disadvantages of localhost and 127.0.0.1

  • Advantages: Relatively simple implementation
  • Cons: Paid, free version has many restrictions

RTMP is a protocol developed by Adobe, generally using Adobe Media Server can be easily built up; With the advent of open source era, some big god developed nginx RTMP plug-in, can also directly use Nginx to achieve RTMP, the biggest advantage of RTMP mode is low latency, after testing delay is generally 1-3 seconds, can be said to be very real-time; The downside is that it was developed by Adobe, and RTMP relies heavily on Flash for playback, and most modern browsers disable the Flash implementation step due to its own security

  1. Install the FFMPEG tool
  2. • Install nginx-rtmp-module for Linux. • Install nginx-rtmp-module for Windows.
  3. Change the nginx configuration
rtmp{ server{ listen 1935; application live{ live on; record off; } application hls{ live on; hls on; hls_path nginx-rtmp-module/hls; hls_cleanup off; }}}Copy the code
  1. Ffmpeg transcoding
ffmpeg -i "RTSP: / / 184.72.239.149 / vod/mp4: / / BigBuckBunny_175k. Mov." " -f flv -r 25 -s 1080*720 -an "RTMP: / / 127.0.0.1:1935 / HLS/mystream"
Copy the code
  1. Video playback
<html> <head> <title>video</title> <! --> <link rel="stylesheet" type="text/css" href="./videojs/video-js.min.css" />
</head>
<body>
<video id="test_video" class="video-js vjs-default-skin vjs-big-play-centered" controls autoplay>
    <source src='RTMP: / / 127.0.0.1:1935 / HLS/mystream' type='rtmp/flv'/> </video> </body> </html> <! -- introducing js --> <scripttype="text/javascript" src="./videojs/video.min.js"></script>
<script type="text/javascript" src="./videojs/videojs-flash.js"></script>
<script>
videojs.options.flash.swf = "./videojs/video-js.swf"
    var player = videojs('test_video', {"autoplay":true});
    player.play();
</script>
Copy the code

Note: When using Google Chrome to play, you need to enable Flash

HLS (HTTP Live Streaming) is a protocol based on HTTP proposed by Apple. HTTP Live Streaming is not a real real-time Streaming system because of the potential time delay corresponding to the size and duration of the media segments. On the client, at least one segment media file can not be played until it is completely downloaded, and it is usually required to download two media files before playing to ensure seamless connection between audio and video in different segments. In addition, the encoder and stream splitter on the server must wait for at least one TS file to be generated before the client can start downloading, which also introduces potential latency. The server software wraps the received stream into a new TS file every cached time, and then updates the M3U8 file. In the m3U8 file, only the indexes of the latest clips are kept, so as to ensure that the audience can see the new content whenever they connect to it, and realize the effect similar to live broadcast. The theoretical minimum delay of this method is the length of one TS file, generally 2-3 TS files. Implementation steps

  1. Install the FFMPEG tool
  2. Ffmpeg transcoding
ffmpeg -i "RTSP: / / 184.72.239.149 / vod/mp4: / / BigBuckBunny_175k. Mov." " -c copy -fHls-hls_time 2.0 - HLS_LIST_size 0 - hlS_wrap 15"D:/Program Files/html/hls/test.m3u8"
Copy the code

Ffmpeg instructions on HLS

  • -hls_time n: set the length of each slice. The default value is 2. The unit is in seconds
  • -hls_list_size n: sets the maximum number of items to be saved in the playlist. If the value is set to 0, pieces of information will be saved. The default value is 5
  • -hls_wrap n: indicates the number of slices to be covered. If the value is set to 0, it will not be covered. The default value is 0. This option avoids storing too many slices on disk and limits the maximum number of slices that can be written to disk
  • -hls_start_number n: set the sequence number value in the playlist to number. The default value is 0
  1. Video playback
<html> <head> <title>video</title> <! --> <link rel="stylesheet" type="text/css" href="./videojs/video-js.min.css" />
</head>
<body>
<div class="videoBox">
    <video id="my_video_1" class="video-js vjs-default-skin" controls>
        <source src="http://localhost:8088/hls/test.m3u8" type="application/x-mpegURL"> 
    </video>
</div>
</body>
</html>
<script type="text/javascript" src="./videojs/video.min.js"></script>
<script type="text/javascript" src="./videojs/videojs-contrib-hls.min.js"></script>
<script>
videojs.options.flash.swf = "./videojs/video-js.swf"
    var player = videojs('my_video_1', {"autoplay":true});
    player.play();
</script>
Copy the code

Jsmpeg is a video player written in JavaScript. It consists of an MPEG-TS splitter, MPEG1 video and MP2 audio decoders, WebGL and Canvas2D renderers, and WebAudio sound output. JSMpeg can load static video over Ajax, and allow MPEG to be sent over WebSockets. The front end uses JS to parse MPEG to continuously draw canvas, including audio. JSMpeg can decode 720p video at 30fps on the iPhone 5S and works with any modern browser (Chrome, Firefox, Safari, Edge). A simple example:

<script src="jsmpeg.min.js"></script>
<div class="jsmpeg" data-url="video.ts"></div>
Copy the code

WebRTC is a set of API that supports web browsers for real-time audio and video, such as: HTML5 calls the camera directly through WebRTC, but if you want to achieve the display of remote video stream, you need to convert RTSP to WebRTC stream for web display.

Refer to the link

RTMP, HTTP-FLV, HLS Introduction to the differences between the HLS,HTTP,RTSP, and RTMP protocols Browser playback RTSP video streaming solution Chrome live broadcast RTSP (Media Source Extensions + H5Stream)

The last

GitHub if there are omissions, please correct!!

If feel helpful to you! Please don’t forget to like or follow! Your attention will be my motivation to move forward! Rush duck!!

“Fearless front end” updates the community excellent technical articles from time to time!