Why explore flv.js live at this time? The reason is that Flash has been disabled by default by major browser manufacturers. The common Flash live broadcast scheme requires users’ consent to use Flash before normal use of the live broadcast function, which is fatal to user experience.

Before introducing flv.js, I’ll introduce some of the common live protocols and give some data from my tests on their latency and performance. If it’s too hard for you to watch, let’s start with some basic concepts of audio and video technology.

Common Live Streaming Protocols

  • RTMP: Bottom basedTCPTo rely on Flash on the browser side.
  • HTTP-FLVBased on:HTTPStreaming IO transfers FLV, depending on browser support for FLV playback.
  • WebSocket-FLVBased on:WebSocketTransmit FLV, depending on browser support for playing FLV.WebSocketBased on theHTTPBuild onWebSocketYou have to build it before you connect itHTTPThe connection.
  • HLS: Http Live Streaming, apple proposed based onHTTPStreaming media transfer protocol.HTML5You can play directly.
  • RTPBased on:UDP, 1 second delay, browser does not support.

Latency and performance data of common live broadcast protocolsThe following data are for reference only

Transfer protocol player delay memory CPU
RTMP Flash 1s 430M 11%
HTTP-FLV Video 1s 310M 4.4%
HLS Video 20s 205M 3%

In browser-supported protocols, delay sort is: RTMP = http-flv = websocket-flv < HLS. Performance sort is the opposite: RTMP > http-flv = websocket-flv > HLS.

It can be seen that the http-FLV protocol is good for live streaming in the browser. The performance is better than RTMP+Flash, and the latency can be as good as or even better than RTMP+Flash.

FLV. Introduction of js

Flv.js is an open source project from Bilibli. It parses FLV files and feeds them to native HTML5 Video tags to play audio and Video data, making it possible for browsers to play FLV without Flash.

FLV. Js advantage

  • Thanks to the browser’s hardware acceleration for native Video tabs, performance is good and hd is supported.
  • It also supports recording and live broadcasting
  • Remove the reliance on Flash

FLV. Js restrictions

  • The video encoding contained in the FLV must beH.264, the audio encoding must beAACorMP3IE11 and Edge browsers do not support MP3 encoding, so it is best to use H.264+AAC encoding in FLV, which is compatible with audio and video services.
  • For recording, rely onNative HTML5 Video tags 和 Media Source Extensions API
  • For live broadcast, rely on the broadcast technology required by recording, and rely onHTTP FLVorWebSocketTo transmit FLV. Among themHTTP FLVData needs to be pulled through streaming IO, which supports streaming IOfetchorstream
  • flv.min.jsThe file size is 164Kb, 35.5Kb after Gzip, and about the same after Flash Player Gzip.
  • Due to the dependenceMedia Source ExtensionsAt present, all browsers under iOS and Android4.4.4 are not supported, that is to say, flv.js is basically unavailable for mobile terminals.

Flv.js relies on a compatible list of browser features

  • HTML5 Video
  • Media Source Extensions
  • WebSocket
  • HTTP FLV: FETCH or stream

FLV. Js principle

The only thing flv.js does is to decode the FLV data from native JS and feed it to native HTML5 Video tags via the Media Source Extensions API. (HTML5 native only supports playback in MP4 / WebM format, not FLV)

Why does FLV.js go around, get FLV from the server, decode it, convert it and feed it to the Video tag? Here’s why:

  1. Compatible with current live broadcast schemes: Most of the audio and video services of current live broadcast schemes use FLV container format to transmit audio and video data.
  2. FLV container format is simpler than MP4 format, parsing faster and more convenient.

Flv.js compatible scheme

Since flv.js is not compatible at present, it is necessary to take into account browsers that do not support flV.js in order to use it in products. The compatibility scheme is as follows:

PC

  1. Prefer HTTP-FLV because it has low latency and good performance. 1080P is smooth.
  2. Use Flash player to play RTMP streams if flv.js is not supported. Flash has good compatibility but poor performance and is disabled by default in many browsers.
  3. You can use HLS if you don’t want to use Flash compatibility, but only Safari supports HLS on PC

The mobile terminal

  1. Http-flv is preferred because of its low latency and the performance of devices that support HTTP-FLV running flv.js is sufficient.
  2. Use HLS if flv.js is not supported, but HLS latency is very high.
  3. HLS also does not support live streaming, because mobile devices do not support Flash.

FLV. Js in actual combat

Said so much introduction and principle, next teach you how to use FLv.js to build a complete live broadcast system. I’ve built a demo for you to play with.

Set up audio and video services

The anchor pushes the stream to the audio and video service, which then forwards it to all connected clients. I recommend LiveGo, which I implemented in the GO language, for your quick build service, as it runs on any operating system.

  1. Download LiveGo and choose the right operating system and bits.
  2. Decompress, executelivegoThe service is started. It starts the RTMP(port 1935) service for the streamer push, and the HTTP-FLV(port 7001) service for playback.

Implement playpage

The React flv.js component reflv is used for fast implementation in the React system. Install NPM I Reflv first, then write the code:

import React, { PureComponent } from 'react';
import Reflv from 'reflv';

export class HttpFlv extends PureComponent {
  render() {
    return (
      <Reflv
        url={`http://localhost:7001/live/test.flv`}
        type="flv"
        isLive
        cors
      />
    )
  }
}Copy the code

Run the above code in a browser. You can’t see the livestream yet because there is no streamer yet.

  • You can use OBS to push streams. Note that OBS is configured properly:

  • You can also useffmpegPush the flow. Push the flow commandffmpeg -f avfoundation -i "0" -vcodec h264 -acodec aac -f flv rtmp://localhost/live/test

Flv.js delay optimization

The livestream delay running as shown above is about 3 seconds, and can be optimized to 1 second. Before teaching you how to optimize, first introduce the running process of live broadcast:

  1. After the host collects audio and video raw data for a period of time, it needs to compress the data because of the huge amount of audio and video raw data:

    • Compress data through H264 video coding
    • Compress audio AAC data through PCM audio encoding

  2. After compression, the compressed data is encapsulated in FLV container format and encapsulated into a FLV TAG

  3. The FLV TAG is then streamed to the audio and video server through the RTMP protocol, and the audio and video server then parses the FLV TAG from the RTMP protocol.

  4. The audio and video server then sends the FLV TAG to the browser through a long link stream established with the browser through HTTP.

  5. Flv. js retrieves FLV tags, parses compressed audio and Video data, and feeds the compressed audio and Video data to Video for playback.

Once we know the process, we know where to optimize:

  • The host terminal collects audio and video raw data for a period of time, and its professional name is GOP. Shortening this collection time (i.e., reducing the GOP length) can optimize the delay, but the disadvantage of doing so is that the video compression rate is not high and the transmission efficiency is low.
  • Turning off the I frame cache of audio and video servers can optimize latency, and the disadvantage is that users will see the first screen of a live broadcast for a longer time.
  • Reducing the buffer of the audio and video server can optimize the latency, but the disadvantage is that the audio and video server processing efficiency is reduced.
  • Reducing the browser-side flv.js buffer can optimize latency, and the downside is that browser-side processing is less efficient.
  • The flv.js Worker is opened on the browser side, and multiple processes run flv.js to improve the parsing speed and optimize the delay. The flv.js configuration code to do so is:
{
          enableWorker: true,
          enableStashBuffer: false,
          stashInitialSize: 128,
}Copy the code

Here is the complete optimized code