The author works for a smart home company, security camera business group, responsible for the core audio and video live broadcast, video playback and other salesman functions. The original player is based on hybrid layer, provided by native, and called by JS layer. The problems are large performance consumption, various interfaces, call responsible, difficult to maintain, tuTK + P2P connection is unstable, unable to transplant PC, etc. So, led by the architecture recently started the research of H5VIDEO streaming media player, share your experience and experience.

Webrtc streaming media server

The main function of a streaming media server is to transfer video files to clients through streaming protocols (RTP/RTSP, MMS, RTMP, weBRTC, etc.) for users to watch online. It can also receive real-time video streams from video capture and compression software, and then broadcast them to the client by streaming protocol.

The WeBRTC streaming media server collects and plays videos based on the WEBRTC protocol.

When weBRTC standard interaction is supported with TURN/ signaling, point-to-point live broadcasting can be provided, and one-to-many (small) live broadcasting can be supported when the bandwidth performance of the device is sufficient. The diagram below:There are generally three schemes for multi-party communication architecture:

In the Mesh solution, multiple terminals are connected in pairs to form a Mesh structure. For example, terminals A, B, and C communicate many-to-many. When terminal A wants to share media (such as audio and video), terminal A needs to send data to terminal B and terminal C respectively. Similarly, if B wants to share media, he needs to send data to A and C, and so on. This scheme requires high bandwidth of each terminal.

MCU (Multipoint Conferencing Unit) solution, which consists of a server and multiple terminals to form a star structure. Each terminal will share their own audio and video streams to the server, the server will be in the same room in all terminals mixed audio and video streams, and finally generated a mixed audio and video stream and then sent to each terminal, so that each terminal can see/hear the audio and video of other terminals. In fact, the server side is an audio and video mixer, which can be very stressful for the server.

Selective Forwarding Unit (SFU) is also composed of one server and multiple terminals. However, different from MCU, SFU does not mix audio and video streams. After receiving the audio and video streams shared by a terminal, SFU directly sends the audio and video streams to other terminals in the room. It is essentially an audio and video routing repeater.

Webrtc supports the Mesh architecture by default. SFU is the current mainstream solution, and most manufacturers adopt SFU architecture model.

Implementation steps

H5 Use the video label to initialize the player.

import React, { memo } from 'react';
import { Button, Flex } from '@leedarson/ui-mobile';
import useWebRTC from './useWebRTC';

const WebRTCPlayer = memo(() => {
  const { handleStart, call, handleStop } = useWebRTC();
  return (
    <>
      <video id="webRTC-player" autoPlay muted />
      <Flex>
        <Button width="30%" onClick={handleStart}>
          START
        </Button>
        <Button width="30%" onClick={call}>
          Call
        </Button>
        <Button width="30%" onClick={handleStop}>
          STOP
        </Button>
      </Flex>
    </>
  );
});

export default WebRTCPlayer;
Copy the code

Websocket is used for authentication and streaming media transmission, and WSS protocol, SSL + WS, is used for security transmission.

import logger from '@leedarson/logger'; class WSWebRTC { constructor(url) { this.ws = new WebSocket(url); this.ws.conn_status = false; } init(localId, remoteId, processSignalingMessage) { this.ws.onopen = () => { const login = { clientId: localId }; this.ws.conn_status = true; this.ws.send(JSON.stringify(login)); }; this.ws.sendFormatMsg = obj => { const m = { peerId: remoteId, payload: obj }; const str = JSON.stringify(m); this.ws.send(str); }; This.ws. Onmessage = event => {if (typeof Event.data === 'string') {// processSignalingMessage(event.data); return; } if (event.data instanceof ArrayBuffer) { const buffer = event.data; logger.log('Received arraybuffer', buffer); this.ws.close(); } logger.log(`Received Message: ${event.data}`, typeof event.data); }; This.ws. Onclose = () => {this.ws. Conn_status = false; }; this.ws.onerror = event => { this.ws.conn_status = false; logger.error('webRTC_Error:', event); }; } } export default WSWebRTC;Copy the code

Use the navigator. MediaDevices. GetUserMedia method for flow

This method prompts the user to grant permission to use media input, which produces a MediaStream containing a track for the requested media type. Note that calls on mobile devices must be based on HTTPS. The main reason is that the browser prevents video streams from being stolen by attacks for security purposes. (There is no restriction on PC.) When the author upgraded the APP, this method directly reported an error when calling because the code was packaged and loaded locally in the APP using http://localhost:3000. The author’s solution is to deploy the front-end page related to the WebrTC player on the HTTPS certificate encrypted server to remotely call to solve the problem.

const handleStart = useCallback(async () => { // 1. Connect the webSocket handleWSConnect (); Try {/ / 2. Obtain video stream channel const stream = await the navigator. MediaDevices. GetUserMedia ({audio: true, video: true,}); localVideo.srcObject = stream; localStreamRef.current = stream; // 3. Call (); } catch (error) {logger.error(' read permission or video stream failed ', error); } }, [call, handleWSConnect, localVideo]);Copy the code

Use the webRTC interface RTCPeerConnection to pull the video stream to play.

const onIceCandidate = useCallback( event => { if (event.candidate) { const msg = { candidate: event.candidate.candidate, sdpMLineIndex: event.candidate.sdpMLineIndex, sdpMid: event.candidate.sdpMid, }; if (answerReadyRef.current || localRole === 'Callee') wsClientRef.current.sendFormatMsg(msg, true); else { qRemoteCandidates.current.push(msg); } } }, [localRole], ); const handleAddStreamEvent = useCallback( event => { const { streams = [], stream } = event; if (callerRef.current.ontrack) { if (remoteVideo.srcObject ! == streams[0]) { const stream0 = streams[0]; remoteVideo.srcObject = stream0; logger.log(` received remote stream`); } } else if (remoteVideo.srcObject ! == stream) { remoteVideo.srcObject = stream; logger.log(` received remote stream`); } }, [remoteVideo], ); / / success callback const onCreateOfferSuccess = useCallback (desc = > {callerRef. Current. SetLocalDescription (desc). Then (() = > { logger.log('successful'); }, error => logger.error('error', error), ); const msg = { type: desc.type, sdp: desc.sdp, }; wsClientRef.current.sendFormatMsg(msg, true); } []); const call = useCallback(() => { // 1. Verify the WSS connection if (! wsClientRef.current || ! wsClientRef.current.conn_status) { logger.error("call is failed case mqtt don't sub ok, please click call later!" ); } callerref. current = new RTCPeerConnection(configuration); const caller = callerRef.current; caller.onicecandidate = onIceCandidate; caller.oniceconnectionstatechange = event => logger.log('ICE state change event: ', event); caller.ontrack = handleAddStreamEvent; if (localStreamRef.current){ localStreamRef.current.getTracks().forEach(track => { caller.addTrack(track, localStreamRef.current); }); } caller.createOffer(offerOptions).then(onCreateOfferSuccess, error => logger.log(error)); }, [configuration, handleAddStreamEvent, onCreateOfferSuccess, onIceCandidate]);Copy the code