Shallow-in shallow-out WebRTC- Video calling

Summary of WebRTC

(Some excerpts from Baidu Encyclopedia)

WebRTC, which stands for Web Real-time Communication, is an API that enables Web browsers to conduct real-time voice or video conversations. It became open source on 1 June 2011 and was included in the W3C recommendation of the World Wide Web Consortium with the support of Google, Mozilla, and Opera.

On January 26, 2021, the World Wide Web Consortium (W3C) and the Internet Engineering Task Force (IETF) announced that Web Real-time Multi-Service Communication (WebRTC) is now an official standard that can bring audio and video communications anywhere on the Web. www.w3.org/TR/webrtc/

In May 2010, Google acquired VoIP software developer Global IP Solutions’ GIPS engine for $68.2 million and renamed it “WebRTC.” WebRTC uses GIPS engine to realize web-based video conferencing, and supports 722, PCM, ILBC, ISAC and other encoding, while using Google’s own VP8 video decoder; It also supports RTP/SRTP transmission.

Google integrated the software into its Chrome browser in January 2012. Also, the FreeSWITCH project claims support for iSAC Audio Codec.

QQ voice technology also comes from Global IP Solutions

WebRTC provides the core technology of video conferencing, including audio and video collection, codec, network transmission, display and other functions, and also supports cross-platform: Windows, Linux, MAC, Android.

The core code is written in C++. The Web platform is just an implementation of WebRTC. The core code is still the C++ version of WebRTC

WebRTC Architecture Diagram (reproduced in network)

As we can see from the architecture diagram, WebRTC helped me implement the audio engine, the video engine, the web delivery engine, encapsulating the upper layer API, and allowing developers to use the browser (Chrome, FireFox… Easily and quickly develop rich real-time multimedia applications, without downloading and installing any plug-in, also do not need to pay attention to multimedia digital signal processing process, only need to write a simple Javascript program to achieve

WebRTC principle

A peer-to-peer channel in a browser that can send any data without passing through a server.

The implementation of WebRTC is to establish a direct connection between the clients without the server transfer, namely P2P. Therefore, they need to know each other’s external network address, and most computers are located after NAT, only a few hosts have external network address. This requires a NAT penetrating technology, such as STUN and TRUN.

WebRTC uses the default STUN server to obtain the extranet address and port of the current host. Chrome uses a default STUN server under Google domain name.

NAT Traversal (piercing)

NAT traversal is required when a connection is established between hosts on a private TCP/IP network that uses NAT devices. NAT behavior is not standardized, traversal technology mostly use a public server, so that anywhere in the world can access the IP address, in RTCPeerConnection use ICE framework to ensure the implementation of NAT traversal RTCPeerConnection.

ICE comprehensive NAT traversal technology

ICE(Interactive Connectivity Establishment) comprehensive NAT traversal technology

The ICE framework incorporates various NAT Traversal technologies such as STUN and TURN. ICE uses STUN to establish a UDP-based connection, then TCP to try HTTP, then HTTPS. If that fails, ICE uses the relayed TURN server.

Do WebRTC on the Web platform

The process for creating a WebRTC

ClientA createOffer
ClientB bindingOffer
ClientB createAnswer
ClientA bindingAnswer, to establishP2PThe connection

Browser-side code implementation

MediaStreamGet local audio and video streams
RTCPeerConnectionAn RTC object is used to establish P2P connections and transmit audio and video streams
RTCDataChannelUsed to transfer custom data.

1. ClientA created`WebRTC`Object and RTC data channel, and bind some events

SDP: Session Description Protocol

// Create the WebRTC object
const pc = new RTCPeerConnection();

// Create a WebRTC data channel to transfer custom data
const dc = pc.createDataChannel("channel");

// The data channel is enabled
dc.onopen = () = > console.log("Data Channel Opened");

// The data channel receives the data event
dc.onmessage = (e) = > console.log("Get the Message:", e.data);

// SDP change event
pc.onicegatheringstatechange = () = > {
   if (pc.iceGatheringState === "complete") {  // Print SPD when it is created
      console.log(JSON.stringify(pc.localDescription)); }}Copy the code

2. ClientA created`Offer`

Using PC. CreateOffer (); To create the Offer, and call setLocalDescription to bind the Offer to the local SDP

When you’re done, you’ll see your Offer printed in the console

Copy this Offer and we can write ClientB code

/** * create Offer *@return {Promise<void>}* /
async function createOffer() {
   const offer = await pc.createOffer();
   await pc.setLocalDescription(offer);
}
Copy the code

3. ClientB binding`Offer`

Write ClientB’s code first

Use similar code to create ClientB

Since ClientA already has a data channel, ClientB does not need to create its own

const pc = new RTCPeerConnection();

let dc = null;

// Similarly listen for SDP changes and print SPD when it is created
pc.onicegatheringstatechange = () = > {
   if (pc.iceGatheringState === "complete") {
      console.log(JSON.stringify(pc.localDescription)); }}// Listen for the data channel event, add the event, save the data channel
pc.ondatachannel = (e) = > {
   console.log("data channel");
   dc = e.channel;
   dc.onmessage = (e) = > console.log("get message", e.data);
   dc.onopen = async (e) => console.log("channel opened");
}
Copy the code

Next, let’s bind the Offer we created earlier

async function setOffer() {
   await pc.setRemoteDescription(JSON.parse(offer to bind));console.log("Offer Set");
}
Copy the code

Call this setOffer to set the Offer created earlier

SetRemoteDescription sets the SDP of the other client

4. ClientB created`Answer`

Modify the ClientB code to create an Answer using createAnswer

And set it to the local SDP with setLocalDescription

We create the Answer immediately after setting the Offer

After the above process, you can see the printed local Answer in the console. Copy this Answer and we go to ClientA to bind it

async function setOffer() {
   await pc.setRemoteDescription(JSON.parse(offer to bind));console.log("Offer Set");
   await createAnswer();
}

async function createAnswer() {
   const answer = await pc.createAnswer();
   await pc.setLocalDescription(answer);
}
Copy the code

5. ClientA binding`Answer`, to establish`P2P`The connection

Going back to ClientA, bind the Answer we just created with the following code

async function setAnswer() {
   await pc.setRemoteDescription(JSON.parse(Answer to be bound));console.log("Answer Set");
}
Copy the code

After executing, we see that in ClientA the console prints a data channel open prompt

The received data channel and data channel open prompt are printed in ClientB

6. Use`RTCDataChannel`Send custom data

The data channel has been turned on and we can send any data directly using the data channel

dc.send("I'm ClientA. Who are you?");
Copy the code

The data transmission is not dependent on any server. This is called peer-to-peer transmission.

7. Enable the local camera

Write two videos in the Html of both clients and take out the standby with the ID


<style>
    .video-div div {
        width: 50%;
    }

    .video-div div video {
        width: 100%;
        background-color: #6D7587;
    }

    .video-div {
        display: flex;
        align-items: center;
        justify-content: center;
        width: 100%;
    }
</style>
<div class="video-div">
    <div>
        <label>Ta in the video</label>
        <video id="video" autoplay="autoplay"></video>
    </div>
    <div>
        <label>Your video</label>
        <video id="myVideo" autoplay="autoplay"></video>
    </div>
</div>

<script>
    const video = document.getElementById("video");
    const myVideo = document.getElementById("myVideo");
</script>
Copy the code

Use the navigator in the two clients. MediaDevices. GetUserMedia open cameras, and will push to the video element

async function openVideo() {
   const stream = await navigator.mediaDevices.getUserMedia({
      video: true.audio: true
   });
   myVideo.srcObject = stream;
}
Copy the code

You can see your handsome face in the video on the right

8. Push audio and video streams to`WebRTC`And receive audio and video streams from the other party

Modify the openVideo function to add code to push streams to WebRTC

async function openVideo() {
   const stream = await navigator.mediaDevices.getUserMedia({
      video: true.audio: true
   });
   myVideo.srcObject = stream;
   stream.getTracks().forEach((track) = > {
      pc.addTrack(track, stream);
   });
}
Copy the code

Enable the camera and push the stream before creating the local SPD

Using asynchrony makes it easier to control the order of calls

async function init() {
   await openVideo();
   await createOffer();
}

init();
Copy the code

Then listen for an onTrack event for RTCPeerConnection, indicating that an audio and video track has been pushed in

Pushes the audio/video stream to another video TAB

pc.ontrack = (e) = > {
   console.log("track");
   video.srcObject = e.streams[0];
}
Copy the code

With these actions, you can now see your handsome face on both video tabs, one of which is posted by the other client

Better WebRTC architecture

1. Unify ClientA and ClientB into one client software

For all the audio, video, and conference products on the market, I’m afraid there is no need to distinguish the two apps between the caller and the called

Write a web page that has both the caller function and the called function

2. The screen is shared

const stream = await navigator.mediaDevices.getDisplayMedia({
   video: {
      cursor: "always"
   },
   audio: {
      echoCancellation: true.noiseSuppression: true.sampleRate: 44100}});Copy the code

3. Import another server

Use a server to forward the SDP, SDP will be according to the network status changes, and the SPD is very long, user exchange SDP is not very convenient If you have your own user system, can be done with a unique user ID identification, users only need to call an identity ID such as a QQ number, use the signaling server for forwarding and SDP, At the same time, it is responsible for other tasks, such as forwarding whether the called user answers the call or not, and forwarding whether the called user hangs up

4. Do an online interview?

Do an online interview?

Post code

The following code incorporates dual clients

<! DOCTYPEhtml>
<html lang="zh">
<head>
    <meta charset="UTF-8">
    <title>WebRTC Video calls</title>

    <style>
        html.body {
            width: 100%;
            height: 100%;
            margin: 0;
            padding: 0;
        }

        .video-div div {
            width: 50%;
        }

        .video-div div video {
            width: 100%;
            background-color: #6D7587;
        }

        .video-div {
            display: flex;
            align-items: center;
            justify-content: center;
            width: 100%;
        }

        .input-div {
            width: 100%;
            margin-top: 50px;
            display: flex;
            align-items: center;
            justify-content: center;
            flex-direction: row;
        }

        .input-div > div {
            flex-direction: column;
        }

        .input-div div {
            margin-top: 30px;
            width: 100%;
            display: flex;
            align-items: center;
            justify-content: center;
        }

        #SDP {
            width: 50%;
            height: 300px;
        }
    </style>
</head>
<body>

<div class="video-div">
    <div>
        <label>Ta in the video</label>
        <video id="video" autoplay="autoplay"></video>
    </div>
    <div>
        <label>Your video</label>
        <video id="myVideo" autoplay="autoplay"></video>
    </div>
</div>

<div class="input-div">
    <div>
        <! -- Create an Offer-->
        <button onclick="createOffer()">Create the Offer</button>

        <! -- Set the area of the offer -->
        <div>
            <label for="offerInput">Input offers:</label>
            <input type="text" id="offerInput"/>
            <button onclick="setOffer()">Set up the offer</button>
        </div>

        <! -- set the answer area -->
        <div>
            <label for="answerInput">Enter the answer:</label><input type="text"
                                                             id="answerInput"/>
            <button onclick="setAnswer()">Set the answer</button>
        </div>
    </div>

    <! -- local SDP-->
    <div>
        <label for="SDP">Your SDP:</label>
        <textarea id="SDP"></textarea>
    </div>
</div>
<script>

    const SDP = document.getElementById("SDP");
    const answerInput = document.getElementById("answerInput");
    const offerInput = document.getElementById("offerInput");
    const video = document.getElementById("video");
    const myVideo = document.getElementById("myVideo");

    /** * Open the camera and add Track to RTCPeerConnection *@return {Promise<void>}* /
    async function openVideo() {
        const stream = await navigator.mediaDevices.getUserMedia({
            video: true.audio: true
        });
        myVideo.srcObject = stream;
        stream.getTracks().forEach((track) = > {
            pc.addTrack(track, stream);
        });
    }

    / / create a RTCPeerConnection
    const pc = new RTCPeerConnection();

    // Monitor ontrack, indicating that a video stream or audio stream is being pushed
    pc.ontrack = (e) = > {
        console.log("ontrack");
        video.srcObject = e.streams[0];
    }

    // This event indicates a change in the local SDP
    pc.onicegatheringstatechange = () = > {
        if (pc.iceGatheringState === "complete") {  // If the SDP is created, show it to him
            SDP.value = JSON.stringify(pc.localDescription); }}// Create a custom data channel
    let dc = null;
    dc = pc.createDataChannel("channel");

    // onMessage indicates that a message has been received
    dc.onmessage = (e) = > console.log("Get the Message:", e.data);

    // Indicates that the custom data channel is enabled
    dc.onopen = () = > console.log("Data Channel Opened");

    // This event indicates that a custom data channel has been passed in, i.e. PeerA has created a custom data channel and PeerB does not need to create it itself
    pc.ondatachannel = (e) = > {
        console.log("Data Channel");
        dc = e.channel;

        // Similarly, let him listen for some events
        dc.onmessage = (e) = > console.log("Get Message", e.data);
        dc.onopen = () = > console.log("Channel Opened");
    }

    openVideo();    // Open video add audio video track to RTCPeerConnection

    /** * Set Answer *@return {Promise<void>}* /
    async function setAnswer() {
        await pc.setRemoteDescription(JSON.parse(answerInput.value));
        console.log("Answer Set");
    }

    /** * set Offer *@return {Promise<void>}* /
    async function setOffer() {
        await pc.setRemoteDescription(JSON.parse(offerInput.value));
        console.log("Offer Set");
        await createAnswer();
    }

    /** * create Answer *@return {Promise<void>}* /
    async function createAnswer() {
        const answer = await pc.createAnswer();
        await pc.setLocalDescription(answer);
    }

    /** * create Offer *@return {Promise<void>}* /
    async function createOffer() {
        const offer = await pc.createOffer();
        await pc.setLocalDescription(offer);
    }

</script>
</body>
</html>
Copy the code

Shallow-in shallow-out WebRTC- Video calling

Summary of WebRTC

(Some excerpts from Baidu Encyclopedia)

WebRTC principle

NAT Traversal (piercing)

ICE comprehensive NAT traversal technology

Do WebRTC on the Web platform

The process for creating a WebRTC

Browser-side code implementation

1. ClientA createdWebRTCObject and RTC data channel, and bind some events

2. ClientA createdOffer

3. ClientB bindingOffer

4. ClientB createdAnswer

5. ClientA bindingAnswer, to establishP2PThe connection

6. UseRTCDataChannelSend custom data