Write this article power: before online education, although the use of third-party development, but the bottom of the use of webRTC technology, has been looking for time to study, recently see the relevant implementation, personally lu code to achieve the principle. Learn the following functions and how to implement them:

This article is written relatively rough, the specific implementation of the combination of source code understanding

Main steps:

  • Start the project and call createOffer to get the offer
  • Call createAnswer (pc.localDescription) and add the desktop stream to this function
  • On the control side, the pc.localDescription obtained above is passed into setRemote and called, while listening for the increase of the stream
  • To implement STUN, see below

MediaStream API

  • Streaming of media content

  • A stream object can contain multiple tracks, including audio and video tracks, and so on

  • It can be transmitted through WebRTC

  • You can play by tag

How do I capture desktop/window streams

async function getScreenStream(){
    const sources = await desktopCapturer.getSources({types: ['screen']})
    navigator.webkitGetUserMedia({
        audio:false.video: {
            mandatory: {
                chromeMediaSource: 'desktop'.chromeMediaSourceId: sources[0].id,
                maxWidth: window.screen.width,
                maxHeight: window.screen.height
            }
        }
        // Successfully capture the first argument placed in callback
    },(stream) = >{
        peer.emit('add-stream', stream)
    }, (err) = > {
        // It must be written, or an error will be reported
        console.log(err)
    })
}
Copy the code

How do I play a media stream object

var video = document.querySelector('video')
video.srcObject = stream
video.onloadedmetadata = function(e) {
 video.play();
}
Copy the code

Desktop streaming is implemented

The simplest transfer process

SDP

Session Description Protocol (SDP) is a Session Description Protocol used to describe multimedia sessions. It is mainly used for communication between negotiation parties and transmission of basic information.

  • The SDP format contains multiple lines, each of which

    =


  • : characters that represent specific attributes, such as v for version


  • : structured text, formatted according to the attribute type, UTF8 encoding

Real coding

Step 1: Create RTCPeerConnection, which is an encapsulated object for establishing P2P connection. Then we will call RTCPeerConnection method to create an offer, which is an SDP. SDP is essentially a protocol. We’re going to talk about SDP in a second, and you can think of it as, we’re going to make an invitation, we’re going to set our invitation to our LocalDescription, and you can think of these three steps very simply, we’re actually creating an initial connection, an initial P2P connection.

Then we need to send the invitation generated by our control terminal, namely our offer, to our puppet terminal through other media, for example, you can directly send the INVITATION SDP to our puppet terminal through wechat or SMS. After our puppet terminal gets the offer, it will also create an object of PeerConnection. In the section on desktop sharing, we actually talked about that we need to capture our desktop stream and add it to our PeerConnection. Then we set the control end as the remote end of our puppet end. To send the PeerConnection corresponding to the offer, we call the setRemoteDescription method. Then to show that we have determined, we call createAnswer, which represents an SDP that I have determined. Then I will also set the SDP in the puppet side, at this time our puppet side will also generate an SDP, we can also through any method, our response TO the SDP to the control side, at this time the control side will also set the SDP as the object it wants to transmit.

At this point, the control side and the puppet side have been set to each other as Remote, at which point a P2P connection will be established. The control side is easy to summarize, we launched the clothes moths invitation, puppet end after determine the invitation, add your own desktop flow to P2P connection, then also return a definitive agreement and in the end we control the set will also determine agreement, this way we will represent the control to the client and puppet P2P connection already can begin.

NAT (Network Address Translation)

P2P data exchange needs to go through the server. For example, in Meituan Daelephant, if you want to send a message, we will pass it to the server and then send it to the corresponding user. If the P2P connection, it will be faster and more secure than the server.

Why want to do a server forwarding, the answer is very much, one reason is that we are in the end-to-end communication, need to know each other’s public IP and port number, in fact this is not an easy thing, because our network environment filled with NAT technology, NAT is a network address translation, why this technology will appear, For those of you who know anything about networks, IPv4 addresses are not enough, and they’re not enough for two main reasons:

  1. The first reason is that IPv4, which is a 32-bit integer, can theoretically support just over 4 billion addresses, a number far smaller than the world’s population.
  2. The second reason is that IP addresses are unevenly distributed geographically. The United States has a lot of IP addresses, and China is very scarce. China has only 0.06 addresses per capita, while Asia, which accounts for 56% of the world’s population, gets only 9%.

Therefore, in order to solve the address problem, NAT emerged. In NAT, each device will have an independent LAN address, and then they will share the same public IP address when connecting with the external network. NAT is responsible for maintaining a mapping table including the local IP port and the external IP port.

How do I get the real IP and port?

The basic method is to establish a connection between the server and one party ClientB. At this time, NAT will establish a mapping of the extranet with a port number. After that, the server can know the IP and port of ClientB’s extranet, and then pass it to ClientA. Finally ClinetA can directly use NAT to make a hole, and then communicate with ClientB. There is already an integrated mechanism in webRTC, that is, STUN service. When ClientA and ClientB want to make P2P connection, it first needs to make a hole through our server, and then send the result of hole drilling to ClientB. ClientB also needs to do a similar operation. In this way, with the help of the server, ClientA and ClientB can get the IP address and port of each other’s real public network.

WebRTC’s NAT penetration is a whole mechanism we call ICE

Interactive Connectivity Establishment (ICE) Creation

  • Session Traversal Utilities for NAT (STUN) is preferred. The NAT Session traverses the application

  • Alternative TURN (Traversal Using Relay NAT), Traversal implemented by NAT

    1. Full Cone NAT – Full Cone NAT

    2. Restricted Cone NAT – Restricted Cone NAT

    3. Port Restricted Cone NAT Indicates the Port Restricted Cone NAT

    4. Symmetric NAT Symmetric NAT

To play a video, you need to change the address

The whole process of STUN

First of all, our control end will initiate an address, and then our STUN service will make the hole, and then return to our control end, at this time the control end knows its IP and port of the external network, and then we need to pass through a certain medium and then send it to the puppet end. This is the same as the SDP transmission of PeerConnection. You can transmit through any medium, such as email or wechat. After the puppet gets the IceEvent, it will add our proxy through the addIceCandidate method. Our puppet end will know the control end of an external IP, similar puppet end will get their OWN IP and port to the control end, the control end add ICE proxy, in this way, our P2P is really established successfully.

Signaling service: a server that transmits messages between webrTCs, connecting both ends

The signaling bearer is used for various forwarding

Based on the webSocket

Establish RTCDataChannel procedure for data transmission

/ / control
var pc = new RTCPeerConnection();
let dc = pc.createDataChannel('robotchannel', {reliable: false});
// Successfully established
dc.onopen = function() {
    console.log('opened')
    peer.on('robot'.(type, data) = > {
        dc.send(JSON.stringify({type, data}))
    })
}
// Receive the message
dc.onmessage = function(event) {
    console.log('message', event)
}
dc.onerror = (e) = > {console.log(e)}
Copy the code
/ / puppet
const pc = new window.RTCPeerConnection();	
pc.ondatachannel = (e) = > {
    console.log('data', e)
       e.channel.onmessage = (e) = > {
        console.log('onmessage', e, JSON.parse(e.data))
       let {type, data} = JSON.parse(e.data)
        console.log('robot', type, data)
        if(type === 'mouse') {
            data.screen = {
                width: window.screen.width, 
                height: window.screen.height
            }
        }
        ipcRenderer.send('robot', type, data)
     }
 }
Copy the code

See your own Git source code

There are three steps to interview speaking:

  1. Obtaining Multimedia Data
  2. Establish P2P connections and transfer multimedia data through signaling
  3. To transmit data