This article is a summary of WebRTC learning. The author is Xie Yan, a front-end engineer of New Oriental Online. She is mainly responsible for the development and maintenance of question bank, live broadcast and other related businesses.

introduce

Google’s WebRTC enables fast, real-time communication between browsers. It allows web applications or sites to set up peer-to-peer connections between browsers for streaming video and audio, or any other data, without the use of intermediaries. WebRTC includes standards that enable users to create peer-to-peer data sharing and conference calls without installing any plug-ins or third-party software.

There are three main data structures

  • MediaStream: Access to audio and video streams
  • RTCPeerConnection: Communication of audio and video data
  • RTCDataChannel: any data other than audio and video data

MediaStream

MediaStream, obtained through getUserMedia, indicates a media data stream, including audio and video.

navigator.mediaDevices.getUserMedia({
  audio: true.video: {
    width: {min: 1024.ieda: 1280.max: 1920},
    height: {min: 576.ideal: 720.max: 1080},
    frameRate: {max: 30}
  }
}).then(stream= > {
  let video = document.createElement('video')
  document.body.append(video)
  video.srcObject = stream
  video.onloadedmetadata = function() {
    video.play()
  }
})
Copy the code

RTCPeerConnection

It can be understood as a super functional socket. RTCPeerConnection is the same as an ordinary socket. There is at least one RTCPeerConnection object at each end of the call. In WebRTC, it is responsible for establishing connection with each end, receiving and sending audio and video data, and ensuring the service quality of audio and video.

Suitable for various browsers

Typically, a script called Adapter.js is added to the display page, which provides a consistent, up-to-date WebRTC API for all browsers.

.<script src="https://WebRTC.github.io/adapter/adapter-latest.js"> </script>.Copy the code

How does RTCPeerConnection work

  • Create an RTCPeerConnection object for each end of the connection and add a local stream to the RTCPeerConnection object, which is obtained from getUserMedia();

  • Obtain local media description information, that is, SDP information, and exchange it with the peer end.

  • Get the network information, the Candidate (IP address and port), and exchange it with the remote end.

    See the RTCPeerConnection example for an example

RTCDataChannel

Non-audio and video data are transmitted through RTCDataChannel, which is a class specially used to transmit non-audio and video data in WebRTC. RTCDataChannel supports a large number of data types, including: String, Blob, ArrayBuffer, and ArrayBufferView. The DataChannel interface represents a bidirectional DataChannel between two nodes that can be set up for reliable or unreliable transmission

Create RTCDataChannel as follows:

.var pc = new RTCPeerConnection(); // Create the RTCPeerConnection object
var dc = pc.createDataChannel("dc", options); // Create RTCDataChannel object parameter 1: label name parameter 2: configuration item.Copy the code

The options configuration items

  • Ordered: Whether the message is delivered in order
  • MaxPacketLifeTime: Indicates the maximum duration of retransmission failure
  • MaxRetransmits: Indicates the maximum number of failed messages to be retransmitted
  • Protocol: indicates the user-defined subprotocol. The default value is null
  • Negotiated: If true, remove the automatic Settings for the other party’s data channel. This also means that you can create your own data channel on the other side with the same ID
  • Id: When Negotiated is true, allows you to provide your own ID to bind to a channel

In addition to the three objects described above, there are two important processes in WebRTC, namely media negotiation and network negotiation.

Media negotiation

Media negotiation is about looking at what codecs are supported on your device and does mine support them? If my equipment also supports it, then we have negotiated successfully.

SDP

Session Description Protocol (SDP) is a Session Description Protocol that describes multimedia sessions. It is mainly used for the communication process between two parties and the transmission of basic information.

SDP has a multi-line <type>=<value> structure

  • Type: character that represents a specific attribute, such as v for version
  • Value: attribute value. The structure is related to type

Such as:

v=0 // SDP version number, always 0
o=- 3409821183230872764 2 IN IP4 127.0. 01. // Description of the session initiator. m=audio9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126 // Audio stream information.Copy the code

How to negotiate

  • First, the communication parties organize their respective media information, such as codec, SSRC of media stream, transport protocol, IP address and port, according to SDP format.
  • Then, the communication parties exchange SDP information through the signaling server, and after receiving each other’s SDP information, find out which media capabilities they support together.
  • Finally, the two parties begin communication in accordance with the agreed media capabilities.

Network consultation

After understanding each other’s media capabilities, the next thing to confirm is their network communication capabilities. Only each other to understand each other’s network, so it is possible to find a communication link.

Let’s assume an example:

The two sides of the communication we call A and B; A is the calling party and B is the called party. C is a relay server, also called a relay server or a TURN server.

One is that we connect directly through P2P, that is, A->B, B ->A; The second is to transfer through the relay server, A->C, C->B, B->C, C->A. For WebRTC, the first one will be preferred. If the connection fails, the second one will be selected.

When WebRTC establishes communications, ICE Candidate is used.

Candidate: The protocol, IP address, and port used by the WebRTC to communicate with the remote terminal are classified into the following three types:

  • The host type is the IP address and port of the local Intranet.
  • SRFLX type, that is, the IP address and port of the external network after NAT mapping.
  • The relay type is the IP address and port of the relay server.

ICE: The process of obtaining candidates of various types, namely, collecting all candidates of host type on the native machine, collecting candidates of SRFLX type by STUN protocol, and collecting candidates of relay type by TURN protocol, It integrates STUN and TURN.

NAT (Net Address Transport)

role

When we access public network resources, NAT first translates the internal IP address to the external IP address, and then sends the request to the server to be accessed. The server returns the result to the public IP address + port, and then forwards the request to the internal host through NAT

background

  • To solve the problem of insufficient IPv4 addresses, multiple hosts share a public IP address and use the internal IP address to communicate with each other. This method greatly alleviates the problem of insufficient IPv4 addresses
  • To solve the security problem, the host is hidden on the internal network, and the outside is blocked by NAT. In this way, it is difficult for hackers to obtain the IP address and port of the host in the public network, so as to achieve the role of protection

species

There are three types of NAT: full cone NAT, IP restricted cone NAT, port restricted cone NAT, and symmetric NAT. Take a perfect cone as an example:

The fully tapered feature is that when a host accesses host B on the extranet through NAT, a “hole” is punched in the NAT, and all hosts that know of this “hole” can communicate with listeners on the Intranet host through it. The so-called “hole” is to establish an extranet mapping table on NAT, in the form of the following structure:

{Internal IP, internal port, mapped Extranet IP, mapped Extranet port}Copy the code

If host and HOST B successfully punch holes, and A and C obtain the external IP address and port of host A and C from HOST B, then A and C can send data to the IP address and port, and the applications listening on the corresponding port on host host can receive the data sent by them.

IP limit cone has one more IP limit than full cone. Only registered IP addresses can pass. The port limit cone has more port limits than the IP limit cone. The major difference between the symmetric NAT and the port-restricted NAT is that if the host accesses A, it will open A new hole in the NAT instead of using the hole opened in the previous access to B. That is, a symmetric NAT uses a different port for each connection and even changes the IP address, while a port-restricted NAT uses the same port for multiple connections. This is the biggest difference between a symmetric NAT and a port-restricted NAT.

STUN (Session Traversal Utilities for NAT)

When you want to access resources on the public network, NAT first translates the internal address of the host into an external address, and then sends the request to the server to access. After processing, the server returns the result to the public IP address and port of the host, and then forwards the result to the internal host through NAT. What STUN does is to obtain the IP address and port of the public network.

Three of the four NAT types are fully cone, IP restricted, and port restricted, but most large companies use the symmetric type, which means the router will only accept connections established by nodes you have previously connected to. Such networks require the TURN technology.

TURN (Traversal Using Relays around NAT)

TURN is an extension of STUN/RFC5389 that allows NAT to bypass the limitations of symmetric NAT.

Media negotiation + network negotiation data switching channel

signaling

Signaling is the process of sending control information between two devices to determine communication protocols, channels, media codecs and formats, and data transmission methods, as well as any required routing information.

Signaling server

Two clients negotiate media information (SDP) and network information (candidate) exchange, need a server, we call signaling server, real applications will also handle room management, personnel in and out of the room and other functions.

WebRTC of actual combat

The flow chart

./ / create the PeerConnection
let peer = new PeerConnection(iceServer);
peer.addStream(this.localStream); . .createOffer(account, peer) {
  / / create the offer
  peer.createOffer({
      offerToRecieveAudio: 1.offerToReceiveVideo: 1
  }).then((desc) = > {
      // setLocalDescription after the offer is successfully created
      peer.setLocalDescription(desc, () = > {
          socket.emit('offer', {'sdp': peer.localDescription, roomid: this.$route.params.roomid, account: account}) }) }) }, ... . socket.on('offer'.v= > {
  // The remote PC saves the offer
  this.peerList[v.account] && this.peerList[v.account].setRemoteDescription(v.sdp, () = > {
    // Create an answer on the remote PC
      this.peerList[v.account].createAnswer().then((desc) = > {
        // Save the answer on the remote end
          this.peerList[v.account].setLocalDescription(desc, () = > {
              socket.emit('answer', {'sdp': this.peerList[v.account].localDescription, roomid: this.$route.params.roomid, account: v.account})
          })
      })
  }, (err) = > {
      console.log(err) }) }) ... .// Save the answer on the local PC
socket.on('answer'.v= > {
    this.peerList[v.account] && this.peerList[v.account].setRemoteDescription(v.sdp, function() {}, (err) = > {
        console.log(err) }) }) ... . socket.on('__ice_candidate'.v= > {
    if(v.candidate){
      // addIceCandidate
        this.peerList[v.account] && this.peerList[v.account].addIceCandidate(v.candidate).catch(() = >{})}})... . peer.onaddstream =function (event) {
  // Take the data stream and render it
  console.log(event.stream) }; . . peer.onicecandidate =(event) = > {
    if(event.candidate) {
        socket.emit('__ice_candidate', {
            'candidate': event.candidate, roomid: this.$route.params.roomid, account: v.account
        })
    }
}
...
Copy the code

WebRTC data statistics

To see all the statistics, enter the Chrome :// webrtc-internals URL in Chrome

Matters needing attention

If you only need to set up a STUN/ Turn server for local LAN testing, but if you want to access the Internet, then you need a cloud host & HTTPS domain name, you can refer to this article: WebRTC Signaling control and STUN/ Turn Server setup

Code sample

Github.com/xieyanxieya…

Refer to the article

STUN/TURN service setup

WebRTC

WebRTC API

Geek time: Building audio and video systems from 0 to 1

Search for “ikoofe” on wechat, follow the public account “KooFE Front-end Team”, and publish front-end technical articles irregularly.