Introduction to related apis

In previous chapters, weBRTC-related important knowledge has been introduced, including the involved network protocol, session description protocol, how to do network penetration, etc. The rest is WebRTC API.

WebRTC communication related API is very many, mainly complete the following functions:

  1. Signaling exchange
  2. Communication candidate address exchange
  3. Audio and video capture
  4. Audio and video sending and receiving

There are too many related apis, in order to avoid too long length, part of the paper uses pseudo-code to explain. The detailed code can be found at the end of the article or on Github. If you have any questions, please leave a comment.

Signaling exchange

Signaling exchange is the key link in WebRTC communication. The information exchanged includes codec, network protocol, candidate address, etc. WebRTC does not specify how signaling is exchanged, leaving it up to the application to decide, for example, WebSocket.

The sender pseudocode is as follows:

const pc = new RTCPeerConnection(iceConfig);
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
sendToPeerViaSignalingServer(SIGNALING_OFFER, offer); // The sender sends a signaling message
Copy the code

The receiver pseudocode is as follows:

const pc = new RTCPeerConnection(iceConfig);
await pc.setRemoteDescription(offer);
const answer = await pc.createAnswer();
await pc.setLocalDescription(answer);
sendToPeerViaSignalingServer(SIGNALING_ANSWER, answer); // The receiver sends a signaling message
Copy the code

Candidate address exchange service

When session descriptions are set locally and media streams are added, the ICE framework starts collecting candidate addresses. After collecting candidate IP addresses, the two sides need to exchange candidate IP addresses and learn the appropriate candidate IP address pairs.

For the exchange of candidate addresses, the aforementioned signaling service is also adopted, with the pseudo-code as follows:

// Set the local session description
const localPeer = new RTCPeerConnection(iceConfig);
const offer = await pc.createOffer();
await localPeer.setLocalDescription(offer);

// Collect audio and video locally
const localVideo = document.getElementById('local-video');
const mediaStream = await navigator.mediaDevices.getUserMedia({ 
	video: true.audio: true
});
localVideo.srcObject = mediaStream;

// Add audio and video streams
mediaStream.getTracks().forEach(track= > {
	localPeer.addTrack(track, mediaStream);
});

// Exchange candidate addresses
localPeer.onicecandidate = function(evt) {
	if(evt.candidate) { sendToPeerViaSignalingServer(SIGNALING_CANDIDATE, evt.candidate); }}Copy the code

Audio and video capture

You can use the getUserMedia interface provided by the browser to collect local audio and video files.

const localVideo = document.getElementById('local-video');
const mediaStream = await navigator.mediaDevices.getUserMedia({ 
	video: true.audio: true
});
localVideo.srcObject = mediaStream;
Copy the code

Audio and video sending and receiving

Add the collected audio and video track through addTrack and send it to the remote end.

mediaStream.getTracks().forEach(track= > {
	localPeer.addTrack(track, mediaStream);
});
Copy the code

The remote end can monitor the arrival of audio and video by listening ontrack and play it.

remotePeer.ontrack = function(evt) {
    const remoteVideo = document.getElementById('remote-video');
    remoteVideo.srcObject = evt.streams[0];
}
Copy the code

The complete code

It consists of two parts: client code and server code.

1. Client code

const socket = io.connect('http://localhost:3000');

const CLIENT_RTC_EVENT = 'CLIENT_RTC_EVENT';
const SERVER_RTC_EVENT = 'SERVER_RTC_EVENT';

const CLIENT_USER_EVENT = 'CLIENT_USER_EVENT';
const SERVER_USER_EVENT = 'SERVER_USER_EVENT';

const CLIENT_USER_EVENT_LOGIN = 'CLIENT_USER_EVENT_LOGIN'; / / login

const SERVER_USER_EVENT_UPDATE_USERS = 'SERVER_USER_EVENT_UPDATE_USERS';

const SIGNALING_OFFER = 'SIGNALING_OFFER';
const SIGNALING_ANSWER = 'SIGNALING_ANSWER';
const SIGNALING_CANDIDATE = 'SIGNALING_CANDIDATE';

let remoteUser = ' '; // Remote user
let localUser = ' '; // Local login user

function log(msg) {
    console.log(`[client] ${msg}`);
}

socket.on('connect'.function() {
    log('ws connect.');
});

socket.on('connect_error'.function() {
    log('ws connect_error.');
});

socket.on('error'.function(errorMessage) {
    log('ws error, ' + errorMessage);
});

socket.on(SERVER_USER_EVENT, function(msg) {
    const type = msg.type;
    const payload = msg.payload;

    switch(type) {
        case SERVER_USER_EVENT_UPDATE_USERS:
            updateUserList(payload);
            break;
    }
    log(` [${SERVER_USER_EVENT}] [${type}].The ${JSON.stringify(msg)}`);
});

socket.on(SERVER_RTC_EVENT, function(msg) {
    const {type} = msg;

    switch(type) {
        case SIGNALING_OFFER:
            handleReceiveOffer(msg);
            break;
        case SIGNALING_ANSWER:
            handleReceiveAnswer(msg);
            break;
        case SIGNALING_CANDIDATE:
            handleReceiveCandidate(msg);
            break; }});async function handleReceiveOffer(msg) {
    log(`receive remote description from ${msg.payload.from}`);
    
    // Set the remote description
    const remoteDescription = new RTCSessionDescription(msg.payload.sdp);
    remoteUser = msg.payload.from;
    createPeerConnection();
    await pc.setRemoteDescription(remoteDescription); // TODO error handling

    // Local audio and video capture
    const localVideo = document.getElementById('local-video');
    const mediaStream = await navigator.mediaDevices.getUserMedia({ video: true.audio: true });
    localVideo.srcObject = mediaStream;
    mediaStream.getTracks().forEach(track= > {
        pc.addTrack(track, mediaStream);
        // pc.addTransceiver(track, {streams: [mediaStream]}); // This will also work
    });
    // pc.addStream(mediaStream); // This works for now, but the interface will be deprecated later

    const answer = await pc.createAnswer(); // TODO error handling
    await pc.setLocalDescription(answer);
    sendRTCEvent({
        type: SIGNALING_ANSWER,
        payload: {
            sdp: answer,
            from: localUser,
            target: remoteUser
        }
    });
}

async function handleReceiveAnswer(msg) {
    log(`receive remote answer from ${msg.payload.from}`);
    
    const remoteDescription = new RTCSessionDescription(msg.payload.sdp);
    remoteUser = msg.payload.from;

    await pc.setRemoteDescription(remoteDescription); // TODO error handling
}

async function handleReceiveCandidate(msg){
    log(`receive candidate from ${msg.payload.from}`);
    await pc.addIceCandidate(msg.payload.candidate); // TODO error handling
}

@param {Object} MSG: {type: 'xx', payload: {}} */
function sendUserEvent(msg) {
    socket.emit(CLIENT_USER_EVENT, JSON.stringify(msg));
}

@param {Object} MSG format, such as {type: 'xx', payload: {}} */
function sendRTCEvent(msg) {
    socket.emit(CLIENT_RTC_EVENT, JSON.stringify(msg));
}

let pc = null;

/** * Invite users to join video chat * 1. Start video collection locally * 2. Exchange signaling */
async function startVideoTalk() {
    // Enable local video
    const localVideo = document.getElementById('local-video');
    const mediaStream = await navigator.mediaDevices.getUserMedia({
        video: true.audio: true
    });
    localVideo.srcObject = mediaStream;

    / / create the peerConnection
    createPeerConnection();

    // Add the media stream to webrTC's audio and video transceiver
    mediaStream.getTracks().forEach(track= > {
        pc.addTrack(track, mediaStream);
        // pc.addTransceiver(track, {streams: [mediaStream]});
    });
    // pc.addStream(mediaStream); // This works for now, but the interface will be deprecated later
}

function createPeerConnection() {
    const iceConfig = {"iceServers": [{url: 'stun:stun.ekiga.net'},
        {url: 'turn:turnserver.com'.username: 'user'.credential: 'pass'}}; pc =new RTCPeerConnection(iceConfig);

    pc.onnegotiationneeded = onnegotiationneeded;
    pc.onicecandidate = onicecandidate;
    pc.onicegatheringstatechange = onicegatheringstatechange;
    pc.oniceconnectionstatechange = oniceconnectionstatechange;
    pc.onsignalingstatechange = onsignalingstatechange;
    pc.ontrack = ontrack;
    
    return pc;
}

async function onnegotiationneeded() {
    log(`onnegotiationneeded.`);

    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer); // TODO error handling

    sendRTCEvent({
        type: SIGNALING_OFFER,
        payload: {
            from: localUser,
            target: remoteUser,
            sdp: pc.localDescription // TODO:}}); }function onicecandidate(evt) {
    if (evt.candidate) {
        log(`onicecandidate.`);

        sendRTCEvent({
            type: SIGNALING_CANDIDATE,            
            payload: {
                from: localUser,
                target: remoteUser,
                candidate: evt.candidate } }); }}function onicegatheringstatechange(evt) {
    log(`onicegatheringstatechange, pc.iceGatheringState is ${pc.iceGatheringState}. `);
}

function oniceconnectionstatechange(evt) {
    log(`oniceconnectionstatechange, pc.iceConnectionState is ${pc.iceConnectionState}. `);
}

function onsignalingstatechange(evt) {
    log(`onsignalingstatechange, pc.signalingstate is ${pc.signalingstate}. `);
}

// Call pc.addTrack(track, mediaStream), the onTrack of the remote peer will trigger twice
Evt.streams [0] actually points to the same mediaStream reference when triggered twice
/ / this behavior is a little strange, making issue has referred to https://github.com/meetecho/janus-gateway/issues/1313
let stream;
function ontrack(evt) {
    // if (! stream) {
    // stream = evt.streams[0];
    // } else {
    // console.log(`${stream === evt.streams[0]}`); // this is true
    // }
    log(`ontrack.`);
    const remoteVideo = document.getElementById('remote-video');
    remoteVideo.srcObject = evt.streams[0];
}

// Click on the user list
async function handleUserClick(evt) {
    const target = evt.target;
    const userName = target.getAttribute('data-name').trim();

    if (userName === localUser) {
        alert('Can't have a video conversation with myself');
        return;
    }

    log(`online user selected: ${userName}`);

    remoteUser = userName;
    await startVideoTalk(remoteUser);
}

/** * update user list * @param {Array} users list, such as [{name: 'xiao Ming ', name:' Xiao Xiao '}] */
function updateUserList(users) {
    const fragment = document.createDocumentFragment();
    const userList = document.getElementById('login-users');
    userList.innerHTML = ' ';

    users.forEach(user= > {
        const li = document.createElement('li');
        li.innerHTML = user.userName;
        li.setAttribute('data-name', user.userName);
        li.addEventListener('click', handleUserClick);
        fragment.appendChild(li);
    });    
    
    userList.appendChild(fragment);
}

@param {String} loginName User name */
function login(loginName) {
    localUser = loginName;
    sendUserEvent({
        type: CLIENT_USER_EVENT_LOGIN,
        payload: {
            loginName: loginName
        }
    });
}

// Process the login
function handleLogin(evt) {
    let loginName = document.getElementById('login-name').value.trim();
    if (loginName === ' ') {
        alert('Username is empty! ');
        return;
    }
    login(loginName);
}

function init() {
    document.getElementById('login-btn').addEventListener('click', handleLogin);
}

init();
Copy the code

2. Server-side code

// Add the WS service
const io = require('socket.io')(server);
let connectionList = [];

const CLIENT_RTC_EVENT = 'CLIENT_RTC_EVENT';
const SERVER_RTC_EVENT = 'SERVER_RTC_EVENT';

const CLIENT_USER_EVENT = 'CLIENT_USER_EVENT';
const SERVER_USER_EVENT = 'SERVER_USER_EVENT';

const CLIENT_USER_EVENT_LOGIN = 'CLIENT_USER_EVENT_LOGIN';
const SERVER_USER_EVENT_UPDATE_USERS = 'SERVER_USER_EVENT_UPDATE_USERS';

function getOnlineUser() {
  return connectionList
  .filter(item= > {
    returnitem.userName ! = =' ';
  })
  .map(item= > {
    return {
      userName: item.userName
    };
  });
}

function setUserName(connection, userName) {
  connectionList.forEach(item= > {
    if(item.connection.id === connection.id) { item.userName = userName; }}); }function updateUsers(connection) {
  connection.emit(SERVER_USER_EVENT, { type: SERVER_USER_EVENT_UPDATE_USERS, payload: getOnlineUser()});  
}

io.on('connection'.function (connection) {

  connectionList.push({
    connection: connection,
    userName: ' '
  });
  
  // Push the list of online users
  // connection.emit(SERVER_USER_EVENT, { type: SERVER_USER_EVENT_UPDATE_USERS, payload: getOnlineUser()});
  updateUsers(connection);

  connection.on(CLIENT_USER_EVENT, function(jsonString) {
    const msg = JSON.parse(jsonString);
    const {type, payload} = msg;

    if (type === CLIENT_USER_EVENT_LOGIN) {
      setUserName(connection, payload.loginName);
      connectionList.forEach(item= > {
        // item.connection.emit(SERVER_USER_EVENT, { type: SERVER_USER_EVENT_UPDATE_USERS, payload: getOnlineUser()});updateUsers(item.connection); }); }}); connection.on(CLIENT_RTC_EVENT,function(jsonString) {
    const msg = JSON.parse(jsonString);
    const {payload} = msg;
    const target = payload.target;

    const targetConn = connectionList.find(item= > {
      return item.userName === target;
    });
    if(targetConn) { targetConn.connection.emit(SERVER_RTC_EVENT, msg); }}); connection.on('disconnect'.function () {
    connectionList = connectionList.filter(item= > {
      returnitem.connection.id ! == connection.id; }); connectionList.forEach(item= > {
      // item.connection.emit(SERVER_USER_EVENT, { type: SERVER_USER_EVENT_UPDATE_USERS, payload: getOnlineUser()});
      updateUsers(item.connection);
    });    
  });
});
Copy the code

Write in the back

WebRTC has a large number of apis, because WebRTC itself is complex. Over time, some oF the apis (including some protocol details) of WebRTC have changed or been abandoned. There are also complications of backward compatibility, such as adding local video to the stream after it is captured. AddStream or addTrack or addTransceiver could be used, and for example, the session description version could be migrated from plan-B to unified plan.

It is suggested to masturbate the code to deepen understanding.

A link to the

2019.08.02 – video – talk – using webrtc

Developer.mozilla.org/en-US/docs/…

onremotestream called twice for each remote stream