Fu Jinting, front end Group of the fourth Profit Center of Wedoctor.
Speak before the notice
WebRTC, short for Web Real-time Communication, is an API that enables Web browsers to conduct real-time voice or video conversations. It became open source on June 1, 2011 and was incorporated into the World Wide Web Consortium’s W3C recommendation with support from Google, Mozilla, and Opera.
First, it is both an API and a protocol.
Second, it’s an API for the browser to make audio and video calls, and it actually has screen sharing capabilities.
Finally, it is now a W3C standard and compatible with major browser vendors.
But if we want to use WebrTC well, we need to understand WebSocket first. Websockets are familiar with social chat, multiplayer games, collaborative editing, video conferencing, location-based applications (maps), and other scenarios that require high real time. We commonly used wechat, QQ, some live broadcast software and so on are also based on Websocket to achieve message and signaling forwarding. So you see here and maybe you’re hesitant in signaling here, so keep going.
Webrtc is a P2P technology, what is P2P? It’s end-to-end, where your audio and video streams go directly from one end to the other without passing through the server.
Without going through the server, that is to say, if the server crashes in the process of going through, can the call continue?
Yes! However, before sending audio and video streams, a P2P connection must be established. Before establishing a connection, the server must forward signaling, which is the identification of both ends of a call.
And if you want to use weBRTC call, you have to transfer signaling, establish a connection. The best way to establish a connection is to use Websocket for signaling forwarding. As we all know, websocket is a channel, at all ends of this channel, can receive any end of the message flow, including the sender himself.
Why not go through the server to directly get the other party’s video and audio stream? Because of the establishment of P2P channel, this P2P in the transmission of signaling has been through, transmission of video and audio streams when what server ah. At this time, there must be some partners expressed doubt, audio and video stream can not pass the server? Yes, I lied to you, do after the server, but just online server forwarding, if we are two or more local end of the same local area network (wlan) webrtc audio streaming forward, do not need to transfer server, but online is likely to need, also may not need, this also involves the concept of a hole.
We might hear a lot of cool words, such as holes, Intranet penetration, NAT traversal, all sorts of fancy things, but it’s actually pretty easy to understand. As we all know, two hosts on two different networks cannot communicate with each other directly, but need to access the public network or their respective gateways. Tunneling, Intranet penetration, and NAT traversal actually mean the same thing, which is to use UDP to connect two hosts on different networks without connecting to the public network. Those of you who have played peanut shells will understand the concept of Intranet penetration.
Local development, two hosts together with a LAN, do not need Intranet penetration, you can communicate directly.
Online development, if can STUN hole success, also do not need to transfer the server. However, there is a probability of failure to make a hole, why? Because there is no public network, no benefits to the operator but the cost of communication, it must be limited. The probability of success in foreign holes is 70%, but less than 50% in China.
Therefore, in order to prevent the failure of hole drilling, we use TURN transfer server to forward streaming media data for a final guarantee. In addition, there is another way for reverse connection, which can also help us to achieve P2P establishment. Its principle is that one party must be connected to the public network, but it also has limitations.
The COTURN relay server consists of STUN and TURN. STUN helps us make holes and TURN helps us forward streaming media data.
## Connect process
All apis below are up to date as of 2021.12.06
Media negotiations begin
A and B connect to each other through websocket through the back-end service and enter the same channel. Both A and B can receive messages and signaling from each other. A Create PeerConnect Creates an RTCPeerConnection instance (webrTC connection instance). A AddStream (getUserMedia) gets the local audio stream and displays it locally. (Open the call first display their own video screen). A calls the CreateOffer method of the WebrTC connection instance to create an offer(SDP format description) that contains A's own media information and codec information. A calls the setLocalDescription method of the WebrTC connection instance to set the offer to A local description and sends A bind Request to the Sturn/Turn relay server to collect candidates. A sends the offer, and the signaling server (the back-end service) forwards it to B. B receives the offer, Create PeerConnect creates an RTCPeerConnection instance (WebrTC connection instance). B AddStream (getUserMedia) gets the local audio stream and displays it locally. (Display your own video screen). B calls the setRemoteDescription method of the WebrTC connection instance to set the offer to its own remote description. B calls the CreateAnswer method of the webrTC connection instance to create an answer(SDP format description), which contains A's own media information and codec information. B calls the setLocalDescription method of the WebrTC connection instance to set the answer to the local description and sends a bind Request to the Sturn/Turn relay server to collect candidates. B sends the Answer, and the signaling server (back-end service) forwards the answer to A. After receiving the answer, A calls the setRemoteDescription method of the WebrTC connection instance and sets the answer as its remote description.Copy the codeMedia negotiation completed
The Sturn /turn server is no longer receiving A bind Request at this time, and responds to A onIceCandidate(candidate) containing A public IP address and port number. A sends the candidate to the signaling server (back-end service), which forwards the candidate to B. B calls the addIceCandidate method of the WebrTC connection instance to add candidate A(A's public IP address and port) to B's list of candidates. The Sturn /turn server no longer receives a bind Request at this time, and responds to B onIceCandidate(candidate), which contains B's public IP address and port. B sends the candidate to the signaling server (back-end service), which forwards the candidate to A. A calls the addIceCandidate method of the WebrTC connection instance to add candidate B(B's public IP address and port) to A's list of candidates. At this point, both A and B have obtained the communication candidate (public IP address and port) of the other party. Conduct P2P to select the optimal line. B calls the webrTC connection instance onAddStream to play A's video and audio stream. A calls the WebrTC connection instance onAddStream to play B's video and audio stream (the call starts and the other party's video is available). If Intranet penetration fails at this point. The Sturn/Turn server will help forward the audio stream because it already has a list of candidates (all IP addresses and ports).Copy the codeDeepen the understanding
Local development, the same LAN, P2P connection establishment does not need coturn transfer server, because there is no need to hole, Websocket channel enables the host to be in the same pipeline, send offer and answer to each other, P2P can be established. Online environment, different Lans, websocket channel make hosts in the same pipe, send offer and answer to each other, want to establish P2P (sturn must be used for NAT traversal), but the operator truncated (failed to establish P2P), use turn to transfer audio stream.Copy the code## I have a question
What is a P2P connection?
P2P streaming media technology, this technology is mainly to solve the server load is too large (the traditional is the server to forward audio stream), multi end does not pass through the server to forward audio stream, but in the network P2P channel. Download resources, live broadcast, audio and video calls, desktop sharing and so on are largely based on this technology.Copy the codeWhat is the nature of an offer? What is SDP?
Offer is a signaling name, and creating an offer is essentially creating an SDP.Copy the codeTo show you the essence of SDP, it is its own media information and codec information
sdp: {
type: 'offer',
sdp: 'v=0\r\n' +
'o=- 890410854023526853 2 IN IP4 127.0.0.1\r\n' +
's=-\r\n' +
't=0 0\r\n' +
'a=group:BUNDLE audio video\r\n' +
'a=extmap-allow-mixed\r\n' +
'a=msid-semantic: WMS EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc\r\n' +
'm=audio 9 UDP/TLS/RTP/SAVPF 111 63 103 104 9 0 8 106 105 13 110 112 113 126\r\n' +
'c=IN IP4 0.0.0.0\r\n' +
'a=rtcp:9 IN IP4 0.0.0.0\r\n' +
'a=ice-ufrag:+b1T\r\n' +
'a=ice-pwd:2MMQo86tKV27zgrrsMhvhGqK\r\n' +
'a=ice-options:trickle\r\n' +
'a=fingerprint:sha-256 A0:F2:F7:C0:BE:1B:8C:EF:6C:42:03:D7:6E:6B:B2:DC:AE:57:F1:F3:DD:67:86:F6:11:F5:5B:44:49:D5:44:9A\r\n' +
'a=setup:actpass\r\n' +
'a=mid:audio\r\n' +
'a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\n' +
'a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\n' +
'a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\n' +
'a=sendrecv\r\n' +
'a=rtcp-mux\r\n' +
'a=rtpmap:111 opus/48000/2\r\n' +
'a=rtcp-fb:111 transport-cc\r\n' +
'a=fmtp:111 minptime=10;useinbandfec=1\r\n' +
'a=rtpmap:63 red/48000/2\r\n' +
'a=fmtp:63 111/111\r\n' +
'a=rtpmap:103 ISAC/16000\r\n' +
'a=rtpmap:104 ISAC/32000\r\n' +
'a=rtpmap:9 G722/8000\r\n' +
'a=rtpmap:0 PCMU/8000\r\n' +
'a=rtpmap:8 PCMA/8000\r\n' +
'a=rtpmap:106 CN/32000\r\n' +
'a=rtpmap:105 CN/16000\r\n' +
'a=rtpmap:13 CN/8000\r\n' +
'a=rtpmap:110 telephone-event/48000\r\n' +
'a=rtpmap:112 telephone-event/32000\r\n' +
'a=rtpmap:113 telephone-event/16000\r\n' +
'a=rtpmap:126 telephone-event/8000\r\n' +
'a=ssrc:1511813723 cname:P0KGpA3OHyfIh1hw\r\n' +
'a=ssrc:1511813723 msid:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc a3daa1c2-1f35-426f-a242-2a0286202c04\r\n' +
'a=ssrc:1511813723 mslabel:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc\r\n' +
'a=ssrc:1511813723 label:a3daa1c2-1f35-426f-a242-2a0286202c04\r\n' +
'm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116\r\n' +
'c=IN IP4 0.0.0.0\r\n' +
'a=rtcp:9 IN IP4 0.0.0.0\r\n' +
'a=ice-ufrag:+b1T\r\n' +
'a=ice-pwd:2MMQo86tKV27zgrrsMhvhGqK\r\n' +
'a=ice-options:trickle\r\n' +
'a=fingerprint:sha-256 A0:F2:F7:C0:BE:1B:8C:EF:6C:42:03:D7:6E:6B:B2:DC:AE:57:F1:F3:DD:67:86:F6:11:F5:5B:44:49:D5:44:9A\r\n' +
'a=setup:actpass\r\n' +
'a=mid:video\r\n' +
'a=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\n' +
'a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\n' +
'a=extmap:13 urn:3gpp:video-orientation\r\n' +
'a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\n' +
'a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\n' +
'a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\n' +
'a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\n' +
'a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\n' +
'a=sendrecv\r\n' +
'a=rtcp-mux\r\n' +
'a=rtcp-rsize\r\n' +
'a=rtpmap:96 VP8/90000\r\n' +
'a=rtcp-fb:96 goog-remb\r\n' +
'a=rtcp-fb:96 transport-cc\r\n' +
'a=rtcp-fb:96 ccm fir\r\n' +
'a=rtcp-fb:96 nack\r\n' +
'a=rtcp-fb:96 nack pli\r\n' +
'a=rtpmap:97 rtx/90000\r\n' +
'a=fmtp:97 apt=96\r\n' +
'a=rtpmap:98 VP9/90000\r\n' +
'a=rtcp-fb:98 goog-remb\r\n' +
'a=rtcp-fb:98 transport-cc\r\n' +
'a=rtcp-fb:98 ccm fir\r\n' +
'a=rtcp-fb:98 nack\r\n' +
'a=rtcp-fb:98 nack pli\r\n' +
'a=fmtp:98 profile-id=0\r\n' +
'a=rtpmap:99 rtx/90000\r\n' +
'a=fmtp:99 apt=98\r\n' +
'a=rtpmap:100 VP9/90000\r\n' +
'a=rtcp-fb:100 goog-remb\r\n' +
'a=rtcp-fb:100 transport-cc\r\n' +
'a=rtcp-fb:100 ccm fir\r\n' +
'a=rtcp-fb:100 nack\r\n' +
'a=rtcp-fb:100 nack pli\r\n' +
'a=fmtp:100 profile-id=2\r\n' +
'a=rtpmap:101 rtx/90000\r\n' +
'a=fmtp:101 apt=100\r\n' +
'a=rtpmap:102 H264/90000\r\n' +
'a=rtcp-fb:102 goog-remb\r\n' +
'a=rtcp-fb:102 transport-cc\r\n' +
'a=rtcp-fb:102 ccm fir\r\n' +
'a=rtcp-fb:102 nack\r\n' +
'a=rtcp-fb:102 nack pli\r\n' +
'a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\r\n' +
'a=rtpmap:121 rtx/90000\r\n' +
'a=fmtp:121 apt=102\r\n' +
'a=rtpmap:127 H264/90000\r\n' +
'a=rtcp-fb:127 goog-remb\r\n' +
'a=rtcp-fb:127 transport-cc\r\n' +
'a=rtcp-fb:127 ccm fir\r\n' +
'a=rtcp-fb:127 nack\r\n' +
'a=rtcp-fb:127 nack pli\r\n' +
'a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\r\n' +
'a=rtpmap:120 rtx/90000\r\n' +
'a=fmtp:120 apt=127\r\n' +
'a=rtpmap:125 H264/90000\r\n' +
'a=rtcp-fb:125 goog-remb\r\n' +
'a=rtcp-fb:125 transport-cc\r\n' +
'a=rtcp-fb:125 ccm fir\r\n' +
'a=rtcp-fb:125 nack\r\n' +
'a=rtcp-fb:125 nack pli\r\n' +
'a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\n' +
'a=rtpmap:107 rtx/90000\r\n' +
'a=fmtp:107 apt=125\r\n' +
'a=rtpmap:108 H264/90000\r\n' +
'a=rtcp-fb:108 goog-remb\r\n' +
'a=rtcp-fb:108 transport-cc\r\n' +
'a=rtcp-fb:108 ccm fir\r\n' +
'a=rtcp-fb:108 nack\r\n' +
'a=rtcp-fb:108 nack pli\r\n' +
'a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\r\n' +
'a=rtpmap:109 rtx/90000\r\n' +
'a=fmtp:109 apt=108\r\n' +
'a=rtpmap:35 AV1/90000\r\n' +
'a=rtcp-fb:35 goog-remb\r\n' +
'a=rtcp-fb:35 transport-cc\r\n' +
'a=rtcp-fb:35 ccm fir\r\n' +
'a=rtcp-fb:35 nack\r\n' +
'a=rtcp-fb:35 nack pli\r\n' +
'a=rtpmap:36 rtx/90000\r\n' +
'a=fmtp:36 apt=35\r\n' +
'a=rtpmap:124 H264/90000\r\n' +
'a=rtcp-fb:124 goog-remb\r\n' +
'a=rtcp-fb:124 transport-cc\r\n' +
'a=rtcp-fb:124 ccm fir\r\n' +
'a=rtcp-fb:124 nack\r\n' +
'a=rtcp-fb:124 nack pli\r\n' +
'a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f\r\n' +
'a=rtpmap:119 rtx/90000\r\n' +
'a=fmtp:119 apt=124\r\n' +
'a=rtpmap:123 H264/90000\r\n' +
'a=rtcp-fb:123 goog-remb\r\n' +
'a=rtcp-fb:123 transport-cc\r\n' +
'a=rtcp-fb:123 ccm fir\r\n' +
'a=rtcp-fb:123 nack\r\n' +
'a=rtcp-fb:123 nack pli\r\n' +
'a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f\r\n' +
'a=rtpmap:118 rtx/90000\r\n' +
'a=fmtp:118 apt=123\r\n' +
'a=rtpmap:114 red/90000\r\n' +
'a=rtpmap:115 rtx/90000\r\n' +
'a=fmtp:115 apt=114\r\n' +
'a=rtpmap:116 ulpfec/90000\r\n' +
'a=ssrc-group:FID 1741155232 1898443615\r\n' +
'a=ssrc:1741155232 cname:P0KGpA3OHyfIh1hw\r\n' +
'a=ssrc:1741155232 msid:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc fb34f344-fbe3-45e9-969d-af4d9fb5bdc4\r\n' +
'a=ssrc:1741155232 mslabel:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc\r\n' +
'a=ssrc:1741155232 label:fb34f344-fbe3-45e9-969d-af4d9fb5bdc4\r\n' +
'a=ssrc:1898443615 cname:P0KGpA3OHyfIh1hw\r\n' +
'a=ssrc:1898443615 msid:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc fb34f344-fbe3-45e9-969d-af4d9fb5bdc4\r\n' +
'a=ssrc:1898443615 mslabel:EHsXxPKkpBfFwyGLbTIFHt4eXe6smVEHN9Yc\r\n' +
'a=ssrc:1898443615 label:fb34f344-fbe3-45e9-969d-af4d9fb5bdc4\r\n'
}
Copy the codeWhat does the setLocalDescription method do? What is the setRemoteDescription?
SetLocalDescription is to make its own webrTC instance clear of its own media information and codec information. SetRemoteDescription is to make the other webrTC instance clear, my media information and codec information.Copy the codeOne offer and one answer. We both know each other’s media information, encoding and decoding information, so that we can negotiate with each other on how we should decode and render your video and audio stream.
Send a request to the Sturn /turn relay server to collect candidates.
The process is a bit complicated. For details, you can read this article: WebRTC TURN Protocol introduction and TurnServer practice.
purpose
Understand weBRTC audio and video collection, desktop collection;
Understand the whole link establishment process of WebSocket and WebrTC;
1V1 text transmission, video call, voice call, screen sharing;
Video call, voice call, screen recording in the process of screen sharing screenshots, voice recording, screen recording, screenshots, voice recording, screen recording online play and download;
Deploy the above functions online;
The process to build
Here, we will draw a basic flow chart of the audio and video creation process.
Basic flow chart
For these signaling, we use WebSocket for forwarding, and here the question is, why not HTTP?
First of all, the demo we’re implementing already has the ability to send plain text messages, using websockets. (Long or short polling too old, poor performance)
Second, the first point can be ignored, HTTP request will be back to the original route, A request to the server, never to B.
But if we are going to use webSocket to forward signaling, it is important to understand that all ends on the same pipe receive this message. So, for the above flowchart, all the little arrows are actually two-way.
At this point, we can control the direction of the push message in the Node service, or in the client side, I choose AB side to control.
Secondly, we developed locally, if using one computer, two browser form, Websocket text message is ok. But audio and video calls don’t work because both the transmission channel and the audio and video equipment (mics, speakers, etc.) conflict. So, we can solve this problem using two computers on the same LAN. In addition, due to webrTC security restrictions, HTTPS (whether online or local) and domain name must be used. We can solve this problem by configuring HTTPS and domain name online and setting browser locally ignoring HTTPS and configuring host file mapping.
Next, we will use Vue and NodeJS to make the demo the fastest and easiest to implement.
Cut the crap and let’s rip!
Open to tear
Show some code
socket-io
Here I use socket. IO, a third party package, to quickly initiate messages and forward signaling. You are advised to use vue-socket. IO to send and receive messages and signaling messages in components.
The websocket of socket-IO is used to establish connections, receive messages, and put the received signaling into vuEX.
Async connectSocket({commit, state, dispatch}) {// LAN let socket = io.connect("http://172.28.74.16:3004"); Online / / / / let the socket = IO. Connect (" https://www.codeting.top:3004 "); socket.on("connect", () => { commit(types.SET_SOCKET, socket); dispatch("handleChatData", state.friends); }); On ("friendMessage", (res) => {if (res) {commit(types.set_message, res); } else {console.log(" problem "); }}); socket.on("apply", (res) => { if (! state.isCalling) { let text = ""; If (res.webRtcType === "video") {text = "video"; } MessageBox(' your friend ${res.username} request with you ${text}, do you agree? ', "confirmButtonText ", {confirmButtonText:" agree ", cancelButtonText: "reject ", type: "warning", } ) .then(() => { const friendInfo = { account: res.userName, _id: res.Id, }; commit(types.SET_FRIENDINFO, friendInfo); commit(types.SET_ISCALLING, true); commit(types.SET_WEBRTCSTATE, "reply"); }) .catch(() => { console.log("9. Refuse to answer "); }); } else { } }); socket.on("reply", (res) => { if (res) { localStorage.setItem("nowRoomId", res.roomId); commit(types.SET_REPLYISAGREE, true); }}); socket.on("1v1ICEA", (res) => { if (res && state.role === "receiver") { commit(types.SET_ECE, res); }}); socket.on("1v1ICEB", (res) => { if (res && state.role === "caller") { commit(types.SET_ECE, res); }}); socket.on("1v1OFFER", (res) => { if (res) { commit(types.SET_ISOFFER, true); commit(types.SET_OFFER, res); }}); socket.on("1v1ANSWER", (res) => { if (res) { commit(types.SET_ISANSWER, true); commit(types.SET_ANSWER, res); }}); },Copy the codeSimilarly, we use socket-io in the Node service
io.on("connect", function (socket) { socket.on('friendMessage', async function (res) { const roomId = res.userId > res.friendId ? res.userId + res.friendId : res.friendId + res.userId io.to(roomId).emit('friendMessage', res) }); socket.on('apply', async function (res) { io.to(res.roomId).emit('apply', res) }); socket.on('reply', data => { io.to(data.roomId).emit('reply', data) }) socket.on('1v1ICEA', data => { console.log('1v1ICEA', data) io.to(data.roomId).emit('1v1ICEA', data) }) socket.on('1v1ICEB', data => { io.to(data.roomId).emit('1v1ICEB', data) }) socket.on('1v1OFFER', Data => {// forward Offer io.to(data.roomid). Emit ('1v1offer', data)}) socket. On ('1v1ANSWER', Data => {// forward answer io.to(data.roomid).emit('1v1answer', data)})};Copy the codeAudio and video capture
The code is similar for video, audio, and screen sharing. So, for example, video capture.
const constraints = {
audio: {
noiseSuppression: true,
echoCancellation: true,
},
video: true,
};
this.localstream = navigator.mediaDevices.getUserMedia(constraints);
let video = document.querySelector("#rtcA");
video.srcObject = this.localstream;
Copy the codeBy using getUserMedia, we can capture a dual-track media stream for audio and video. We pass in a parameter named Constraints, which can be configured to capture audio or video.
Assign values to the video TAB from the collected dynamic media stream, and our own image will be displayed on the web page.
Similarly, if you are collecting audio, you simply set audio in the constraints parameter to false.
Computer screen capture, just replace the getUserMedia API with getDisplayMedia.
Transfer process
After the video initiator collects the media stream, it needs to send the Apply signaling to the receiver. This signaling asks the receiver whether to accept a video call.
If it is connected, the receiving end will collect its own audio and video dual-track media stream, initialize PeerConnection, and exile the media into this pipeline. If the ICE candidate information is collected, it will be sent to the other end, and its consent signaling reply will be sent to the video initiator.
If the connection is rejected, the receiver sends a rejection message to the video initiator.
The receiving end receives the rejection and stops the collection of video and audio streams.
At this time, the receiving end receives the connection, initializes peerConnection, and banishes its own media into this pipe. If the ICE candidate information is collected, it will send it to the other party. Create an offer(including SDP), place the offer locally, and send the offer to the video receiver.
The video receiver receives the offer, puts it on its remote end, creates an answer, puts the answer on the local end, and sends it to the video initiator.
The video initiator receives the answer and stores the answer on the remote end.
props: { visible: Boolean, friendInfo: Object, webRtcType: String, }, data() { return { showDialog: false, localstream: {}, peer: null, isToPeer: false, iceServers: { iceServers: [ { url: Turn: "www.codeting.top:3478", for the domain name the credential: / / XXXXX "XXXXX", / username/password: "xx", / / account},], sdpSemantics: "plan-b", }, bufferFriend: [], bufferMy: [], mediaRecorder: {}, startRecordVideo: true, }; }, watch: { visible(val) { this.handleVisible(val); if (! val) { this.$store.commit("chat/SET_ISCALLING", false); }}, "$store.state.chat.replyIsAgree"(v) { if (v && this.$store.state.chat.role === "caller") { let roomId = getLocalStorage("nowRoomId").nowRoomId; this.initPeer({ roomId, webRtcType: "video" }); this.createOffer({ roomId, webRtcType: "video" }); }}, "$store.state.chat.isEce"(v) { if (v && this.$store.state.chat.role === "receiver") { if (this.$store.state.chat.ece) { this.onIce(this.$store.state.chat.ece); } } if (v && this.$store.state.chat.role === "caller") { if (this.$store.state.chat.ece) { this.onIce(this.$store.state.chat.ece); }}}, "$store.state.chat.isOffer"(v) { if (v && this.$store.state.chat.role === "receiver") { if (this.$store.state.chat.offer) { this.onOffer(this.$store.state.chat.offer); }}}, "$store.state.chat.isAnswer"(v) { if (v && this.$store.state.chat.role === "caller") { if (this.$store.state.chat.answer) { this.onAnswer(this.$store.state.chat.answer); } } }, }, methods: { handleVisible(val) { this.showDialog = val; this.$emit("update:visible", val); }, async apply() { let constraints = null; if (this.webRtcType === "video") { constraints = { audio: { noiseSuppression: true, echoCancellation: true, }, video: true, }; } else { constraints = { audio: true, video: false, }; } this.localstream = await navigator.mediaDevices.getUserMedia(constraints); let video = document.querySelector("#rtcA"); video.srcObject = this.localstream; const userId = getCookie(); const friendId = this.friendInfo._id; let roomId = userId > friendId ? userId + friendId : friendId + userId; this.$store.state.chat.socket.emit("apply", { webRtcType: this.webRtcType, roomId: roomId, userName: getLocalStorage("account").account, id: userId, }); }, reply(roomId) { this.$store.state.chat.socket.emit("reply", { roomId, webRtcType: this.webRtcType, }); }, async createP2P(data) { await this.createMedia(data); }, async createMedia(data) { try { let constraints = null; if (this.webRtcType === "video") { constraints = { audio: { noiseSuppression: true, echoCancellation: true, }, video: true, }; } else { constraints = { audio: true, video: false, }; } this.localstream = await navigator.mediaDevices.getUserMedia( constraints ); let video = document.querySelector("#rtcA"); video.srcObject = this.localstream; } catch (e) { console.log("getUserMedia: ", e); } await this.initPeer(data); }, initPeer(data) { let PeerConnection = window.RTCPeerConnection || window.mozRTCPeerConnection || window.webkitRTCPeerConnection; this.peer = new PeerConnection(this.iceServers); this.peer.addStream(this.localstream); window.streamMy = this.localstream; this.peer.onicecandidate = (event) => { if (event.candidate && this.$store.state.chat.role === "caller") { this.$store.state.chat.socket.emit("1v1ICEA", { ... data, sdp: event.candidate, }); } if (event.candidate && this.$store.state.chat.role === "receiver") { this.$store.state.chat.socket.emit("1v1ICEB", {... data, sdp: event.candidate, }); }}; this.peer.onaddstream = (event) => { window.streamFriend = event.stream; this.isToPeer = true; let video = document.querySelector("#rtcB"); video.srcObject = event.stream; }; }, async createOffer(data) { try { let offer = await this.peer.createOffer({ offerToReceiveAudio: 1, offerToReceiveVideo: 1, }); await this.peer.setLocalDescription(offer); this.$store.state.chat.socket.emit("1v1offer", { ... data, sdp: offer, }); } catch (e) { console.log("createOffer: ", e); } }, async onIce(data) { try { await this.peer.addIceCandidate(data.sdp); } catch (e) { console.log("onAnswer: ", e); } }, async onOffer(data) { try { await this.peer.setRemoteDescription(data.sdp); let answer = await this.peer.createAnswer(); await this.peer.setLocalDescription(answer); this.$store.state.chat.socket.emit("1v1answer", { ... data, sdp: answer, }); } catch (e) { console.log("onOffer: ", e); } }, async onAnswer(data) { try { await this.peer.setRemoteDescription(data.sdp); } catch (e) { console.log("onAnswer: ", e); }}},Copy the codePlay each other’s screen
At this point, both the receiver and the originator listen for ICE candidate information and, if collected, send it to the other side. Once monitored, the dynamic media stream of the other party is assigned to B and played out.
if (event.candidate && this.$store.state.chat.role === "receiver") { this.$store.state.chat.socket.emit("1v1ICEB", { ... data, sdp: event.candidate, }); }Copy the codeScreenshots and audio and video recording screen
Screenshot: We can use Canvas to capture pictures on the Web level by using the relevant method getContext(“2d”). DrawImage.
let picture = document.querySelector("#picture");
let rtcA = document.querySelector("#rtcA");
picture.getContext("2d").drawImage(rtcA, 0, 0, 200, 120);
Copy the codeAudio/Video/screen recording: Use MediaRecorder to save our media stream or the other party’s media stream to an array.
let that = this; let options = { mineType: "video/webm; code=vp8", }; if (! MediaRecorder.isTypeSupported(options.mineType)) { console.error(`${options.mineType}is not supported`); } try { this.mediaRecorder = new MediaRecorder(window.streamFriend, options); } catch (error) {console.error(error, "failed "); return; } / / trigger events when data is valid. This mediaRecorder. Ondataavailable = function (e) {if (e && e.d ata && e.d ata. Size > 0) { that.bufferFriend.push(e.data); }}; this.mediaRecorder.start(10);Copy the codePlay audio and video recording screen
Just assign the saved static media stream to the video TAB
let recplayer = document.querySelector("#recplayer");
let blob = new Blob(this.bufferFriend, { type: "video/webm" });
recplayer.src = window.URL.createObjectURL(blob);
recplayer.srcObject = null;
recplayer.controls = true;
recplayer.play();
Copy the codeDownload audio, video and screen recording
In the same way, we can download audio and video streams.
download(videoName) { var blob = new Blob(this.bufferFriend, { type: "video/webm" }); var url = window.URL.createObjectURL(blob); var downloadLink = document.createElement("a"); downloadLink.href = url; downloadLink.style.display = "none"; downloadLink.download = `${videoName}.webm`; // Can be named in any format such as mp4/webm, etc., where webM format can be used to play video with a browser downloadlink.click (); document.body.removeChild(downloadLink) },Copy the codeLocal effects
Desktop sharing
Video call
online
There are two important conditions for deploying weBRTC: domain name and HTTPS, which need to be configured.
Our Node service is not only HTTPS + domain name, websocket also needs the more secure WSS protocol, we need to configure WSS for our websocket.
As mentioned earlier, the reason why local development can be successful and effective is that the internal network communicates directly with each other rather than through the public network, so there is no penetration of the internal network.
If we want to do this online, we have to configure the Coturn relay server. For details about how to configure the centos kernel and ubuntu kernel, see this section.
defects
After development and launch, the following problems can be found.
Environment, equipment, signal overflows, algorithmic incompatibilities, acoustics and line echoes, network congestion, and delays caused by erratic packet transmission rates.
We can reduce noise echo and delay by incorporating algorithms and improving the quality of the hardware.
For noise, the noiseSuppression: True can be set when collecting audio, which reduces the noise of some environments and equipment.
For echo, set echoCancellation: True when collecting audio, which can remove the echo.
Algorithms, devices and networks will take care of the rest.
In this aspect of exploration, I will stop here, you can continue to study how WebRTC transmission is to ensure the quality of audio and video service, study how mature applications are to solve these three difficulties.
The special effects
There are many special effects you can use during a video call. Face beautification, stickers and so on.
However, on the web side of WeBRTC, the field of video effects is very latent. The reason for this is js performance issues.
The easy way to do this is to use a Canvas canvas and apply a filter to our video image without changing the media stream per se. The transfer to the remote end is still non-efficacious. Of course, we can control the remote video effects through websocket, but since the video stream does not change, if the other party downloads the video stream, it will still play without effects.
Another option is as follows, I won’t go into details here, but you can think about how to do it (simple effects and stickers below).
Many people video
We need to create n-1 PeerConnection connections because we are sharing video with n-1 people, everyone. However, it will involve the issue of who takes the initiative to send offer. We can have new members send offers to other n-1 members, or we can have n-1 members send offers to new members. Here we can generate PeerConnection and offer in a traversal manner. Of course, many people call to see your server on the top of the live.
Here we unwittingly use a well-known multi-terminal communication scheme called Mesh, where two people communicate to form a Mesh. In addition to the communication scheme of Mesh, there is also MCU. The MCU scheme mainly mixes audio and video streams of all terminals in the same room and sends them to each terminal. In this way, the pressure of the server is actually very huge. In addition, there is an SFU communication scheme. After receiving audio and video streams from a terminal, the transfer server forwards them to other terminals.
conclusion
After the above series of understanding, thinking, construction, development, deployment and so on, we have some preliminary understanding and understanding of WEBRTC. For this aspect of research and exploration we should continue to go deep. To satisfy our curiosity and thirst for knowledge, improve our technology in this field and enrich our overall knowledge system, why not?
Finally, all the above content comes from the data, personal experiments and personal summary. I hope you can correct any mistakes in the article.
reference
www.html5rocks.com/en/tutorial…
zhuanlan.zhihu.com/p/7102543
zhuanlan.zhihu.com/p/71025431
zhuanlan.zhihu.com/p/26796476
okokokokok
The annual nuggets account voting began ~ give us a chance to debut micro medical university front end technology ~ welcome to vote together, forward ~
Go to the online diagnosis and treatment platform of Wedoctor Internet hospital, make a quick consultation, and find a top three doctor for you in 3 minutes.
