Websocket compatibility

DEMO

The basic structure

Client:

//./client/index.html var socket = new WebSocket("ws://localhost:3000/test/123"); var timmer; socket.onopen = function (evt) { console.log('open',evt); timmer = setInterval(function(){ socket.send(new Date().getSeconds()); }}, 1000); Onmessage = function (evt) {console.log(' MSG ',evt); }; Onerror = function (evt) {// failed to connect console.log('e',evt); clearInterval(timmer); };Copy the code

Server:

//./server/index.js
const Koa = require('koa');
const route = require('koa-route');
const websockify = require('koa-websocket');

const app = websockify(new Koa());

// Regular middleware
// Note it's app.ws.use and not app.use
app.ws.use(function(ctx, next) {
    // return `next` to pass the context (ctx) on to the next ws middleware
    return next(ctx);
});
  
// Using routes
app.ws.use(route.all('/test/:id', function (ctx) {
    // `ctx` is the regular koa context created from the `ws` onConnection `socket.upgradeReq` object.
    // the websocket is added to the context on `ctx.websocket`.
    ctx.websocket.send('Hello World');
    ctx.websocket.on('message', function(message) {
      // do something with the message from client
          console.log(message);
    });
}));

app.listen(3000);
Copy the code

Add a heartbeat mechanism

Note that ping and Pong are not used at either end, but both. We can say this, Pong: Are you there? Ping: I'm here. When the other party pong you must immediately respond to the ping. If the other person is already talking to you, you should put off pong. Don't say, "Are you there?" [Eye roll]Copy the code

Client:

/ /. / client/index. HTML var socket = new WebSocket (ws: / / 127.0.0.1:3000 / test / 123 "); var socketLive = false; var heartCheckTimer; var timmer; function heartCheck(){ clearTimeout(heartCheckTimer); heartCheckTimer = setTimeout(function(){ if(socketLive){ socketLive = false; socket.send('pong'); heartCheck(); }else{console.log(' The other party is down, ready to restart '); }}, 3000); } socket.onopen = function (evt) { console.log('open',evt.data); socketLive = true; var s = 0 timmer = setInterval(function(){ socket.send(++s); },1000) heartCheck(); }; Onmessage = function (evt) {console.log(' MSG ',evt.data); socketLive = true; heartCheck(); if(evt.data == 'pong'){ socket.send('ping'); }}; Onerror = function (evt) {// failed to connect console.log('e',evt); clearInterval(timmer); }; Onclose = function (evt) {// failed to connect console.log('close',evt); clearInterval(timmer); };Copy the code

Server:

//./server/index.js var heartCheckTimer; function heartCheck(socket){ clearTimeout(heartCheckTimer); heartCheckTimer = setTimeout(function(){ if(socket.socketLive){ socket.socketLive = false; socket.send('pong'); heartCheck(socket); }else{console.log(' The other party is down, ready to restart '); }}, 3000); } app.ws.use(route.all('/test/:id', function (ctx) { ctx.websocket.socketLive = true; heartCheck(ctx.websocket); setInterval(function(){ ctx.websocket.send('Hello World'); },1000) ctx.websocket.on('message', function(message) { console.log(message); ctx.websocket.socketLive = true; if(message == 'pong'){ ctx.websocket.send('ping'); } heartCheck(ctx.websocket); }); ctx.websocket.on('close', function(message) { console.log(message); console.log("close"); }); ctx.websocket.on('error', function(message) { console.log(message); console.log("error"); }); }));Copy the code

Pay attention to

If the domain of your service is HTTPS, the WebSocket protocol used must also be WSS, not WS. If one side of the browser (client) or server suddenly cuts off the network, the other side cannot be heard through the event. So the heartbeat mechanism is used not just to keep the connection going, but to make sure the other person is alive.Copy the code

Why do I write notice here? Because at the beginning of my local test, I found that the close event of the other party would be triggered either by closing the browser, refreshing, closing the TAB page, or calling the close function on the browser side, or exiting the process, throwing an exception, or calling the close function on the server side. And I started the Websocket service under the NodeJS server environment, and found that no matter how long it took, even if no data was sent, the connection still existed (for example, setting a timer to send a message after a few hours was successful). This led me to the “we don’t need a heartbeat mechanism” illusion (Illusion 1: It won’t disconnect; Illusion 2: Disconnect can be reconnected by listening for the close event.

I later tested it on my phone and found that when the phone’s WIFI was turned off, the server didn’t trigger any events, meaning he didn’t know when the browser was disconnected. And after introducing nginx as a reverse proxy, if there is no data interaction, it will automatically disconnect after a while.

The body of the

When does websocket disconnect

Again: if an exception is too sudden, such as a network outage, one party has no time to notify the other party to shut down, the other party will not trigger the shutdown event.Copy the code

Client Cause

For example, disconnect the Internet, close the browser, close the TAB, refresh the page and so on.

Server Cause

The server directly exits the process, throws an error, or restarts the process.

If the server is abnormally disconnected and restarted, the client needs to re-connect to the server using a new WebSocket

On the client side:

The first failed connection will prompt:

index.html:10 WebSocket connection to 'ws://xxxx' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

This triggers the onError and onClose callbacks of the WebSocket instanceCopy the code

If the server closes after a successful connection and the client executes socket.send, the Chrome console displays an error message (other browsers do not) : WebSocket is already in CLOSING or CLOSED state.

And point the error stack to my code: socket.send

The console will tell you that WebSocket is closing or is already closed. Attention! Try catch, window.onerror, window.addEventListener(‘error’,function(){}))

When the server unexpectedly fails or shuts down (calling websocket.terminate()), browsers behave differently:

The browser	onerror	onclose
chrome	x	Square root
firefox	The server reports an error √	Server error or shutdown √
IE10	Square root	Square root

In Chrome, onError is not raised, but onclose. In Firefox, onError and onClose are triggered when the server reports an error, and onClose is triggered when the server closes. In IE10, when the server reports an Error, both onError and onclose functions will be triggered, and the message WebSocket Error: Network Error 12030 is displayed, indicating that the connection with the server is terminated unexpectedly.Copy the code

So between onError and onClose, it’s a good choice to just use onClose;

Why do we need a heartbeat mechanism?

On my Node service, I found that webSocket can still be connected even if there is no heartbeat. After collecting the Internet, I found that Nginx will actively disable WebSocket

We turn on the reverse proxy with NgniX:

server { listen 80; server_name node-test.com; location / { proxy_pass http://localhost:3000; proxy_next_upstream error timeout invalid_header http_500 http_502 http_503; proxy_set_header Host $host; Proxy_http_version 1.1; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto https; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Origin ""; }}Copy the code

Sure enough, the close event is triggered after a short time, so wait until it is closed to reconnect, better to use a heartbeat mechanism to keep it open.

summary

Websocket has no other means to listen for exceptions except for its own error and close status

On the browser side, using onclose to handle port conditions can effectively solve browser compatibility problems

Nginx will automatically port websocket connections at specified times

Websocket transfers files

Server sends files:

The server reads the Buffer using fs.readfilesync and sends it to the browser, which receives a Blob:

let content = fs.readFileSync(path.resolve('./test.txt'));
ctx.websocket.send(content);
Copy the code

Blob {size: 10, type: “”}

If you put Buffer data into an object and serialize it:

ctx.websocket.send(JSON.stringify({foo:content}));
Copy the code

Browser then print results: {” foo “: {” type” : “Buffer”, “data” :,98,99,100,13,10,65,66,67,68 [97]}}, for an array of data, if the data is a picture, how to convert the url? The two methods

The first method:

let url = URL.createObjectURL(new Blob([new Uint8Array(fileObj.foo.data)])); // This address is only a temporary reference addressCopy the code

Release urls when not in use: url.revokeobjecturl (url)

The second method:

let B = new Blob([new Uint8Array(fileObj.foo.data)]); var reader = new FileReader(); reader.readAsDataURL(B); reader.onload = function (e) { console.info(reader); let url = reader.result; / / base64 format}Copy the code

Socket. Send (new File([“a”.repeat(100)], “test.txt”)); The server receives a Buffer:

IsTrusted: true, wasClean: false, code: 1006, Reason: “”, type: “close”,… }

On my computer, RangeError: Max Payload Size exceeded will be triggered whenever the transmitted file is greater than or equal to 100M

That’s where shard uploads come in

shard

The core of sharding upload lies in the fact that Blob type data can be partitioned by slice. Then, how to ensure the correct transmission order after sharding? As mentioned above, when the server sends the fragmented data to the browser, it can put the added information in the empty object together with the Buffer data, and send it after serialization.

The browser cannot serialize the Blob type json.stringfy, but converts it to Base64.

On the server side, after receiving base64 format data, you need to do some processing, key code:

// base64Data = message.replace(/^data:\w+\/\w+; base64,/, ""); let dataBuffer = Buffer.from(base64Data, 'base64'); fs.writeFile("./server_save/a.png", dataBuffer, function(err) { console.log(err); });Copy the code

Instead of serialization, we now write additional information directly to the Blob or Buffer

On the browser side, we convert the additional information (ordinals) into bloBs, which are then combined with the sharded BLOBs

First, we agreed on the structure: serial number + delimiter + sharded data

We then send a delimiter to the server first. For example, I use — as the delimiter:

var splitB = new Blob(['---']);
socket.send(splitB);
Copy the code

Then I send the agreed data structure:

Var blobToSend = new Blob([123,splitB,sliceBlob]) var blobToSend = new Blob([123,splitB,sliceBlob]Copy the code

The back end receives data of type Buffer:

let splitIndex = message.indexOf(splitBuffer); // Message is the data passed from the front end; Let indexNum = message.slice(0,splitIndex); Let chunkData = message.slice(splitIndex + splitBuffer.length)// Get fragment dataCopy the code

All that remains is to agree on what to start, receive, and how to concatenate shards:

var TestB = a.target.files[0]; Var splitB = new Blob(['-- ']); socket.send(new Blob(['00000'])); // start, this is the convention to start data socket.send(splitB); // pass the separator let chunk = 5000; Let chunkNums = math.ceil (testb.size / 5000); // Let chunkNums = math.ceil (testb.size / 5000); For (var I = 0; i < chunkNums; i++){ let chunkData; if(i == chunkNums - 1){ chunkData = TestB.slice(i*chunk); Else {chunkData = testb. slice(I *chunk,(I +1)*chunk); // Last chunk}else{chunkData = testb. slice(I *chunk,(I +1)*chunk); } let TestB2 = new Blob([i,splitB,chunkData]); The console. The log (' transmission ', I); socket.send(TestB2); } socket.send(new Blob(['11111'])); // End, this is the agreed end dataCopy the code

In this process, if the amount of data is too large, the shard will break up into many pieces. Executing a for loop to send data blocks the JS process and UI process. So we need to do some processing

The introduction ofrequestIdleCallback(Compatibility is not good)

for(var i = 0; i < chunkNums; i++){ (function(i){ requestIdleCallback(function(){ let chunkData; if(i == chunkNums - 1){ chunkData = TestB.slice(i*chunk); Else {chunkData = testb. slice(I *chunk,(I +1)*chunk); // Last chunk}else{chunkData = testb. slice(I *chunk,(I +1)*chunk); } let TestB2 = new Blob([i,splitB,chunkData]); The console. The log (' transmission ', I); socket.send(TestB2); if(i == chunkNums - 1){ socket.send(new Blob(['11111'])); }})})(I)}Copy the code

There are two problems with this code:

RequestIdleCallback does not guarantee the order of execution, which is the end of the signalsocket.send(new Blob(['11111'])), not necessarily last (for example, the third shard data may be interrupted due to other time-consuming work, and the interrupted function is not executed until the other shard data is transmitted, as shown in the screenshot above).
No pause function, such as abnormal pause, active pause

Since the execution order problem can not be solved, so another way of thinking, ensure that all data transmission is completed before sending signals. This requires the introduction of a variable to determine whether the transfer is complete. Here I use promise + a variable Sign that keeps increasing:

let p = new Promise((resolve,reject)=>{ var Sign = 0 for(var i = 0; i < chunkNums; i++){ (function(i){ requestIdleCallback(function(){ let chunkData; if(i == chunkNums - 1){ chunkData = TestB.slice(i*chunk); Else {chunkData = testb. slice(I *chunk,(I +1)*chunk); // Last chunk}else{chunkData = testb. slice(I *chunk,(I +1)*chunk); } let TestB2 = new Blob([i,splitB,chunkData]); The console. The log (' transmission ', I); process_span.innerText = ++Sign; socket.send(TestB2); if(Sign == chunkNums){ resolve(1); } }) })(i) } }); p.then((r)=>{ console.log(r); socket.send(new Blob(['11111'])); // end console.log(" end "); });Copy the code

To do this, it seems that INSTEAD of using a for loop to send shard data, I need to use a recursive approach:

let p = new Promise((resolve,reject)=>{ var Sign = 0; window.uploadData = function uploadData(i){ let chunkData; if(i == chunkNums - 1){ chunkData = TestB.slice(i*chunk); Else {chunkData = testb. slice(I *chunk,(I +1)*chunk); // Last chunk}else{chunkData = testb. slice(I *chunk,(I +1)*chunk); } let TestB2 = new Blob([i,splitB,chunkData]); The console. The log (' transmission ', I); ++Sign requestAnimationFrame(()=>{ process_span.innerText = Sign; }); socket.send(TestB2); if(Sign == chunkNums){ resolve(1); return; } currentIndex = i; CancelIdleCallBackId cancelIdleCallBackId cancelIdleCallBackId cancelIdleCallBackId cancelIdleCallBackId cancelIdleCallBackId cancelIdleCallBackId CancelIdleCallBackId = requestIdleCallback(function(){uploadData(++ I)}); } uploadData(0); }) p.then((r)=>{ console.log(r); socket.send(new Blob(['11111'])); // end console.log(" end "); });Copy the code

Why requestAnimationFrame was introduced in the above code (incompatible with IE9 and below) : "Avoid changing the DOM in idle callbacks. By the time the idle callback is executed, the current frame has been drawn and all layout updates and calculations have been completed. If you make changes that affect the layout, you might force the browser to stop and recalculate, which, on the other hand, is unnecessary. If you need to change the DOM callback, it should use the Window. The requestAnimationFrame () to dispatch it."Copy the code

I find that requestIdleCallback pauses when Chrome switches to other tabs or shrinks the browser, which is not appropriate and I want you to pause it while it’s running in the background. Since requestIdleCallback is executed when the browser has time left after each frame is rendered, the original TAB should stop rendering after switching the notepad, so requestIdleCallback is also paused, as is requestAnimationFrame. (Firefox is slow, IE is not, and even older Versions of Edge don’t support requestIdleCallback.)

If I replace requestIdleCallback with setTimeout, setTimeout immediately becomes quite slow once I apply the same method to my browser, even if I set the second parameter to 0 (same as chrome, Firefox, IE).

How can I make setInterval also work when a TAB is inactive in Chrome? I decided to try Web Workers, which happens to be the method mentioned below.

To consider compatibility, Web Workers is still recommended (not compatible under IE9).

Core code:

//./client/index.html
var worker = new Worker('./js/work.js');

inputF.onchange = function(a){
    worker.postMessage(a.target.files[0]);
}
Copy the code

//./client/js/work.js var self = this; Var socket = new WebSocket (" ws: / / 127.0.0.1:3000 / test / 123 "); self.addEventListener('message', function (e) { var TestB = e.data; var splitB = new Blob(['---']); socket.send(splitB); // Start by passing the separator let chunk = 50000; Let chunkNums = math.ceil (testb.size/chunk); // Let chunkNums = math.ceil (testb.size/chunk); // Round up console.log(chunkNums); self.postMessage(JSON.stringify({ name:'total_span', value:chunkNums })); let p = new Promise((resolve,reject)=>{ var Sign = 0; self.uploadData = function uploadData(i){ let chunkData; if(i == chunkNums - 1){ chunkData = TestB.slice(i*chunk); Else {chunkData = testb. slice(I *chunk,(I +1)*chunk); // Last chunk}else{chunkData = testb. slice(I *chunk,(I +1)*chunk); } let TestB2 = new Blob([i,splitB,chunkData]); The console. The log (' transmission ', I); // process_span.innerText = ++Sign; ++Sign; requestAnimationFrame(()=>{ self.postMessage(JSON.stringify({ name:'process_span', value:Sign })); }); socket.send(TestB2); if(Sign == chunkNums){ resolve(1); return; } currentIndex = i; CancelIdleCallBackId = setTimeout(function(){//requestIdleCallback undefined uploadData(++ I)},1); } uploadData(0); }) p.then((r)=>{ console.log(r); socket.send(new Blob(['11111'])); // end console.log(" end "); }); }, false);Copy the code

Note: requestIdleCallback indicates in the Web worker that the maximum size of an undefined array is 2^32-1, so you can't split it into this many pieces (it usually doesn't).Copy the code

The above perfect solution to the background run + fragment upload function

If you want to refer to demo, my Github address is here

subsequent

A further feature is to find missing shards and re-request them
The fragment size is automatically adjusted according to the network condition

Websocket library for additional nodes

Socket. IO making 51.6 stars

If the client needs to use the library, the server needs to use the library accordingly. Currently, node.js, Java, C++, Swift, Dart are supported

Features (a brief translation of the official website to the features + explanation)

Reliability, can be used for: proxy, load balancing, personal firewall, antivirus software
Automatic reconnection support
Disconnection detection
Binary support (browser: ArrayBuffer, Blob; Node. Js: ArrayBuffer, Buffer)
Simple and convenient API
cross-browser
Multiplex support (i.e., create any object in the unit of namespace, easy to manage, but still use the same socket connection)
Support Room (support namespace-based grouping, for group chat, etc.)

The principle of maintaining long connections

The Engine.IO library starts with a long connection (LongPolling) and attempts to upgrade the connection (to webSocket). In order to see what the connection looks like, I will start with websocket = undefined; , you can then see the following screenshot on the console:

You can see that in addition to the first 5 requests, there will be 2 requests each time, one of which will be pending for a period of time. If the server has data to push over during this period, the request will be successful, otherwise after this period, the server will automatically request success and make two more requests. A quote is posted below:

LongPolling Browser/UA sends a Get request to the Web server. At this point, the Web server can do two things. First, if the server has new data that needs to be sent, it immediately sends the data back to Browser/UA. Send the Get request to the Web Server immediately; Second, if the server has no new data to send, unlike the Polling method, the server does not immediately send a response to Browser/UA. Instead, the server holds the request until new data arrives and then responds to it. Of course, if the server data is not updated for a long time, the Get request will time out after a period of time. Browser/UA receives the timeout message and immediately sends a new Get request to the server. Then repeat the process in turn. Although this method reduces network bandwidth and CPU utilization to some extent, it still has defects. For example, if the data update rate on the server is fast, the server must wait for Browser's next Get request after sending a packet to Browser. In order to pass the second updated packet to Browser, the fastest time Browser can display real-time data is 2×RTT (round trip time), and this should not be acceptable to users in the case of network congestion. In addition, due to the large amount of header data of HTTP packets (usually more than 400 bytes), but the data really needed by the server is very little (sometimes only about 10 bytes), such packets are periodically transmitted on the network, which inevitably wastes network bandwidth. From the above analysis, it would be nice to have a new network protocol in Browser that supports two-way communication between the client and server, and that has a less bulky header. WebSocket is to shoulder such a mission on the stage.Copy the code

Ws dead simple 15.2 stars

Express-ws source code can be found using THE WS library, koA-websocket is also

socket.io vs ws

Differences between socket. IO and websockets

The resources

Nodejs message push socket. IO versus WS

socket.io

Reprinted: WebSocket principle and server construction

Blob you don’t know

Talk about the binary family of JS: Blob, ArrayBuffer, and Buffer

requestIdleCallback

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Websocket practical study notes

Websocket compatibility

DEMO

The basic structure

Add a heartbeat mechanism

Pay attention to

The body of the

When does websocket disconnect

Client Cause

Server Cause

Why do we need a heartbeat mechanism?

summary

Websocket transfers files

shard

Instead of serialization, we now write additional information directly to the Blob or Buffer

subsequent

Websocket library for additional nodes

Socket. IO making 51.6 stars

The principle of maintaining long connections

Ws dead simple 15.2 stars

The resources

Websocket practical study notes

Websocket compatibility

DEMO

The basic structure

Add a heartbeat mechanism

Pay attention to

The body of the

When does websocket disconnect

Client Cause

Server Cause

Why do we need a heartbeat mechanism?

summary

Websocket transfers files

shard

Instead of serialization, we now write additional information directly to the Blob or Buffer

subsequent

Websocket library for additional nodes

Socket. IO making 51.6 stars

The principle of maintaining long connections

Ws dead simple 15.2 stars

The resources

Related Posts

Various problems you may encounter when installing Homebrew

DNS and TCP

Wechat small program to upload multiple local pictures to the server and picture preview