preface

Recently, I summarized what I learned in the real-time communication section of building two systems.

This is a series of articles, temporarily mainly conceived in four parts

  • Websocket (1) Websocket protocol
  • (2) Distributed Websocket cluster
  • IO source code and WS-wrapper

The body of the

This is a simple schematic of the toy I’m building. The real-time communication part is taken out as a Websocket node, forming a simple distributed system. Then through Redis Pub/Sub to do the communication between Websocket clusters and the communication between Websocket nodes and Restful API nodes (for example, after users call Restful API to publish articles, notify Websocket to push new messages to the front end with red dots).

The Left Ear Rat of distributed Systems: The Difficulties of distributed systems from amazon’s experience

This article focuses on distributed Websocket clustering solutions, with a working Demo at the end.

Websocket Distributed cluster solution

In this blog post, we eventually want to build a Websocket cluster for real-time communication with clients, such as chat rooms. Sure, we could build a Websocket server with a simple demo and have all the clients connect to the machine, but what if the chat room has a lot of interaction? For example, I went to douyu to check the request, and it can be seen from the name that it has established a WS connection called Danmuproxy.douyu.com, as shown in the picture below.

Live barrage

So, if I only use one server, how do I support a chat room that might have 100,000 people joining at the same time? Obviously we need A solution, such as balancing the traffic load among different servers and providing A communication mechanism for each server to synchronize messages (otherwise user A can connect to server A, and user B can face server B, and neither of them can receive messages).

In fact, the name of the proxy in danmuproxy.douyu.com that Douyu is connected to can be inferred from its name that douyu is also distributing traffic.

Websocket cluster

As mentioned in the previous section, these Websocket servers need to share information (as do servers that need to do Session sharing, of course) because of the different load balancing from normal HTTP servers. This meant that the client’s interaction with the Websocket server was stateful, and we needed to keep each client’s connection data in memory. When we want to implement distribution, we need to share this information across machines, so we need a Publish/Subscribe broker. Here’s an example.

Suppose we now use Redis as our solution, then we now have three Websocket servers WS1, WS2, and WS3. Then three users were connected to each server. One of the users on the WS1 machine sends a message to the chat room. In your Websocket server logic, you first store the message to the database for persistence (such as history messages). This message is then pushed to The chat room’s channel based on something like The channelId (The implementation of Websocket’s channel will be covered in more detail in The next article), let’s say The channelId is called “The☆World”.

Now you have the data safely stored in DB, and you publish an event to your Pub/Sub Broker (Redis Channel) to notify other interested parties (other Websockets or API servers, etc.). Therefore, the other two servers WS2 and WS3 were interested in this part, so they also listened to this Redis channel through the script. They would be notified, and then each server would request query to the DB for updates and emit messages to the corresponding channel on the Websocket.

This is what you can see, a scale-out Websocket cluster using Pub/Sub Brooker.

You can also see the advantages of clustering, high scalability and high availability.

implementation

This implementation uses my high-configuration Ali Cloud domestic server, a relatively low Ali Cloud 9 yuan student server and redis on the high-configuration server.

Nginx load balancing

First configure Nginx to do load balancing, the following is my configuration, just a Demo did not do WSS related.

Server-side implementation

The code is all on Github.

The Demo code is also very short

const WebSocket = require('ws');
const publicIp = require('public-ip');
const uuidv1 = require('uuid/v1');
const redis = require("redis");
const config = require('./config');

const sub = redis.createClient(config.DB.REDIS_PORT, config.DB.REDIS_HOST);
const pub = redis.createClient(config.DB.REDIS_PORT, config.DB.REDIS_HOST);

if (config.DB.REDIS_PASSWORD) {
  sub.auth(config.DB.REDIS_PASSWORD);
	pub.auth(config.DB.REDIS_PASSWORD);
}

const wss = new WebSocket.Server({ port: 2333 });

const ip2name = {
  '47.94.233.234': 'The High distribution stronghold of The King of Liang'.'115.28.68.89': '9 servers of The King of Liang',}let sockets = {};

wss.on('connection'.function connection(ws) {
  const uuid = uuidv1();
  ws.uuid = uuid;
  sockets[uuid] = ws;
  ws.on('message'.function incoming(message) {
    // Publish messages to other servers
    pub.publish('channel'.`${ws.uuid}>${message}`);
    console.log(`publish to channel:  ${ws.uuid}>${message}`)
    // Broadcast to the local socket
    wss.clients.forEach(function each(client) {
      if(client ! == ws && client.readyState === WebSocket.OPEN) { client.send(` from${ws.from || '????? '}The user${ws.uuid}Sent:${message}`); }}); }); publicIp.v4().then(ip= > {
    console.log(ip);
    ws.from = ip2name[ip] ? ip2name[ip] : 'unknown';
    ws.send('The server you are connecting to is${ws.from}`);
  });
});


// Listen for messages sent by other servers
sub.on('message'.function(channel, message) {
  console.log(`channel ${channel}.${message}`)
	if (channel == 'channel')
	{
    var messageArr = message.split('>');
    var uuid = messageArr[0]
		var wsFrom = sockets[uuid];
		var content = messageArr[1];

    // If the socket is not the local server
    if(! wsFrom) { wss.clients.forEach(function each(client) {
        client.send('Users from other servers${uuid}Sent:${content}`); }); }}}); sub.subscribe('channel');
Copy the code

The effect

You can try this in the console with the following code, the server may shut down later.

var socket = new WebSocket('ws://websocket-demo.lwio.me');

// Listen for messages
socket.addEventListener('message'.function (event) {
    console.log('Received', event.data);
});

// socket.send('keke')
Copy the code

Afterword.

Update on April 1, mom, today aliyun has been calling the police, you see me Redis directly exposed to the public network to give me a wave is it. Studied studied, bowed to the principal big man.

References:

Scaling WebSockets