We’ll discuss the controversial topic of “stealing” computing resources from web visitors’ browsers. There’s a lot of talk about how to use browsers to mine digital currency, but I’m not going to get into that. I’m just going to talk about an efficient way to use computing resources.

Web browsers are increasingly capable of executing code. The growth of JavaScript, the advent of WebAssembly, improved GPU access, and the evolution of threading models have combined to make browsers as powerful as computers. With the rise of the browser digital currency miner, I’ve also been thinking about how to consolidate the world’s computing resources into a single entity — a supercomputer made up of web visitors’ browsers.

Like a normal computer cluster, all of the supercomputer’s computing nodes work together to solve a problem. But unlike normal computer clusters, these compute nodes are temporary (as site visitors come and go), and they can’t previously talk to each other (no cross-site requests).

Here’s an example that comes to mind:

On the right is the supercomputer that controls the server. On the left is a browser to a website, one of the nodes in the supercomputer, which also shows its CPU metrics.

The supercomputer’s problem is to find the original value of a given hash. As can be seen from the figure, a total of 23 nodes participated in the calculation, and 380,204,032 hashes were calculated and compared, among which visitors from the United States contributed 50% of the processing capacity.

Here we mainly use WebSocket technology to establish persistent connections between servers and compute nodes. These connections are used to coordinate the behavior of nodes, making them cooperative entities. WebSocket can transfer code and collaboration messages, making everything possible.

The advent of Websockets dramatically changed the behavior of Web clients. The client connects to the web site, executes pre-defined JavaScript, and after the WebSocket connection is established, it can execute any other JavaScript script.

On the right is the supercomputer control server and on the left is the Web client receiving dynamic instructions.

If an App uses a WebView, JavaScript can run directly into the App. That is, code transmitted through WebSocket can skip the WebView and go directly to the App.

On the right is the supercomputer control server and on the left is the Web App receiving instructions. As you can see, the instructions penetrate directly into the App layer.

The rest is nothing new. Apps can get instructions through THE C&C protocol (Botnet Command and Control), web pages can get JavaScript scripts dynamically after the first load, and WebSocket is really dynamic (unlike the polling pull mode of Ajax). It runs across multiple browsers and devices and has full access to the runtime environment.

Therefore, we can completely transfer instruction code to compute node through WebSocket, of course, can also be used to pass messages, to achieve distributed coordination.

Six years ago, I developed a distributed password cracker (http://ben.akrin.com/?p=1424) based on OpenMPI called Crackzor. Password cracking is a very typical distributed problem. It is very simple to guess the password by permutation and combination of characters. I rewrote Crackzor in JavaScript and used WebSocket instead of OpenMPI.

Still, every distributed problem is different, and Crackzor isn’t the solution to every problem. Crackzor’s magic lies in its flexibility. It breaks up a character array space into chunks and apportions those chunks among compute nodes. Given the problem to solve and the start and end of the iteration, the nodes are ready to work without having to provide them with character permutations, so there is no network bandwidth bottleneck.

JavaScript uses a single-threaded model by default, with code delivered to the client via WebSocket, using only one CPU core by default. Most of today’s computer cpus are multi-core, so we have to find a way to use them all.

And here comes a savior: the Web Worker. HTML5 provides this feature, which greatly simplifies multithreading implementations. However, we still need to solve a problem. The Web Worker documentation tells us to load a script file from a file, but our code is transferred over WebSocket and stored in memory, so we can’t execute the code directly by specifying a script file.

We solve this problem by wrapping the code as a Blob object:

var worker_code = 'alert( "this code is threaded on the nodes" ); ' window.URL = window.URL || window.webkitURL; var blob; try { blob = new Blob([worker_code], {type: 'application/javascript'}); } catch (e) { window.BlobBuilder = window.BlobBuilder || window.WebKitBlobBuilder || window.MozBlobBuilder; blob = new BlobBuilder(); blob.append(worker_code); blob = blob.getBlob(); } workers.push( new Worker(URL.createObjectURL(blob)) ) ;Copy the code

The WebSocket server does most of the subsequent coordination, keeping track of node access and exit, as well as whether a node is performing computing tasks, and assigning tasks to nodes as they become available.

The server needs to be up and running all the time, handling connections from nodes. However, the supercomputer may need to solve a different problem every day. To do this, I wrote a function that reads the file and executes the code in the file. This function is called by a process.

function eval_code_from_file() { if( ! file_exists("/tmp/code") ) { console.log( "error: file /tmp/code does not exist" ) ; } else { var code = read_file( "/tmp/code" ) ; code = code.toString() ; eval( code ) ; } } process.on('SIGUSR1', eval_code_from_file.bind() );Copy the code

With this function, next time I can kill the old process and use the new process to load the new code. This is thanks to the flexibility of JavaScript, which allows you to run any code at any time, as long as you have full access to the runtime environment.

Assigning tasks to nodes is also as simple as having the client register a callback function when connecting to the server and execute code in the callback function. Such as:

Client:

var WebSocket_client=io.connect("http://WebSocket_server.domain.com"); 
WebSocket_client.on( "eval_callback",function(data){data=atob(data),eval(data)}.bind() ) ;Copy the code

Server side:

client_socket.emit( "eval_callback", new Buffer("alert('this code will run on the client');" ).toString("base64") ) ;Copy the code

So far:

  1. All temporary nodes (the Web browser of the site user) connect to the WebSocket server

  2. The process signals the WebSocket server to execute the new code

  3. The new code contains new problems that nodes need to solve

  4. Does the new code tell the WebSocket server how to coordinate nodes

  5. Once a node has solved the problem, move on to the next problem

Now we know how to build a supercomputer using a Web browser. For reasons such as readability, security, and complexity, I don’t want to make all of my code public. However, if anyone is interested, please feel free to contact me. I’d be happy to share more ideas.

When splitting tasks, they should not be too large. Because nodes are temporary, interruptions are highly likely if the task becomes too heavy. Most Web browsers refuse to execute or abort code that consumes too much resources, and small tasks can be completed in seconds without interruption.

Using JavaScript to implement MD5:

https://gist.github.com/josedaniel/951664

Record the average time taken by the nodes to solve the problem and exclude the slow nodes so as not to affect the overall performance of the “supercomputer”.

Original link: http://ben.akrin.com/?p=5997