Protobuf is a format specification (RPC communication protocol) for serializing and deserializing objects. Other serialization methods include hessian, thrift, and so on. They are similar to JSON, but somewhat different. Here’s how to use Protobufs in Node.js and how to use it in Express.
Note that ❗️ RPC can be built directly over TCP or HTTP. For RPC, this is all the same, it’s all about stuffing the contents of the communication into the message.
In fact, HTTP is one of the most commonly used communication protocols to carry RPC. And we can transfer text protocols like XML and JSON over HTTP, as well as binary protocols like Protobuf and Thrift, and that’s not a problem. Common REST apis can be encapsulated as RPC. Of course, HTTP protocol is cumbersome, but it is better penetration, supporting facilities are complete, but also relatively simple and intuitive, or more welcome. The famous GRPC, for example, is transmitted over HTTP.
Protobuf is introduced
With a Protobuf, the biggest difference from unstructured formats such as JSON or XML is that you must define data types for protobufs, the most common way is to define.proto files. Here is an example of a.proto file that defines User as an object with two attributes: the name string; The age integer.
package userpackage;
syntax = "proto3";
message User {
string name = 1;
int32 age = 2;
}
Copy the codeThere are several packages that deal with Protobuf in NPM, the most downloaded of which is ProtobufJS. So, we use it to demonstrate this in this article. First, import protobufjs and call load() to tell Protobufjs to load the user.proto file:
const protobuf = require('protobufjs');
run().catch(err= > console.log(err));
async function run() {
const root = await protobuf.load('user.proto');
const User = root.lookupType('userpackage.User');
}
Copy the codeNow that Protobufjs knows about the User type, you can use User to validate or serialize objects. The user.verify () function is called to verify that the object’s name and age attribute types are correct:
console.log(User.verify({ name: 'test'.age: 2 })); // null
console.log(User.verify({ propertyDoesntExist: 'test' })); // null
console.log(User.verify({ age: 'not a number' })); // "age: integer expected"
Copy the codeNote that properties defined by protobuf are not required by default. So in the example above, verify() does not report an error for an object that has no name attribute defined.
Encoding and Decoding
While Protobufs can do basic data validation, it is primarily used for serializing and deserializing objects. To serialize an object, you can call user.encode (obj).finish(), which returns a buffer containing the protobuf representation of the object.
const buf = User.encode({ name: 'Bill'.age: 30 }).finish();
console.log(Buffer.isBuffer(buf)); // true
console.log(buf.toString('utf8')); // Gnarly string that contains "Bill"
console.log(buf.toString('hex')); // 0a0442696c6c101e
Copy the codeTo deserialize objects, you can call decode() :
const buf = User.encode({ name: 'Bill'.age: 30 }).finish();
const obj = User.decode(buf);
console.log(obj); // User { name: 'Bill', age: 30 }
Copy the codeProtobufs are commonly used to transfer data over a network as an alternative to JSON/XML. One reason is that a serialized Protobuf is much smaller because it does not need to include attribute names. For example, the following shows the size difference between using Protobuf and JSON to represent data, respectively:
const asProtobuf = User.encode({ name: 'Joe'.age: 27 }).finish();
const asJSON = JSON.stringify({ name: 'Joe'.age: 27 });
asProtobuf.length; / / 7
asJSON.length; // 23, 3x bigger!
Copy the codeBecause Protobuf knows the keys of the object in advance in the.proto file, Protobufs can be more space-efficient than JSON. While you can use patterns such as Mongoose Aliases to shrink JSON keys, you can never shrink them to the same size as protobufs because protobufs don’t serialize keys at all!
HTTP send Protobufs
Protobufs are valuable only if the data is stored in a file or sent over the network. So let’s look at how protobufs can be used to handle HTTP requests and responses. Here is an example of using Axios to send protobufs to an Express server:
const axios = require('axios');
const express = require('express');
const protobuf = require('protobufjs');
const app = express();
run().catch(err= > console.log(err));
async function run() {
const root = await protobuf.load('user.proto');
const doc = { name: 'Bill'.age: 30 };
const User = root.lookupType('userpackage.User');
app.get('/user'.function(req, res) {
res.send(User.encode(doc).finish());
});
app.post('/user', express.text({ type: '* / *' }), function(req, res) {
// Assume `req.body` contains the protobuf as a utf8-encoded string
const user = User.decode(Buffer.from(req.body));
Object.assign(doc, user);
res.end();
});
await app.listen(3000);
let data = await axios.get('http://localhost:3000/user').then(res= > res.data);
// "Before POST User { name: 'Bill', age: 30 }"
console.log('Before POST', User.decode(Buffer.from(data)));
const postBody = User.encode({ name: 'Joe'.age: 27 }).finish();
await axios.post('http://localhost:3000/user', postBody).
then(res= > res.data);
data = await axios.get('http://localhost:3000/user').then(res= > res.data);
// "After POST User { name: 'Joe', age: 27 }"
console.log('After POST', User.decode(Buffer.from(data)));
}
Copy the codeconclusion
Protobufs is a good choice for transferring data over a network, especially in combination with GRPC. Protobufs provides minimal type checking because decode() will report errors on type exceptions, and Protobufs handles much smaller data than JSON! However, the downside is that Protobuf is not human-readable (unlike a text format like JSON or XML) and can’t decode data without.proto files. Protobufs is harder to debug and use, but if you care about the size of the data sent over the network, protobufs can be very useful!
You can also find sample code for this article on Github.
reference
- Working with Protobufs in Node.js