Nodejs BFF of RPC

An overview of the

The generation of BFF

The generation of BFF is closely related to two factors: one is background microservitization and domain; Two is the front end of the application across the trend. The former makes the backend become domain cohesive, while the latter makes the same application present inevitable differences at different ends. The development of two directions leads to an obvious Gap between background services and foreground applications.

Assuming that the back-end interface is thoroughly domatized, some logic in the front-end application needs to acquire and assemble multiple back-end interfaces in the front-end code, and then arrange them. This is too heavy for front-end applications, and the logic is hard to maintain.

Assuming that this logic for UI-assembled data is put in the background, it impedes the domain of the background.

So the BFF layer was created to fill the Gap between the background and the changeable front.

Functional positioning of BFF

As can be seen from the above, the BFF layer should have the following core functions:

Interface aggregation and orchestration

In addition to this core function, the BFF layer usually provides:

Authentication (usually by accessing third-party services)
Parameter calibration and clipping

In different enterprise application architectures, the functions of the BFF layer may be slightly different, for example, authentication may be carried off at the gateway layer. But generally not beyond the above range of functions.

Interface aggregation and orchestration

When the request goes to the BFF layer, the traffic has already gone to the service background of the enterprise. The interface for aggregation and orchestration is often provided in the form of RPC. Then the first thing to be done in the BFF layer is the call to the RPC interface.

RPC is a general protocol for internal service invocation. There are many specific implementations, such as Dubbo and gRPC.

Without loss of generality, we first define an implementation of the RPC protocol and then implement it in code.

To implement the RPC protocol, the following steps are required:

Communication protocol. We’re using TCP here
Data protocol. We need to design the message size, the key is how to determine the message length, codec

Message frame

We have written the message body in the following format:

Length + / n + content

The length is the byte length of payload. We use the simplest serialization/deserialization as the codec scheme, using the format of {data: {}, error: {}} to represent the decoded message, so the codec function is:

export function encode(payload) {
  return JSON.stringify(payload)
}
export function decode(payload) {
  try {
  	return JSON.parse(payload)
  } catch(err) {
    log.error('decode error: ', err)
    return { error: err }
  }
}
Copy the code

Buffer generation method in RPC communication:

export function buildDataBuffer(payload) {
  const encoded = encode(payload)
  // The length of the message +'/n'+payload
  return Buffer.byteLength(encoded) + '\n' + encoded;
}
Copy the code

After determining the size of the message, we have the length determination and decoding rules, and we can determine the logic of parsing the payload from the received buffer data:

/** * Convert buffer data to payload information *@param {*} Data Data transmitted through RPC *@param {*} LengthObj is used to store buffer data. The structure is as follows: {bufferBytes: undefined, // The received buffer data getLength: True, // Identifies whether the byte length field needs to be resolved: When the message body is too large, on('data') will be fired multiple times, so it is needed as a container to store buffer data. Initialize this container outside of connection.on('data') to ensure that multiple triggered data events can be stored through it
export function parseRpcResponse(data, lengthObj) {
  // Retrieve the buffer message
  if (lengthObj.bufferBytes && lengthObj.bufferBytes.length > 0) {
    // If the length is greater than 0, the data has been received, and the new data is concatenated after it
    const tmpBuff = Buffer.alloc(lengthObj.bufferBytes.length + data.length);

lengthObj.bufferBytes.copy(tmpBuff, 0);
data.copy(tmpBuff, lengthObj.bufferBytes.length);

lengthObj.bufferBytes = tmpBuff;




} else {
lengthObj.bufferBytes = data;
}




var [fnDatas, finished] = parseBufferDatas.call(lengthObj);
return [fnDatas, finished]
}
export function parseBufferDatas() {
var datas = [];
// Indicates whether the message body has been parsed
let finished = false;
var i = -1;




Parse the payload intercepted according to the length in length + '/n' + payload
var parseBufferData = function () {
if (this.getLength === true) {
// find the length before \n
i = getNewlineIndex(this.bufferBytes);
if (i > -1) {
this.length = Number(this.bufferBytes.slice(0, i).toString());
this.getLength = false;
this.bufferBytes = clearBuffer(this.bufferBytes, i + 1); }}// Wait until the length of the data reaches the length, then start parsing
if (this.bufferBytes &amp; &amp;this.bufferBytes.length &gt; =this.length) {
  const dataStr = this.bufferBytes.slice(0.this.length).toString();
  // After processing the first data, we need to see if there are any unparsed data to continue parsing
  this.getLength = true;

  let parsedData
  try {
    parsedData = decode(dataStr)
  }
  catch (e) {
    log.e(&#39; ERROR PARSE: &#39; , e, dataStr);return;
  }

  datas.push(parsedData);
  // Put the rest into the cache object
  this.bufferBytes = clearBuffer(this.bufferBytes, this.length);

  // Continue with the following instructions
  if (this.bufferBytes &amp; &amp;this.bufferBytes.length &gt; 0) {
    parseBufferData.call(this);
  } else {
    // After parsing the data, set it to true
    finished = true}}}; parseBufferData.call(this);
return [datas, finished];
}
Copy the codeparseBufferData.call(this);
return [datas, finished];
}

In RPC communication, the client needs to describe the methods and parameters called, and its data format is designed as follows:

function buildClientRequestPayload(fnName, args) {
  var id = idGenerator();
  const payload = { id: id, fn: fnName, args }
  return buildDataBuffer(payload)
}
Copy the code

After receiving the message, the RPC communication server invokes the corresponding method and returns the result in the following form:

/ * * *@param {*} * {* reqData: {id: 'x', fn: 'fnName ', args: []} Request * data? : method execution result * MSG? : Error description * error? : Error message *} */
function buildResponse(oPayload) {
  const responsePayload = { id: oPayload.reqData.id, ... oPayload }delete responsePayload.reqData
  return buildDataBuffer(responsePayload)
}
Copy the code

RPC Server

The Rpc Server consists of the following parts:

Receives messages from the client
The message is parsed and the corresponding method is invoked
Returns the result

When starting an RPC Server, you need to describe the methods on the current server. We designed the following RPC Server instance description:

import RPCServer from '.. /rpc/server.js';


const port = 5665;




const rpc = new RPCServer({
combine: function (a, b) {
return a + b
},
longString: function () {
// construct a 2m string
let s = '1'
const n2m = (2 << 20)
for (let i = 0; i < n2m; i++) {
s += '1'
}
s += 'over.'
return s
}
});




rpc.listen(port);
Copy the coderpc.listen(port);

Combine and longString are the methods provided by this service. Based on the above interface, we designed RPCServer as follows:

class RPCServer {
  constructor(services, logger) {
    this.services = services;
    this.listen = (port) = > {
      this.getServer();
      this.server.listen(port, () = > {
        console.log(`server running on port ${port}`)}); }this.getServer = () = > {
      const self = this;
      const server = net.createServer(function (c) {
        // To store the data to be parsed
        const lengthObj = {
          bufferBytes: undefined.getLength: true.length: -1
        };

    c.on(&#39; data&#39; , getOnDataFn(c, lengthObj, self)); });this.server = server;
}
this.close = () =&gt; {
  this.server.close(); }}}function getOnDataFn(connection, lengthObj, rpcInstance) {
return function (data) {
const [fnDatas, finished] = parseRpcResponse(data, lengthObj)
if (finished) {
fnDatas.forEach(fData= >fnExecution(fData, connection, rpcInstance)); }}; }function fnExecution(reqData, c, rpcInstance) {
if(! rpcInstance.services[reqData.fn]) { c.write(buildResponse({ reqData,msg: 'No corresponding method found'.error: { code: 'UNKNOWN_COMMAND'}}))return
}




const args = reqData.args;
try {
const fn = rpcInstance.services[reqData.fn]
const argList = Array.isArray(args) ? args : [args]
const data = fn.apply({}, argList)
c.write(buildResponse({ reqData, data }))
}
catch (error) {
c.write(buildResponse({ reqData, error, msg: 'Execution method error'})); }};Copy the codeconst args = reqData.args; try { const fn = rpcInstance.services[reqData.fn] const argList = Array.isArray(args) ? args : [args] const data = fn.apply({}, argList) c.write(buildResponse({ reqData, Data}))} catch (error) {c.write(buildResponse({reqData, error, MSG: 'error')); }};

ParseRpcResponse is the data parsing method described above.

Request management in the BFF layer

We use Express to start a service as a BFF layer

import express from 'express';
import bodyParser from 'body-parser';
import RpcClient from '.. /client.js';


const rpcClient = new RpcClient()
const app = express()
app.use(bodyParser.json());




app.post('/'.async (req, res) => {
const { fn, args } = req.body
const response = await rpcClient.proxy({
host: 'localhost'.port: 5665,
fn,
args,
})
if (response.data) {
res.send(response.data)
return
}
res.send(response)
})




app.listen(3000.() = > {
console.info('listening on port 3000.')})Copy the codeapp.listen(3000, () => {
console.info('listening on port 3000.')
})

RpcClient is the class we need as the client, using its proxy method for the remote invocation.

RPC Client

Let’s implement the client used in the BFF, which has been identified in the message physique as needing to provide proxy methods for initiating requests and handling responses. The main things to be done are:

Request a connection to the server

const connection = net.createConnection(port, host);
Copy the code

Send method description

connection.on('connect'.function () {
  After sending the message / / connection, buildClientRequestPayload see before
  connection.write(buildClientRequestPayload(fn, args));
})
Copy the code

The parse response returns data

RpcClient.prototype.proxy = function proxy({ host, port, fn, args, } = {}) {
  return new Promise((resolve, reject) = > {
    // Establish a connection
    const connection = net.createConnection(port, host);

let success = false
// A container for remote data, similar to server logic
const lengthObj = {
  bufferBytes: undefined.getLength: true.length: -1
}
// connection.on(' connect' , () => {})...
// Parse the remote response
connection.on(&#39; data&#39; .function (data) {
  try {
    // If the data exceeds the length of the socket's single message, it will be received several times, which is triggered repeatedly in one communication. Therefore, finished is used to indicate whether the communication is complete. The communication is returned only after the communication is complete
    // The length of all messages in the FINISHED message header is greater than that in the header
    const [result, finished] = parseRpcResponse(data, lengthObj)
    if (finished) {
      success = true
      connection.end()
      resolve(getDataFromRpcResponse(result))
    }
  } catch (err) {
    resolve({ error: err, msg: & #39; parse data error.&#39; })}})/ /... Omit other events such as error})}Copy the code})}

This completes the BFF layer’s ability to communicate with RPC services. See Github Repo for detailed code

Once RPC communication with microservices is implemented, interface aggregation and choreography are relatively simple and will not be covered here.

reference

nodejs-light-rpc
Github.com/QxQstar/nod…