Node.js communicates with HBase

Recently, WE are working on a data retrieval platform, one of which uses HBase. Unlike other common databases, when you install the database and download the Node.js package, you will find that almost all the packages are unavailable, and there are few topics about Node.js and HBase, which can be very frustrating as a beginner to HBase.

HBase non-cluster environment construction

Download the HBase 2.4.4 bin installation package

The curl - o https://downloads.apache.org/hbase/2.4.4/hbase-2.4.4-bin.tar.gzCopy the code

Starting the HBase Service

./bin/start-hbase.sh
Copy the code

Start the Thrif2 Server service

./bin/hbase thrift2
Copy the code

The Thrift2 server listens to port 9090

HBase Thrift2 client compiled

Download, compile, and install Thrift

Curl - o https://downloads.apache.org/thrift/0.14.2/thrift-0.14.2.tar.gz tar - ZXVF thrift - 0.14.2. Tar. Gz CD thrift - 0.14.2 configure make make installCopy the code

Download the HBase thrift file

https://github.com/apache/hbase/blob/rel/2.4.4/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thr iftCopy the code

Note: Thrift files may vary from HBase version to HBase version, and each version may need to be recompiled and retuned to the Node.js side of the code (this is also a complaint).

Compile thrift to generate hbase_types.js and thBaseservice.js

thrift --gen js: node hbase.thrift
Copy the code

Node.js communicates with HBase

Node.js can communicate with HBase using REST and Thrift RPC. I prefer RPC, so I use the Thrift2 solution (Thrift is an upgraded version).

Create links

const thrift = require("thrift"); const HBase = require(".. /gen-nodejs/THBaseService"); // const HBaseTypes = require(".. /gen-nodejs/hbase_types"); const transport = thrift.TBufferedTransport; const protocol = thrift.TBinaryProtocol; const connection = thrift.createConnection("localhost", 9090, { transport: transport, protocol: protocol, }); connection.on("error", function (err) { console.log("error:", err); }); connection.on("close", function (err) { console.log("close"); }); const client = thrift.createHttpClient(HBase, connection);Copy the code

Operating database

Take creating namespaces, data tables, adding row data, and obtaining row data as examples

const namespace = "ns"; const tableName = "test"; const cf = "cf"; // create Namespace client.createNamespace({ name: namespace }, function (err) { if (err) { console.log("create namespace error:", err); } else { console.log("create namespace success"); }}); Client.createtable ({tableName: {ns: namespace, qualifier: tableName}, columns: [{name: cf }], }, null, function (err) { if (err) { console.log("create table error:", err); } else { console.log("create table success"); }}); Client. Put (' ${namespace}:${tableName} ', {row: "192", columnValues: [{family: "cf", qualifier: "value", value: "hhhh", }, { family: "cf", qualifier: "value1", value: "hhhh", }, { family: "cf", qualifier: "value2", value: "hhhh", }, ], }, function (err) { if (err) { console.log("put column error", error); } else { console.log("put column success"); }}); Get (' ${namespace}:${tableName} ', {row: "192" }, function (err, data) { if (err) { console.log("Get error:", err); } else { console.log("Get result: " + JSON.stringify(data)); }});Copy the code

Simple packaging

Take the example of getting a single piece of data

Function formate(rowData) {const obj = {}; obj.rowkey = rowData.row ? rowData.row.toString() : null; rowData.columnValues.forEach(column => { const family = column.family.toString(); const qualName = column.qualifier.toString(); obj[family] = obj[family] || {}; obj[family][qualName] = column.value.toString() }); return obj; }... client.getAsync = promisify(client.get); // Promise const res = await client.getAsync(' ${namespace}:${tableName} ', {row: "192"}); console.log('res', formate(res))Copy the code

In earlier versions, interface parameters in THBaseService. Js were constrained by the types in hbase_types.js. Now, this version is compatible, and you can pass parameters using hbase_type or javascript objects directly. For example, the following two forms are equivalent.

const res = await client.createNamespaceAsync(new HBaseTypes.TNamespaceDescriptor({ name: "ns" }));
const res1 = await client.createNamespaceAsync({ name: "ns" })
Copy the code

To improve

  1. Introduce the concept of connection pooling
  2. During the test, it was found that the communication link would be broken if it was not communicated within a short period of time, and the heartbeat mechanism might be introduced
  3. The members of the department are mainly familiar with the monGO structure, encapsulated as the operational form of Mongo

Follow my wechat official account “SUNTOPO WLOG”, welcome to leave a comment and discuss, I will reply as much as possible, thank you for reading.