Node.js is designed to do fast and efficient network I/O. Its event-driven flow makes it ideal for an intelligent agent, often acting as the glue between back-end systems and the front end. Node was designed to do just that, but in the meantime, it has been successfully used to build traditional Web applications: an HTTP server that provides responses to HTML pages or JSON messages and uses a database to store data.

While other platforms and languages have more mature Web frameworks and prefer to use open source relational databases such as MySQL or PostgreSQL, most Node Web frameworks (Express, Hapi, etc.) do not mandate the use of any particular database, or even any type of database at all. Yesterday in the “Brief introduction to front-end anomaly monitoring platform implementation scheme” article mentioned LevelDB, today with you to introduce this ultra-high performance key-value database LevelDB, the same type of more popular MongoDB, this article will not introduce MongoDB.

Know LevelDB

LevelDB is a high performance key-value storage engine designed by Jeff Dean and Sanjay Ghemawat. LevelDB is a high performance key-value storage engine designed by Jeff Dean and Sanjay Ghemawat. LevelDB is a high performance key-value storage engine designed by Jeff Dean and Sanjay Ghemawat. This open source project is currently a C++ library that supports handling billion-level key-value data persistence. In excellent performance, the memory footprint is also very small, and a lot of data is stored directly on disk, which can be understood as space for time.

Design ideas

Sequential writes perform much better than random writes on normal mechanical disks, and LevelDB is designed to take advantage of this feature. LevelDB data is stored on disk using the LSM-tree structure. Lsm-tree converts random disk write to sequential disk write, greatly increasing the write speed. To achieve this, the idea of LSM-tree is to split the index Tree structure into a large Tree and a small Tree. The smaller one is resident in memory and the larger one is persisted to disk to jointly maintain an ordered key space. Write operations operate on trees in memory first, and as the trees in memory grow, merge operations with trees in disk are triggered, and merge operations themselves only have sequential writes. As shown below:

LevelDB static structure is composed of six parts:

  • MemTable(wTable): Memory data structure, implemented as SkipList. Accept the user’s read/write request, where new data changes are written first.
  • Immutable MemTable(rTable): When the size of MemTable reaches the preset threshold, it becomes Immutable. MemTable only accepts read operations but no writes. Subsequent Flush operations are flushed to disks by background threads.
  • SST Files: Sorted String Table Files, disk data storage Files. It is divided into Level0 to LevelN layers, and each layer contains multiple SST files with data in order. Level0 is obtained directly from Immutable Memtable Flush, and data from each other layer is obtained from the previous Compaction.
  • Manifest FilesThe Manifest file records the distribution of SST files at different levels, the maximum and minimum keys of a single SST file, and other meta information required by LevelDB. Because LevelDB supports snapshot and requires multiple versions to be maintained, multiple Manifest files may exist at the same time.
  • Current File: Because there may be multiple Manifest files, Current records the Current Manifest file name.
  • Log Files (WAL): Log file used to prevent data loss from MemTable.

Bold arrows indicate the direction of the flow in which data is written:

  1. Let’s start with MemTable.
  2. When the size of MemTable reaches a set threshold, it is converted to Immutable MemTable.
  3. Immutable Table is asynchronously flushed to disk by background threads and becomes an SST file on Level0.
  4. Under certain conditions, a background thread triggers a Compaction between Level0 and LevelN files.
The flow direction of read operations is similar to that of write operations:
  1. Read MemTable, if present, return.
  2. Read Immutable MemTable, if present, return.
  3. Read Level0 to Leveln sequentially, return if any exists.
  4. Return does not exist.

LevelDB is designed to meet the requirements of the LevelDB database. LevelDB is designed to meet the requirements of LevelDB. LevelDB is designed to meet the requirements of LevelDB. If you want to know more about the design principle, you need to learn the design algorithm of skip lists, which will not be expanded here.

Skip list is an alternative data structure of balanced tree, but different from red black tree, the realization of the balance of skip list is based on a randomized algorithm, that is to say, the insertion and deletion of skip list is relatively simple.

LevelDB features:

  • Keys and values are arbitrary length byte arrays;
  • Store key sorting data;
  • Provide basicput(key, value),get(key)anddelete(key)Interface;
  • Support batch operation to atomic operation;
  • You can create a snapshot of a data panorama and allow you to look up data in the snapshot.
  • The data can be traversed through a forward (or backward) iterator (the iterator implicitly creates a snapshot);
  • Automatically compress data using Snappy.
  • Portability;

Based on the above features, LevelDB is used as a data store for many blockchain projects. Here’s how to use LevelDB in a project, using the code in the project pretender-service.

Install LevelDB

The project is based on Node.js, and the base environment needs to be configured itself. Execute the following command:

npm install level --save
Copy the code

LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB: LevelDB

npm install level-sublevel --save
Copy the code

You also need a module that generates a unique ID for the data, and here we install cuID.

npm install cuid --save
Copy the code

Once installed, you will use it in your application, creating a levelDb class./ SRC /db/ leveldb.js:

Add a referenced dependency library

const level = require("level");
const sublevel = require("level-sublevel");
Copy the code

Declare two variables in the constructor as follows:

class LevelDb {
    constructor(dbPath, options = {}) {
        this.options = options;
        this.db = sublevel(level(dbPath, { valueEncoding: "json" }));
    }
}
module.exports = LevelDb;
Copy the code

Here, you create a singleton module that exports the LevelDB database. You need the Level module first and use it to instantiate the database and provide it with a path dbPath. This path is the directory where LevelDB stores the data files. This directory is dedicated to LevelDB and may not exist at the beginning. The path defined here is passed in when it is declared externally. ValueEncoding defines the value format as JSON. The following formats are supported: HEX, UTF8, Base64, binary, ASCII, and UTF16LE.

Add LevelDB action method

Now add methods to the class defined above. Common methods are PUT, GET, DELETE, Batch, and find. This paper is just a basic operation, the actual project needs to design the data reasonably.

The LevelDB database object this.db is created in the constructor and its methods can be called.

put

Add or update data.

put(key, value, callback) { if (key && value) { this.db.put(key, value, (error) => { callback(error); }); } else { callback("no key or value"); }}Copy the code

get

Gets the data for the specified key.

get(key, callback) { if (key) { this.db.get(key, (error, value) => { callback(error, value); }); } else { callback("no key", key); }}Copy the code

delete

Delete the data of the specified key

delete(key, callback) { if (key) { this.db.del(key, (error) => { callback(error); }); } else { callback("no key"); }}Copy the code

batch

A powerful feature of LeveLDB is that it allows you to group operations in a batch to be automatically executed.

batch(arr, callback) { if (Array.isArray(arr)) { var batchList = []; arr.forEach(item); { var listMember = {}; if (item.hasOwnProperty("type")) { listMember.type = item.type; } if (item.hasOwnProperty("key")) { listMember.key = item.key; } if (item.hasOwnProperty("value")) { listMember.value = item.value; } if ( listMember.hasOwnProperty("type") && listMember.hasOwnProperty("key") && listMember.hasOwnProperty("value") ) { batchList.push(listMember); } } if (batchList && batchList.length > 0) { this.db.batch(batchList, (error) => { callback(error, batchList); }); } else { callback("array Membre format error"); } } else { callback("not array"); }}Copy the code

Now that you have a LevelDB base action class, it’s time to apply it to your project.

The use of LevelDB

A key-value database has a unique id for each record in a relational database. A key-value database needs to maintain a unique key value that is similar to an ID. The design of this key value also affects whether the data query is traversed. Here we just do a simple example, the complete code is as follows:

"use strict"; require(".. /src/utils/logger.js")(2); const { dbConfig } = require(".. /config"); const LevelDB = require(".. /src/db/levelDb"); const path = require("path"); const cuid = require("cuid"); const dbHelper = new LevelDB( path.resolve(__dirname, dbConfig.path, dbConfig.folder) ); // Add user information const Administrators = [{name: "QuintionTang", email: "[email protected]", password: "123456", id: "ckoyhjqbj0000mzkd1o63e31p", }, { name: "JimGreen", email: "[email protected]", password: "123456", id: "ckoyhjqbk0001mzkdhuq9abo4", }, ]; const keyPrefix = "administrator"; Console. info("====> Start inserting data "); const administratorsKeys = []; for (const item of administrators) { const uid = item.id; const keyName = `${keyPrefix}_${uid}`; item.id = uid; dbHelper.put(keyName, item, (error) => { if (error ! == null) { administratorsKeys.push(keyName); }}); } console.info("====> Start looking for data "); / / find the uid for ckoyhjqbj0000mzkd1o63e31p user information const findUid = "ckoyhjqbj0000mzkd1o63e31p"; const findKeyName = `${keyPrefix}_${findUid}`; dbHelper.find( { prefix: findKeyName, }, (error, result) => { console.info(result); });Copy the code

Next, execute the sample code:

cd examples
node index.js
Copy the code

The result is as follows:

Looking at database at the root will generate the database file.

conclusion

This article briefly introduces the design principle of LevelDB, complete the definition of LevelDB class in the project, and implement a simple data insert and query. In the future, more complex business logic will be completed in the project pretender-service and basic CURD will be realized. Please pay attention to the project dynamics.