NebulaGraphCommunity first published by NebulaGraph, Follow NebulaGraphCommunity to take a look at technical practice in the dachang library.

1 background

Text query capabilities based on external full-text search engines are already supported in Nebula 2.0. Before introducing this feature, let’s briefly review the Nebula Graph’s architectural design and storage model to make it easier to describe in the following sections.

1.1 Introduction to the Nebula Graph Architecture

As shown in the figure, the Storage Service has three layers. The Store Engine is a stand-alone Local Store Engine that provides get, PUT, scan, and delete operations on local data. The interfaces are contained in the KVStore/kvEngine. h file, allowing you to customize the Local Store plugin to your own specifications. Nebula provides a Store Engine based on the RocksDB implementation.

On top of the Local Store Engine is our Consensus layer, which implements Multi Group Raft, where each Partition corresponds to a Raft Group and the Partition is our data shard. Nebula’s sharding strategy currently uses static hashing, which is described in the next section, Schema. When creating a SPACE, you need to specify the number of partitions. Once the number of partitions is set, it cannot be changed. Generally speaking, the number of partitions must meet future service expansion requirements.

On top of Consensus, the top layer of Storage Services, is our Storage Interfaces, which define a set of graph-specific apis. These API requests are translated at this layer into a set of KV operations for the corresponding Partition. It is the existence of this layer that makes our Storage Service become a true graph Storage, otherwise, Storage Service is just a KV Storage. The main reason Nebula did not present KV as a separate service was that the diagram query process involved a lot of calculations that would use the diagram’s Schema. The KV layer does not have a data Schema concept, making it easier for the design to push down calculations.

1.2 Nebula Graph Storage Introduction

Nebula Graph has modified the storage structure for dots, edges, and indexes in 2.0, which we’ll review briefly. With an explanation of the storage structure, you can also get a basic overview of Nebula Graph’s data and index scanning principles.

Nebula Data Storage Structure

The Nebula Data store contains the “points” and “edges” stores that are based on the KV model. Here we focus on the Key storage structure, which is shown below

  • Type: 1 byte, indicating the key type. The current type is vertex, edge, index, system, and so on.
  • PartID: 3 bytes, indicating a data fragment partition. This field is used to scan the entire partition data based on the prefix during partition redistribution
  • VertexID: n bytes. The outgoing edge indicates the ID of the source point, and the incoming edge indicates the ID of the target point.
  • Edge Type: 4 bytes, indicating the type of the edge. If the value is greater than 0, it indicates the outgoing edge. If the value is less than 0, it indicates the incoming edge.
  • Rank: 8 bytes, used to handle multiple edges of the same type. Users can set according to their own needs, this field can beStore transaction time,Transaction serial number, orSome sort weight.
  • PlaceHolder: 1 byte, invisible to the user, used for future distributed transactions.
  • TagID: 4 bytes, representing the type of tag.
1.2.1.1 Storage structure
Type (1 byte) PartID (3 bytes) VertexID (n bytes) TagID (4 bytes)
1.2.1.2 Storage structure of edges
Type (1 byte) PartID (3 bytes) VertexID (n bytes) EdgeType (4 bytes) Rank (8 bytes) VertexID (n bytes) PlaceHolder (1 byte)

Nebula Indexed Storage Structure

  • Props binary (n bytes) : the props property value in tag or edge. If the property is NULL, 0xFF is populated.
  • Nullable bitset (2 bytes) : Indicates whether the prop property is NULL. There are 2 bytes (16 bits) in total. Therefore, an index can contain a maximum of 16 fields.
1.2.2.1 Tag Index Storage structure
Type (1 byte) PartID (3 bytes) IndexID (4 bytes) props binary (n bytes) nullable bitset (2 bytes) VertexID (n bytes)
1.2.2.2 Edge Index Storage Structure
Type (1 byte) PartID (3 bytes) IndexID (4 bytes) props binary (n bytes) nullable bitset (2 bytes) VertexID (n bytes) Rank (8 bytes) VertexID (n bytes)

1.3 Reasons for borrowing third-party full-text search engines

As can be seen from the above storage structure reasoning, if we want to conduct text fuzzy query on a prop field, we need to perform a full table scan or full index scan, and then filter row by row. Therefore, the query performance will be greatly reduced. It is very likely to run out of memory before the scan is completed. In addition, designing the Nebula indexed storage model as an inverted index model suitable for text search would be a departure from the original design principles of Nebula indexing. After much research and discussion, the text search was done by an external third-party full-text search engine, reducing the cost of development for the Nebula kernel while maintaining query performance.

2 goals

2.1 features

Version 2.0 only supports text search for LOOKUP. This means using Nebula’s internal index to complete the text search capabilities of A LOOKUP with the help of a third-party full-text search engine. For third-party full-text engines, only some basic data import and query functions are used at present. Nebula’s current capabilities need to be refined for complex, text-only query computations, and we look forward to hearing from the broader community. Currently supported text search expressions are as follows:

  • Fuzzy query
  • The prefix queries
  • Wildcard query
  • Regular expression query

2.2 performance

The performance mentioned here is data synchronization performance and query performance.

  • Data synchronization performance: Since we are using a third-party full-text search engine, it is inevitable to save a copy of the data in the third-party full-text search engine as well. After Nebula’s data import performance was proven to be lower than the data import performance of the third-party full-text search engine, we implemented an asynchronous data synchronization scheme for the third-party full-text search engine without compromising Nebula’s own data import performance. The specific data synchronization logic is described in the following sections.
  • Data query performance: As mentioned earlier, Nebula’s text search is a nightmare without the help of a third-party full-text search engine. At presentLOOKUPWith text search supported through third-party full-text engines, which inevitably perform slower than Nebula’s native index scans, and sometimes even the third-party full-text engines themselves, we needed oneAging mechanismTo ensure query performance. namelyLIMIT 和 TIMEOUT, will be described in detail in the following sections.

3 noun Explanation

The name of the instructions
Tag For attribute structures on points, a vertex can attach multiple tags, identified by a tagId.
Edge Like tag, edge is an attribute structure for edges, identified by edgeType.
Property The value of an attribute on a tag or edge, whose data type is determined by the structure of the tag or edge.
Partition A Storage Engine containing multiple partitions is the Nebula Graph’s smallest logical Storage unit. A Partition is divided into leader and follower roles. Raftex ensures data consistency between the leader and follower.
Graph space Each Graph space is a separate business Graph unit, and each graph space has its own set of tags and edges. A Nebula Graph cluster can contain multiple Graph Spaces.
Index The indexes shown below refer to the property indexes at the dots and edges of the Nebula Graph. Its data type depends on tag or Edge.
TagIndex Tag based index, a tag can create multiple indexes. Since composite indexes are not currently supported, an index can only be based on one tag.
EdgeIndex Index created based on edge. Similarly, an edge can create multiple indexes, but an index can only be based on one edge.
Scan Policy One query statement can have multiple index scanning modes. However, the specific scan mode is determined by scan Policy.
Optimizer Optimize query conditions, such as sorting, splitting, merging subexpression nodes in the expression tree of the WHERE clause. Its purpose is to obtain higher query efficiency.

4 Implementation Logic

ElasticSearch is the third party full text search engine that we are currently compatible with. This chapter mainly describes ElasticSearch.

4.1 Storage Structure

4.1.1 DocID

partId(10 bytes) schemaId(10 bytes) encoded_columnName(32 bytes) encoded_val(max 344 bytes)
  • PartId: the partition ID corresponding to Nebula, which is not available in the current 2.0 release and is used for future query push-down and ES routing.
  • SchemaId: The tagId or EdgeType corresponding to Nebula.
  • Encoded_columnName: corresponds to column name in tag or EDGE. An MD5 encoding is used to avoid incompatible characters in ES DocID.
  • Encoded_val has a maximum size of 344 bytes because prop Value uses base64 encoding to resolve the problem of visible characters in prop that docId does not support. The actual val size is limited to 256 bytes. Why is the length limited to 256? At the beginning of the design, the main purpose is to complete the text search LOOKUP function. Based on Nebula’s own index, the length of the index is also limited, similar to the traditional relational database MySQL, where the recommended field length is 256 characters. Therefore, the length of the third search engine is also limited to 256. Full-text search for long text is not supported here.
  • ES has a maximum docId of 512 bytes and currently has about 100 bytes of reserved bytes.

4.1.2 Doc Fields

  • Schema_id: Corresponds to Nebula’s tagId or EdgeType.
  • Column_id: The encoding of the column in nebula Tag or EDGE.
  • Value: corresponds to the value of an attribute in the Nebula native index.

4.2 Data Synchronization Logic

Leader & Listener

The logic of asynchronous data synchronization is briefly introduced in the previous chapters, which will be described in detail in this chapter. Before you introduce yourself, let’s meet Nebula’s Leader and Listener.

  • Leader: Nebula is itself a horizontally scalable distributed system with the distributed protocol RAFT. A Partition can have multiple roles in a distributed system, such as Leader, Follower, and Learner.When new data is written to the Follower and Learner, the Leader initiates a synchronization event to synchronize the WAL to the Follower and Learner. When a network or disk exception occurs, the partition role changes accordingly. Thus the data security of distributed database is guaranteed. The Leader, Follower, and Learner are all controlled in the Nebula – Storaged process, and their system parameters are determined by the configuration parametersnebula-stoage.confDecision.
  • Listener: Different from Leader, Follower, and Learner, Listener is controlled by a separate process whose configuration parameters are specified bynebula-stoage-listener.confDecision. As a Listener, the Listener passively receives the WAL from the Leader, periodically parses the WAL, and invokes the data insertion API of the third-party full-text engine to synchronize data to the third-party full-text search engine. Nebula supports ElasticSearchPUT 和 BULKInterface.

Let’s look at the data synchronization logic:

  1. Insert Vertex or Edge through Client or Console
  2. The graph layer calculates the associated partition from the Vertex ID
  3. The Graph layer through storageClient willINSERTThe request is sent to the Leader of the relevant Partition
  4. Leader parsingINSERTRequest, and synchronize WAL to the Listener
  5. The Listener periodically processes the newly synchronized WAL, parses the WAL, and obtains the value of the tag or Edge attribute whose field type is String.
  6. Assemble tag or Edge metadata and attribute values into ElasticSearch compatible data structures
  7. Through the ElasticSearchPUT 或 BULKInterface to ElasticSearch.
  8. If the write fails, go back to Step 5 and retry the failed WAL until the write succeeds.
  9. After data is successfully written, the Log ID and Term ID are recorded and used as the start value for the next WAL synchronization.
  10. Go back to the timer at step 5 and process the new WAL.

In the above steps, stop WAL synchronization if the ElasticSearch cluster is down or the Listener process is down. When the system recovers, data synchronization continues with the Log ID of the last successful data synchronization. Here is a suggestion that dbAs should use external monitoring tools to monitor the running status of ES in real time. If ES is in an invalid state for a long time, the log volume of the Listener will soar and normal query operations cannot be performed.

4.3 Query Logic

According to the figure above, the key steps of text search are “Send Fulltext Scan Request” → “Fulltext Cluster” → “Collect Constant Values” → “IndexScan Optimizer”.

  • Send Fulltext Scan Request: generates a Fulltext index query Request based on the query condition, schema ID, and Column ID.
  • Fulltext Cluster: sends a query request to ES and obtains the query result of ES.
  • Collect Constant Values: Use the returned query results as Constant Values to generate a query expression within Nebula. For example, the original query request is for an attribute value starting with “A “in the C1 field. If the returned result contains two results,” A1 “and “A2”, the neubla expression will be parsed in this stepC1 == "A1" OR C1 == "A2".
  • IndexScan Optimizer: Identifies the optimal Nebula internal Index based on RBO and generates the optimal execution plan based on the newly generated expression.
  • In the “Fulltext Cluster” step, there may be slow query performance or massive data returns, which we provide hereLIMIT 和 TIMEOUTMechanism to interrupt ES terminal query in real time.

5 shows

5.1 Deploying an External ES Cluster

For the deployment of ES cluster, I will not go into details here, I believe that you are familiar with. After the ES cluster is successfully started, we need to create a generic template for the ES cluster, which has the following structure:

{
 "template": "nebula*",
  "settings": {
    "index": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties" : {
            "tag_id" : { "type" : "long" },
            "column_id" : { "type" : "text" },
            "value" :{ "type" : "keyword"}
        }
  }
}
Copy the code

5.2 Deploying the Nebula Listener

  • Modify configuration parameters based on the actual environmentnebula-storaged-listener.conf
  • Start the Listener:./bin/nebula-storaged --flagfile ${listener_config_path}/nebula-storaged-listener.conf

5.3 Registering ElasticSearch Client Connection Information

Nebula > SIGN IN TEXT SERVICE (127.0.0.1:9200); nebula> SHOW TEXT SEARCH CLIENTS; + -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- + | Host | Port | + -- -- -- -- -- -- -- -- -- -- -- -- -- + | + -- -- -- -- -- - "127.0.0.1" | 9200 | + -- -- -- -- -- -- -- -- -- -- -- -- -- + | + -- -- -- -- -- - "127.0.0.1" | 9200 | + -- -- -- -- -- -- -- -- -- -- -- -- -- + | + -- -- -- -- -- - "127.0.0.1" | 9200 | + -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- - +Copy the code

5.4 Creating a Nebula Space

CREATE SPACE basketballplayer (partition_num=3,replica_factor=1, vid_type=fixed_string(30));
 
USE basketballplayer;
Copy the code

5.5 add the Listener

Nebula > ADD LISTENER ELASTICSEARCH 192.168.8.5:46780192168 8.6:46780; nebula> SHOW LISTENER; +--------+-----------------+-----------------------+----------+ | PartId | Type | Host | Status | + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- + | 1 | "ELASTICSEARCH" | | "[192.168.8.5:46780]" "ONLINE" | + -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- + | 2 | "ELASTICSEARCH" | | "[192.168.8.5:46780]" "ONLINE" | + -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- + | 3 | "ELASTICSEARCH" | "[192.168.8.5:46780]"  | "ONLINE" | +--------+-----------------+-----------------------+----------+Copy the code

5.6 Creating a Tag, Edge, and Nebula Index

In this case, it is recommended that the length of field name be less than 256. If services permit, it is recommended that the type of field name in Player be set to fixed_string and the length of field name be less than 256.

nebula> CREATE TAG player(name string, age int);
nebula> CREATE TAG INDEX name ON player(name(20));
Copy the code

5.7 Inserting Data

nebula> INSERT VERTEX player(name, age) VALUES \
  "Russell Westbrook": ("Russell Westbrook", 30), \
  "Chris Paul": ("Chris Paul", 33),\
  "Boris Diaw": ("Boris Diaw", 36),\
  "David West": ("David West", 38),\
  "Danny Green": ("Danny Green", 31),\
  "Tim Duncan": ("Tim Duncan", 42),\
  "James Harden": ("James Harden", 29),\
  "Tony Parker": ("Tony Parker", 36),\
  "Aron Baynes": ("Aron Baynes", 32),\
  "Ben Simmons": ("Ben Simmons", 22),\
  "Blake Griffin": ("Blake Griffin", 30);
Copy the code

5.8 the query

nebula> LOOKUP ON player WHERE PREFIX(player.name, "B");
+-----------------+
| _vid            |
+-----------------+
| "Boris Diaw"    |
+-----------------+
| "Ben Simmons"   |
+-----------------+
| "Blake Griffin" |
+-----------------+
Copy the code

Problem tracking and solving skills

In the process of setting up the system environment, some steps may lead to the abnormal operation of functions. In the previous user feedback, I summarized three types of possible errors, and summarized the skills for analyzing and solving the problems as follows

  • The Listener cannot be started or does not work properly after being started
    • Check the Listener configuration file to ensure that the ListenerIP:PortConflicts with the existing Nebula – Storaged
    • Check the Listener configuration file to ensure that the MetaIP:PortCorrect. This has to be consistent with nebula- Storaged
    • Check the Listener configuration file to ensure that the pIDS and logs directories are independent and do not conflict with the nebula- Storaged directory
    • After the startup is successful, the configuration is modified due to incorrect configurations. After the restart, the system still fails to work properly. In this case, you need to clear meta metadata. The commands for doing this are available at nebula’s help manual: Documentation link.
  • Data cannot be synchronized to the ES cluster
    • Check whether the Listener receives the WAL from the Leadernebula-storaged-listener.confIn the configuration file- listener_pathIs there a file in the directory of
    • Open the vlog (UPDATE CONFIGS storage:v=3), and check whether the CURL command in the log is successfully executed. If there is an error, it may be an ES configuration error or an ES version compatibility error
  • There is data in the ES cluster, but the correct result cannot be queried
    • Also open vlog (UPDATE CONFIGS graph:v=3Do you want to do something about the CURL command
    • During query, only lowercase characters can be identified, but not uppercase characters. It could be the template creation error for ES. Refer to the Nebula Help manual to create: Documentation links.

7 TODO

  • Full-text indexing for a specific tag or edge
  • REBUILD of full-text indexes

Ac graph database technology? To join the Nebula Exchange group, please fill out your Nebula card and Nebula Assistant will bring you into the group

Want to share graph database technology with other big companies? NUC 2021 Is waiting for you to communicate: NUC 2021 registration portal