Use TiKV to build distributed Redis service

What is a Redis

Redis is an open source, high performance, support a variety of data structures in memory database, has been widely used in database, cache, message queue and other fields. It has rich data structure support, such as String, Hash, Set, and Sorted Set, with which users can build their own high-performance applications.

Redis is very fast, probably the fastest database in the world, and although it uses memory, it also provides some persistence mechanisms and asynchronous replication mechanisms to keep data secure.

The shortage of the Redis

Redis is pretty cool, but it has some problems:

Memory is expensive and not infinite, so it’s impossible to store large amounts of data on a single machine.
Asynchronous replication does not guarantee Redis data security.
Redis provides transaction mode, but it does not satisfy ACID properties.
Redis provides clustering support, but also does not support distributed transactions across multiple nodes.

So sometimes, we need a more powerful database, which may not match Redis in terms of latency, but has enough features, such as:

Rich data structures
High throughput, acceptable delay
Strong data consistency
Horizontal scaling
Distributed transaction

Why TiKV

About four years ago, I began to solve some of the problems Redis encountered above. The most intuitive way to persist data is to keep it on hard disk rather than in memory. So I developed LedisDB, a database that uses the Redis protocol to provide rich data structures, but puts the data in RocksDB. LedisDB was not fully Redis compliant, so later on, my colleagues and I went ahead and created RebornDB, a fully Redis compliant database. Both LedisDB and RebornDB can store a larger amount of data because they both store data on hard disks. But they still don’t provide ACID support, and while we can provide clustering support via CODIS, we don’t do a good job of supporting globally distributed transactions.

So we needed another approach, and fortunately, we had TiKV.

TiKV is a high performance key-value database that supports distributed transactions. It only provides a simple key-value API, but based on key-value, we can construct our own logic to create more powerful applications. For example, we built TiDB, a distributed relational database compatible with MySQL based on TiKV. TiDB supports SQL features by mapping database schemas to key-values. So for Redis, we can also adopt the same approach – build a service that supports Redis protocol and map Redis data structure to key-value.

How to start

The whole architecture is very simple, all we need to do is to build a Proxy of Redis, which will parse the Redis protocol, and then map the Redis data structure to key-value.

Redis Protocol

Redis Protocol is called Redis Serialization Protocol (RESP). It is of text type, easy to read, and easy to parse. It uses “RN” as the delimiter for each line and uses a different prefix to represent different types. For a simple String, for example, the first byte is “+”, so an “OK” line is “+OKrn”. Most of the time, the client will use the most common request-Response model to interact with Redis. The client first sends a request and then waits for Redis to return the result. The request is an Array containing bulk Strings elements, and the return value may be any RESP type. Redis also supports other communication methods:

Pipeline – This is where the client continuously sends multiple requests to Redis and waits for Redis to return a result. Push – The client subscribes to a channel on Redis, from which the client is continually pushed.

Here is a simple example of a client sending the LLEN mylist command to Redis:

C: *2\r\n
C: $4\r\n
C: LLEN\r\n
C: $6\r\n
C: mylist\r\n

S: :48293\r\n
Copy the code

The client sends an array with two BULK Strings, the first of length 4 and the second of length 6. Redis returns an integer of 48293. As you can see, RESP is very simple, naturally, and writing a RESP parser is very easy.

The author created a Go library, Goredis, based on which we can easily resolve the RESP from the connection, a simple example:

// Create a buffer IO from the connection.
br := bufio.NewReaderSize(conn, 4096)
// Create a RESP reader.
r := goredis.NewRespReader(br)
// Parse the Request
req := r.ParseRequest()
Copy the code

The function ParseRequest returns a parsed Request of [][]byte type, the first field being the function name, such as “LLEN”, and the following fields being the command arguments.

TiKV transaction API

Before we Begin, the authors will give a simple example of using the TiKV transaction API. We call Begin to start a transaction:

txn, err := db.Begin()
Copy the code

The Begin function creates a transaction. If something goes wrong, we need to check for err.

After we start a transaction, we can do many operations:

Value, err := txn.Get([]byte(" key ")) // Do something with value andthenUpdate the newValue to the key.txn. Put([]byte(" key "), newValue)Copy the code

Above we get the value of a key and update it to the new value. TiKV uses an optimistic transaction model, which caches all changes locally and then collectively commits them to the Server.

// Commit the transaction
txn.Commit(context.TODO())
Copy the code

Like any other transaction, we can roll back this transaction:

txn.Rollback()
Copy the code

If two transactions operate on the same key, they conflict. One transaction commits successfully and the other fails and rolls back.

Map Data structure to TiKV

Now that we know how to parse the Redis protocol and how to operate within a transaction, the next step is to support Redis data structures. Redis mainly has 4 data structures: String, Hash, Set and Sorted Set, but for TiKV, it only supports key-value, so we need to map these data structures to key-value.

First, we need to distinguish between different data structures. An easy way to do this is to add a Type flag after the key. For example, we can add ‘s’ to String, so a String key “ABC” in TiKV is actually “abCS”.

For other types, we may need to consider more. For example, for Hash types, we need to support the following operations:

HSET key field1 value1
HSET key field2 value2
HLEN key
Copy the code

A Hash has many fields, and I sometimes wonder how many Hash fields there are, so for TiKV, not only do we need to combine the Hash key with the field to make a key for TiKV, We also need another key to hold the length of the Hash, so the Hash layout is similar:

Key + 'H' -> length key + 'F' + fielD1 -> value key + 'F' + field2 -> valueCopy the code

If we don’t save length, then if we want to know the length of the Hash, we have to scan the Hash every time to get all the fields, which is not efficient. But if we use another key to hold the length, any time we add a new field, we need to update the value of that length, which is also an overhead. For my part, I prefer to use another key to save length, because HLEN is a high frequency operation.

example

We can clone the String and Hash operations and compile them as follows:

git clone https://github.com/siddontang/redis-tikv-example.git $GOPATH/src/github.com/siddontang/redis-tikv-example

cd $GOPATH/src/github.com/siddontang/redis-tikv-example
go build
Copy the code

Before running, we need to start TiKV, refer to instruction, and execute:

./redis-tikv-example
Copy the code

This example listens on port 6380, which we can then connect to using any Redis client, such as redis-CLI:

Redis - cli - p 6380 127.0.0.1:6380 >setK1 a OK 127.0.0.1:6380> get k1"a"127.0.1:6380 > hset k2 a (integer) 1
127.0.0.1:6380> hget k2 f1
"a"
Copy the code

The end of the

Several companies have built their own Redis Server based on TiKV, and there is an open source project called TIDIS that does the same. Tidis is pretty well established, so if you want to replace your own Redis, try it out.

As you can see, TiKV is actually a foundational component on which many other applications can be built. If you’re interested in what we’re doing, please feel free to contact me at [email protected].

Author: Tang Liu

The original link: www.jianshu.com/p/b4dee8372…