scenario

In the scenario of high concurrency of big data, using a single Redis instance will be very demanding, no matter how high the performance of Redis is.

First of all, the larger the amount of data, the larger the memory occupied by Redis, further resulting in the RDB file is too large, this situation will make the master/slave full synchronization time too long, at the same time, when the instance restarts, loading too large RDB will also make the startup time longer.

Secondly, in terms of CPU usage, a single instance of Redis can only use one CPU core, and a core should have too much data, which will also seem inadequate.

Therefore, a clustering solution is needed to spread the huge amount of data from one instance to multiple instances, between the popularity of Redis and the official support for their own clustering solution, and third parties are developing their own cluster-supporting components, such as Codis.

Codis is developed using Go language and acts as a proxy between Redis and client using Redis protocol. Therefore, the client can directly connect Codis and send instructions to it. Codis is responsible for forwarding instructions to Redis and finally receiving the returned results and returning them to the client.

The Redis instances of Codis agent constitute a Redis cluster. When the cluster space is insufficient, it can be dynamically expanded and continue to increase the Redis instances. At the same time, the SDK used by the client does not need to make any changes, just need to change the original connection Redis to Codis.

Codis itself can also adopt a cluster to ensure its high availability. Since it is stateless and only responsible for forwarding content, adding multiple Codis without side effects can also ensure the improvement of QPS. When one OF the Codis fails, others can also be used.

The principle of

Codis forwards specific keys to specific Redis instances, and each instance in the cluster holds a portion of the Key to reduce the pressure on other instances, while the data of all instances adds up to a complete piece of information.

By default, Codis is divided into 1024 slots. Each Redis instance in the cluster has a certain number of slots. Codis maintains the mapping between slots and Redis instances in memory.

The default number of slots is 1024 and can be changed. If the cluster has a large number of nodes, you can increase the number.

When receiving a key sent from the client, Codis performs a CRc32 operation on the key to produce a hash value.

Then mod the hash integer value to 1024(number of slots) to obtain a remainder, which is the slot to which the Key will be stored. With the remainder, you can find which Redis instance the Key should be sent to.

Pseudo code:

Hash = crc32(command.key) # Hash = hash % 1024 # redisInstance = slots[slot] Redis. Do (commandCopy the code

The cluster slots are synchronized

The mapping relationship between Redis and slots exists in the memory of Codis, so Codis cluster needs to consider to ensure the synchronization of slot mapping relationship in each node. Therefore, Zookeeper and Etcd distributed configuration storage middleware are adopted by Codis to persist slot mapping relationship and ensure data synchronization between Codis clusters.

As shown in the following figure, Codis stores slot relationships in Zookeeper and provides a Dashboard to observe and modify slot relationships. When changes occur, Codis Proxy monitors the changes and resynchronizes slot relationships.

Its capacity

If the existing cluster does not meet service requirements, you need to add new instances to the cluster. In this case, the slot mapping relationship needs to be reassigned to new nodes.

The SLOTSSCAN command is added to Codis to scan all the keys in the slot to be migrated, and then migrate each key to the new node one by one.

During the migration process, Codis continues to provide services externally. At this time, a request is sent to the slot that is being migrated. Since the slot now corresponds to the old and new nodes, Codis cannot determine whether the key is migrated from the old node to the new node.

In this case, Codis immediately forces a single migration of the current key, and when the migration is complete, the request is forwarded to the new Redis instance.

Pseudo code:

slot_index = crc32(command.key) % 1024
if slot_index in migrating_slots:
    doMigratingKey(command.key)
    redis = slots[slot_index].new_redis
else:
    redis = slots[slot_index].redis
Copy the code

SLOTSSCAN, like the Scan command of Redis itself, cannot avoid the duplication of scanned data, but this does not affect the correctness of migration, because a single key is deleted from the old instance immediately after migration and cannot be scanned again.

Automatic slot balancing

If it is too troublesome to manually maintain the slot mapping relationship each time a new instance is added, Codis provides automatic balancing. This function will observe the number of slots corresponding to each Redis instance when the system is relatively idle. If the number is unbalanced, it will automatically balance and migrate data.

disadvantages

Codis has brought expansion benefits to Redis, but it has also caused some side effects.

Transactions not supported

A transaction may operate on multiple keys, but transactions can only be done in a single instance, but because keys are scattered across different instances, Codis does not support transactions.

Do not support the rename

Rename names a key to another key, but the two keys may hash to different slots in different instances. Therefore, rename is not supported.

Official list of unsupported directives: github.com/CodisLabs/c…

Expansion CARDS,

In the process of capacity expansion, Codis directly migrates the whole key to the data, such as a hash structure. Codis will directly hGEtall all the content and use HMset to put it into the new node.

If the hash content is too large, it will cause congestion. It is recommended that the total size of a single hash structure not exceed 1MB. For service purposes, you can divide large data into multiple small ones by buckets to achieve a compromise.

Network overhead

Because Codis acts as a network Proxy between the client and the Redis instance, the network overhead is naturally higher and the performance is slightly lower than connecting directly to Redis.

Middleware o&M cost

The Codis cluster configuration requires the use of Zk or Etcd, which means that the introduction of Codis cluster also introduces other middleware, increasing the cost of operation and maintenance machine resources.

advantages

Codis offloads the issue of distributed consistency to a third party (ZK or Etcd), eliminating maintenance, reducing implementation code complexity,

In order to achieve decentralization, The official Cluster of Redis introduced Raft and Gossip protocols, as well as a large number of configuration parameters that need to be tuned.

Batch to obtain

For batch operations, such as using MGET to get the values of multiple keys that may be scattered across multiple instances, Codis groups the keys by the instance they are in, calls MGET for each instance, and returns the sum to the client.

Other features

Codis provides Dashboard interfaces and CODIS-Fe to manage clusters, add groups, add nodes, perform automatic balancing, and view slot status and corresponding Redis instances. These functions make O&M easier.

To the history

Codis was created to compensate for the fact that Redis did not officially provide the concept of clustering. Now Redis officially provides the function of clustering, and official support naturally has advantages over third-party support.

At the same time, third-party software also need to pay attention to the official release of new features, and Cluster is certainly compatible with new features in real time, so it is more recommended to use the official Cluster, Codis as a knowledge point to understand, some ideas and Cluster are overlapping.