www.infoq.cn/article/K7d…

Key value design

1. Design the key name

  • (1) [Suggestion] : readability and manageability

Prefixed with the business name (or database name) (to prevent key collisions) and separated by colons, such as business name: table name: ID

 Copy code 

ugc:video:1
Copy the code
  • (2) [Suggestion] : Simplicity

On the premise of ensuring semantics, the length of keys is controlled. When there are too many keys, the memory usage cannot be ignored. For example:

 Copy code 

User :{uid}:friends:messages:{mid} simplified to u:{uid}:fr:m:{mid}.Copy the code
  • (3) [Mandatory] : Do not contain special characters

Counter example: contains Spaces, line breaks, single and double quotation marks, and other escape characters

Detailed analysis

2. The design value

  • (1) [Mandatory] : reject bigKey (prevent network adapter traffic and slow query)

The number of hash, list, set, and zset elements should not exceed 5000.

Counterexample: a list of 2 million elements.

Use HSCAN, SSCAN, and ZSCAN to gradually delete non-string Bigkeys. At the same time, avoid automatic deletion of bigKey expiration time (for example, if a Zset of 2 million is set to expire in 1 hour, the DEL operation will be triggered. Blocks, and does not fail to occur in slow queries, find methods, and delete methods

Detailed analysis

  • (2) [Recommendation] : Select the appropriate data type.

For example: entity types (control and use data structure memory encoding to optimize configuration, such as Ziplist, but also pay attention to the balance between memory savings and performance)

Example:

 Copy code 

set user:1:name tomset user:1:age 19set user:1:favor football
Copy the code

Is:

 Copy code 

hmset user:1 name tom age 19 favor football
Copy the code

3. Control the life cycle of the key. Redis is not a garbage can.

It is recommended that you use expire to set the expiration time (if permitted, the expiration time can be scattered to prevent cluster expiration). For data that does not expire, focus on idletime.

2. Command usage

1. [Recommended] Run the O(N) command to check the number of N

Examples such as hgetall, lrange, smembers, zrange, and sinter are not unusable, but the value of N needs to be specified. Hscan, SSCAN, and ZSCAN can be used instead of traversal.

2. [Recommended] : Disable the command

Prohibit the use of keys, flushall, and flushdb online. Disable the command by using the Redis rename mechanism, or use scan to process the command increments.

3.【 recommended 】 Use select wisely

Redis multi-database is weak, using numbers to distinguish, many clients support poor, at the same time multi-business multi-database is actually single thread processing, there will be interference.

4. [Recommended] Use batch operations to improve efficiency

 Copy code 

Native commands: for example, mget and mset. Non-native commands: You can use pipelines to improve efficiency.Copy the code

However, it is important to control the number of elements in a batch operation (for example, up to 500, which also depends on the number of elements in bytes).

Note the difference:

 Copy code 

A native is an atomic operation; a pipeline is a non-atomic operation. Pipelines can package different commands; native pipelines require both client and server support.Copy the code

5. [Suggestion] Redis transaction function is weak, so it is not recommended to use it too much

Redis has weak transaction functionality (no rollback support), and the clustered version (self-developed and official) requires that the key for a transaction be in one slot (which can be resolved using hashtag).

6. [Suggestion] Redis cluster version has special requirements for using Lua:

  • Call /pcall/redis/keyarray/key array/key array/key array/key array/key array/key array/key array/key array/key array/key array/key array/key array/key array “-ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array”
  • 2. All keys must reside in one slot. Otherwise, error is returned, “-err eval/evalsha Command keys must in the same slot”.

7. [Suggestion] Do not use the monitor command for a long time if necessary.

Three, the use of the client

1. [‘ t? : n]

Avoid multiple applications using one Redis instance

Example: irrelevant business split, public data do servitization.

2. Age: 33

Use the database with connection pool, you can effectively control the connection, while improving efficiency, standard use:

 Copy code 

Run the following command: Jedis Jedis = null; try { jedis = jedisPool.getResource(); Jedis.executecommand ()} catch (Exception e) {logger.error("op key {} error: "+ LLDB message (), key, e); } finally {// Note that the connection is not closed. In JedisPool mode, Jedis is returned to the resource pool. if (jedis ! = null) jedis.close(); }Copy the code

The following is the article of JedisPool optimization method:

  • Summary of common Jedis exceptions
  • JedisPool Optimized the resource pool

3. [Suggestions]

In case of high concurrency, it is recommended that clients add the fuse breaker function (for example, Netflix Hystrix).

4. Age: 43

Set a reasonable password, if necessary can use SSL encryption access (Aliyun Redis support)

5. [Suggestions]

Select maxmemory-policy and set the expiration time based on the service type.

The default policy is volatile- LRU, that is, when the maximum memory is exceeded, the lRU algorithm is used to delete expired keys to ensure that the data will not be deleted. However, OOM problems may occur.

Other strategies are as follows:
  • Allkeys-lru: Deletes keys according to the LRU algorithm, regardless of whether the timeout attribute is set, until enough space is made.
  • Allkeys-random: randomly remove allkeys until enough space is left.
  • Volatile -random: Deletes expired keys randomly until sufficient space is available.
  • Volatile – TTL: Deletes the recently expired data based on the TTL attribute of the key value object. If not, retreat to Noeviction strategy.
  • Noeviction: Won’t exclude any data, reject all write operations and return client error message “(error) OOM command not allowed when used memory”, Redis will only respond to read operations.

Iv. Relevant tools

1. [Recommended] : Synchronize data

For data synchronization between Redis, run the redis-port command

2. [选 自 : BBC

Redis big Key search tool

3. [Recommendation] : Hot key search (internal implementation uses Monitor, so it is recommended to use it for a short time)

Facebook’s redis – faina

 Copy code 

Ali Cloud Redis has solved the hot key problem at the kernel level, welcome to use.Copy the code

Five Appendices: Delete bigKey

 Copy code 

1\. The following operations can be accelerated using pipeline. 2\. Redis 4.0 already supports asynchronous deletion of keys, welcome to use it.Copy the code

1. Delete Hash: HScan + hdel

 Copy code 

public void delBigHash(String host, int port, String password, String bigHashKey) { Jedis jedis = new Jedis(host, port); if (password ! = null && !" ".equals(password)) { jedis.auth(password); } ScanParams scanParams = new ScanParams().count(100); String cursor = "0"; do { ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams); List<Entry<String, String>> entryList = scanResult.getResult(); if (entryList ! = null && ! entryList.isEmpty()) { for (Entry<String, String> entry : entryList) { jedis.hdel(bigHashKey, entry.getKey()); } } cursor = scanResult.getStringCursor(); } while (!" 0".equals(cursor)); // Delete bigkey jedis.del(bigHashKey); }Copy the code

2. Delete List: ltrim

 Copy code 

public void delBigList(String host, int port, String password, String bigListKey) { Jedis jedis = new Jedis(host, port); if (password ! = null && !" ".equals(password)) { jedis.auth(password); } long llen = jedis.llen(bigListKey); int counter = 0; int left = 100; While (counter < llen) {// Cut 100 jedis.ltrim(bigListKey, left, llen); counter += left; } // Finally delete key jedis.del(bigListKey); }Copy the code

3. Set delete: ssCAN + srem

 Copy code 

public void delBigSet(String host, int port, String password, String bigSetKey) { Jedis jedis = new Jedis(host, port); if (password ! = null && !" ".equals(password)) { jedis.auth(password); } ScanParams scanParams = new ScanParams().count(100); String cursor = "0"; do { ScanResult<String> scanResult = jedis.sscan(bigSetKey, cursor, scanParams); List<String> memberList = scanResult.getResult(); if (memberList ! = null && ! memberList.isEmpty()) { for (String member : memberList) { jedis.srem(bigSetKey, member); } } cursor = scanResult.getStringCursor(); } while (!" 0".equals(cursor)); // Delete bigKey jedis.del(bigSetKey); }Copy the code

4. SortedSet delete: zscan + zrem

 Copy code 

public void delBigZset(String host, int port, String password, String bigZsetKey) { Jedis jedis = new Jedis(host, port); if (password ! = null && !" ".equals(password)) { jedis.auth(password); } ScanParams scanParams = new ScanParams().count(100); String cursor = "0"; do { ScanResult<Tuple> scanResult = jedis.zscan(bigZsetKey, cursor, scanParams); List<Tuple> tupleList = scanResult.getResult(); if (tupleList ! = null && ! tupleList.isEmpty()) { for (Tuple tuple : tupleList) { jedis.zrem(bigZsetKey, tuple.getElement()); } } cursor = scanResult.getStringCursor(); } while (!" 0".equals(cursor)); // Delete bigKey jedis.del(bigZsetKey); }Copy the code

Author’s brief introduction

Fu Lei, the principal of Kuaishou Cache, is responsible for tens of thousands of Redis guarantee and architecture related work of the company. Former Redis database technology expert of Aliyun. Published the technical book Redis Development and Operation and Maintenance, douban score 9.0. Open source Redis private cloud platform Cachecloud(Github Star 4000). Personal public account is “Redis development operation combat”, ID: redisDevops.