Hello everyone, this is the Java backend of Lebyte Education.

In this article I want to talk to you about Redis best practices.

Your project may have been using Redis for a long time, but you may still encounter some of the following problems:

• Why is my Redis memory growing so fast?

• Why is my Redis operation delayed?

• How to reduce the frequency of Redis failures?

• What should be paid attention to in daily operation and maintenance of Redis?

• How do you plan resources when deploying Redis?

• What indicators should Redis monitor focus on?

These questions become especially important as your project increasingly relies on Redis.

At this point, you desperately need a “best practice guide” **.

In this article, I will give you a “comprehensive” analysis of Redis best practice optimization from the following seven dimensions:

• memory

• performance

• Highly reliable

• Daily operations

• Resource planning (erp)

• monitoring

• security

At the end of this article, I’ll give you a complete list of best practices that will help you use Redis “gracefully,” whether you’re a business developer or DBA operations.

This article has a lot of dry material. I hope you can read it patiently.

How to use Redis to save more memory?

First, let’s look at Redis memory optimizations.

It is well known that the reason why Redis performs so well is that its data is stored in “memory”, so accessing data in Redis is extremely fast.

However, in terms of resource utilization, the memory resources of the machine are relatively expensive compared to disk.

When your business application stores very little data in Redis, you may not care much about memory resource usage. But as your business grows, your business stores more and more data in Redis.

If you don’t have a memory optimization strategy in place, the memory footprint of Redis will start to swell as the business starts to grow.

Therefore, it is necessary to develop a reasonable memory optimization strategy in advance to improve resource utilization.

So what can you do to save more memory when using Redis? Here are my six tips for you to follow:

1) Control the length of the key

The simplest and most direct way to optimize memory is to control the length of the key.

When developing a business, you need to estimate in advance the number of keys written to the entire Redis. If the number of keys is in the millions, then too long key names will take up too much memory.

Therefore, you need to keep the key definition as short as possible while keeping it simple and clear.

For example, if the original key is user:book:123, you can optimize it to u:bk:123.

In this way, your Redis can save a lot of memory. The optimization of memory is very direct and efficient.

2) Avoid storing bigkeys

In addition to controlling the length of the key, you also need to pay attention to the size of the value. If you store a large number of bigkeys, Redis memory will grow too fast.

In addition, clients have performance issues when reading or writing Bigkeys (more on this later).

So, if you want to avoid storing bigkeys in Redis, my advice to you is:

• String: the size must be smaller than 10KB

• List/Hash/Set/ZSet: The number of elements should be less than 10,000

3) Choose the right data type

Redis provides a rich set of data types that are implemented with optimized memory usage. Specifically, one data type corresponds to multiple data structures to achieve:

For example, String and Set store int data in integer encoding. Hash and ZSet are stored in a ziplist when the number of elements is small (configurable). When a large amount of data is stored, Hash and ZSet are converted to Hash and hop tables.

The reason for the author’s design is to further save memory resources.

You can then take advantage of these features to optimize Redis memory when storing data. My advice to you is as follows:

• String, Set: stores data of the int type as far as possible

• Hash and ZSet: The number of stored elements is lower than the conversion threshold to save memory in a compressed list

4) Use Redis as a cache

Redis data is stored in memory, which means its resources are limited. When you use Redis, use it as a cache, not as a database.

So, as much as possible, the data your application writes to Redis is set to expire.

When a service application fails to find data in Redis, it loads the data from the back-end database to Redis.

This allows Redis to keep only the frequently accessed “hot data”, and the memory utilization is high.

5) Instance set maxMemory + elimination policy

Even though your Redis keys are set to expire, if your business application is writing a lot and expires for a long time, the Redis memory will still grow rapidly over a short period of time.

If you do not control the memory limit of Redis, you can also use too much memory resources.

For this scenario, you need to estimate the amount of business data in advance, and then set the maxMemory control instance memory limit for this instance to prevent Redis from continuously ballooning memory.

When maxMemory is configured, you also need to set the data elimination strategy, and how to choose the elimination strategy depends on your business characteristics:

• Volatile -lru/allkeys-lru: The most recently accessed data is retained preferentially

• Volatile -lfu/allkeys-lfu: Retain the most frequently accessed data first (version 4.0+ support)

• Volatile – TTL: indicates that the data that is about to expire is preferentially eliminated

• Volatile -random/allkeys-random: randomly eliminates data

6) Write the data into Redis after compression

The above scheme basically covers all aspects of Redis memory optimization.

If you want to further optimize Redis memory, you can compress data in your business application and then write it to Redis (for example, using snappy, gzip, etc.).

Of course, compressing stored data, which the client needs to decompress while reading, consumes more CPU resources during this period, and you need to make a trade-off based on the actual situation.

The above is “save memory resources” in the practical optimization, is not relatively simple?

Let’s look at “performance” optimizations.

How can Redis continue to deliver high performance?

When you decide to introduce Redis to your system, there’s probably one thing that matters most: performance.

We know that a single Redis can achieve 10W QPS, which means that if there is a delay during use, it is not what we expected.

Therefore, when using Redis, how to continuously use its high performance and avoid operation latency is also our focus.

I’ve compiled 13 tips for you in this regard:

1) Avoid storing bigkeys

In addition to using too much memory, storing bigkeys can have a significant impact on Redis performance.

Since Redis processes requests in a single thread, when your application writes a Bigkey, more time is spent on “memory allocation”, which increases latency. Similarly, deleting a Bigkey takes time to “free up memory”.

Also, when you read the Bigkey, you will spend more time on “network data transfer”, which will queue up later requests and degrade Redis performance.

Therefore, your business application should try not to store bigkeys to avoid operation latency.

If you really need to store bigKeys, you can split bigKeys into smaller stores.

2) Enable lazy-free

If you can’t avoid storing bigkeys, I recommend turning on Redis’s lazy-free mechanism. (Version 4.0+ supported)

When this mechanism is enabled, Redis removes a bigkey, and the time-consuming operation of freeing memory is carried out in the background thread, thus minimizing the impact on the main thread.

3) Do not use excessively complex commands

Redis is a single-threaded model for processing requests, and in addition to bigkey operations causing subsequent requests to be queued, this can also happen when executing overly complex commands.

Because executing complex commands consumes more CPU resources, other requests in the main thread have to wait, which also causes queuing delays.

Therefore, you should avoid aggregative commands such as SORT, SINTER, SINTERSTORE, ZUNIONSTORE, ZINTERSTORE, etc.

For this aggregation-like operation, I suggest you put it on the client side and don’t let Redis do too much computing.

4) Pay attention to the size of N when running O(N)

Avoid using overly complex commands, and you can rest easy?

The answer is no.

When you execute O(N), you also need to pay attention to the size of N.

If too much data is queried at a time, network transmission takes a long time and the operation delay increases.

LRANGE key 0-1 / HGETALL/SMEMBERS/ZRANGE key 0-1 should not be foollessly executed for container types (List/Hash/Set/ZSet) with an unknown number of elements.

When querying data, you should follow the following principles:

1. Query the number of data elements (LLEN/HLEN/SCARD/ZCARD)

2. If the number of elements is small, you can query all data at a time

3. Element number very much, partial query data (LRANGE/HASCAN/SSCAN/ZSCAN)

5) Pay attention to DEL time complexity

You read that correctly, when deleting a key, if the posture is not correct, it may also affect Redis performance.

To delete a key, we usually use the DEL command. Recall, what do you think is the time complexity of DEL?

The O (1)? Not necessarily.

When you delete a String key, the time complexity is O(1).

But when you delete a List/Hash/Set/ZSet key, it’s O(N), where N is the number of elements.

If you delete a key, the more elements there are, the slower the DEL will be!

The reason is that when you delete a large number of elements, you need to reclaim memory for each one in turn, and the more elements you have, the longer it takes!

Also, this process is executed in the main thread by default, which can block the main thread and cause performance problems.

If you delete a key with a large number of elements, how do you handle it?

My advice to you is to delete in batches:

• List type: Perform LPOP/RPOP multiple times until all elements are removed

• Hash/Set/ZSet: Run HSCAN/SSCAN/SCAN to query elements, and then run HDEL/SREM/ZREM to delete each element in sequence

Didn’t expect that? A small delete operation can cause performance problems if you are not careful, and you need to be careful when doing this.

6) Batch commands replace single commands

When you need to manipulate multiple keys at once, you should use batch commands.

Compared with multiple single operations, batch operations can significantly reduce the number of I/OS going back and forth between the client and server.

So my advice to you is this:

• String/Hash uses MGET/MSET instead of GET/SET and HMGET/HMSET instead of HGET/HSET

• Other data types use pipelines, which package multiple commands to be executed on the server at once

7) Avoid concentration of expired keys

Redis cleans up stale keys in a timed + lazy manner, and this process is performed in the main thread.

If your business has a large number of expired key sets, Redis also runs the risk of blocking the main thread while cleaning expired keys.

To prevent this from happening, you can add a random time to the set expiration time to break up the expiration time of these keys, thus reducing the impact of the central expiration on the main thread.

8) Use long connection operation Redis, and properly configure connection pool

Your business should operate Redis with long connections and avoid short connections.

When a short connection is used to operate Redis, three TCP handshakes and four TCP waves are required each time, which also increases the operation time.

At the same time, your client should use connection pool to access Redis, and set reasonable parameters. If you do not operate Redis for a long time, you need to release connection resources in time.

9) Use db0 only

Although Redis provides 16 dB, I only recommend using DB0.

Why is that? I summarize the following three reasons:

1. When working with multiple DB data on a single connection, the need to perform SELECT first each time puts additional pressure on Redis

2. The purpose of having multiple dB’s is to store data along different lines of business, so why not split up multiple instance stores? Split multiple instance deployments so that multiple lines of business do not interfere with each other and improve Redis access performance

3. Redis Cluster only supports DB0. If you want to migrate to Redis Cluster later, the migration cost is high

10) Use read/write split + sharding cluster

If you have a large number of business read requests, you can deploy multiple slave libraries to achieve read and write separation so that the slave libraries of Redis share the read pressure and improve performance.

If your business has a high volume of write requests that a single Redis instance cannot support, then you need to use a sharded cluster to share the write load.

11) AOF is not enabled or AOF is configured to flush disks per second

If you are not sensitive to data loss, I suggest that you do not enable AOF to prevent THE performance of Redis from being slowed down by AOF writing disks.

If you do need to enable AOF, I recommend that you configure appendfsync everysec to put persistent flush operations into background threads to minimize the performance impact of Redis writing to disks.

12) Deploy Redis on physical machines

Redis uses the method of creating child processes to persist data.

Creating a child process invokes the operating system fork call, which takes time depending on the system environment.

Virtual machine environments take much longer to fork than physical machines, so your Redis should be deployed on physical machines whenever possible.

13) Disable the large memory page mechanism of the operating system

The Linux operating system provides a large memory page mechanism. Each time an application requests memory from the operating system, the application unit is 2MB instead of 4KB.

What problems does this cause?

When Redis persists data, a child process is forked, and the primary and child processes share the same memory address space.

When the main process needs to modify existing data, it copies On Write. During this process, the memory needs to be allocated again.

If the memory application unit is changed to 2MB, the memory application time will increase. If a large number of write operations are performed on the main process and the original data needs to be modified, the operation delay will increase during this period.

So, to avoid this problem, you need to turn off the memory page mechanism on the operating system.

Ok, so these are the practical optimizations for “high performance” in Redis. If you are very concerned about Redis performance, you can optimize it in combination with these aspects.

Let’s look at how Redis “reliability” is guaranteed.

How to ensure the reliability of Redis?

I would like to remind you that it is not difficult to ensure the reliability of Redis, but it is difficult to achieve “continuous stability”.

I’ll take you through the best practices for Redis reliability in three dimensions: resource isolation, multiple copies, and failover.

1) Deploy instances by line of business

The first step in improving reliability is resource isolation.

You are better off deploying Redis instances along different lines of business so that when one instance fails, it does not affect the rest of the business.

This kind of resource isolation program, the implementation cost is lowest, but the effect is very large.

2) Deploy the primary and secondary clusters

If you only use the standalone version of Redis, there is a risk that the machine will go down and the service will not be available.

Therefore, you need to deploy a “multi-replica” instance, that is, a master/slave cluster, so that when the master goes down, there are still slave libraries available, avoiding the risk of data loss and reducing the duration of service outages.

When deploying a master/slave cluster, you also need to be aware that the master/slave libraries need to be distributed on different machines to avoid cross-deployment.

The reason for doing this is that, normally, the primary library of Redis is responsible for all the read and write traffic, so we must prioritize the stability of the primary library and not affect the primary library even if the secondary machine fails.

In addition, when you need to perform routine maintenance on Redis, such as regular data backups, you can do it only on the slave server. This will only consume resources on the slave server and avoid the impact on the master server.

3) Properly configure primary/secondary replication parameters

When deploying a primary/secondary cluster, improper parameters may cause problems in the primary/secondary replication:

• The primary and secondary replication is interrupted

• A full replication is initiated from the secondary library, and the performance of the primary library is affected

My advice to you in this regard is as follows:

1. Set a reasonable Repl-backlog parameter. An excessively small Repl-backlog may cause the risk of full replication when the primary/secondary replication is interrupted in the case of large write traffic

2. Set a proper slave client-output-buffer-limit: When the slave library replication problem occurs, too small buffers will overflow the slave library buffer, resulting in replication interruption

4) Deploy sentinel cluster to realize automatic failover

Only the primary and secondary nodes are deployed, but they cannot automatically switch over when a failure occurs, so you also need to deploy a sentinel cluster for “automatic switching” when a failure occurs.

In addition, multiple sentinel nodes need to be distributed on different machines, with an odd number of instances, to prevent sentinel election failure and affect the switching time.

These are the optimizations to ensure Redis is “highly reliable” practices, and as you may have noticed, these are optimizations for the deployment and operations layers.

On top of that, you may be doing some “day-to-day operations” for Redis. What do you need to look out for?

What should be paid attention to in daily operation and maintenance of Redis?

If you are a DBA operations person, there are 6 things you should pay attention to when running Redis.

1) Disable KEYS/FLUSHALL/FLUSHDB

Executing these commands will block the Redis main thread for a long time, so you must disable it.

If you do want to use these commands, my advice to you is:

• SCAN for replacement KEYS

• Version 4.0+ uses FLUSHALL/FLUSHDB ASYNC, which flushes data in a background thread

2) Set the sleep time when scanning instances on the line

Whether you are using SCAN instances on the SCAN line or bigkey statistical analysis of instances, I recommend that you always set sleep time when scanning.

This prevents high instance OPS from causing performance jitter on Redis during scanning.

3) Run the MONITOR command with caution

Sometimes when you troubleshoot Redis problems, you will use MONITOR to see what commands Redis is executing.

However, if you have high Redis OPS, the Redis output buffer will continue to grow during the execution of MONITOR. This will severely consume Redis memory resources, and even cause instance memory to exceed maxmemory, causing data to be flushed out.

So you should be careful to use MONITOR as little as possible.

4) The slave library must be set to slave-read-only

Your slave library must be set to the slave-read-only state to prevent data from being written to the slave library, resulting in data inconsistency between the master and slave.

In addition, if the slave library is in a non-read -only state, if you are using Redis below 4.0, it has the following Bug:

Data with an expiration date is written from the library. Periodic cleanup and memory release are not performed.

This will cause a memory leak from the library! This issue is not fixed until version 4.0, so you need to be careful when configuring slave libraries.

5) Properly configure timeout and TCP-Keepalive parameters

If your Redis maxClients parameter is set to a small size, the client may not be able to establish new connections with the server.

This problem is caused by the fact that Redis assigns a Client FD to the client every time it establishes a connection with the server.

The server does not immediately release the client FD when there is a problem with the client/server network.

When will it be released?

Redis has a scheduled task that periodically checks whether the idle time of all clients exceeds the configured timeout value.

If Redis does not enable TCP-Keepalive, the server will not clean up and release the client FD until the configured timeout period.

If a large number of new connections are added before cleaning, the Redis server may hold more Client FDS than maxClients, and new connections will be rejected.

In this case, my optimization advice to you is as follows:

1. Do not configure timeout too high: let the server clean up invalid client FDS as soon as possible

2. Redis Tcp-Keepalive enabled: The server periodically sends TCP heartbeat packets to the client to check connection connectivity. When the network is abnormal, the zombie client FD can be cleared as soon as possible

6) When adjusting maxMemory, pay attention to the adjustment order of master and slave libraries

Redis versions below 5.0 have a problem with data flushing if the memory from the library exceeds maxMemory.

In some cases, the slave library may reach MaxMemory before the master library (for example, when the MONITOR command is executed in the slave library, the output buffer consumes a large amount of memory), and then the slave library starts to flush out data, resulting in inconsistencies between the master and slave libraries.

To avoid this problem, when adjusting MaxMemory, it is important to pay attention to the order in which the primary and secondary libraries are modified:

• Maxmemory: Modify the slave library first, then modify the master library

• Reduce maxMemory: modify the primary library first, then modify the secondary library

It was not until Redis 5.0 that Redis added a configuration replica-ignore-maxmemory, which by default does not weed out data when the secondary library exceeds maxmemory.

Ok, these are the “daily operation” Redis need to pay attention to, you can check the various configuration items missing, see what needs to be optimized.

Next, let’s take a look at what needs to be paid attention to to ensure Redis “security”.

How to guarantee Redis security?

In any case, in the Age of the Internet, security issues must be we need to keep alert.

You’ve probably heard of security issues with Redis being injected with executable scripts and then given root privileges on machines because you didn’t take security risks into account when deploying Redis.

My advice to you in this regard is:

1. Do not deploy Redis on servers accessible to the public network

2. The default port 6379 is not used in deployment

3. Start the Redis process as a common user

4. Restrict directory access to Redis profiles

5. You are advised to enable password authentication

6. Disable/rename dangerous commands (KEYS/FLUSHALL/FLUSHDB/CONFIG/EVAL)

As long as you get these things right, you can basically keep the security risks of Redis under control.

So far, we have analyzed Redis best practice optimizations for memory, performance, reliability, and day-to-day operations.

In addition to the above, you also need to “prevent” in advance.

How can Redis problems be prevented?

To prevent Redis problems in advance, you need to do two things:

1. Rational resource planning

2. Perfect monitoring and early warning

Let’s start with resource planning.

When deploying Redis, you can avoid many of the problems caused by insufficient resources if you plan ahead. My advice to you in this regard is as follows:

1. Ensure that the machine has sufficient CPU, memory, bandwidth, and disk resources

2. Capacity planning should be made in advance, and half of the memory resources should be reserved for the master library machine to prevent the memory shortage of the master library machine caused by the network failure of the master and slave machines

3. You are advised to limit the memory size of a single instance to less than 10 GB. Large instances may be blocked during primary/secondary full synchronization and RDB backup

Now let’s see how monitoring works.

Monitoring and early warning is an important link to improve stability. Perfect monitoring and early warning can expose problems in advance, so that we can react quickly and minimize problems.

My advice to you on this is:

1. Monitor the CPU, memory, bandwidth and disk of the machine, and timely alarm when resources are insufficient. Any insufficient resources will affect the performance of Redis

2. Set a reasonable slowlog threshold and monitor it, and alarm the excessive slowlog in time

3. When the monitoring component collects Redis INFO information, it uses long connections to avoid frequent short connections

4. Monitor the running time of instances, focusing on expired_keys, EVicTED_keys, and LATest_FORK_USEC indicators. These indicators may block when increased for a short time

conclusion

To sum up, this article takes you through a comprehensive analysis of Redis best practice optimization path, including memory resources, high performance, high reliability, daily operation and maintenance, resource planning, monitoring, security 7 dimensions.

Here I have drawn a mind map for you to refer to in practice.

This practice is optimized and further divided into two dimensions: business development and operation and maintenance.

It is marked at the “mandatory”, “recommended”, and “reference” levels, so that when you practice optimization, you will have a better understanding of what to do and what to analyze in the context of the actual business scenario.

The implementation rules for these levels are as follows:

• Mandatory: it must be strictly observed, otherwise it will do great harm

• Recommended: Compliance is recommended to improve performance, reduce memory, and facilitate o&M

• Reference: according to the business characteristics of reference implementation

If you are a business developer, you need to understand the operation mechanism of Redis, such as the execution time complexity of each command, data expiration strategy, data elimination strategy, etc., and use reasonable commands and optimize them according to business scenarios.

If you are a DBA, you need to be proactive in resource planning, operations, monitoring, and security.

Afterword.

If you have the patience to read this, you should have a new understanding of how to “use” Redis well.

This article focuses on Redis best practices, and I’d like to talk to you a little bit more about “best practices”.

If you are working with other middleware, such as MySQL or Kafka, instead of Redis, do you have any optimization ideas when using these components?

You can also use the following dimensions of this article:

• performance

• reliability

• resources

• operations

• monitoring

• security

You can think about what you need to pay attention to in these dimensions with MySQL and Kafka.

In addition, from a learning skills perspective, we should think and explore “best practices” as much as possible during software development.

Because only in this way can we constantly urge ourselves to think, put forward higher requirements for ourselves, and make continuous progress.

The article reprinted the joy byte

** Finally, I would like to recommend you some super detailed Java self-study courses of B station: **

MySQL database: BV1tK4y197JC

NodeJs project: BV1SK4y197G3 JavaScript complete tutorial: BV1yf4y1Y7oM