All the data in Redis is stored in memory, which is getting cheaper now, but the price of a hard disk is a fraction of that of a hard disk installed on a normal computer. Memory is still very precious, take my Tencent cloud server for example, is currently 1 core 2G, but to upgrade to 4G, you need to pay more than 1000 dollars. It feels like I could buy a 1TB hard drive for that much money… That’s the gap. So, how to make reasonable and efficient use of Redis memory becomes very important. First, we should know where Redis memory is mainly consumed, then how to manage memory, and finally how to do memory optimization of Redis. In this way, you can store more data with less memory and lower costs.

1. Memory consumption

How to check memory consumption in Redis? You can use the info command to view Redis memory consumption indicators to better analyze memory. After executing the command, there are several important indicators:

The property name Attributes that
used_memory The amount of memory allocated by the Redis allocator is the amount of memory used by all data stored in Redis
used_memory_human Return in a readable formuser_memory
used_memory_rss Total physical memory occupied by Redis processes
used_memory_peak used_memoryPeak usage
used_memory_peak_human Readable format returnsused_memory_peak
used_memory_lua The amount of memory consumed by the Lua engine
mem_fragmentation_ratio used_memory_rss/used_memoryRatio, memory fragmentation rate
mem_allocator The memory allocator used by Redis, jemalloc by default 
Used_memory: 1038744104 used_memory_human: 990.62 M used_memory_rss: 5811122176 used_memory_peak: 4563077088 Used_memory_peak_human: 4.25 G used_memory_lua: 35840 mem_fragmentation_ratio: 5.59 mem_allocator: libcCopy the code

Note the mem_fragmentation_ratio value:

Mem_fragmentation_ratio > 1 indicates that the extra names are not used for data storage but are consumed by memory fragmentation. The larger the difference is, the more serious the memory fragmentation rate is. Mem_fragmentation_ratio < 1 occurs when Redis swaps memory to hard disk (used_memory > when the maximum memory available, Redis writes old and invalid data to hard disk). This situation requires special attention, the hard disk speed is much slower than memory, Redis performance will become poor, even dead.

1.1. Division of memory consumption

Redis memory mainly includes: object memory + buffer memory + own memory + memory fragment.

1. Object memory

Object memory is the largest part of Redis memory, which stores all user data. All data in Redis uses key-value data type. Each time a key-value pair is created, two objects, key object and value object, are created. Key objects are strings, value objects are stored in five types of data – String, List, Hash, Set, and Zset. Each storage mode occupies different memory due to its different length and data type.

2. Buffer memory

Mainly includes: the client buffer, replication backlog buffer, AOF buffer client buffer: ordinary client connection (a large number of connections), from the client (mainly copy, long distance across the room, or have more than one from the nodes under the master node), subscribe to the client (release subscription function, production is greater than the consumer can cause backlog) replication backlog buffer: The reusable fixed size buffer provided after version 2.8 is used for partial replication. The default is 1MB, mainly for master/slave synchronization. AOF buffer: Persistent, written to the buffer and then synchronized to disk according to the response policy. Memory consumption is usually small, depending on the number of commands written and rewriting time.

3. Memory fragmentation

The current alternative allocators are Jemalloc, glibc, tcMALloc. By default, jemalloc has a high memory fragmentation problem: a large number of update operations such as Append and Setrange; A large number of expired keys are removed and the space freed cannot be used effectively. Solutions: data alignment, safe restart (high availability/master/slave switchover).

4. Own memory

It mainly refers to the memory consumption of the child process created by Redis during AOF/RDB rewrite. Linux has copy-on-write technology, in which the father and child processes share the same physical memory page. When the parent process writes a request, it will make a copy of the page to be modified to complete the write operation.

2. Manage memory

Redis defaults to unlimited memory usage. Therefore, when using maxMemory, try to configure maxMemory, set the upper limit of Redis memory, to prevent the system from running out of memory due to the unlimited use of Redis. It is important to note that maxMemory is configured with the amount of memory actually used by Redis, usED_memory. Due to memory fragmentation, the actual memory usage is larger than usED_memory. Redis can perform memory adjustments dynamically:

config set maxmemory 6GBCopy the code

Configuring a memory reclamation policy The memory reclamation mechanism of Redis is mainly reflected in the following two aspects:

Processing of expired data Triggers a reclaiming policy when memory usage reaches maxMemory

     1. Delete the expiration key

Lazy delete: When will it be executed? When a client reads a key with a timeout attribute, if the expiration time set by the key value is exceeded, it will be deleted and return null. The main purpose of this is to save CPU cost, there is no need to maintain a separate TTL linked list to deal with the deletion of expired keys. However, there is a problem with using this approach alone. What if the current key value is never accessed again? You’re not going to delete it? No, that would cause a memory leak. So how does Redis solve this? Redis provides a deletion mechanism for scheduled tasks to complement this.

    

     2. Delete the scheduled task

Redis maintains a scheduled task internally, which runs ten times per second by default. Delete logic as shown below:



    

     Memory overflow control policy

Redis memory usage reaches its upper limitmaxmemoryAfter, will be based onmaxmemory-policyRedis supports the following 6 policies:

strategy instructions
noeviction Default policy that does not delete any data, rejects all write operations and returns client error messages(Error) OOM Command not allowed when used memory
volatile-lru Only the keys with the timeout attribute are deleted based on the LRU algorithm. If there are no keys that can be deleted, the system rolls back tonoevictionstrategy
allkeys-lru For all keys, delete according to the LRU algorithm until sufficient memory is reclaimed
allkeys-random Delete all keys at random until enough space is left
volatile-random For keys with expired attributes, delete them until enough space is available
volatile-ttl Deletes recently expired data based on the TTL property of the key-value object. If not, fall back to the Noviction strategy
3. Memory optimization

     Hashtable

    

Redis all data stores are key-value data types. The overall structure is the hashtable implemented by Redis itself. There are two Hashtables in Redis, but only one of them is used to store data, and the other one is used for expansion. Copy the data from the old Hashtable into another hashtable in a progressive fashion. Why do you do it incrementally? Because Redis is single-threaded, data migration is time consuming, so the migration process should not affect other uses of Redis. So it’s incremental.



Therefore, the structure of hashtable becomes very important. The design of Hashtable is realized in the way of array and linked list. One-dimensional array is an array structure, and two-dimensional is a linked list structure, and the one-dimensional array stores a pointer to the first data in the list.

Struct dictht {dictEntry** table; // 2d long size; // The length of the array long used; Struct dictEntry {void* key; struct dictEntry {void* key; void* val; dictEntry* next; // Link next entry}Copy the code

     RedisObect object

All value objects in Redis are internally defined as redisObject structures. The structure is as follows:

struct RedisObject { int4 type; // 4bits int4 encoding; // 4bits int24 lru; // 24bits int32 refcount; // 4bytes void *ptr; // 8bytes, 64-bit system} robj; .Copy the code

Each field is described in detail below:

The field name instructions
type The type of the object to store. Redis has 5 data types:String.List.Hash.Set.Zset, can be accessed throughtype {key}Command to check the type of the object, return the value object type, all key objects areStringtype
encoding The internal encoding format used in Redis for data storage will be discussed later
lru Records the last time the object was accessed, when configuredmaxmemoryAfter that, the LRU algorithm is used to delete the relevant key value, which can be passedobject idletime {key}Check the time when the key was accessed last time. Also throughscan + object idletimeCommand to batch query keys that have not been used for a long time to delete keys that have not been used for a long time, reducing memory usage.
refcount Records the number of times the current object is referenced. The object can be recycled according to the current field. When the refCount is 0, the object can be safely recycled and can be usedobject refcount {key}View the current object reference.
*ptr Relates to the data content of the object. If the object is an integer and ranges from 0 to 9999, the data is stored directly. If the object is an integer and ranges from 0 to 9999, the data is stored directly.

     Simple Dynamic String (SDS)

    

In Redis, string objects are often used, for example, to execute a list command,lpush queue "redis" "list" "queue", will be created firstqueueKey string, and then create a linked list object that contains three string objects internally. Several other structures store data without strings. The structure of the string in Redis is also the structure defined by Redis itself. The structure diagram is as follows:

struct SDS { int8 capacity; // 1byte int8 len; // 1byte int8 flags; // 1byte byte[] content; // Inline array with length of capacity}...Copy the code

SDS has several features:

The time complexity is O(1), because there is known length, unknown length, string length support safe binary data storage, used to save byte array internal space pre-allocation mechanism, reduce the number of memory redistribution lazy deletion mechanism, the space after the string reduction is not released, as and allocated space reserved

Therefore, minimize the number of string appending operations to avoid memory reallocation and increase the memory fragmentation rate. Instead, try to use direct insertion. The preallocation of strings does not double the capacity each time. The preallocation rules are as follows:

  • The len attribute is equal to the actual size of the data, free is equal to 0, and no preallocation is made.
  • After the modification, if the available free space is insufficient and the data size is less than 1MB, the storage capacity is doubled each time.
  • After the modification, if the existing free space is insufficient and the data size is greater than 1MB, pre-allocate 1MB of data space each time. If the amount of data is too large to double, it is likely to cause insufficient memory. All data larger than 1MB is allocated 1MB space each time, which is also based on this reason.

Here is a simple example, such as setting the value of a simple point in Redis. The structure of Redis memory is as follows (see sample) :



     Code optimization

Redis has 5 different storage formats:String.List.Hash.Set.Zset, but different types will have different implementation of the underlying data structure when stored in Redis. Similar to the collectionArrayListThe underlying implementation is array, whileLinkedlistThe underlying implementation is the same as the linked list structure, the encoding is different, the memory is occupied and the efficiency of reading and writing is also different. The following is a list of different encodings for different Redis constructs:



     embstrwithrawThe difference between

Before we talk about the differences between the two encoding formats, there is a cache structure between the CPU and memory in modern computers, which is used to balance the efficiency of the CPU with the relatively slow fetch. L1 Cache, L2 Cache, and L3 Cache. When the CPU wants to access the memory, it will first look in the cache to see if there is any, if not, it will go to the memory to find, and then put it in the cache. The minimum unit of the cache is usually 64 bytes. The minimum unit of the cache is called 64 consecutive bytesCache line.

In Redis, everyvalueObjects all have oneredisObjectObject header, for the Redis string object, gets when reading data*ptrPointer, and then find the SDS object to point to. If the object is far away, it will affect the efficiency of Redis reading. Therefore, a special coding structure is designed in Redis, which isembstr, itredisObjectThe request header and the SDS object are close together, and then the whole is placed in a cache line, so that the relevant data can be directly obtained from the cache during the query, improving the efficiency of the query.



As mentioned above, a cache line is 64 bytes long. If you want to store an object in a cache line, you must first limit the overall length to 64 bytes per request headerredisObjectTake up 16 bytes, while SDS has at least 3 of its own occupied, and Redis will be with\ 0So the length of data left for input becomes (64-16-3-1) =44 bytes. When the length of data stored exceeds 44 bytes, it becomesrawThe encoding form of.

That’s all for this blog post, which is currently under research and will be updated in the future…