The info command can display a lot of information about the Redis server, including basic server information, CPU, memory, persistence, client connection information and so on. Memory is a parameter that displays only memory-related information.

Some important notes in the returned results are as follows:

(1) USED_memory: the total amount of memory (in bytes) allocated by the Redis allocator, including the virtual memory used (i.e. Swap); The Redis allocator is described later. Used_memory_human is just more user-friendly.

Used_memory_rss: the amount of memory (in bytes) occupied by the Redis process, which is consistent with the value seen by the top and ps commands; In addition to the memory allocated by the allocator, usED_memory_RSS includes memory needed by the process itself to run, memory fragments, and so on, but not virtual memory.

Thus, usED_memory and USED_memory_rss, the former from the Redis perspective, and the latter from the operating system perspective. The difference between the two is that on the one hand, memory fragmentation and Redis process running require memory, making the former may be smaller than the latter; on the other hand, the existence of virtual memory makes the former may be larger than the latter.

Due to the large amount of Redis data in practical applications, the memory occupied by the process running at this time is much smaller than the amount of Redis data and memory fragments. Therefore, the ratio of USED_memory_RSS to USED_memory becomes a parameter to measure the memory fragmentation rate of Redis. This parameter is mem_fragmentation_ratio.

(3) mem_fragmentation_ratio: memory fragmentation ratio. This value is the ratio of USED_memory_rss to USED_memory.

Mem_fragmentation_ratio is generally greater than 1, and a larger value indicates a larger proportion of memory fragments. Mem_fragmentation_ratio <1 indicates that Redis uses virtual memory. The media of virtual memory is disk, and the speed of virtual memory is much slower than that of memory. If virtual memory is insufficient, check it in time. Such as adding Redis nodes, increasing the memory of Redis servers, optimizing applications, etc.

In general, mem_fragmentation_ratio around 1.03 is a healthy state (for Jemalloc); The mem_fragmentation_ratio value in the screenshot above is large because no data has been stored to Redis, and the Redis process itself runs in memory that makes USED_memory_rss much larger than USED_memory.

(4) mem_allocator: memory allocator used by Redis, specified at compile time; It can be libc, jemalloc, or tcMALloc. The default is jemalloc. The default Jemalloc is used in the screenshot.

As a database, data is the most important part; This portion of memory is counted in usED_memory.

Redis uses key-value pairs to store data, where values (objects) are of five types: string, hash, list, set, and ordered set. These five types are provided externally by Redis. In fact, within Redis, each type may have two or more internal encoding implementations; In addition, when Redis stores objects, it does not directly throw data into memory, but packs objects in various ways, such as redisObject, SDS, etc. The details of data storage in Redis will be highlighted later in this article.

Buffer memory includes client buffer, replication backlog buffer, AOF buffer, etc. Where, the client buffer stores the input/output buffer of the client connection. The replication backlog buffer is used for partial replication functions; The AOF buffer is used to hold the most recent write command during AOF rewrite. You don’t need to know the details of these buffering until you understand the functionality; This portion of memory is allocated by Jemalloc and therefore counted in USED_memory.

Redis specifies the memory allocator at compile time; The memory allocator can be LIBC, JEMalloc, or TCMALloc, and the default is jemalloc.

Jemalloc, the default memory allocator for Redis, does a relatively good job of reducing memory fragmentation. In the 64-bit system, jemalloc divides the memory space into three ranges: small, large and huge. Each range is divided into many small memory block units; When Redis stores data, it selects the most appropriate memory block for storage.

The memory units divided by Jemalloc are as follows:

Image source: http://blog.csdn.net/zhengpeitao/article/details/76573053

For example, if you need to store a 130-byte object, Jemalloc puts it into a 160-byte memory cell.

Each field of a redisObject has the following meanings and functions:

(1) type

The type field indicates the type of the object, which is four bits long. Currently, the value can be REDIS_STRING, REDIS_LIST, REDIS_HASH, REDIS_SET, or REDIS_ZSET.

When we execute the type command, we get the type of the object by reading the type field of the RedisObject. As shown below:

(2) encoding

Encoding Indicates the internal encoding of the object, which is four bits.

For each type supported by Redis, there are at least two internal encodings, such as int, EMbstr, and RAW for strings. With the Encoding attribute, Redis can set different encoders for different usage scenarios, making Redis much more flexible and efficient. Taking the list object as an example, there are two encoding methods: compressed list and double-endian linked list. If there are fewer elements in a list, Redis tends to use compressed lists for storage, because compressed lists take up less memory and can load faster than double-ended lists. When the list object has a large number of elements, the compressed list is transformed into a double-ended list that is better suited for storing a large number of elements.

Using the object encoding command, you can view the encoding mode of the object, as shown in the following figure:

The encoding methods and application conditions corresponding to the five object types will be introduced later.

(3) the lru

Lru records the last time an object was accessed by a command program, and the number of bits it occupies varies according to versions (for example, 24 bits in version 4.0, 22 bits in version 2.6).

The idling time of an object can be calculated by comparing the LRU time with the current time. The object idletime command displays the idling time (in seconds). The object idletime command is special in that it does not change the LRU value of the object.

In addition to printing the lRU value through the object idletime command, it is also related to Redis memory reclamation: If MaxMemory is enabled and Redis uses volatile- LRU or Allkeys-LRU, Redis will preferentially free the object with the longest idle time when Redis memory usage exceeds the value specified by maxMemory.

(4) refcount

Refcount and shared objects

Refcount counts the number of times the object is referenced. The type is an integer. Refcount is used for reference counting and memory reclamation of objects. When a new object is created, refCount is initialized to 1; Refcount increases by 1 when a new program uses the object; Refcount is reduced by 1 when the object is no longer in use by a new program; When refCount goes to 0, the memory occupied by the object is freed.

Objects that are used multiple times in Redis (refcount>1) are called shared objects. Redis to save memory, new programs do not create new objects when some objects appear repeatedly, but still use the old objects. The object that is reused is a shared object. Currently, shared objects only support string objects with integer values.

Concrete implementation of shared objects

Redis’s shared objects currently only support string objects with integer values. This is actually a balance between memory and CPU (time) : sharing objects reduces memory consumption, but it takes extra time to determine whether two objects are equal. For integer values, the complexity of operation is judged to be O(1). For ordinary strings, the judgment complexity is O(n). For hashes, lists, sets, and ordered sets, the judgment complexity is O(n^2).

Although shared objects can only be string objects with integer values, they can be used by all five types (such as elements of hashes, lists, and so on).

For the current implementation, Redis server initialization, will create 10000 string objects, values are 0 to 9999 integer values; These shared objects can be used directly when Redis needs to use string objects with values from 0 to 9999. The number 10000 can be changed by adjusting the value of the parameter REDIS_SHARED_INTEGERS (in 4.0, OBJ_SHARED_INTEGERS).

You can run the object refcount command to view the number of references to the shared object, as shown in the following figure. The result page confirms that only integers between 0 and 9999 will be used as shared objects.

(5) the PTR

The PTR pointer points to concrete data, such as in the previous example, set Hello World, where the PTR points to SDS containing the string world.

(6) Summary

To sum up, the structure of a redisObject is related to object type, encoding, memory reclamation, and shared objects. A redisObject is 16 bytes in size:

24 bit bit 4 + 4 bit + + 4 byte + 8 byte = 16 byte.

Where buf stands for byte array, which is used to store strings. Len indicates the used length of BUF, and free indicates the unused length of BUF. Here are two examples.

Image credit: Redis Design & Implementation

As can be seen from the structure of SDS, the length of buF array =free+len+1 (where 1 represents the null character at the end of the string); So, the space occupied by an SDS structure is: the length of free+ the length of len+ the length of buF array =4+4+ Free +len+1=free+len+9.

(2) Comparison between SDS and C string

SDS adds free and len fields to the C string, bringing many benefits:

• Get string length: SDS is O(1), C string is O(n)
• Buffer overflow: When using the C string API, it is easy to overflow the buffer if the string length increases (such as strcat operations) and you forget to reallocate memory. SDS records the length, and the corresponding API automatically reallocates memory when it may cause buffer overflows, eliminating buffer overflows.
• Reallocation of memory when changing strings: For C strings, if you want to change the string, you must reallocate memory (free and then requisition), because without reallocation, increasing the string length will cause memory buffer overflow and decreasing the string length will cause memory leak. As for SDS, len and free can be recorded, so the association between string length and spatial array length is removed, and optimization can be carried out on this basis: the space pre-allocation strategy (that is, allocating more memory than is actually needed) greatly reduces the probability of reallocating memory when the string length increases; The lazy space free strategy greatly reduces the probability of reallocating memory when the string length decreases.
• Accessing binary data: SDS does, C strings do not. Because the C string is marked by a null character as the end of the string, and for some binary files (such as pictures, etc.), the content may contain an empty string, so the C string cannot be accessed correctly. SDS uses len as the end of the string, so there is no such problem.

In addition, because BUF in SDS still uses C strings (that is, ending with ‘\0’), SDS can use some of the functions in the C string library; Note, however, that this is only possible when SDS is used to store textual data, not binary data (‘ \0 ‘is not necessarily the ending).

(3) SDS and C string application

Redis uniformly uses SDS instead of C strings when storing objects. The set Hello world command, for example, stores hello and world as SDS. The sadd mySet member1 member2 member3 command, both the key (” myset “) and the elements in the collection (” member1 “, “member2” and “member3”), are stored as SDS. In addition to storing objects, SDS is also used to store various buffers.

The C string is used only when the string does not change, such as when printing logs.

(1) Overview

Strings are the most basic type, because all keys are strings, and elements of several other complex types are strings.

The value cannot exceed 512MB.

(2) Internal coding

There are three internal encodings of string types, which can be used in the following scenarios:

• Int: a long integer of 8 bytes. When the string value is an integer, the value is represented by the long integer.
• Embstr: a string of <=39 bytes. Both EMbSTR and RAW use redisObject and SDS to store data. The difference is that the use of EMBSTR allocates memory space only once (thus redisObject and SDS are contiguous), whereas raw allocates memory space twice (redisObject and SDS, respectively). So compared to raw, embSTR has the advantage of allocating less space on creation, freeing less space on deletion, and having all the data of an object together for ease of finding. The disadvantages of embstr are also obvious. If the length of the string increases and memory needs to be reallocated, the entire redisObject and SDS need to be reallocated, so embSTR in Redis is implemented as read-only.
• Raw: The string contains more than 39 bytes

An example is shown below:

The length of embstr from raw is 39; Because redisObject is 16 bytes long, SDS is 9+ string length; So when the string length is 39, embstr is exactly 16+9+39=64, and Jemalloc can allocate exactly 64 bytes of memory.

(3) Coding conversion

When int data is no longer an integer, or the size exceeds long’s range, it is automatically converted to RAW.

In the case of embstr, since its implementation is read-only, any modification to an EMbstr object is converted to RAW before it is modified. Therefore, whenever an EMbstr object is modified, the modified object must be raw, regardless of whether it reaches 39 bytes. An example is shown below:

(1) Overview

Hash (as a data structure) is not only one of the five object types provided by Redis (along with strings, lists, collections, and ordered combinations), but also the data structure used by Redis as a key-value database. For the sake of illustration, when “inner hash” is used later in this article, it represents one of the five object types provided externally by Redis; The use of “outer hash” refers to the data structure Redis uses as a key-value database.

(2) Internal coding

The internal encoding used for inner hashing can be ziplist and Hashtable. The outer hashes of Redis only use HashTable.

The compressed list was introduced earlier. Compared with the hash table, the compressed list is used for the scene with a small number of elements and small length. Its advantage lies in centralized storage, saving space; Meanwhile, although the operation complexity of elements has changed from O(n) to O(1), the operation time is not significantly inferior due to the small number of elements in the hash.

Hashtable: A Hashtable consists of one dict structure, two Dictht structures, one dictEntry pointer array (called bucket), and multiple dictEntry structures.

Under normal circumstances (that is, when the Hashtable is not rehash), the relationship between the parts is as follows:

Adapted from: Redis Design and Implementation

The following sections are introduced from the bottom up:

dictEntry

The dictEntry structure is used to hold key-value pairs. The structure is defined as follows:

?

The functions of each attribute are as follows:

• Key: a key in a key-value pair;
• Val: a value in a key-value pair, implemented using a union(common body) that stores either a pointer to a value, a 64-bit integer, or an unsigned 64-bit integer;
• Next: points to the next dictEntry, which is used to resolve hash conflicts

On 64-bit systems, a dictEntry object accounts for 24 bytes (key/val/next each accounts for 8 bytes).

bucket

A bucket is an array, and each element of the array is a pointer to the dictEntry structure. The size of the bucket array in Redis is calculated as follows: the smallest 2^n larger than dictEntry; For example, if there are 1000 dictentries, the bucket size is 1024; If there are 1500 dictentries, the bucket size is 2048.

dictht

Dictht structure is as follows:

?

The function description of each attribute is as follows:

• The table property is a pointer to a bucket;
• The size attribute records the size of the hash table, that is, the size of the bucket;
• Used records the number of dictentries used;
• The sizemask attribute is always size-1. This attribute, along with the hash value, determines where a key is stored in the table.

dict

In general, the function of ordinary hash table can be realized by using dictht and dictEntry structures. But the Redis implementation also has a dict structure on top of the Dictht structure. The following explains the definition and role of the dict structure.

Dict has the following structure:

?

The Type and privData attributes are designed to accommodate different types of key-value pairs and are used to create polymorphic dictionaries.

The HT and treHashidx attributes are used for rehash, that is, when the hash table needs to be expanded or shrunk. Ht is an array of two items, each pointing to a dictht structure, which is why Redis hash has 1 dict and 2 Dictht structures. Normally, all data is stored in dict ht[0], and HT [1] is only used for rehash. When the dict rehash is performed, all the data in HT [0] is rehash to HT [1]. Then assign HT [1] to HT [0] and empty HT [1].

Therefore, hashing in Redis has a dict structure in addition to dictht and dictEntry, partly to accommodate different types of key-value pairs, and partly for rehash.

(3) Coding conversion

As mentioned earlier, the inner hash in Redis may use either a hash table or a compressed list.

A compressed list is used only if the following two conditions are met: the number of elements in the hash is less than 512; All key-value pairs in the hash have key and value strings of less than 64 bytes. If one condition is not met, hash tables are used; And encoding can only be converted from a compressed list to a hash table, not the other way around.

The following figure shows the characteristics of the Redis inner hash encoding transformation:

Where encoding stands for the type of contents stored. Although contents is int8_T, it actually stores the value int16_t, INT32_t, or int64_t, depending on encoding. Length indicates the number of elements.

The integer set is suitable for all the elements of the set are integers and the number of elements in the set is small. Compared with the hash table, the integer set has the advantage of centralized storage and space saving. Meanwhile, although the operation complexity of elements has changed from O(n) to O(1), the operation time is not significantly inferior due to the small number of sets.

(3) Coding conversion

A collection of integers is used only if both conditions are met: the number of elements in the collection is less than 512; All elements of the set are integer values. If one condition is not met, hash tables are used; And the encoding can only be converted from a set of integers to a hash table, not the other way around.

The following figure shows the characteristics of the collection encoding transformation:

Now that you know about Redis’s memory model, here are a few examples to illustrate its application.

Running result: 8247552

The error between the theoretical value and the resulting value is 1.2 parts per million, which is good enough for calculating how much memory is needed. The error is due to the fact that Redis allocated some bucket space before we inserted 90,000 pieces of data, and that bucket space was not yet used.

As a comparison, if the length of key and value is increased from 7 bytes to 8 bytes, the corresponding SDS becomes 17 bytes and Jemalloc will allocate 32 bytes, so the number of bytes occupied by each dictEntry also changes from 80 bytes to 112 bytes. In this case, the memory occupied by the 90,000 key-value pairs is estimated to be 90000*112 + 131072*8 = 11128576.

Verify the code in Redis as follows (only modify the code that inserts data) :

?

Running results: 11128576; Accurate estimation.

For other types other than string types, the estimation of memory usage is similar and needs to be combined with the specific encoding method.

Memory fragmentation rate is an important parameter, which is of great significance to redis memory optimization.

If the memory fragmentation rate is too high (jemalloc is normal around 1.03), it indicates that there are many memory fragments and serious memory waste. In this case, you can consider restarting the Redis service to rearrange data in memory to reduce memory fragmentation.

If the memory fragmentation rate is less than 1, redis has insufficient memory and some data uses virtual memory (swap). Since virtual memory is much slower to access than physical memory (2-3 orders of magnitude), redis can be slow to access. Therefore, you must try to increase physical memory (by increasing the number of server nodes, or by increasing the memory on a single machine), or reduce the data in Redis.

In order to reduce the data in Redis, in addition to selecting appropriate data types and using shared objects, a reasonable maxmemory-policy should be set. When the memory reaches a certain amount, the memory will be reclaimed according to different priorities.