An overview of the

We usually use Redis at the user level, we may operate a key-value pair without thinking to conveniently access data, feel very convenient. But do you know how that data is stored and encoded behind the scenes? Understanding this will guide us to use Redis more effectively. At the beginning of this article, we will combine Redis source code to explore the internal coding mechanism of Redis five data types one by one.

  • Experimental environment: Redis 4.0.10

Note: This article was published on My public account CodeSheep. You can subscribe by holding down or scanning the heart below ↓ ↓ ↓

Redis data type internal encoding overview

For the five commonly used Redis data types (String, Hash, List, Set, sorted Set), each data type provides at least two internal encoding formats, and the choice of the internal encoding mode of each data type is completely transparent to users. Redis adaptively selects a better internal encoding format based on the amount of data.

To see the internal ENCODING of a key, use the OBJECT ENCODING keyname directive, for example:

127.0.0.1:6379 > 127.0.0.1:6379 >setFoo bar OK 127.0.0.1:6379> 127.0.0.1:6379> object encoding foo // Check the encoding of a Redis key"embstr"127.0.0.1:6379 > 127.0.0.1:6379 >Copy the code

Each key value of Redis is stored in a C language structure named redisObject. The code is as follows:

The explanation is as follows:

  • type: Indicates the data type of the key value, including String, List, Set, ZSet, and Hash
  • encoding: indicates the internal encoding mode of the key value, from Redis source code, there are several values as follows:
#define OBJ_ENCODING_RAW 0 /* Raw representation */
#define OBJ_ENCODING_INT 1 /* Encoded as integer */
#define OBJ_ENCODING_HT 2 /* Encoded as hash table */
#define OBJ_ENCODING_ZIPMAP 3 /* Encoded as zipmap */
#define OBJ_ENCODING_LINKEDLIST 4 /* No longer used: old list encoding. */
#define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */
#define OBJ_ENCODING_INTSET 6 /* Encoded as intset */
#define OBJ_ENCODING_SKIPLIST 7 /* Encoded as skiplist */
#define OBJ_ENCODING_EMBSTR 8 /* Embedded sds string encoding */
#define OBJ_ENCODING_QUICKLIST 9 /* Encoded as linked list of ziplists */
Copy the code
  • refcount: indicates the number of keys referenced. That is, a key can be referenced by multiple keys

This article we from Redis the most basic String type of internal coding to explore!

Internal encoding of String

String is the most basic data type of Redis. The encoding of a string object in Redis can be int, RAW, or embstr.

  • Int encoding: holds a 64-bit signed integer of type long
  • Embstr encoding: Saves the string whose length is less than 44 bytes
  • Raw encoding: Saves a string larger than 44 bytes

Let’s do an experiment and actually see:

What happens is that Redis uses different encodings based on the key values given by the user, and this is completely transparent to the user!

Redis uses the structure SDS (” simple dynamic string “) to store strings. The code defines five SDS structures:

struct __attribute__ ((__packed__)) sdshdr5 {
    unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr8 {
    uint8_t len; /* used */
    uint8_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr16 {
    uint16_t len; /* used */
    uint16_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr32 {
    uint32_t len; /* used */
    uint32_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr64 {
    uint64_t len; /* used */
    uint64_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
Copy the code

It can be seen that, except for the different data types of the structure field, its field meanings are similar, where:

  • len: Length of string (actual length used)
  • alloc: Specifies the size of allocated memory
  • flags: flag bit. The lower three digits indicate the type. The remaining five digits are unused
  • buf: character array

With these basic data structures in mind, let’s look at the example above:

  • set foo 123
  • set foo abc
  • set foo abcdefghijklmnopqrstuvwxyzabcdeffasdffsdaadsx

How does Redis store data in these three situations?

INT encoding format

Example Set foo 123

When the contents of a string key can be represented as a 64-bit signed integer, Redis stores the key as a long, which corresponds to the OBJ_ENCODING_INT encoding type.

The internal memory structure of the OBJ_ENCODING_INT encoding type can be graphically represented as follows:

This means that if the set string has a key value between 0 and 10000, it can point directly to the shared object without creating a new object. Now the key takes up no space!

Therefore, when executing the following command:

set key1 100
set key2 100
Copy the code

Both key1 and key2 refer directly to a shared redisObject that Redis has already created, as follows:

Before source code, no secret, we compare the source code below, to understand the above process

EMBSTR encoding format

Example: set foo ABC

Redis uses OBJ_ENCODING_EMBSTR encoding to store strings smaller than 44 bytes.

As you can easily see from the above code, Redis applies OBJ_ENCODING_EMBSTR to key values of strings of length less than 44. EMBSTR, as the name implies, stands for embedded string. In terms of memory structure, that is, the string SDS structure and its corresponding redisObject are allocated in the same contiguous memory space, just as the string SDS is embedded in the redisObject. This can be clearly seen from the following code:

Thus, the memory structure of the key set by the instruction set foo ABC looks like this:

RAW encoding format

Command examples: set foo abcdefghijklmnopqrstuvwxyzabcdeffasdffsdaadsx

As in the instruction example, Redis changes the internal encoding of the key value to OBJ_ENCODING_RAW when the string key value is an extremely long string of length greater than 44. This differs from OBJ_ENCODING_EMBSTR in that the memory of the dynamic string SDS is no longer contiguous with the memory of its dependent redisObject. To set foo abcdefghijklmnopqrstuvwxyzabcdeffasdffsdaadsx as an example, the key value of memory structure is shown below:

This is the end of the most basic String data type of the internal encoding situation, how, or very good understanding of it!

We will continue to examine the internal encoding of Hash data types in Redis.

Remember after

Due to the limited ability, if there is a mistake or improper place, please also criticize and correct, study together!

  • My Personal Blog

  • The author’s more original articles are here, welcome to watch

The author has more SpringBt practice articles here:

  • A vegetable chicken’s six – year road to technology blogging
  • Spring Boot Logging framework practices
  • Spring Boot Admin2.0 Unpacking experience
  • Spring Boot application monitoring actual combat
  • The SpringBoot application is deployed in an external Tomcat container
  • ElasticSearch in SpringBt practice
  • A preliminary study on Kotlin+SpringBoot joint programming
  • SpringBoot elegant coding: Lombok plus

If you are interested, take some time to read some of the author’s articles on containerization and microservitization:

  • Use K8S technology stack to create personal private cloud serial articles
  • Nginx server configuration from a detailed configuration list
  • Docker container visual monitoring center was built
  • Use ELK to build Docker containerized application log center
  • RPC framework practice: Apache Thrift
  • RPC framework practice: Google gRPC
  • Establishment of microservice call chain tracking center
  • Docker containers communicate across hosts
  • Preliminary study on Docker Swarm cluster
  • Several guidelines for writing dockerFiles efficiently

To subscribe to CodeSheep, long press or scan the heart below for more practical, readable, and reproducible original articles