Performance testing is different from functional testing. Functional testing only needs to complete the established test cases and control the progress, so that the test can pass. Performance testing is not. In the process of pressure survey, environmental barrier removal, problem location analysis and tuning all need deep knowledge and experience. Pressure test execution is only a basic skill, excellent performance test engineers are fed out of hundreds of system performance tests and multiple rounds of large-scale activities, just like e-commerce double 11 and the opening of the insurance industry, etc. Some time ago combined with practice, learning and sorting out some knowledge points of JVM, this time to understand redis in a simple way

Redis learning is divided into 4 modules

  • Redis memory model
  • Redis persistence scheme and selection
  • High availability of Redis
  • Redis common performance analysis and optimization

Redis memory model

What can understanding the Redis memory model help you do? 1. Estimate the memory usage of REDIS and make reasonable use of resources. 2

How is redis memory divided

  • Store the data

As an in-memory database, data is the most important part of the storage data is reflected in usED_memory. Redis uses key-value to store data, which contains five types: string, hash, list, set and ordered set. Of course, Redis does not store the data directly into memory, it wraps the object, as described below.

  • Memory occupied by the Redis process

This memory footprint is small, a few megabytes, negligible and not counted in USED_memory.

  • The buffer memory

Client cache ** refers to the input/output cache of all TCP connections to Redis. The input cache is used to temporarily save the commands sent by the client, and the server pulls the commands from the input cache. The maximum cache space of a single client is 1 GB. You can run the client list command to view the commands.

** Replication backlog ** A reusable and fixed size buffer used for incremental replication, default 1MB, controlled by the REPL_backlog_size parameter. The buffer policy is FIFO. The primary node has one buffer and the secondary nodes share one buffer. This buffer is released when the master node is not connected to or disconnected from the slave node.

** AOF overwrite buffer ** Flush stores write commands during AOF overwrite and writes the buffer’s data to an AOF file when AOF overwrite is complete. This buffer size depends on when AOF is executed and how many commands are written during AOF rewrite.

  • Memory fragments

Memory fragmentation occurs when Redis allocates and reclaims physical memory. For example, a 5KB object is stored in an 8KB memory block, and the remaining 3KB becomes a memory fragment that cannot be redistributed. Memory fragmentation is not counted in USed_memory. You can run the mem_fragmentation_ration command to check the memory fragmentation ratio. The larger the value is, the larger the ratio is.

How is data stored in Redis

Related knowledge: Jemalloc memory allocator, simple dynamic string (SDS), 5 object types and internal encoding, RedisObject

         

Image: searchdatabase.techtarget.com.cn/7-20218/

DictEntry: one key one value stores Pointers to key and value, and next points to the next dictEntry.

Key: stored in the SDS structure RedisObject: Value is stored in the RedisObject structure rather than directly as a string. Type is the type of value, and PTR refers to the address of the object.

** jemalloc** : The above storage structure is allocated by Jemalloc memory for storage, redis default memory allocator, other not introduced. Jemalloc divides the memory space into three types: small, large, and large, each of which in turn divides many small memory blocks into units. Figure: For example, a 161-byte object is stored in a 192-byte memory cell

RedisObject: redisObject not only stores object functions, but also supports object types, internal encoding, memory reclamation, and shared objects.

Definition of RedisObject:

Type: indicates the object type. The value is 4 bits. (Type “key”) encoding: internal encoding of the object, 4bit (Object encoding “key”) LRU: time when the object was last accessed. If Redis is volatile/ allkeys-lRU and the redis memory usage is greater than maxMemory, the object idleTime “key” will be freed. Refcount: Records the number of times an object is referenced. Used for memory reclamation. The value is 1 when a new object is created, +1 when used by a new program, and -1 is no longer used by a new program. The value is 0 and the object is released. PTR: SDS pointing to the specific data

A RedisObject is 16 bytes in size.

Simple Dynamic String (SDS) structure:

Len: Indicates the used length of the data. Free: unused data length. Buf: byte array.

An SDS structure takes up space of free+len+9 bytes.

Redis’s five object types and internal encodings

Redis supports five object types, each of which supports at least two encodings.

Multiple coding benefits:

  • Interface and implementation separation, need to increase or change the encoding mode users do not receive impact.
  • Switch codes according to different business scenarios to improve redis efficiency.

There have been a lot of introductions on digging gold. It is recommended to read the official document and Redis Design and Implementation.

Redis persistence scheme and selection

Not many words on the official documents, the summary is still relatively complete, they are not repeated car.

Redis high availability

Still official documentation, master-slave mode and Sentinel mode

Here we go!!

Redis common performance analysis and optimization

Mainly from three aspects: memory, delay, big key

memory

Estimated memory capacity:

I have searched all kinds of data to calculate the amount of memory occupied by the storage structure of each data format, but later I found that the Redis capacity estimate on the official website directly hit me.

Memory usage:

To view memory usage, focus on the following parameters:

Used_memory: The total amount of memory (in bytes) allocated by the Redis allocator, equivalent to used_memory_human

Sed_memory_rss: The amount of memory (in bytes) occupied by the Redis process, which is the same as the value seen by the top and ps commands.

The former is from the Redis perspective, the latter is from the operating system perspective

Mem_fragmentation_ratio: memory fragmentation ratio

Used_memory_rss/used_memory is greater than 1, and the larger the value is, the larger the memory fragmentation ratio is. <1, Redis uses virtual memory.

It is ok for the memory fragmentation rate to be slightly greater than 1. This value indicates that the memory fragmentation rate is relatively low, and also indicates that no memory swapping occurs in Redis. However, if the memory fragmentation rate exceeds 2, then Redis is consuming 200% of the actual physical memory required, of which 100% is the memory fragmentation rate. If the memory fragmentation rate is less than 1, it indicates that the Redis memory allocation exceeds the physical memory, and the operating system is performing a memory swap.

What can I do if the memory fragmentation ratio is high?

  1. Restart the Redis server (recommended), but the restart involves the problem of snapshot data recovery, see the Redis Persistence scheme and selection section
  2. Increase the available physical memory

How to alleviate it?

  1. Optimization using jEMalloc characteristics. For example, if the length of the key is 8 bytes, SDS is 17 bytes and Jemalloc allocates 32 bytes. At this point, if the key length is reduced to 7 bytes, SDS is 16 bytes and JEMalloc allocates 16 bytes. The space occupied by each key can be reduced by half.
  2. Use reasonable data types. If it is an integer, Redis saves more space by using an int (8 bytes) instead of a string.
  3. Shared objects. With shared objects, you can reduce object creation (and RedisObject creation), saving memory space. You can increase the number of shared objects by adjusting the REDIS_SHARED_INTEGERS parameter.

Memory usage

If the memory usage of the Redis instance exceeds the maximum available memory, used_memory > the maximum available memory, then the operating system swaps memory with swap space, writing the old or unused contents of memory to the hard disk to set aside new physical memory for new or active pages. Then Redis and applications that rely on data from Redis will suffer a significant performance impact.

When the snapshot function is enabled and triggered, Redis forks a child process to write a copy of the data in the current memory to the hard disk. Therefore, if the snapshot function is triggered when the current memory usage exceeds 45% of the available memory, the memory may lose data. If there are a lot of frequent updates on the instance at this point, the problem becomes even worse. To avoid this problem, set “maxMemory” to 45% of the available memory of the system, because it takes twice as much memory to copy the entire data set at snapshot time. That is, if 45% is currently used, it will become 95%(45%+45%+5%) during snapshot time. Five percent of that is set aside for other expenses. If snapshot is not enabled, maxMemory can be set to 95% of the available memory of the system.

delay

You can check the Redis slow log when you notice a sudden increase in access latency

Setting Method 1 Config Command setting:

Slowlog-log-slower than 5000 # Slowlog-max-len 1000 slowlog-max-len 1000 slowlog-max-len 1000Copy the code

Setting Method 2 Setting the redis. Conf file:

Slowlog-log-lower-than 1000 # Maximum number of logs slowlog-max-len 1000Copy the code

Viewing slow logs:

SHOW LEN # Clear the log SLOWLOG RESETCopy the code

If the number of service requests is not high but the CPU usage of the Redis instance is high, it is likely that the use of a complex command is causing this, which can be viewed via info STATS –> total_commands_processed

If Redis suddenly has a wave of delay at a certain point in time, and the delay is very regular, you need to consider whether there is a large number of key expiration sets. Set the key expiration time based on service characteristics

  • Volatile – lRU: Discard the least recently used data from a set with an expiration date (server.db[I].expires) (recommended)
  • Volatile – TTL: Selects expired data from a set (server.db[I].expires) to be discarded
  • Volatile -random: Selects any data from a set with an expiration date (server.db[I].expires) to be discarded
  • Allkeys-lru: Cull least recently used data from dataset (server.db[I].dict) (recommended)
  • Allkeys-random: Random selection of data from a dataset (server.db[I].dict)
  • No-enviction: Data expulsion is prohibited

Redis is enabled to automatically generate RDB and AOF overrides, so it is possible to generate RDB and AOF overrides in the background, resulting in an increase in Redis access latency. After these tasks are completed, the latency disappears. This is usually the result of performing the RDB and AOF rewrite tasks. Run the info command to view the latest_fork_usec time of the last fork execution, in microseconds.

To avoid this situation, we need to plan the data backup cycle. It is recommended to perform the backup on the secondary node, and preferably during the low peak period. Redis is recommended to be deployed on physical machines to reduce the impact of fork. In addition, if the AOF mechanism is enabled, improper policies can cause performance problems.

AOF provides three disk flushing mechanisms to ensure the security of writing files to disks:

  1. Appendfsync always: Flush disks every time data is written, which has the greatest impact on performance and occupies a high DISK I/O footprint and has the highest data security
  2. Appendfsync Everysec: Flush disks once every 1 second, relatively small impact on performance, data loss of at most 1 second when a node is down (recommended)
  3. Appendfsync no: Flush disks based on the OPERATING system (OS) mechanism, which has minimal impact on performance and low data security. Data loss during node downtime depends on the OS flush mechanism

The big key

In terms of big key, we borrowed some knowledge points we sorted out before, which have not been practiced yet. We will sort them out after we have optimization experience