Dealing with difficult issues

Flushdb to prevent aof from overwriting: Config set auto-aof-rewrite-percentage 1000 config set auto-aof-rewrite-min-size 100000000000 Deletes flush contents in the primary and secondary AOF files *1 $8 flushall redis-check-aof Checks whether the aOF file is correctly restarted after the aOF file is manually rewritten to the master node and data is recovered

Bigkey processing: debug object key Displays the serializedLength property, indicating the number of bytes after serialization of the value corresponding to the key. For example: debug object www:common:zxt Value at:0x7f3b7e9fd2a0 refcount:1 encoding:int serializedlength:3 lru:6487054 Lru_seconds_idle :3211982 The unit is bytes

How to improve fork time: 1. Check latest_fork_usec info stats: Check latest_fork_usec in seconds. In Linux, the parent process shares the same physical memory page. When the parent process processes a write request, the parent process creates a copy of the page to be modified, while the child process shares the entire parent process memory snapshot during the fork operation

Transparent Huge Pages(THP) of Linux is enabled by default. It supports Huge page allocation (2MB), which can reduce the speed of fork creation, but results in the unit of page replication from 4KB to 2MB, which greatly increases the memory consumption of the parent process during rewriting. Suggested that echo never > / sys/kernel/mm/transparent_hugepage/enabled

No-appendfsync-on-rewrite: no fsync operation will be performed during AOF rewrite. However, in extreme cases, data may be lost during the entire AOF rewrite. The AOF_delayed_fsync indicator is convenient for locating AOF blocking problems

Redis allocates an input buffer to each client, temporarily saving the command sent by the client, while redis pulls the command from the buffer and executes it. (Qbuf stands for query buffer.) The buffer size is not fixed, it is dynamically adjusted according to the input content. Only the buffer size of each client should not exceed 1 gb. If the buffer size of a client exceeds 1 GB, the client will be shut down. The used_memory_human buffer is too large. 1. The processing speed of Redis cannot keep up with the input speed of the input buffer, and each command entering the input buffer contains a large number of bigkeys. 2

The method of monitoring input buffer abnormality is as follows: 1. Run the client list command periodically to collect qBUf and qbuf-free, find abnormal connections and analyze which client has problems (this method is slow to execute, and frequent execution may block Redis) 2. 127.0.0.1:6379> Info Clients Find the maximum input buffer set to 10 MB or higher

Clients

connected_clients:2 client_longest_output_list:0 client_biggest_input_buf:200 blocked_clients:0

Output buffer: Different from the input buffer, the client-output-buffer-limit can be set to divide into ordinary clients, publish-subscribe clients, Slave client normal: normal client slave:slave client pubsub: publish and subscribe client If the output buffer used by the client exceeds and lasts for a second, the client is immediately shut down if the output buffer used by the client exceeds, Client-output-buffer-limit Monitoring methods are as follows: 1. Client list View obL, OLL, and OMEM. 2. Info clients View client_longest_output_list:0

Default Normal clients are not limited because they don’t receive data without asking (in a push way), but just after a request, So only asynchronous clients may create a scenario where data is requested faster than it can read. The output buffer has a high probability of exceptions, while the input buffer has a low probability of exceptions. Obl represents the length of the fixed buffer (storing the execution results of commands that return relatively small values), ool represents the length of the dynamic buffer list (storing the results of commands that return large values), Omem indicates the total number of bytes in the output buffer. Output buffer = ool + obl

The slave output buffer can be increased as appropriate: if the master node has a large write, the slave client may have a large output buffer. If the slave client connection is killed due to the overflow of the output buffer, it will cause repeated reconnection. Frequent memory jitter may be the result of a large output buffer. The monitor command increases the output buffer in high concurrency cases

The client status is as follows: Age and idle (in seconds) of the client list indicate the connection time and the last idle time. If age is idle, the connection is in idle state

Config get maxClients Specifies the maximum number of connections. If the number of connections exceeds maxclients, new connections will be rejected. Info Clients Check the number of current client connections. S indicates slave. O indicates that the monitor command is being executed

The client elegantly-named setName | client getName set for the current connection client name and name, the default value is nil client kill IP: port client pause the timeout: Block client timeout Seconds. Once this command is sent, all clients will block

Monitor is used to monitor the commands being executed by Redis

Timeout Indicates the timeout period for detecting the client. 0 indicates that tcp-Keepalive is not detected. If this parameter is set to 60, TCP activity is detected every 60 seconds to prevent a large number of dead connections from consuming system resources

Tcp-backlog: Queues accepted connections after three handshakes. This parameter indicates the queue size. The default value is 511

Connected_clients Indicates the number of client connections to the current Redis node. If maxClients is exceeded, connections will be rejected

Info STATS ‘reject_Connections allows you to view the number of rejected client connections

The memory size of the primary node suddenly increases. Check whether the primary and secondary data are consistent. See dbsize results are the same info clients screen is caused by the master node client buffer memory spurt redis – cli client list | grep -v “omem = 0” (because most of the processing speed is fast enough, the client omem 0) omem larger, Client kill Kills the connection

Periodic client timeout problem: The client periodically times out and the server has no obvious exception but some slow query operations. Analysis: Network cause: Check whether the network periodically fails. Redis itself: Check the redis log statistics client: Check whether the slow query logs correspond to the timeout period