preface

Redis knowledge summarized a map, share with you:

1.1 introduction of Redis

1.1.1 to introduce

Redis is an open source, network-enabled, memory-based, optionally persistent key-value storage database written in ANSI C. Development of Redis was sponsored by Redis Labs beginning in June 2015, and from May 2013 to June 2015, its development was sponsored by Pivotal. Prior to May 2013, its development was sponsored by VMware. Redis is the most popular key-value pair storage database, according to DB-Engines.com, a monthly ranking website.

Redis uses in-memory datasets.

Multiple data types are supported.

Runs on most POSIX systems, such as Linux, *BSD, OS X, etc.

Redis作者： Salvatore Sanfilippo

The author making: github.com/antirez/red…

1.1.2 Obtaining and Helping Software

Official website: redis.io

Official versions download address: download. Redis. IO/releases /

Redis 英 文 标 题 : redisdoc.com

Chinese website 1: redis.cn

Chinese website 2: www.redis.net.cn

1.1.3 Redis features

High-speed read and write, rich data types

Support persistence, multiple memory allocation and reclamation strategies

Support for weak transactions, message queuing, message subscription

Supports high availability and distributed sharding cluster

1.1.4 Comparison of Enterprise Cache Database Solutions

Memcached:

Advantages: High-performance read/write, single data type, client-side distributed cluster, consistent hash multi-core structure, and multi-threaded read/write performance.

Disadvantages: No persistence, node failure may lead to cache penetration, distributed data needs to be implemented by clients, data synchronization is difficult across rooms, and architecture expansion complexity is high

Redis:

Advantages: High-performance read/write, multi-data type support, data persistence, high availability architecture, customized virtual memory, distributed fragmented cluster, and high single-thread read/write performance

Disadvantages: Multithreaded reads and writes are slower than Memcached

Tair: Official website: tair.taobao.org

Advantages: High-performance read and write, supports three storage engines (DDB, RDB and LDB), supports high availability, supports distributed sharding cluster, and supports caching of almost all Taobao businesses.

Disadvantages: The read and write performance is slower than that of the other two products on a single machine.

1.1.5 Redis Application Scenario

Data Cache, Web Session Cache

Leaderboard app

Message queue, publish subscription

Appendix – Enterprise application of Redis

1.2 Redis simple deployment

1.2.1 Typical Installation – Single Instance

System Environment description

[root@Redis ~]# cat /etc/redhat-release 
CentOS release 6.9 (Final)
[root@Redis ~]# uname -r 
2.6.32-696.el6.x86_64
[root@Redis ~]# sestatus 
SELinux status:                 disabled
[root@Redis ~]# /etc/init.d/iptables status
iptables: Firewall is not running.
[root@Redis ~]# hostname -I 
10.0.0.186 172.16.1.186
Copy the code

Install redis

/ root @ Redis ~ # CD/usr/local / [root @ Redis local] # wget HTTP: / / http://download.redis.io/releases/redis-3.2.10.tar.gz [root@Redis local]# tar XZF Redis-3.2.10.tar. gz [root@Redis local]# \rm Redis-3.2.10.tar. gz [root@Redis local]# mv Redis -3.2.10 redis [root@Redis local]# CD redis/ [root@Redis redis]# makeCopy the code

The Redis installation is complete

1.2.2 Starting the first Redis instance

Create a client soft connection

[root@Redis src]# ln -s /usr/local/redis/src/redis-cli /usr/bin/
Copy the code

Simple boot method, use the default configuration

[root@Redis ~]# cd /usr/local/redis/src
[root@Redis src]# ./redis-server &
Copy the code

Writing configuration files

1. Simplify the configuration file

[root@Redis redis]# cp redis.conf{,.bak}
[root@Redis redis]# grep -Ev '^$|#' redis.conf.bak > redis.conf
[root@Redis redis]# cp redis.conf /etc/
Copy the code

2. Edit the configuration file

[root@Redis ~]# cat /etc/redis.conf bind 127.0.0.1 10.0.0.186 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize yes supervised no pidfile /var/run/redis_6379.pid loglevel notice logfile "/var/log/redis_6379.log" dir /usr/local/redis/data/ # ··· ·Copy the code

Description of the configuration file: www.cnblogs.com/zhang-ke/p/…

3. Write startup scripts (for CentOS 6.x)

View Code Redis management script

Note: Self-written scripts pay attention to execution permissions.

1.2.3 Redis Multi-instance configuration

Note: This multi-instance configuration is based on the single-instance configuration

Create and enter the program directory

[root@Redis redis]# mkdir /application/redis  -p
[root@Redis redis]# cd /application/redis/
Copy the code

Modifying a Configuration File

For I in 0 1 2 do # to create multiple instances (port) directory mkdir -p 638 $I # replication initiator to each instance \ cp/usr/local/redis/SRC/redis server. - /application/redis/638$I / Note: Sed -i "/dir/s#.*#dir /application/redis/638$I /#g" Sed -i "s#6379#638$I #g" /application/redis/638$I /redis  sed -i '/protected-mode/s#yes#no#g' /application/redis/638$i/redis.conf doneCopy the code

Restart the instance

for i in 0 1 2 
  do
  /application/redis/638$i/redis-server /application/redis/638$i/redis.conf 
done
Copy the code

Connect the redis

[root@Redis redis]# redis-cli -h 10.0.0.186 -p 6379
10.0.0.186:6379>
Copy the code

1.2.4 Redis.conf Configuration Description

Background running:

daemonize no/yes

Default port:

port 6379

Check whether the AOF log switch is enabled:

appendonly no/yes

Log file location

logfile /var/log/redis.log

RDB persistent data file:

dbfilename dump.rdb

Specify an IP address for listening

Bind 10.0.0.51 ip2 ip3 ip4

Ban protected mode

Protected-mode Yes /no (protected mode, local access only)

Increase the requirepass {password}

requirepass root

Used in redis-CLI

Auth {password} for authentication

1.2.5 Online Configuration Change

Get the current configuration

CONFIG GET *

Changing the Running configuration

CONFIG SET loglevel "notice"

Change the password to null

10.0.0.186:6379> config set requirepass ""
10.0.0.186:6379> exit
10.0.0.186:6379> config get dir
1) "dir"
2) "/usr/local/redis/data"
Copy the code

1.3 Redis Data Persistence

1.3.1 Persistence Policy

Redis provides several different levels of persistence: one is RDB and the other is AOF.

RDB persistence

A point-in-time snapshot of a data set can be generated at a specified time interval.

AOF persistence

Log all write commands executed by the server and restore the data set by re-executing these commands when the server starts. All commands in the AOF file are saved in Redis format, and new commands are appended to the end of the file.

Redis can also rewrite AOF files in the background so that the size of the AOF file does not exceed the actual size needed to store the state of the dataset. Redis can also use both AOF and RDB persistence. In this case, when Redis restarts, it will use AOF files to restore the dataset in preference, since AOF files usually hold more complete datasets than RDB files.

You can even turn off persistence so that data only exists while the server is running.

1.3.2 RDB Persistence

The advantages of RDB

⚔ RDB is a very compact file that holds Redis data sets at a point in time. This type of file is perfect for backup: for example, you can back up an RDB file every hour for the last 24 hours and also back up an RDB file every day of the month. This way, if you run into problems, you can always revert the dataset to a different version.

⚔ RDB is perfect for Disaster Recovery: it’s a single file, and the content is so compact that it can be sent (encrypted) to another data center, or to Amazon S3.

⚔ RDB maximizes the performance of Redis: the only thing the parent has to do to save the RDB file is fork out a child, which then handles all subsequent save work without the parent performing any disk I/O operations.

⚔ RDB can recover large data sets faster than AOF.

The disadvantage of RDB

If you need to avoid losing data in the event of a server failure, the RDB is not for you.

Although Redis allows you to set different save points to control how often RDB files are saved, it is not an easy operation because RDB files need to hold the state of the entire data set. So you’ll probably save your RDB file at least once every 5 minutes. In this case, you could lose several minutes of data in the event of a malfunctioning outage.

Each time the RDB is saved, Redis forks () out a child process that does the actual persistence. In large data sets, fork() can be time-consuming, causing the server to stop processing the client in so-and-so milliseconds; If the data set is very large and CPU time is very tight, this stop time can even take a full second. Although AOF overrides also require forking (), the durability of the data is not compromised regardless of the interval between AOF overrides.

1.3.3 AOF Persistence

AOF advantages

Using AOF makes your Redis more durable: you can use different fsync strategies: no fsync, fsync per second, fsync every time you write. With the default fsync per second policy,Redis still performs well (fsync is handled by background threads and the main thread does its best to handle client requests), and you can lose up to 1 second of data in the event of a failure.

Redis can automatically rewrite the AOF in the background when the AOF file becomes too large: the rewritten new AOF file contains the minimum set of commands needed to restore the current data set. The entire rewrite operation is absolutely safe because Redis continues to append commands to existing AOF files while creating new AOF files, and the existing AOF files will not be lost even if an outage occurs during the rewrite.

Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

AOF files orderly store all writes to the database in the Redis protocol format, so the contents of AOF files are easy to read and parse. Exporting AOF files is also very simple: For example, if you accidentally execute the FLUSHALL command, but as long as the AOF file hasn’t been overwritten, you can restore the data set to its pre-flushall state by stopping the server, removing the FLUSHALL command at the end of the AOF file, and restarting Redis.

AOF shortcomings

AOF files are usually larger than RDB files for the same data set. Depending on the fsync strategy used, AOF may be slower than RDB.

Fsync per second performance is still very high under normal conditions, and turning off fsync allows the AOF to be as fast as the RDB, even under high loads. However, RDB can provide more guaranteed maximum latency when handling large write loads.

AOF has had a bug in the past where an AOF file could not restore the dataset as it was saved when it was reloaded due to certain commands. (The blocking command BRPOPLPUSH, for example, has caused such a bug.) Test suites have added tests for this situation: they automatically generate random, complex data sets and reload them to make sure everything is fine.

Although this kind of bug is not common in AOF files, RDB bugs are almost impossible by comparison.

1.3.4 How do I Choose which Persistence Mode to Use

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence features.

If you care deeply about your data, but can still afford to lose it within minutes, you can use RDB persistence only.

There are many users who only use AOF persistence, but this is not recommended: In addition to providing periodic SNAPSHOTS for database backups and restoring data sets faster than AOF, RDB also avoids the aforementioned AOF bugs.

Note: For all the reasons mentioned above, redis will likely integrate AOF and RDB into a single persistence model in the future (this is a long-term plan).

1.3.5 Persistency of Snapshots

By default, Redis saves database snapshots in a binary file named dump.rdb. You can set Redis to automatically save the dataset once the condition of “at least M changes to the dataset in N seconds” is met.

You can also manually tell Redis to SAVE the dataset by calling SAVE or BGSAVE.

For example, the following setting will automatically save the data set once Redis meets the requirement that at least 1000 keys have been changed within 60 seconds: Save 60 1000

This type of persistence is called snapshotting.

When Redis needs to save the dump. RDB file, the server does the following:

🎣 Redis calls forks. Has both parent and child processes.

The 🎣 child process writes the data set to a temporary RDB file.

🎣 When the child process finishes writing to the new RDB file, Redis replaces the original RDB file with the new one and deletes the old one.

This way of working allows Redis to benefit from copy-on-write mechanisms.

1.3.6 AOF Persistence

File to append only (append-only file, AOF)

The snapshot feature is not very durable: if Redis malfunctions for some reason, the server loses data that was recently written to and still hasn’t been saved to the snapshot. While data durability may not be the most important consideration for some programs, snapshots may not be suitable for programmers looking for full durability.

Starting with version 1.1, Redis added a completely durable persistence method: AOF persistence.

You can turn on the AOF function by modifying the configuration file: appendOnly Yes

From now on, whenever Redis executes an imperative to change the data SET (such as SET), the command is appended to the end of the AOF file.

This way, when Redis restarts, the program can recreate the dataset by re-executing the commands in the AOF file

1.3.7 AOF Log Rewriting

Because AOF works by continually apending commands to the end of a file, the size of an AOF file gets bigger as more commands are written.

For example, if you call INCR 100 times on a counter, the AOF file would need to use 100 entries just to hold the current value of that counter. In practice, however, just one SET command is enough to hold the current value of the counter, and the remaining 99 records are virtually redundant.

To handle this situation, Redis supports an interesting feature: the ability to rebuild AOF files without interrupting the service client. By executing the BGREWRITEAOF command, Redis generates a new AOF file that contains the minimum of commands needed to rebuild the current data set. Redis 2.2 You need to run the BGREWRITEAOF command manually.

1.3.8 How durable is AOF?

You can configure how often Redis fsync data to disk. There are three ways:

👽 Fsync is performed every time a new command is appended to the AOF file: very slow and very safe

👽 fsync once per second: fast enough (similar to using RDB persistence) and only 1 second of data lost in the event of a failure.

👽 never fsync: hands data to the operating system for processing. The faster and less safe option.

👽 The recommended (and default) measure is fsync once per second, a fsync policy that balances speed and security.

1.3.9 What if the AOF file is corrupted?

The server may stop when a program is writing to an AOF file, and if the stop results in corrupt AOF files, Redis will refuse to load the AOF file when it restarts to ensure that data consistency is not compromised. When this happens, you can fix the faulty AOF file by:

🎑 creates a backup of the existing AOF files.

🎑 use the redis-check-aof program to fix the original aof file: $redis-check-aof — fix

🎑 use diff -u to compare the restored AOF file with the original AOF file backup to see the differences between the two files. (optional)

🎑 Restart the Redis server and wait for the server to load the repaired AOF file and restore data.

1.3.10 Interaction between AOF and RDB

In Redis with version 2.4 or higher, BGREWRITEAOF cannot be executed during BGSAVE execution.

Conversely, BGSAVE cannot be executed during BGREWRITEAOF execution. This prevents both Redis background processes from doing a lot of I/O to the disk at the same time.

If BGSAVE is executing and the user explicitly calls the BGREWRITEAOF command, the server will reply with an OK status and inform the user that BGREWRITEAOF is scheduled to be executed: Once BGSAVE is executed, BGREWRITEAOF will officially begin.

When Redis is started, if both RDB persistence and AOF persistence are enabled, the program will use AOF files to recover the dataset in preference, since AOF files usually hold the most complete data.

1.3.11 Backing up Redis Data

Redis is very friendly for data backup because you can copy RDB files while the server is running:

Once an RDB file is created, no changes are made. When the server creates a new RDB file, it saves the contents of the file in a temporary file. When the temporary file has been written, rename(2) replaces the original RDB file atomically with the temporary file.

This means that copying RDB files is perfectly safe at all times.

🎍 Create a cron job to back up one RDB file to a folder every hour and one RDB file to another folder every day.

🎍 Ensure that snapshots are backed up with date and time information, and use the find command to delete expired snapshots each time you execute a periodic task script: for example, you can keep snapshots every hour within the last 48 hours and daily snapshots within the last one or two months.

🎍 At least once a day, back up your RDB outside of your data center, or at least outside of the physical machine on which you run your Redis server.

1.3.12 RDB Persistent Configuration

RDB persists the basic configuration

Modifying a Configuration File

save 900 1 save 300 10 save 60 10000

The configurations represent:

• 1 change in 900 seconds (15 minutes)

• 10 changes in 300 seconds (5 minutes) • 10,000 changes in 60 seconds • Memory data is persisted to disk when the configuration time defined above is reached.

Advanced configuration of RDB persistence

stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir ./clsn/redis/data/6379

The above configurations indicate:

• If the background backup process fails, does the main process stop writing? If the main process is not stopped, data inconsistency may occur. • Whether to compress the exported RDB file if the SIZE of the RDB is too large. • Whether to verify the integrity of the RDB to verify the consistency of the version when importing data during RBD recovery • The name of the exported RDB file • The path to save the RDB

1.3.13 AOF Persistence Configuration

AOF persists the basic configuration

appendonly yes/no appendfsync always appendfsync everysec appendfsync no

The configurations represent:

• Whether to enable the AOF log function

• Every command is immediately synchronized to aOF

• Write once per second

• Write to the operating system. The operating system determines the buffer size and writes to the AOF.

AOF Advanced configuration for persistence

no-appendfsync-on-rewrite yes/no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb

The configurations represent:

• Do you want to stop aOF synchronization when the RDB snapshot is being exported

• The aOF file size is rewritten when the growth rate is 100% compared to the last rewrite. Disadvantages: It will be rewritten several times when the business starts.

• AOF files, at least over 64M, rewritten

1.3.14 Switching between RDB and AOF

In Redis 2.2 or above, it is possible to switch from RDB to AOF without rebooting:

1. Create a backup of the latest dump. RDB file.

2. Put the backup in a safe place.

3. Run the following two commands:

Appendonly yes redis-cli config set save ""

4. Ensure that write commands are correctly appended to the end of the AOF file.

Execute instructions

The first command executed turns on AOF: Redis blocks until the initial AOF file is created, after which Redis continues processing command requests and begins appending write commands to the end of the AOF file.

The second command is used to disable the RDB function. This step is optional, or you can use both RDB and AOF if you prefer.

Note: Don’t forget to turn on the AOF feature in redis.conf! Otherwise, after the server restarts, the previous configuration SET through the CONFIG SET will not take effect, and the program will start the server with the original configuration.

1.4 Redis management practice

1.4.1 Basic Data types

1.4.2 Global Key Operations

1.4.3 String (String)

String is the most basic type of redis. Each key corresponds to a value. The maximum storage capacity of a key is 512MB.

Application scenarios

General count: number of micro blog, number of fans, etc.

1.4.4 Hash (Dictionary)

We can think of the Hashes type in Redis as a map container with String keys and String values.

So this type is very good for storing information about value objects. For example, Username, Password, and Age. If the Hash contains very few fields, that type of data will also take up very little disk space. Each Hash can store 995,701,749 key-value pairs.

Application Scenarios:

Stores some changed data, such as user information.

1.4.5 LIST

The List type is a linked List of strings sorted by insertion order. Like a normal linked list in a data structure, we can add new elements to its head (left) and tail (right).

At insert time, if the key does not exist, Redis creates a new linked list for that key. Conversely, if all the elements in the linked list are removed, the key will also be removed from the database.

The maximum number of elements that a List can contain is 4294967295.

Application scenarios

Message queue system, such as Sina Weibo

In Redis we use the resident cache for our latest Twitter ID, which is always updated. However, it is limited to 5000 ids, so the function that gets the ID will always ask Redis. Only if the start/count parameter is outside this range do you need to access the database.

The system does not “flush” the cache in the traditional way, and the information in the Redis instance is always consistent. SQL databases (or other types of databases on the hard disk) are triggered only when the user needs to fetch “distant” data, and the home page or first comment page does not bother the hard disk database.

1.4.6 SET(SET)

The Set type is seen as an unsorted collection of characters. The maximum number of elements that a Set can contain is 4294967295. If you add the same element more than once, the Set will keep only one copy of that element.

Application Scenarios:

In the microblog application, all the followers of a user can be stored in a collection, and all the fans can be stored in a collection.

Redis also provides the set of intersection, union, difference and other operations, which can be very convenient to achieve such as common concerns, common preferences, second degree friends and other functions. For all of the above set operations, you can also use different commands to choose whether to return the results to the client or save the set to a new set.

1.4.7 SortedSet (Ordered Set)

Each member of a Sorted- set is associated with a score, which Redis uses to sort the members of the set from smallest to largest. Members are unique, but scores are repeatable.

Collections are implemented by hashing tables, so adding, deleting, and searching are O(1) complexity. The maximum number of members in a collection is 232-1 (4294967295, each collection can store more than 4 billion members).

Application Scenarios:

In this case, we need to set the sorted set value to the sorted set score. In this case, we need to set the sorted set value to the sorted set score. To set specific data to the corresponding value, you only need to execute the ZADD command one at a time.

1.4.8 Message Mode

Redis generally publishes messages in two modes:

• Queuing Mode

• Publish-subscribe

Task queue:

As the name suggests, it’s a “message queue.” There are two types of entities that interact with a task queue: producer and consumer. The producer puts the tasks that need to be processed into the task queue, and the consumer continuously reads the task information from the task independence and executes it.

Benefits of task queues:

• Loose coupling.

Producers and consumers simply write the code in the agreed task description format.

• Easy to expand.

In multi-consumer mode, consumers can be distributed across multiple servers, thus reducing the load on a single server.

1.4.9 Redis publishing subscription

In fact, from the mechanism of Pub/Sub, it is more like a broadcast system, with multiple subscribers subscribing to multiple channels and multiple publishers publishing messages to multiple channels. It’s as simple as this:

• Subscriber: a radio that can receive multiple channels and display them in a queue

• Publisher: Radio station, can send messages to different FM channels

• Channel: FM channels with different frequencies

1.4.9.1 Publish and subscribe model

One Publisher, multiple Subscriber models

As shown in the figure below, it can be used as a message queue or a message pipe.

Main applications: notifications and announcements.

Multiple publishers, one Subscriber model

PubSub can be made into an independent HTTP interface, each application sends messages to the Channel as Publisher, and the Subscriber end performs corresponding business logic after receiving the message, such as database writing and display, etc.

Main applications: leaderboards, voting, counting.

Multiple publishers, multiple Subscriber models

It can send messages to different channels and receive them by different Subscriber.

Main application: group chat, chat.

1.4.9.2 Practice publish and subscribe

Publish subscribe practice commands

Note: When a client subscribes to a channel, it can only receive subsequent messages published to the channel. The previously sent messages are not cached. Provider and Consumer must be online at the same time.

1.4.9.3 Comparison of Message Queue Systems

After executing the subscription command, the client enters the subscription state and can only receive SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, and PUNSUBSCRIBE commands.

An open subscription client cannot receive messages before the channel because Redis does not persist published messages.

Compared to many professional message queuing systems (such as Kafka and RocketMQ), Redis is a bit rough with its publish and subscribe, for example, failing to implement message stacking and backtracking. But winning in simple enough, if the current scenario can tolerate these shortcomings, is also a good choice.

1.4.10 Redis Transaction Management

Transactions in Redis are a similar concept to transactions in a relational database, but there are differences.

The SQL statement following a relational database transaction failure does not execute, and one command in Redis fails, leaving the rest of the commands to execute as usual.

In Redis, a transaction is started using multi, which is equivalent to begin\start transaction, exec commit transaction, or discard de-queue command (non-rollback operation).

Redis compared to mysql

1.4.10.1 Redis Transaction command

Transaction Execution Example

ZADD salary 2000 user1 ZADD salary 3000 user2 ZRANGE salary 0 -1 WITHSCORES MULTI ZINCRBY salary 1000 user1 ZINCRBY salary -1000 user2 EXEC

1.4.10.2 Locking mechanism in Redis transactions

Example: I’m buying a Ticket -1, money-100

There’s only one ticket, and if someone else buys it after I’m multi and before I’m exec, then ticket becomes 0.

How can I observe this scenario and no longer submit:

Pessimistic thoughts:

The world is full of danger, there must be someone and I grab, lock the ticket, only I can operate.

Optimistic thoughts:

There’s not a lot of competition, so I just have to be careful if someone changes the value of the ticket.

In Redis transactions, optimistic locks are enabled and only responsible for monitoring that the key has not been changed.

1.4.10.3 Redis Service Management Commands

1.4.11 Redis Slow Log Query

Slow log is the logging system Redis uses to record query execution times.

Slow logs are stored in memory and read and write speeds are very fast

They can be modified dynamically by overwriting the redis.conf file or using the CONFIG GET and CONFIG SET commands

Slowlog-log-slower -than 10,000 slowlog-log-slower CONFIG SET slowlog-log-slower than 100 CONFIG SET slowlog-max-len 1000 Slowlog-max 1000 slowlog-max 1000 slowlog-max 1000 slowlog-max 1000 slowlog-max 1000 slowlog-max CONFIG GET slow* SLOWLOG GET SLOWLOG RESET

1.5 Redis primary/secondary Replication

1.5.1 Redis Replication Feature

⚗ Use asynchronous replication.

⚗ A primary server can have multiple secondary servers.

⚗ slave servers can also have their own slave servers.

⚗ replication does not block the primary server.

⚗ can use replication to keep the primary server from persisting and the secondary server from persisting.

Data security for replication when primary persistence is turned off

When configuring Redis replication, it is strongly recommended to enable persistence on the primary server. Otherwise, deployed services should avoid automatic pull-up due to delays and other issues.

To help understand the dangers of automatic pull when the primary server turns off persistence, consider the following example that results in total loss of data from the primary and secondary servers:

1. Assume that node A is the primary server and persistence is turned off. And nodes B and C copy data from node A

2. Node A crashes and is restarted by the automatic pull-up service. Since persistence on node A is turned off, there is no data after the restart

3. Nodes B and C will copy data from node A, but A’s data is empty, so it will delete its own data copy.

Even using Sentinel for Redis high availability is dangerous when persistence is turned off on the primary server and the automatic pull-up process is enabled. The primary server may be pulled up so quickly that Sentinel does not detect that the primary server has been restarted during the configured heartbeat interval, and then performs the data loss process described above.

Data security is extremely important at all times, so the master server should be prohibited from automatically pulling up while turning off persistence.

1.5.2 Principle of Primary/Secondary Replication

There are two types of redis master/slave synchronization (or both phases) : full and partial.

When the master and slave are first connected, they are fully synchronized. After full synchronization, partial synchronization is performed. Of course, slave can initiate full synchronization at any time if needed.

The Redis strategy is to try partial synchronization first anyway, and if this fails, require full synchronization from the slave machine and start BGSAVE… After BGSAVE ends, RDB files are transferred. If successful, the slave machine is allowed to partially synchronize and transfer the data in the backlogged space.

The following diagram summarizes the mechanism of master-slave synchronization:

Master/slave replication principle:

1. The secondary server sends the SYNC command to the primary server.

2. The primary server receiving the SYNC command calls the BGSAVE command, creates an RDB file, and uses a buffer to record all subsequent write commands.

3. When the primary server finishes executing the BGSAVE command, it sends the RDB file to the slave server, which receives and loads the file.

4. The primary server sends all write commands stored in the buffer to the secondary server for execution.

Command propagation

After the master server completes synchronization with the slave server, for each write command executed by the master server, it sends the write command that is executed to the slave server for execution, this operation is called command propagate.

Command propagation is a continuous process: as long as replication continues, command propagation continues, keeping the state of the primary and secondary servers consistent.

SYNC and PSYNC in 1.5.3 Replication

Prior to Redis 2.8, slave servers that reconnected after disconnection always performed a full resynchronization operation.

Starting with Redis 2.8, Redis uses PSYNC instead of SYNC. The biggest improvement over SYNC is that PSYNC implements partial resync: In the event that the master and slave servers are disconnected and reconnected, PSYNC allows the master server to synchronize only the data that was missing during the disconnection to the slave server, rather than re-synchronizing the entire database to the slave server, whenever possible.

1.5.4 Replication Consistency Problem

In the read/write separation environment, the client sends the write command SET N 10086 to the master server. After executing the write command, the master server replies to the client and propagates the write command to the slave server.

The client receiving the reply continues to send the read command GET n to the slave server, and because of the network state, the client’s GET command reaches the slave server faster than the SET command propagated by the primary server.

Because the value of n from the server key has not been updated, the client will get an incorrect (expired) value of n from the server.

1.5.5 Improved Replication Security

The primary server performs write operations only when there are at least N secondary servers. Starting from Redis 2.8, you can configure the primary server to execute write commands only when there are at least N secondary servers connected to the primary server to ensure data security.

However, because Redis uses asynchronous replication, the write data sent by the master server is not necessarily received by the slave server, so there is still the possibility of data loss.

Ensure data security by setting the following parameters:

min-slaves-to-write <number of slaves> min-slaves-max-lag <number of seconds>

1.5.6 Redis master/slave Replication practice

Multiple instances are configured when Redis is installed

Prepare two or more redis instances 6380/redis-server 6380/redis.conf 6381/redis-server 6381/redis.conf 6382/redis-server 6382/redis.conf

Example configuration file:

Bind 127.0.0.1 10.0.0.186 port 6380 daemonize yes pidfile /var/run/redis_6380.pid loglevel notice logfile "/var/log/redis_6380.log" dbfilename dump.rdb dir /application/redis/6380/ appendonly no appendfilename "appendonly.aof" appendfsync everysec slowlog-log-slower-than 10000 slowlog-max-len 128 protected-mode no

Activation:

./6380/redis-server ./6380/redis.conf ./6381/redis-server ./6381/redis.conf ./6382/redis-server ./6382/redis.conf

Replication Environment description

Primary node: 6380 Secondary node: 6381, 6382

Enable primary/secondary (on instance 6381 6382)

Redis -cli -p 6381/6382 SLAVEOF 127.0.0.1 6380

The redis master/slave replication is now complete

1.5.7 Redis Master/Slave Replication Management

Primary/secondary replication status monitoring: Info replication

Slave switchover: Slaveof no one

1.6 Redis HA Practices (Redis Sentinel)

Redis. IO /topics/sent…

Redis-sentinel is an official high availability (HA) solution recommended by Redis. When using Redis as a master-slave high availability solution, if the Master crashes, Redis itself (including many of its clients) does not automatically implement the Master/slave switchover. Redis-sentinel itself is an independent process that monitors multiple master-slave clusters and automatically switches over when the master is down.

Sentinel is a monitor that determines what actions should be performed based on the identity and status of the monitored instance.

1.6.1 Redis Sentinel function

📣 Monitoring:

Sentinel constantly checks to see if your primary and secondary servers are working properly.

📣 Notification:

Sentinel can send notifications to administrators or other applications via the API when a monitored Redis server has a problem.

📣 Automatic Failover:

When a primary server fails, Sentinel starts an automatic failover operation. It upgrades one of the secondary servers of the failed primary server to the new master server and replicates the other secondary servers of the failed primary server. When a client tries to connect to a failed primary server, the cluster also returns the address of the new primary server to the client, allowing the cluster to use the new primary server in place of the failed server.

1.6.2 Server Connection

Discover and connect to the primary server

Sentinel discovers the master server through a user-given profile.

Sentinel is created with the monitored master server

Two network connections:

🐱🏍 command connection Used to send commands to the primary server.

🐱🏍 Subscription link is used to subscribe to the specified channel, thereby discovering

Monitor other Sentinels for the same master server.

Discover and connect to the slave server

Sentinel automatically gets the addresses of all slave servers by sending the INFO command to the primary server.

Like the master server, Sentinel creates command and subscription connections with each discovered slave server.

Other sentinels found

Sentinel announces its presence to other Sentinels by sending a “HELLO” message containing the IP address, port number, and ID of the Sentinel to the monitored master and slave servers via a command connection. At the same time, Sentinels receive “HELLO” messages from other Sentinels via subscription connections to discover other Sentinels monitoring the same master server.

Sentinel1 makes Sentinel2 and Sentinel3 find themselves by sending a HELLO message, and the other sentinels do the same.

Links between multiple Sentienl

Sentinels only create command connections with each other for communication. Because there are already master and slave servers that mediate the sending and receiving of HELLO messages, no subscription connections are created between Sentinels.

1.6.3 Checking the Instance Status

Sentinel uses the PING command to check the status of instances: if an instance does not return a reply within a specified period of time, or if it returns an incorrect reply, the instance is considered offline by Sentinel.

There are two different definitions of downsizing in Redis Sentinel:

🥓 Subjective offline (Subjectively Down, or SDOWN for short) refers to the offline judgment of a server made by a single Sentinel instance.

🥓 Objectively Down (ODOWN) refers to the fact that Sentinel instances are judged SDOWN on the same server and communicated with each other through Sentinel is-master-down-by-ADDR. The resulting server offline judgment. (A Sentinel can ask another Sentinel if it considers a given server offline by sending the Sentinel IS-master-down-by-addr command to the other Sentinel.)

If a server does not return a valid reply to the Sentinel that pinged it in the amount of time specified in the master-down-after-milliseconds option, Sentinel then marks the server as subjective offline.

1.6.4 FAILOVER FAILOVER

A failover operation consists of the following steps:

1. The primary server was found to be offline.

2. Ballot election based on Raft Leader Election protocol

3. If the election fails, try again after twice the set failover timeout. If elected, perform the following steps.

4. Select a slave server and upgrade it to the master server.

5. Send the SLAVEOF NO ONE command to the selected slave server to convert it to the master server.

6. The updated configuration is propagated to all other Sentinels, who update their own configurations, through the publish-subscribe function.

7. Send the SLAVEOF command to slave servers that have taken the master offline to replicate the new master.

8. Leader Sentinel terminates the failover operation when all slave servers have started replicating the new master.

1.6.5 configuration sentinel

Create program directory

cd /application mkdir 26380 cp /usr/local/redis/src/redis-sentinel ./26380/ cd 26380

Editing a Configuration File

Vim sentinel.conf port 26380 dir "/ TMP "sentinel monitor myMaster 127.0.0.1 6380 2 Sentinel down-after-milliseconds mymaster 60000 sentinel config-epoch mymaster 0

Start the sentinel

./redis-sentinel ./sentinel.conf

Configuration File Description

Sentinel sentinel monitor myMaster 127.0.0.1 6370 2 #{2 indicates how many sentinels agree} # sentinel auth-pass myMaster root Sentinel down-after-milliseconds mymaster 15000 times sentinel fail-over -timeout mymaster 900000 # The two configurations need to be the same: Sentinel leader-epoch myMaster 1 Sentinel config-epoch myMaster 1Copy the code

1.6.6 Running the Sentinel command

1.6.7 Sentinel publishing and subscription information

Clients can think of Sentinel as a Redis server that only provides subscriptions: You cannot PUBLISH to the server, but you can SUBSCRIBE or PSUBSCRIBE to receive notifications of events by subscribing to a given channel.

A channel can receive events with the same name as the channel. For example, a channel named +sdown can receive events when all instances enter the subjective sDown state.

All event information can be received by executing the PSUBSCRIBE * command.

The following is a list of channels and information formats that clients can subscribe to:

The first English word is the name of the channel/event, the rest is the format of the data.

Note that when the format contains the word Instance Details, it means that the information returned by the channel is contained in the package

Contains the following to identify the target instance:

The content after the @ @ character is used to specify the primary server, which is optional, and is only used if the instance specified by the content before the @ character is not the primary server.

1.7 Redis cluster

1.7.1 Redis cluster

Redis cluster is an installation that can share data between multiple Redis nodes.

Redis clustering does not support Redis commands that require processing of multiple keys at the same time, because executing these commands requires moving data between multiple Redis nodes, and under high loads, these commands can degrade the performance of the Redis cluster and lead to unpredictable behavior.

Redis clusters provide a degree of availability through partitions: the cluster can continue to process command requests even if some nodes in the cluster fail or fail to communicate. The ability to automatically split data to multiple nodes.

The ability to continue processing command requests when a portion of the cluster fails or cannot communicate.

1.7.2 Redis Cluster Data Sharing

Redis clustering uses data sharding instead of consistency hashing: A Redis cluster contains 16384 hash slots in which each key in the database belongs. The cluster uses the formula CRC16(key) % 16384 to calculate which slot the key belongs to. The CRC16(key) statement is used to calculate the CRC16 checksum of the key.

Node A handles hash slots 0 through 5500.

Node B handles hash slots 5501 through 11000.

Node C handles hash slots 11001 through 16384.

The formula for calculating slots

The cluster uses the formula CRC16(Key) & 16383 to calculate which slot the key belongs to.

1.7.3 Cluster Operation Mechanism

All redis nodes are ping-pong with each other and use binary protocols internally to optimize transmission speed and bandwidth.

Nodes fail only when more than half of the master nodes in the cluster detect failures.

The client is directly connected to the Redis node without the intermediate proxy layer. The client does not need to connect to all nodes in the cluster, but to any available node in the cluster

Map all physical nodes to slot [0-16383]. Cluster maintains node<->slot<-> keys

In order to make the cluster work properly even when some nodes go offline or cannot communicate with the majority of nodes, the Redis cluster uses the master-slave replication function for nodes: Each node in a cluster has 1 to N replicas. One replica is the master node and the other N-1 replicas are slaves.

In the previous examples of nodes A, B, and C, if node B went offline, the cluster would not function properly because the cluster could not find A node to handle hash slots 5501 through 11000.

If at the time of cluster creation (or at least before node B goes offline) we add slave node B1 to primary node B, then when primary node B goes offline, the cluster sets B1 as the new primary node and replaces the offline primary node B. Continue processing hash slots 5501 through 11000 so that the cluster does not fail due to primary node B going offline.

However, if both nodes B and B1 go offline, the Redis cluster will still stop working.

The replication feature of the cluster reuses the SLAVEOF command code, so the replication behavior of cluster nodes is exactly the same as that of the SLAVEOF command.

1.7.4 Cluster Failover

1. In a cluster, nodes check for other nodes to go offline.

2. When a primary node goes offline, other primary nodes in the cluster failover the primary node.

3. In other words, the nodes of the cluster integrate sentinel-like features such as offline detection and failover.

4. Because Sentinel is a stand-alone monitor, and cluster functions such as downline detection and failover are integrated into nodes, they run in very different modes, so the cluster implementation does not reuse Sentinel code even though the two functions are very similar.

Two cases of executing commands in a cluster

The command is sent to the correct node:

The slot in which the key to be processed by the command happens to be the responsibility of the node receiving the command, which executes the command as if it were a stand-alone Redis server.

Slot description:

7000: slot 0 ~ 5000

7001: slot 5001 ~ 10000

7002: slot 10001 ~ 16383

The date key is located in slot 2022, which is in the charge of node 7000. The command is executed directly.

The command was sent to the wrong node:

If the node that receives the command is not the node in the slot where the processing key resides, the node will return a redirection error to the client, telling the client which node to run the command on. The client will re-send the command to the correct node based on the error information.

The key date is in slot 2022, which is in charge of node 7000, but the error is sent to node 7001, and 7001 returns a steering error to the customer.

The client redirects to node 7000 based on the steering error and resends the command

1.7.5 On steering errors

Nodes in the cluster tell each other which slots they are responsible for.

Each node in the cluster records which node is responsible for 16,384 slots, forming a slot table.

When receiving a command request, the node checks whether the slot where the key resides is processed by the node through the slot table:

✍ If it is, the node executes the command directly;

✍ if it is not, then the node extracts the correct node address information from the slot table and returns a steering error.

1.7.6 Configuring a Cluster

preparation

#EPEL source install Ruby support

yum install ruby rubygems -y

Use domestic sources

gem source -a http://mirrors.aliyun.com/rubygems/ -remove https://rubygems.org/ # gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ # gem install redis -v 3.3.3 gem sources -lCopy the code

The configuration file

A Redis cluster consists of multiple instances of Redis running in cluster mode. The cluster mode of instances must be enabled by configuration. Instances in cluster mode can use cluster-specific functions and commands.

Here is an example cluster configuration file with the fewest options:

port 7000 cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 appendonly yes

Create program directory

cd /application/redis mkdir 7000 7001 7002 7003 7004 7005

Copies of the application

For I in 0 1 2 3 4 5 do cp/usr/local/redis/SRC/redis – server / 700 $I done to create the configuration file

for i in 7000 7001 7002 7003 7004 7005 do echo "port $i cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 appendonly yes" > $i/redis.conf done

Start the Redis cluster

for i in 7000 7001 7002 7003 7004 7005 do cd $i ./redis-server ./redis.conf & cd .. / done

Create the cluster

CD /usr/local/redis/src/. /redis-trib.rb --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:127.0.0.1 7004:7005

Given that the command of the redis-trib.rb program is create, this means that we want to create a new cluster.

The –replicas 1 option means that we want to create a slave node for each primary node in the cluster.

1.7.7 Cluster Management

Write data to view cluster status

redis-cli -c -p 7000 set foo bar get foo

Resharding practice

CD /usr/local/redis/src/. /redis-trib.rb reshard 127.0.0.1:7000

State of the cluster

redis-cli -p 7000 cluster nodes | grep master

failover

redis-cli -p 7002 debug segfault

Check the status

redis-cli -p 7000 cluster nodes | grep master

Adding new nodes

. / redis - trib. Rb add -node 127.0.0.1:127.0.0.1 7006:7000

Deleting a node

redis-trib del-node ip:port '<node-id>'

Before deleting the master node, use reshard to remove all slots of the master node and then delete the current node

Add a slave node

Rb add-node --slave --master-id $[nodeid] 127.0.0.1:7008 127.0.0.1:7000./redis-trib.rb add-node --slave --master-id $[nodeid] 127.0.0.1:7008 127.0.0.1:7000

1.7.8 Status Description

How long has it been since the cluster last sent a PING command to a node that no response has been received?

The last time the node returned PONG.

Configuration epoch of nodes: Refer to the Redis cluster specification for details.

The network connection of this node is as follows: Connected.

Slots currently contained in a node: For example, 127.0.0.1:7001 currently contains hash slots 5960 to 10921.

Redis 1.8 API

1.8.1 PHP uses Redis

Tar ZXVF 2.2.7. Tar. Gz CD phpredis 2.2.7 / application/PHP/bin/phpize. / configure --with-php-config=/application/php/bin/php-config make && make install echo 'extension="redis.so"' >> /application/php/lib/php.ini service php-fpm restart service nginx restart

Connection test code

`[root@clsn ~]# cat /application/nginx/html/check.php

String manipulation

`
connect (127.0.0.1, 6379); echo “Connection to server sucessfully”; Redis −>set(“tutorial−name”,”Redistutorial”); Echo “Storedstringinredis::”. Redis ->set(“tutorial-name”, “redis tutorial”); Echo “Stored string in redis:: “. Redis −>set(“tutorial−name”,”Redistutorial”); // Get stored data and print echo”Storedstringinredis::”. Redis –

get(“tutorial-name”);

? > `

1.8.2 Python connecting to Redis

Installing software Packages

[root@Redis ~]# yum install python-pip ipython -y [root@Redis ~]# pip install redis

test

[root@Redis ~]# ipython
In [1]: import redis

In [2]: clsn = redis.StrictRedis(host='localhost', port=6379, db=0)

In [3]: clsn.set('blog','blog.nmtui.com')
Out[3]: True

In [4]: clsn.get('blog')
Out[4]: 'blog.nmtui.com'
Copy the code

The last

I here organized a Redis database related information documents, Spring series of family, Java systematic information (including Java core knowledge points, interview topics and 20 years of the latest Internet real questions, e-books, etc.) friends who need to pay attention to the public number [procedures Yuan Xiao wan] can obtain.