Redis is an open source (BSD-licensed), in-memory data structure storage system that can be used as a database, cache, and messaging middleware. It supports many types of data structures, such as strings (hashes), lists (Lists), sets (sets), sorted sets (sorted sets) and range queries, bitmaps, Hyperloglogs and Geospatial index radius query. Redis is built with replication, LUA scripting, LRU eviction, transactions and different levels of disk persistence, And provides high availability through Redis Sentinel and Automated partitioning (Cluster).

Redis is multi-threaded after version 6.0, single-threaded for users before version 6.0, and multi-threaded internally.

Redis is an open source, written in C, memory based persistent operation, key-value database, support a variety of programming languages API

Linux install Redis

  1. Downloading the Installation package
  2. Move the installation package to the /opt root directory on the same level as home
  3. Unzip the file,Sudo tar - ZXVF redis - 6.0.9. Tar. Gz

  1. Install GCC (if not)
sudo apt update
sudo apt install build-essential
gcc --version
Copy the code
  1. Run the make command
sudo make
sudo make install
Copy the code

  1. The default redis installation path is /usr/local/bin

  2. Copy the redis.conf file (from the decompression path) to the custom folder (yconfig) in the default redis installation path

  1. Redis is not started in the background by default. You need to modify the configuration file

  1. Start the Redis service

  1. Stop the Redis service

On the Redis-CLI client, enter shutdown

Redis basics

Redis has 16 databases and uses database 0 by default

select num   	# select database
DBSIZE    		Check the database size
flushdb			Clear the current database
flushall		Clear all databases
exists	name	# check whether the current key exists
move name 1 	Remove the current key
expire name 10 	# set key expiration time
ttl	name		# check the remaining time of the current key
type name		Check the current key type
Copy the code

Redis operates on memory, not CPU, but machine memory and network bandwidth.

Why is Redis so fast?

A: Redis processes all data in memory, so it’s fastest to do it through a single thread. With multithreading, the CPU context is switched (which is time consuming).

Three special data types

Geospatial

Usage scenario: Locate and measure the distance

There are only six commands

# geoAdd adds the geographic location
# Rule: Earth polar coordinates cannot be added
# key value (latitude longitude name)GEOADD China: City 116.40 39.90 Beijing GEOPOS China: City nameGet the latitude and longitude of the specified position
geodist china:city beijing shanghai Get the distance between two points
# Georadius takes a given latitude and longitude as the center to find elements within a certain radius
georadius china:city 110 30 1100 km
georadiusbymember china:city beijing 100 km # find the elements within the specified range
geohash
Copy the code

The underlying principle of GEO is to use sort Sets

bitmaps

Bit storage, which can be stored using Bitmaps if there are only two states of an object

setbit
getbit 
Copy the code

hyperloglog

Redis2.8.9 updates the hyperloglog data structure

Hyperloglog is an algorithm for cardinal statistics (using hyperloglog to count visits if fault tolerance is allowed)

PFADD mykey a b c d e
PFCOUNT mykey
PFMERGE mykey3  mykey1 mykey2
Copy the code

The transaction

Redis transaction is essentially a set of commands, these commands will be serialized, and in the process of transaction execution will ensure one-time, sequential, exclusive!

Redis single commands guarantee atomicity, but transactions in Redis do not.

Redis transactions have no concept of isolation levels

All commands are executed in a transaction, not directly, but only when the execution command is initiated. Exec

Redis transactions are divided into three phases:

  1. Start transaction (multi)
  2. Command to join ()
  3. Execute transaction (exec)
127.0.0.1:6379 > multi OK 127.0.0.1:6379 >setK1 v1 QUEUED 127.0.0.1:6379 >setK2 v2 QUEUED 127.0.0.1:6379 >setK3 v3 QUEUED 127.0.0.1:6379> get k2 QUEUED 127.0.0.1:6379>exec
1) OK
2) OK
3) OK
4) "v2"
Copy the code

To cancel a transaction with a discard, none of the commands in the transaction queue will be executed.

Compiled exception, all commands in the transaction will not be executed.

Run time exception, the transaction error command will not be executed, other commands are executed normally.

Pessimistic locking: As the name implies, thinking that something can go wrong at any time, so you lock up whatever you do

Optimistic locking: Very optimistic, thinking that there is no problem at any time, so it will not lock, update the data will determine whether someone has changed the data in the meantime.

  • Verify with version
  • Access to the version
  • Compare version when updating

Through the watch command to achieve Redis optimistic lock operation, unwatch to unlock.

When watch is applied to a transaction, the transaction will fail if it is found that the monitored object has been modified

  • The first step is to unlock it through Unwatch
  • The second step is to get the latest value and listen through Watch
  • The third part is executing transactions through exec.

Jedis

If you use Java to manipulate Redis, use jedis dependencies

Testing:

  1. Connecting to a Database
  2. Operation command
  3. disconnect

The test code

public class TestPing {
    public static void main(String[] args) {
        //1. new a jedis object
        Jedis jedis = new Jedis("127.0.0.1".6379);
        // All the commands we learned before become methods hereSystem.out.println(jedis.ping()); }}Copy the code

If the output is PONG, the test is successful

Jedis operates transactions

public class TestPing {
    public static void main(String[] args) {
        //1. new a jedis object
        Jedis jedis = new Jedis("127.0.0.1".6379);
        jedis.flushDB();
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("hello"."world");
        jsonObject.put("name"."yang");
        String s = jsonObject.toJSONString();
        // All the commands we learned before become methods here
        // Start the transaction
        Transaction multi = jedis.multi();
        try {
            multi.set("user1",s);
            multi.set("user2",s);
            multi.exec();// Execute a transaction
        }catch (Exception e){
            e.printStackTrace();
            multi.discard();// If there is an error, close the transaction
        }finally {
            System.out.println(jedis.get("user1")); System.out.println(jedis.get("user2")); jedis.close();// Close the connection}}}Copy the code

SpringBoot integrate redis

After SpringBoot2.x, the original jedis used is replaced with lettuce.

Jedis: The underlying direct connection, multi-thread operation is not secure, if you want to avoid insecurity, use Jedis Pool connection pool, more like BIO mode.

Lettuce: With Netty, instances can be shared between multiple threads. There is no thread insecurity. Can reduce thread data, more like NIO mode.

  1. Importing dependent dependencies

            <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
Copy the code

  2. Configure the connection

    spring:
  redis:
    host: 127.0. 01.
    port: 6379

Copy the code

  3. test

            //opsForValue Operation string
/* redisTemplate.opsForValue(); redisTemplate.opsForZSet(); RedisConnection connection = redisTemplate.getConnectionFactory().getConnection(); connection.flushDb(); * /
        redisTemplate.opsForValue().set("gg"."gameover");
        System.out.println(redisTemplate.opsForValue().get("mykey"));
        System.out.println(redisTemplate.opsForValue().get("gg"));
        redisTemplate.getConnectionFactory().getConnection().flushDb();
        System.out.println(redisTemplate.opsForValue().get("gg"));
Copy the code

Custom RedisTemplate

To be added

Redis. Conf

The Redis service is started from a configuration file

Unit is case insensitive

Like other configuration XML files that can be imported through import in Spring configuration files, Redis imports through includ.

Internet NETWORK

bind 127.0.0.1   # Bound IP address
protected-mode yes	# Protected mode
port 6379		Set the port number
Copy the code

GENERAL GENERAL

daemonize yes	# run as daemon, default is no, we need to manually set to yes

pidfile /var/run/redis_6379.pid	If running in the background, we need to specify a PID file

# log
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel notice

logfile ""  The file location name of the log

databases 16	Number of databases, default is 16

always-show-logo yes	Is the logo always displayed

Copy the code

The snapshot

Persist to a file,.rdb.aof, based on how many operations are performed within a specified time

Redis is an in-memory database. If it is not persisted, the data will be lost when power is lost.

save 900 1  If at least one key has been changed within 900 seconds, then the operation is persisted.
save 300 10 # same as above
save 60 10000

stop-writes-on-bgsave-error yes	If something goes wrong, whether to continue working

rdbcompression yes	Compress RDB files, enabled by default

rdbchecksum yes		Check the RDB file while maintaining it

dir ./		# RDB file save path, default is the current directory

Copy the code

security

You can set a password for Redis. Redis has no password by default

config get requirepass Get the redis password
config set requirepass "123456"  Set password to 12456
Verify the password after setting it
auth 123456  
Copy the code

The client

maxclients 10000 # set the maximum number of clients that can connect to Redis
maxmemory <bytes> #redis configures maximum memory capacity
maxmemory-policy noeviction	There are 6 processing strategies after the memory reaches the upper limitVolatile - lRU: lRU only for keys whose expiration time is set (default value) 2. Allkeys-lru: delete keys of lRU algorithm 3. Volatile -random: delete keys that are about to expire randomly 4. Volatile - TTL: delete items that are about to expire. Noeviction: never expire, return errorCopy the code

APPEND ONLY MODE (AOF configuration)

appendonly no 	The aOF mode is disabled by default. The RDB persistence mode is used by default

appendfilename "appendonly.aof"	The name of the persistence file

# appendfSync always # Every change is synchronized, consuming performance
appendfsync everysec	Perform synchronization once per second
# appendfsync no

Copy the code

Redis persistence

RDB (Redis DataBase)

What is RDB?

Writes a collective snapshot of data in memory to disk at a specified interval. When it recovers, the snapshot file is read directly into memory.

Redis forks a word process for persistence, writing data to a temporary file that will be replaced by the last persistent file after the persistence process is complete. The main process does not perform any IO operations during the entire process. This ensures extremely high performance. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB is more efficient than AOF. The downside of RDB is that data can be lost after the last persistence. Our default is RDB, and we generally don’t need to change this configuration! The file saved by RDB is dump.rdb

Conditions that trigger the RDB mechanism

  1. The RDB rule is triggered if the save rule is met
  2. The flushall command also triggers the RDB rule
  3. Exiting Redis also generates RDB files

The backup automatically generates a dump. RDB file

How do I restore RDB files

When redis starts, it automatically reads dump. RDB and restores the data in it.

Check the location where you want to exist

127.0.0.1:6379> config get dir
1) "dir"
2) "/usr/local/bin"		# If there is a dump. RDB file in this file, it will be automatically restored on startup127.0.0.1:6379 >Copy the code

Advantages:

  1. And large-scale data recovery
  2. Data integrity requirements are not high

Disadvantages:

  1. A certain amount of time is required for the process to operate, and if Redis is down, the last chance to modify the data is lost.
  2. The fork process takes up some memory

AOF (Append Only File)

Write down all of our commands, and execute all of the commands when we restore them

Each write operation is recorded in the form of a log, and all instructions executed by Redis are recorded (no read operations are recorded). Only files can be appended but files cannot be overwritten, and Redis will read the file to rebuild the data when it starts. In other words, when Redis restarts, all write instructions are executed in order according to the contents of the log file to complete the data recovery.

The file saved by AOF is appendone.aof

If BOTH AOF and RDB are enabled, which strategy does Redis use?

AOF is based on command append, while RDB is based on snapshot. According to the policy, a snapshot of data is saved at intervals. By comparison, AOF updates more frequently. Data is more complete, so if AOF and RDB exist at the same time,Redis will restore the database state from AOF file preferentially, and if AOF is closed, it will restore from RDB.

If the AOF file has an error, Redis cannot start. We need the redis-check-aof –fix command to fix the faulty AOF file.

Redis publishes subscriptions

Three elements: publisher queue subscriber

test

# the subscriber127.0.0.1:6379 > subscribe kuangshenshuoSubscribe to a channel kuangshenshuo
1) "subscribe"
2) "kuangshenshuo"(3)integer) 1, 1)"message"    # Monitor and read the information pushed by publishers
2) "kuangshenshuo"
3) "hellokuang"
1) "message"   # message
2) "kuangshenshuo"	# Which channel is it from
3) "hello,redis"	# Message content

# the publisher127.0.0.1:6379 > publish kuangshenshuo"hellokuang"
(integer) 1
127.0.0.1:6379> publish kuangshenshuo "hello,redis" # Post information on the specified channel
(integer6379 > 1 127.0.0.1) :Copy the code

The principle of

Redis implements the publish and subscribe function through subscribe, psubscribe, publish and other commands.

After subscribing to a channel with the subscribe command, a dictionary is maintained in Redis-server. The keys of the dictionary are channel by channel, and the value is a linked list, which stores all clients that subscribe to this channel. The key to the subscribe command is to add a client to the subscription list of the corresponding channel.

Redis primary/secondary replication

concept

Primary/secondary replication refers to the replication of data from one Redis server to other Redis servers. The former is called the master node (master/leader) and the latter is called the slave node (slave/follower). == Data replication is one-way and can only go from primary node to secondary node ==. The Master mode is mainly written, while the Slave mode is mainly read.

By default, each Redis server is the primary node; And a master node can have multiple slave nodes (or none), but a slave node can only have one master node.

Primary/secondary replication provides the following functions: 1. Data redundancy: Primary/secondary replication implements hot backup of data and is a data redundancy mode other than persistence. 2. Fault recovery: When the primary node is faulty, the secondary node provides services for rapid fault recovery. It’s actually redundancy of services. 3. Load balancing: on the basis of master/slave replication, with read/write separation, the master node can provide write service, and the slave node can provide read service (that is, when Redis data is written, the application connects to the master node and when Redis data is read, the application connects to the slave node) to share server load; Especially in the scenario of less write and more read, the concurrency of Redis server can be greatly increased by sharing the read load among multiple slave nodes. High availability cornerstone: In addition to the above functions, master-slave replication is the foundation upon which sentry and clustering can be implemented, hence master-slave replication is the foundation of High availability in Redis.

In general, to use Redis in engineering projects, only one Redis is absolutely not (down), the reasons are as follows: 1, from the structure, a single Redis server will occur a single point of failure, and – the server needs to deal with all the request load, the pressure is large; 2, from the capacity, a single Redis server memory capacity is limited, even if a Redis server memory capacity of 256G, also can not use all memory as Redis storage memory, generally speaking, a single Redis maximum use of memory should not exceed 20G.

Commodities on e-commerce websites are usually uploaded once and browned for countless times, which means “read more and write less”. For this scenario, we can make this architecture as follows: |

Environment configuration

Configure only the slave library, not the master library.

View information about the current library127.0.0.1:6379 > info replication# Replication
role:master  # Role host
connected_slaves:0  The number of slave machines is 0
master_replid:d345a13ec64eb0821ca44c66703aeb10e99ea35d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Copy the code

Copy the three configuration files (redis.conf) and modify the corresponding information

  1. The port number

  2. Pid name

  3. Log file name

  4. Dump. RDB file name

After modification, start three Redis servers

Configure the host on the host and enter the host name on the CLI

Slaveof Host IP address Host port numberCopy the code

The actual master/slave configuration should be configured in the configuration file so that it is permanent. Our command configuration is temporary.

details

  • The master machine can write, the slave machine can read.
  • All data in the host is automatically saved by the slave
  • After the host is disconnected, the slave machine is still connected to the host. After the host is restarted, the written data can still be read from the slave host
  • If the master/slave configuration is configured in command line mode, the slave machine becomes the host after being disconnected. If it becomes a slave machine again, the data of the master machine can be read immediately.

The principle

After connecting to the master successfully, the Slave sends the sync command

The master receives the command to start the background saving process and collects all the commands received to modify the data set. After the background process is complete, the master sends the entire data file to the slave and completes a complete synchronization.

Full replication: After receiving the database file data, the slave service saves it and loads it into memory.

Incremental replication: The master sends all the new collected modification commands to the slaves in turn to complete the synchronization.

However, a full synchronization (full copy) will be performed automatically whenever the master is reconnected

Slaveof No One can be used to convert a slave machine to a host

The guard mode

Redis2.8 provides sentinel mode to solve manual switching between master and slave libraries. Sentinel can automatically monitor whether the host fails in the background. If it fails, it will automatically transfer from the slave library to the master library according to the number of votes.

Sentinel mode is a special mode, first of all Redis provides sentinel command, sentinel is a process that runs independently. The idea is that sentry monitors multiple instances of Redis running by sending commands and waiting for the Redis server to respond.

The role of sentinels:

  • Send commands to the Redis server to return to monitor its running status, including master and slave servers.
  • When the sentry detects that the master is down, it automatically switches the slave to master and notifies the other slave servers in public-subscribe mode to modify their configuration files and make them switch hosts.

However, one sentry process monitoring the Redis server is not enough, so multiple sentries are used to monitor each other. This creates the multiple sentinels pattern.

If the primary server detects the result immediately and Sentry 1 detects the result first, the system does not perform failover immediately but sentry 1 considers that the primary server is unavailable. This phenomenon is called subjective offline. When the sentries at the back also detect that the primary server is unavailable and the number reaches a certain value, the sentries vote to select other secondary servers as the primary server (the vote is initiated by any sentries). According to the voting result, the sentries perform failover operations. After the switch is successful, each sentinel will switch the host from the secondary server monitored by himself through the publish and subscribe mode, which is called objective offline process.

1. Configure the sentinel configuration file sentinel.conf

Host port 1
# the number 1 indicates that if the host dies, the one with the most votes will be selected as the host based on the results of the voteSentinel Monitor Myredis 127.0.0.1 6379 1Copy the code

2. Activate the sentry

Run the redis-sentinel yconfig/sentinel.conf command to start the server as if it were a redis server

Sentinel mode rule: if the host dies, the competent host is selected from the slave based on the vote result. If the host returns after the vote, it can only be merged to the new host slave.

Advantages:

  1. The Sentinel cluster, based on the master-slave replication mode, has all the advantages of a master-slave configuration
  2. The master and slave can be switched automatically, and the fault can be transferred, improving the system availability
  3. Sentinel mode is an upgrade from master to slave mode, manual to automatic

Disadvantages:

  1. Redis is not good for online expansion
  2. Sentinel mode is cumbersome to configure

Redis cache penetration and avalanche

Cache penetration (not found)

concept

The user wants to query a data, and finds that the redis in-memory database does not have it, that is, the cache does not hit, so he goes to the database to query, and finds that there is no data. So this query failed, when a lot of users, the cache did not hit, so they went to request the persistent layer database, which will give the persistent layer database a lot of pressure, when the database can not withstand, at this time, it is equivalent to the occurrence of cache penetration.

The solution

1. Bloom filter

A Bloom filter is a data structure that stores all possible query parameters in the hash format. The parameters are verified at the control layer and discarded if they do not meet the requirements, thus avoiding the query pressure on the underlying storage system.

2. Cache empty objects

When the storage layer is not hit, even the returned empty object is cached, and an expiration time is set. Later access to the data is retrieved from the cache, protecting the back-end data source.

This approach also has the following problems:

  1. If null values are cached, it means that the cache needs more space to store keys, most of which may be null values.
  2. Even if the expiration time is set for null values, data at the cache layer and storage layer will have inconsistent time Windows for a period of time, which affects services that require consistency.

Cache breakdown (too many queries, cache expired)

Breakdown is a cache key very hot (hot), and then kept carrying big concurrent, big concurrent concentrated on a visit to this key, when the key in the instant of failure, continuous big concurrent will wear out the cache, direct request database, query the latest data and write back cache, this time cause instantaneous pressure is too large, It’s like cutting a hole in a barrier.

The solution

1. Set the expiration date of hotspot data

From the cache level, the expiration time is not set, so there are no problems caused by hot key expiration

2. Add a mutex lock

Distributed lock: The distributed lock ensures that only one thread queries back-end services for each key. Other threads do not have the permission to obtain the distributed lock, so they only need to wait. This approach shifts the pressure of high concurrency to distributed locks, which is very challenging for distributed locks.

Cache avalanche

Cache avalanche refers to a period in which the cache collectively expires.

The solution

1. Redis high availability

Add more Redis servers (cluster)

2. Traffic limiting degrade

After cache invalidation, the number of threads that can read the database write cache is controlled by locking or queuing. For example, only one thread is allowed to query data and write to the cache for a key, while the other threads wait. Focus on a few key features and stop serving the others.

3. Preheat data

In this way, the data will be loaded into the cache. Before large concurrent access occurs, different cache keys will be loaded manually. Different expiration times will be set to ensure that the cache failure time is as even as possible.