Redis

Distributed cache

First, use scenarios

1. Database cache

Cache – Replication level of raw data for fast access

The accessed content is cached, and the first cache is accessed again. If the cache matches the returned data, the database is not matched, and the cache is backfilled

2. Improve system response

Redis data is stored in memory, while database data is mostly stored in hard disk. The access speed of memory is much faster than that of hard disk, and compared with database, a large number of read/write requests can be processed instantly

3. Split sessions

When the architecture uses Tomcat cluster and Tomcat for Session replication, (1) performance loss during replication and (2) real-time synchronization cannot be guaranteed. Therefore, sessions can be unified in Redis. This allows multiple Tomcats to share Session messages

4. Optimistic locking

Use Redis watch + INCR implementation

jedis1.watch(redisKey);
String redisValue = jedis1.get(redisKey);
int valInteger = Integer.valueOf(redisValue);
String userInfo = UUID.randomUUID().toString();

// There are no seconds left
if (valInteger < 20) {
    Transaction tx = jedis1.multi();
    tx.incr(redisKey);
    List list = tx.exec();
    // Second success failure returns an empty list instead of empty
    if(list ! =null && list.size() > 0) {
    	System.out.println("User: + userInfo + ", seckill succeeded! Number of current successes:" + (valInteger + 1));
    }
    // The version has changed
    else {
    	System.out.println("User: + userInfo + ", seckill failed"); }}/ finished/SEC
else {
	System.out.println("We've got 20 seconds to kill, seconds to kill.");
}
Copy the code

5. Distributed locks (pessimistic locks)

If you need to control the concurrency of multiple processes (JVMS) timing (serialization), you can use Redis setNX implementation

If value does not exist, it is an atomic operation

127.0.0.1:6379> setnx name zhangf # If name does not exist assignment (integer) 1 127.0.0.1:6379> setnx name zhaoyun # Assignment failed again (integer) 0 127.0.0.1:6379 > get the name "zhangf"Copy the code
127.0.0.1:6379> set age 18 NX PX 10000 # if there is no assignment validity 10 seconds OK 127.0.0.1:6379> set age 20 NX # assignment failure (nil) 127.0.0.1:6379> 127.0.0.1:6379> set age 30 NX PX 10000 # set age OK 127.0.0.1:6379> get age "30"Copy the code

Implement distributed locking yourself

/** * Use the redis set command to obtain a distributed lock@paramA lockKey can be a lock@paramRequestId requestId, which ensures the same uuid+threadID *@paramExpireTime Expiration time to avoid deadlock *@return* /
public boolean getLock(String lockKey,String requestId,int expireTime) {
    // NX: ensure mutual exclusion
    If the lockKey is valid, a process is using a distributed lock
    String result = jedis.set(lockKey, requestId, "NX"."EX", expireTime);
    if("OK".equals(result)) {
    	return true;
    } 
    return false;
}
Copy the code
public static boolean releaseLock(String lockKey, String requestId) {
    String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
    Object result = jedis.eval(script, Collections.singletonList(lockKey),Collections.singletonList(requestId));
    if (result.equals(1L)) {
        return true;
    } 
    return false;
}
Copy the code

Use Redission to implement distributed locking

public class DistributedRedisLock {
    
    // Get the Redisson object from the configuration class
    private static Redisson redisson = RedissonManager.getRedisson();
    private static final String LOCK_TITLE = "redisLock_";
    
    / / lock
    public static boolean acquire(String lockName){
        // Declare the key object
        String key = LOCK_TITLE + lockName;
        // Get the lock object
        RLock mylock = redisson.getLock(key);
        // Add a lock and set the lock expiration time to 3 seconds to prevent deadlocks
        mylock.lock(2.3,TimeUtil.SECOND);
        // Lock succeeded
        return true;
    }
    
    // Lock release
    public static void release(String lockName){
        // It must be the same key as the lock
        String key = LOCK_TITLE + lockName;
        // Get the object
        RLock mylock = redisson.getLock(key);
        // Release the lockmylock.unlock(); }}Copy the code

6. MyBatis level 2 cache

Two, the interpretation of the lock

  1. Pessimistic lock – I think there will be a problem, so lock first – poor performance

    • synchronized

    • Row and table locks in a database

  2. Optimistic lock. – Anyone can do it, but if I do it, you don’t. – High performance

    • Seconds kill scenes

Read/write mode of cache

1. Cache Aside Pattern – Redis

  • Read: the cache is read first, but the cache does not read the database. The cache retrieves data from the cache and returns a response

  • More: Update the database first, then delete the cache

Reasons for not updating the cache:

  • If the cache is a complex structure (hash or list), traversal is required to increase the complexity

  • Update but not delete may cause dirty read

2. Read/Write Through Pattern – Guava Cache

Application processes only operate on the cache, which operates on the database

3. Write Behind Caching Pattern – EVCache

The application process only updates the cache, and the cache is “asynchronously” batch updated to the database – it cannot be synchronized in real time and may even lose data

4. Data types

1. String Indicates the character String

  1. string

  2. digital

  3. Floating point Numbers

  4. Optimistic lock – Watch + INCR

  5. Distributed lock – setnx

2. List the List

Store ordered, repeatable elements, get head, tail records very fast

  1. Stack, queue,
  2. List of users, list of products, list of comments

3. Set the Set

Unordered, unique element

  1. Concerned Users
  2. Random draw (SPOP command)

4. SortedSet

Element is unique and can be sorted by fraction

Underlying implementation: skip list

  1. Click on the leaderboard
  2. Sales list
  3. Pay attention to the leaderboard

5. Hash Hash

Field and value mapping table of String type

  1. object
  2. Table data mapping

6. Bitmap Bitmap

Value can only be 0 or 1

  1. User sign in
  2. Counting active users
  3. Find the online status of the user

7

Use z-order curve, Base32 encoding, GeoHash algorithm to store latitude and longitude

  1. Record location
  2. Calculation of distance
  3. The man near the

8. Stream Data flows

Persist message queues

Five, expired, elimination strategy

1. maxmemory

  1. [Default] Disable eviction – can be used as DB – Too much data may crash
  2. [Recommended] Set this parameter to 3/4 of the physical memory

If maxmemory is set, maxmemory-policy should be configured

2. Maxmemory-policy (delete policy)

  1. Time to delete

    • Using a timer to delete an expired Key – Not recommended

  2. Lazy to delete

    • If the access Key has expired, it is deleted

  3. Active delete – Randomly selected key-value pairs, traversal is too time-consuming

    • No-enviction – Do not delete (default)

    • Allkeys-lru – furthest used (depending on the time used) – usually used

    • Volatile -lru – Selects the data that is most old from the set expiration date

    • Allkeys – LFU – Least recently used (see number of uses)

    • Volatile – lFU – Selects the least recently used data from which expiration time has been set

    • Allkeys-random -random – used when the request pressure is expected to be evenly distributed

    • Volatile -random – Randomly selected data from which expiration time has been set

    • Volatile – TTL – Selects the minimum TTL value

Redis uses lazy delete + active delete by default

3. expire (TTL)

Data survival time

Persistence

The goal is to quickly recover data rather than store it

AOF records process, RDB only results

1. RDB snapshot (default)

Process: The parent process forks the child process (the parent process blocks). The child process creates the RDB document, generates a snapshot document from the parent process memory, and replaces the original document atomically

Advantages:

  • Use binary compression, small space, convenient transmission

Disadvantages:

  • Data integrity cannot be guaranteed and all data after the snapshot will be lost

  • Fork The child process blocks. If the data is too large, it will fail to respond to the request for a short time. (To avoid this, close RDB and enable AOF.)

2. AOF operation logs

Record the commands you run in the AOF document

Advantages:

  • Data security, do not lose data

Disadvantages:

  • The performance is low

Save mode (hard disk)

  1. AOF_FSYNC_NO: not saved

    AOF is saved to disk only if (1) Redis is turned off, (2) AOF is turned off, and (3) the operating system write cache is flushed

  2. AOF_FSYNC_EVERYSEC: Save once per second [default]

    Data loss for a maximum of 2 seconds

  3. AOF_FSYNC_ALWAYS: Save one command once (not recommended)

    A maximum of one command data is lost

rewrite

AOF file commands to delete and merge, thin the file, and the whole process is absolutely safe

7, Redis weak transaction

After the transaction is started

(1) Syntax error – command queue clearance

(2) Run error [type error.. etc.] – Correct command run, no rollback

1. Redis ACID

  • Atomicity: A command in a queue that is either run or not run

  • Consistency: The state before and after a transaction must be consistent. Redis is an AP model. Real-time consistency cannot be guaranteed in a cluster, only final consistency can be guaranteed

  • Isolation: Commands run sequentially, but within a transaction, it is possible to run commands from other clients

  • “Durability” : They have persistence but do not guarantee data integrity

2. Transaction command

When the monitored field is changed by another client, the command queue that is enabled after monitoring is cleared

  1. Watch: Monitor key (shared within client)
  2. Multi: enables the transaction. Subsequent commands are added to the command queue
  3. Exec: Runs the command queue
  4. Discard: Clears the command queue
  5. Unwatch: Clear monitoring key (shared within client)
127.0.0.1:6379> watch s1 # Command queue will be cleared when the monitored field is changed by another client OK 127.0.0.1:6379> multi # OK 127.0.0.1:6379> set s1 555 # QUEUED 127.0.0.1:6379> exec # run command (nil) 127.0.0.1:6379> get s1 # run command failed because the command queue was empty 222 127.0.0.1:6379> unwatch # Unwatch key OKCopy the code

3. The Lua script

With strong atomicity, the script running process does not allow to insert new commands, so the running time should be as short as possible

Eight, to optimize

  1. Avoid large keys (value > 100K) → Remove small keys
  2. Avoid key*, hgetall… Equal full operation
  3. RDB changed to AOF, or even turned off
  4. A pipeline is used to add multiple pieces of data
  5. Using Hash storage
  6. Limit the memory size to avoid swap or OOM errors

High availability

1. Primary/secondary replication (read/write separation)

  1. The Lord can write, but never can

  2. The Lord is dead, never the Lord

  3. Enable it from the server using replicaof

2. The guard

When the primary goes offline, Sentinel will upgrade the secondary server to the primary server

Initialize the

  1. The Sentinel sends the info command to the Master to obtain the address of the Slave server. Then the Sentinel sends the info command to the Master and Slave to obtain the Redis status

  2. The Sentinels subscribe to the Hello channel to Master and Slave and send their own messages to the channel so that sentinels can be aware of each other

Sentinel Leader election

  1. Sentinel sends heartbeat connections to Redis every second, and if there is no response, it will be regarded as [subjective offline], and send search commands to other sentinels. If all sentinels with quorum consider this Redis offline, it will be judged as [objective offline].

  2. When the Master goes offline objectively, the Leader Sentinel will be selected to run the Redis [failover] using Raft protocol

Raft

The election begins and all nodes are followers. Maintain Follower status if RequestVote (vote for me) and AppendEntries (elected Leader) requests are received

If no request is received within a period of time (randomly), the Candidate will change his identity to the Candidate and run for the Leader. If he gets more than half of the votes, he will become the Leader.

If the Leader is not elected at last, Term + 1 starts the next round of election

failover

  1. Select the Slave to replace the original Master and have the other slaves copy the new Master

    Selection criteria (1) slave-priority maximum (2) Replication offset maximum (3) run_id Minimum [Minimum restart]

  2. Return the new Master address to the client

  3. Update Redis. Conf, sentinel.conf for all Redis

3. Codis (Proxy)

Advantages:

  • The client is transparent and interacts with Codis in the same way it interacts with Redis

  • Supports online data migration

  • Support high availability (Redis, Proxy)

  • Automatic data balancing

  • 1024 Redis instances are supported

Disadvantages:

  • Some commands are not supported

  • With only one Codis, performance drops by 20%

  • Using its own Redis branch, out of sync with the original

4. Redis Cluster

advantage

  1. A high performance

    • Multiple primary nodes, load balancing, and read/write separation

  2. High availability

    • Master/slave replication, Raft election

  3. Easy extension

    • Nodes can be added or removed without shutdown

    • Data fragmentation

  4. native

    • No other agents or tools are required and fully compatible with stand-alone Redis

Failure to determine

  • More than half of the primary nodes are down (unable to vote)

  • The primary and secondary nodes of a zone go offline at the same time (slot discontinuity)

A copy of the drift

The node group that has the most slave nodes in the cluster is migrated to the primary and secondary node groups at a single point

10. High concurrency

1. Cache penetration

Searching for data whose Key does not exist in the cache will search the database through the cache, causing the DB to be overloaded

Solution:

  1. If the result is null, the cache will be set to a shorter TTL, and the cache will be cleared after the database insert

    Problem: Caching null values takes up space

  2. Use a Bloom filter First use a Bloom filter to check whether the Key exists. If the Key does not exist, check the cache and DB

2. Cache avalanche

A large number of caches fail at the same time. As a result, the client directly searches for the DB and the DB is overloaded

Solution:

  1. Spread out the Key expiration period
  2. Set level 2 cache (local cache) – Data inconsistencies may exist
  3. High availability (read-write separation)

3. Cache breakdown

The hot Key fails, causing a DB record to be accessed in a large number of seconds

Solution:

  1. Use the distributed lock SETnx to keep the DB safe by keeping other threads in a wait state
  2. The Key does not set a timeout period, and the validate-lru expiration policy is used

4. Data consistency – Delayed dual Deletion

  1. Delete the cache after updating the DB, read the data and fill the cache
  2. Delete the cache again 2 seconds later
  3. Set the cache expiration time to 10 seconds or 1 hour
  4. If the cache fails to be deleted, it will be recorded in a log, extracted with a script and deleted (1 day).

5. The big Key

  1. Value is a String type and can be stored in MongoDB or CDN. If Redis is required, the Value can be stored separately in one master and multiple slave architecture
  2. Hash, set, zset, and list contain too many elements. You can hash the Key to generate a new Key and split the Key
  3. Delete using unlink instead of del command