Hello, I am the south orange, ten thousand method immortal door of the door, just from the kyushu world through the earth, because of the impact of space-time turbulence lead to my magic all lost, now have to pass this platform to the vast xiuzhen genius to borrow strength. Every “like” and “attention” from you will help me to come back to kyushu world. Watch for WX: Southern Orange RYC.

After I go back, we are all the elders of ten thousand method immortal door, I will give you countless days of material to treasure, everyone such as dragon, the whole people soar.

As an immortal world of zongmen, of course there will be such a thing as zongmen Dabi.

And as our leading role, Li Xiaogeng will naturally be in which the brilliance.

Ten thousand faxian Gate, Zong Gate arena 3, base group elimination.

“Yield!”

With a flick of his sleeve, our hero Li Xiaogeng nodded toward his opponent who had flown out of the ring.

“This is the end of the contest in ring three. Winner, Xiao Li Geng, Cloud Palace.” The referee gave Li Xiaogeng a sign with a straight face.

“This person also too depend on!”

“Grass (I love you) uses dharma tools as soon as he comes to the stage, giving us ordinary disciples no way to live!”

“The Temple of Clouds is a group of big dogs. In the arena of jiedan period, the two disciples of the temple of Clouds with the Oracle of the Refining apparatus zong killed all sides. I did not expect the story here is the same.”

Li Xiaogeng heard these discussions, the corners of the mouth revealed a helpless smile, the use of unique hairspray hand gently whole the whole hair style, looked at the discussion of the audience said: “thank you for your support, I will continue to work hard!

“But he’s so handsome, I forgive him.”

“Yeah, seeing his face, it’s okay to disrespect fair play like that, right?”

“Grass is just a plant.”

Go back a week.

Xiao Xiao Dian, Li Xiaogeng as always in the cloud xiao Xiao gallantry.

“Master, next week is Zumen Dabi. You must be very tired from doing research in the Wanfa Institute every day. My little apprentice is here to relieve your shoulders.”

Yun Xiaoxiao looked at the fawning Li Xiaogeng, light hummed once: “rest assured, everyone has a share, Redis for you line?”

“Redis? The magic device that claims to solve most distributed problems?”

“Yes, if you want it, I’ll test you. Do you know what the five basic data structures of Redis are?”

“Easy, master, just listen to me.”

First, basic knowledge of Redis

1. 1 String The value is a String

The most basic data type in Redis, one key corresponds to one value

Application:

1. Cache: The classic usage scenario is to put common information, strings, pictures or videos and other information into Redis. Redis acts as the cache layer and mysql acts as the persistence layer to reduce the read and write pressure of mysql. 2. Counter: Redis is a single-thread model, one command is executed before the next one is executed, and the data can be dropped to other data sources one step at a time. 3. Session: Achieve session shared copy code through RedisCopy the code

1, 2 Hash

For a HashMap in Java, it is itself a KV pair structure, such as value={{field1,value1},…… FieldN,valueN}}, very easy to understand

Application:

A HashMap serves as a cache, which is more space-efficient than string for maintaining cache information, and is suitable for storing duplicate code such as user information, video information, and so onCopy the code

The underlying implementation is dict

1, 3 List

Redis’s LinkedList is the Java equivalent of LinkedList

Application:

1. List can be used as both queue and stack in Redis. Its insert and delete operations are very fast and the time complexity is 0(1), but the index positioning is very slow and the time complexity is O(n). 2. It can be used as the timeline of micro-blog. Some people post micro-blog and add lpush to the timeline to display new list information. 3, can achieve blocking queue, left in right out of the queue group to complete the design copy codeCopy the code

The underlying list is implemented using quickList

In earlier designs, ziplist was used for small or small numbers of elements in a list object, and two-way linkedList was used for large or large numbers of elements in a list object.

Both types of storage have advantages and disadvantages

  • The bidirectional linkedList is convenient for push and pop operations at both ends of the table. It has low complexity in inserting nodes, but it has a large memory overhead. First, it holds two Pointers on each node in addition to data; Secondly, each node of bidirectional linked list is a separate memory block, the address is not continuous, more nodes will easily generate memory fragmentation.
  • Ziplist stores on a contiguous piece of memory, so storage is efficient. However, it is not conducive to modification operations, as insert and delete operations require frequent memory requisition and freeing. Especially if the Ziplist is long, one realloc may result in a large number of data copies.

After the 3.2 update, the underlying implementation of list became quickList

QuickList is a hybrid of zipList and linkedList, which splits the linkedList into segments, each of which is stored compactly using zipList, and multiple Ziplists are connected together using bidirectional Pointers.

1, 4

The collection of Redis is the Java equivalent of HashSet, with unordered and unique key-value pairs.

Its internal implementation is equivalent to a special dictionary in which all values are NULL. When the last element in the set is removed, the data structure is automatically deleted and memory is reclaimed. Copy the codeCopy the code

Application:

1. Tag: Add a tag to the user, or the user adds a tag to the message, so that those with the same tag or similar tag can recommend things or people to follow. 3, can be used to store in a certain activity winning user ID, because there is to repeat the function, can ensure that the same user will not win twice. Copy the codeCopy the code

1, 5 Zset ordered set

It is similar to the combination of Java SortedSet HashMap. On the one hand, it is a set, which ensures the uniqueness of the internal value. On the other hand, it can assign a score to each value, representing the sorting weight of the value. Its internal implementation uses a data structure called a skip list.

From this picture, we can see that the bottom layer of the skip list is a sequential linked list. Every other node has an upper pointer to the next node, and the layer recurses upwards. This design is similar to the tree structure, so that the search of the linked list can reach the time complexity of binary search.

Skiplist does not require a strict correspondence between the number of levels on the upper and lower lists. Instead, he randomly assigns a number of levels to each node. For example, in the figure above, the random number of layers from the third node is 4, so it is inserted into the space of the fourth layer, while the random number of layers from the fourth node is 1, so it only exists in the space of the first layer.

  • When there is less data, zset is implemented by a Ziplist, just as before the list bottom layer

A ZIplist is a sequential storage structure composed of a series of specially encoded contiguous memory blocks. Similar to an array, a ZIplist is stored contiguously in memory, but unlike an array, the size of memory occupied by each element of a ziplist can be different to save memory. Ziplist records the necessary offset information in each node so that it can jump to the previous or the next node.

  • When there is a lot of data, zset is implemented by dict, which queries data for scores, and skiplist, which queries data for scores

In addition to these five basic data structures, Redis also has more professional data structures HyperLogLog (algorithm of radix statistics), Geo (geographical location series), Pub\Sub (message queue), Pipeline (Pipeline), BloomFiler (Bloom filter), which are useful in different places. Some of which I will describe below.

1, 6 Pipeline

Multiple I/O round trips can be reduced to one, provided there is no causal relationship between instructions executed by Pipleline

Pipeline can send multiple commands at one time and return the results at one time after execution. Pipeline can reduce the round-trip delay time by reducing the communication times between the client and Redis. Moreover, the principle of pipeline implementation is queue, and the principle of queue is first-in-first-out. This ensures that the data is sequential.

Note: The Pipeline mechanism can optimize throughput, but does not provide atomicity/transaction assurance

“Oh, not bad oh, but also can draw inferiors.Is not early dozen Redis idea?” Cloud xiao Xiao playfully staring at Li Xiaogeng’s eyes.

But who is Li Xiaogeng? “Master, you also know that all I have to do is go to the library. I have dabble in redis books these days. If you don’t mind, you can test me again.”

“Grow skill, small geng classmate!” Yun Xiaoxiao is interested.

When accepting Li Xiaogeng as an entry-level student, 50 percent (actually 100 percent) was based on his face, and the remaining 50 percent was based on his solid foundation, which was completely different from the ordinary gas refining period. Li Xiaogeng himself is also very ambitious, the entrance of half a year will break through the foundation period, and not impatient, step by step practice of ten thousand method fairy gate three years of foundation planning. Learning modern law, although quickly master all kinds of advanced skills can quickly break through, but if there is no solid foundation, then the future road will certainly encounter all kinds of problems.

“So LET me test you, how does Redis implement distributed lock?”

“I have a word to say. I beg your attention.”

Cluster and distributed locks

2, 1 Redis to implement distributed lock

With the rapid development of Internet technology, more and more single architectures have been transformed into distributed architectures. Distributed architectures can indeed improve performance and efficiency, but also bring data consistency problems.

Distributed lock is a special weapon to solve data consistency in distributed architecture. Distributed lock can be used safely only when it meets the following three aspects:

  • Exclusivity: Only one client can acquire the lock at a time. No other client can acquire the lock at the same time

  • Avoid deadlocks: This lock must be released (normally or abnormally) after a limited time.

  • High availability: The mechanism for acquiring or releasing locks must be highly available and perform well

At present, I know about the distributed lock there are three mainstream ways to achieve, respectively zookpeer, Redis, and local database, today I will introduce how to use Redis to achieve distributed lock.

The locking mechanism based on Redis mainly relies on the atomic operation of Redis itself

Setnx fights for the lock and adds the expiration time with expire

It is as simple as that. If you are afraid of something wrong, such as sudden outage before setting the expiration time during lock scrambling, you can directly use the RedisTemplate packaged by Jedis to combine setnx and EXPIRE into one command.

“But are distributed locks really secure? What if the server goes down?”

“Of course, the distributed lock used by such a single Redis is not very secure.” Li Xiaogeng seemed to know that Cloud xiao Xiao would ask, the corners of the mouth inadvertently with a smile.

First, the problem of single points of failure is inevitable

Second, because the client that uses the lock, and redis server, are not together ah! Time is delayed, we can only rely on the TTL command of Redis to query the remaining time of the lock, and then judge whether the lock times out according to the remaining time. However, in common computer systems, it is difficult to obtain a reliable time.

  • The system may synchronize time with the time server.
  • The virtual machine may adjust the time,
  • The JVM GC can cause time pauses

The emergence of RedLock solves this problem to a certain extent, and its execution process is as follows:

  1. The client gets the current time and generates a random value as the value of the lock (to get a more accurate time).
  2. Attempting to acquire the same lock on all 5 Redis in sequence (using the same key and the same random value as a single redis lock) requires itself to set a relatively small timeout (e.g. 5-50ms). Prevent wasting too much time on a dead Redis. If one is unavailable, start trying the next one as soon as possible
  3. The client calculates how long it took to acquire the lock by subtracting the time obtained in step 1 from the current time. If the client acquires most of the lock on Redis (3 to 5 5’s) without exceeding the lock timeout, the lock is considered successful
  4. If the lock is acquired successfully, the valid time is calculated by the lock timeout – the time it took to acquire the lock
  5. If that fails, try unlocking all redis (the unlocking is a lua script that removes a key if the key’s value is a random value generated in step 1)

Of course, it doesn’t solve the problem, but redis locks can only go wrong in extreme cases, and you can safely use Redis clustering or RedLock if you don’t need to be very precise and only need to be most reliable.

“You say you use the RedLock tool, what if I just hit your Redis unit? How else do you keep distributed locks robust? In addition, we also use distributed locks to ensure that the identity of the disciples in and out is not stolen. Use them the way you want. If the results are not accurate enough, then for other schools, there is no difference between our system and sieve.” Yun Xiaoxiao waved from inside the pot to get a few small oranges, looked to the third wall outside of you, as if in the prompt you can drink saliva move.

“Then there’s the Redis cluster.” Li Xiaogeng pushed his glasses.

2, 2Redis cluster

Cluster synchronization mechanism

There is a master-slave mechanism in Redis. A master node corresponds to one or more slave nodes. The master node provides data access, while the slave node pulls data backup from the master node. Redis master/slave synchronization

  • 1. During the first synchronization, the secondary server sends the SYNC command to the primary server. After receiving the command, the primary server performs bgsave and records subsequent modification operations to the memory buffer
  • 2. After receiving the RDB image, the replication node loads the RDB image to the memory and notifies the primary node
  • 3. Subsequent incremental data can be synchronized through AOF logs, similar to database binlogs

At the same time, after version 2.8, Redis can automatically determine whether full synchronization or incremental synchronization is required, which is relatively efficient. Incremental synchronization actually means sending the PSYNC () command to the primary server when a new replication is started after full synchronization. (runid is the id of the primary server for the last replication. Offset is the replication offset from the server. The master server uses these two parameters to determine which synchronization to perform, whether the server ID is the same as the local server, and whether the replication offset is in the buffer.

High availability:

  • The Redis Sentinal (Sentinel mode) cluster focuses on high availability and automatically promotes the slave to master in the event of a master outage to continue service
  • Redis Cluster Cluster focuses on scalability. When a single Redis memory is insufficient, clusters are used to fragment storage

“It was waiting for me here. Li Xiaogeng, if I give you a set of Redis cluster, how do you realize your plan?” Cloud xiao Xiao laughed, originally just want to give a single Redis method, and then with Li Xiaogeng’s own solid foundation, there is hope in the foundation period than win the top, did not expect his goal has been redis cluster.

“No hurry, I read three sets of redis clustering scheme from the book, and asked master to help demonstrate one or two.” Li Xiaogeng also know cloud xiaoxiao know their own purpose, even if the thick skin began to slightly red, with a pair of sword eyebrows under a pair of slender peach blossom eyes, cloud Xiaoxiao saliva does not live up to the flow down.

Yun Xiaoxiao stretched, and then one finger: “Sword!” .

Numerous small red and white flying swords appeared from all corners of the room and surrounded her. Here was the cluster of Redis that Li Xiaogeng had been longing for.

“The next command of the cluster to you, see you perform ~”

Li Xiaogeng learns to have the kind to learn the kind: “today I Li Xiaogeng will return to the land immortal realm.”

Third, the realization of Redis cluster

3. 1 traditional master-slave mode

Probably all clusters have a master-slave mode

One purpose of the master-slave mode is to back up data so that when a node is damaged (i.e. unrecoverable hardware damage), the data can be easily restored because of the backup. Another function is load balancing. All clients accessing a node will definitely affect the working efficiency of Redis. With master and slave nodes, query operations can be completed by querying slave nodes.

In the master-slave mode, a Master can have multiple Slaves. According to the default configuration, the Master node can read and write while the slave node can only read and cannot write.

If the default configuration is changed, the slave can write data, but this is meaningless, because the data written to the slave will not be synchronized to other slaves, and if the master node is changed, the data on the slave will be overwritten immediately.

The failure of the slave node does not affect the read and write of other slave nodes and the master node. Data will be synchronized from the master node after the restart. If the master node is down, the read on the slave node is not affected and Redis does not provide write services. After the master node is started, Redis provides write services again.

So, we can see that the master and slave of Redis and Zookeeper are completely different! It can’t vote!

This disadvantage can be significant, especially for production environments where services cannot be stopped for a moment, so there is no master-slave mode in a typical production environment. Hence the following Sentinel model.

3. 2Sentinel mode (Sentinel mode)

The sentinel mode is used with the master slave mode. The master slave cannot elect itself, so we add a sentinel. When sentinel finds that the master node has failed, sentinel elects another master from the slave node.

The sentry’s role is to monitor the health of the Redis system. Its functions include the following two.

(1) Monitor the normal operation of the primary and secondary servers. (2) When the primary server fails, the secondary server will be automatically converted to the primary server.Copy the code

Wouldn’t that make everyone happy?

How sentinels work:

  • 1. Each Sentinel process sends a PING command to the Master server, Slave server and other Sentinel processes in the cluster at a frequency of once per second.
  • The Sentinel process marks an instance as SDOWN if it has taken longer than the value of the down-after-milliseconds option since it last responded to PING.
  • 3. If a Master server is marked as a subjective offline (SDOWN), all Sentinel processes monitoring the Master server confirm that the Master server is indeed in the subjective offline state at a rate of once per second

The Master server is marked as ODOWN when a sufficient number of Sentinel processes (greater than or equal to the value specified in the configuration file) confirm that the Master server is in a subjective SDOWN state within the specified time period.

  • 4. In general, each Sentinel process sends INFO commands to all Master and Slave servers in the cluster every 10 seconds.

When the Master server is marked as ODOWN by the Sentinel process, the Sentinel process sends INFO commands to all slaves of the offline Master server once every 10 seconds instead of once every second.

  • 5. The objective offline status of the Master server is removed if there are not enough Sentinel processes to allow the Master server to go offline. If the Master server reping the Sentinel process and returns a valid response, the subjective offline status of the Master server will be removed.

Sentinel mode can basically meet the needs of general production, with high availability. However, when the amount of data is too large to be stored in a single server, the master-slave mode or Sentinel mode cannot meet the requirements. At this time, the stored data needs to be fragmented and stored in multiple Redis instances, which is the Cluster mode.

3. 3Cluster mode

The emergence of cluster is to solve the problem of limited capacity of single Redis and distribute Redis data to multiple machines according to certain rules.

Redis-cluster adopts a no-center structure and has the following characteristics:

  • All redis nodes are ping-pong with each other and use binary protocols internally to optimize transmission speed and bandwidth.

  • Node failures take effect only when more than half of the nodes in the cluster have detected failures.

  • The client is directly connected to the Redis node without the need for an intermediate proxy layer. The client does not need to connect to all nodes in the cluster, but to any available node in the cluster.

Cluster can be said to be a combination of Sentinel and master-slave mode. Through cluster, the master-slave and master re-selection functions can be realized. Therefore, if three copies and three sharding are configured, ninety-six Redis instances are needed. Because Redis data is allocated to different machines in the cluster according to certain rules, when the data volume is too large, new machines can be added for capacity expansion. This mode is suitable for the cache requirements of large data volume. When the data volume is not large, sentinel can be used.

Quote a big picture to intuitively show what is redis-cluster

Each request to the Redis-cluster is routed, and the route can be randomly sharded using Hash (or other methods), but a full Hash is likely to result in shard death. Therefore, the method of ** consistency hashing (automatic cache migration)+ virtual node (automatic load balancing)** is proposed to solve the problem

The principle of consistency hashing is to place all the master nodes on a ring, and then a key comes along. It is also the hash value, and then compares the hash value on the corresponding points of the ring (each point has a Hash value) to see where the hash value falls. After falling on the top of the ring, it rotates clockwise to find the nearest node, and data is stored and read on this node.

The advantage of consistent hashing is that the failure of any master will only affect the data on the previous master, because if you go clockwise, all the data on the previous master will not be found. If the master goes down, you will continue to follow the clockwise direction to the next master. Only part of the data is lost.

Under the command of Li Xiaogeng, the REIDS cluster also slowly became his appearance, while in the air into *, while becoming *, very happy.

“Don’t worry, Geng, there are a few other things you need to know about Redis.”

4. Redis advanced knowledge

4, 1 Asynchronous queue

Redis’s job is to cache, but because of its versatility, it works well as a queue, with some blocking apis that allow it to queue messages; In addition, other features of message queuing, such as FIFO (first in, first out), are also easy to implement, requiring only a List object to fetch data from the beginning and stuff data from the end

  • In Redis, if you have a List structure as a queue, an Rpush production message, an LPOP consumption message, and if there is no LPOP message, you can sleep for a while and try again, which is equivalent to the producer consumption pattern. The List also has a directive called blPOP that blocks until a message arrives when there is no message.
  • Pub \sub topic subscriber mode, can realize 1 pair of N message queue, realize production once, consumption many times. However, it also has disadvantages, if the pub\sub theme subscribers mode, consumer offline case, the message will be lost, rather than using MQ directly

4, 2 delay queue

In Redis, you can use sorted-set to do delayed queues

zadd key score1 value1 score2 value2

  • Socre is the execution time, key is the queue name, and value is the data
  • The consumption queue loop obtains (zrangebyScore) data less than or equal to the current time and the smallest score from the sorted-set polling process
  • If you don’t get data, go to sleep and get it

However, Redis’s delayed queue cannot return an ACK, so you need to implement it yourself

4. 3 Persistence

Redis has two types of persistence, RDB and AOF

Because RDB takes a long time and is not real-time enough, a large amount of effective data will be lost during downtime, so AOF is needed to work with it. When redis instance restarts, RDB persistence files will be used to rebuild memory. AOF is then used to replay the recent operation instructions to fully restore the state before the restart.

RDB mechanism

RDB persistence refers to writing a snapshot of an in-memory data set to disk at a specified interval. This is also the default persistence mode, which is to write the in-memory data as a snapshot to a binary file named dump.rdb by default. The RDB provides three mechanisms to trigger persistence

  • 1. Save trigger mode: The client initiates a Save request

    During the execution of the save command, Redis cannot process other commands until the RDB process is complete

  • 2. Bgsave trigger mode – The client initiates a BGSave request

    When this command is executed, Redis asynchronously takes snapshots in the background and responds to client requests at the same time

  • 3. Automatic trigger

    The automatic triggering is done by our configuration file, in the redis. Conf file, you can go to see it, I won’t write too much here

AOF mechanism

Full backups are always time consuming (random legends are always good). Sometimes we offer a more efficient way, AOF, where Redis appends every write command it receives to a file via the write function, that is, logging.

Like RDB, AOF has three synchronization mechanisms:

  • 1. Always: Synchronous persistence Every data change is recorded immediately. Poor disk performance but good data integrity
  • 2, Everysec: synchronization per second, asynchronous operation, record per second if the downtime within one second, there is data loss
  • 3. No: never synchronize

The mechanism of Redis itself is AOF persistence enabled. If AOF files exist, AOF files are loaded first. If AOF file does not exist, load RDB file. After loading the AOF\RDB file, Redis starts successfully. When there is an error in the AOF\RDB file, Redis fails to start and prints an error message.

Don’t ask AOF or RDB, my rule of thumb is to use both. RDB synchronizes quickly, but loses up to five minutes of content. AOF synchronizes slowly, but loses up to 1s of content per second. The lost content can also be retrieved through logs.

Resolve the impact of machine outages on data loss

You can configure the sync attribute in AOF logs. If the performance is not required, you need to sync the disk for each write command to avoid data loss. However, it is unrealistic to sync the disk every time for high performance.