“Let’s Go to The Factory together” series -Redis Sentinel, persistence, master and slave, hand tear LRU

omg

Write this issue is actually more tangled, I wrote before the more easy to understand, one is THAT I know these points, two before I was in the e-commerce company avalanche, breakdown what is still a scene to contact. But I don’t have much experience with Redis clustering online, and I can’t do that in a company environment, so I finally took a look at the official website and some information (which I will post later in this article), and I wrote this post.

I am a warm man, I can’t do it, so I have this one. Please don’t spray me when I am late for the next one. Besides, the next one is still the final chapter of Redis, and I still need to think about it. So if there is anything wrong, please leave a message Diss me, it is really hard to write this post, but here is a video I shot at 2 am one day, ADHD boy.

As I said before, I will post the third post from the second post to 300 likes

I’m not lying to you. I’m exhausted. No, let’s do it.

Don’t point a praise sorry I, this time don’t white fuck me!

---

The body of the

The last few “hang dozen interviewer” haven’t seen the small partner can review (obviously wrote two said as if a lot of same)!

• The Slapping Interviewer series -Redis Basics
• The Slapping Interviewer series – Cache Avalanche, Breakdown, and penetration

We all know that the introduction of a technology is convenient for development, to solve the various problems, but also can bring the corresponding problems, technology is a double-edged sword, the introduction of the cluster may cause many problems, such as: how to ensure the high availability cluster, how data synchronization and so on, we said few words, let’s welcome the next victims show for us.

The interview began

Three fat, middle-aged men in plaid shirts walk up to you with three scratched Macs, looking at their bald hair and thinking they must be top architects. And still be 3, but fortunately I read ao bing to write “hang dozen interviewer” series, abdomen has poem book gas from China, fundamentally virtual all not virtual good cut.

Hello young man, I have asked you the basic knowledge and some common caching questions, can you tell me why Redis is so fast?

Hello, handsome and charming interviewer, let’s take a look at the fundamental difference between a relational database and Redis.

Redis uses a KV database based on memory and single process and single thread model, which is written by C language. The official data provided by Redis can reach 100,000 + QPS (query times per second).

• Completely memory based, most requests are purely memory operations, very fast. It’s, it’s in memory, it’s like a HashMap, but the advantage of a HashMap is that it’s O(1) time to find and time to operate;

• The data structure is simple and the data operation is simple. The data structure in Redis is specially designed.

• Using single thread, avoid unnecessary context switch and competition conditions, there is no multi-process or multi-threaded switching caused by CPU consumption, do not need to consider the problems of various locks, there is no lock release operation, there is no possible deadlock caused by the performance consumption;

• Use multiplex I/O multiplexing model, non-blocking IO;

• Different from using the underlying model, Redis directly built the VM mechanism by itself, because the general system call system function, will waste a certain amount of time to move and request;

May I ask what a context switch is?

I can for example: I remember to have a friend WeChat asked me a context switch is what, why threads may not safe, so I said, just as you are reading an English book, you can see the first ten pages will not find a word to read, you add a bookmark, then go to the dictionary, after a while you come back to read from a bookmark, ok so far do not have what problem.

If you read it alone, there is no problem, but when you go to check, other friends curious about what you are reading, he will flip your book, and then slip, oh huo, when you look at the book is not the page you read. I don’t know if I have explained clearly so far, and why the thread is not safe, it is because you are a person how to look at it is ok, but more people to change the operation of a book data will be messy. Maybe my explanation is crude, but the idea is the same.

So it’s single threaded. We have multi-core servers now. Isn’t that wasteful?

Yes, it’s single-threaded, but we can open multiple instances of Redis on a single server.

Since the single opportunity bottleneck is mentioned, how do you solve this bottleneck?

We use the cluster deployment mode, namely Redis cluster, and the primary/secondary synchronization read/write separation is similar to the primary/secondary synchronization of Mysql. Redis cluster supports N Redis master nodes. Each master node can be mounted to multiple Slave nodes.

This allows Redis to expand horizontally. If you want to support a larger cache of data, expand the number of master nodes horizontally so that each master node can hold more data.

Oh? So the question is, how do they interact with each other? And how does Redis persist? Redis data is stored in memory. It will be lost after a power failure or restart.

Yes, persistence is an important part of Redis high availability. Because of the feature of Redis data in memory, persistence must be available. I have learned that there are two ways of persistence.

• RDB: The RDB persistence mechanism is used to periodically persist data in Redis.
• AOF: The AOF mechanism logs each write command to a log file in append-only mode. Since this mode is append only, there is no disk addressing overhead, so very quickly, a bit like Mysql’s binlog.

Two ways can the Redis persistence data in memory to disk, and then the data backup to go elsewhere, RDB is more suitable for cold standby, AOF is more suitable for hot standby, such as an electricity company in hangzhou I have these two data, I a backup to my hangzhou node, a backup to Shanghai again, if you can’t avoid the natural disasters happen, Also won’t two places all hang together, this disaster recovery is also the remote disaster, the earth destruction he can’t.

Tip: When both mechanisms are on, Redis will rebuild data using AOF by default, as AOF data is more complete than RDB data.

What are the pros and cons of these two mechanisms?

Let me start with RDB

Advantages:

He can generate multiple data files, each data file respectively represents a moment Redis data inside, this way, is there a feel very suitable for cold standby, complete data set up operational timing tasks, timing synchronization to the remote server, such as ali’s cloud services, so that once the online hang up and you want to restore data, how many minutes ago Just go to the remote end and make a copy of the previous data.

The impact of RDB on Redis performance is minimal, as it only forges a child process for persistence while synchronizing data, and it recovers data faster than AOF.

Disadvantages:

RDB files are snapshot files and are generated every five minutes or more by default, which means that you are likely to lose all the data in the five minutes between one synchronization and the next. AOF loses up to one second of data, and the data integrity is determined.

When RDB generates a snapshot of a large file, the client may pause for a few milliseconds or even a few seconds. When your company is doing a seckill, it happens to fork a child process to generate a large snapshot.

Let’s talk about AOF

Advantages:

As mentioned above, RDB generates snapshots every five minutes, but AOF runs through a background thread fsync once a second, which loses up to one second of data.

AOF writes data to log files in append-only mode. It writes data in append mode, which reduces disk addressing overhead, improves writing performance, and prevents file damage.

The AOF log is recorded in a very readable way. This feature is suitable for emergency recovery in case of catastrophic data erasures. For example, if the company’s interns flushall data through the flushall, you can immediately copy a copy of the AOF log file as long as the background rewrite has not occurred. Delete the last flushall command.

If you want to install Redis on a server you bought, don’t tell me about it. Aobin is a deceitfully deceitfully manipulating man, and I crashed the server.

Disadvantages:

The AOF file is larger than the RDB for the same data.

When AOF is enabled, Redis supports lower QPS than RDB does. It has to asynchronously refresh the log every second. Fsync is still very high performance, but I remember it’s the same for ElasticSearch. Don’t directly to a fight a, then I will tell you that the performance may be low to no way to use, we can think about why yo.

So what’s the choice?

If you use RDB alone, you will lose a lot of data. If you use AOF alone, your data recovery is not as fast as RDB. When do you use RDB first time to recover, and then AOF to complete the data? Cold and hot backup together is the king of a highly robust system in the Internet era.

By the way, I heard you mention high availability. Are there any other ways for Redis to ensure high availability of clusters?

!!!!!!!!! Dizzy oneself to bury oneself a hole (actually is the morning ready, deliberately throw out this word to wait for him to ask, in case he does not ask).

Pretend to think for a moment (not so long that you think you really don’t). Oh, I remember, and sentinel.

Sentinel must be robust with three instances. Sentinel + master/slave does not guarantee data loss, but it does guarantee high availability of the cluster.

Why do we have to have three instances? Let’s see what happens to two sentries.

If either sentry s1 or sentry S2 thinks you are down, they switch and elect a sentry to perform the failure, but most sentries need to be up and running.

So what’s wrong with that? M1 is down, S1 is not down that’s OK, but the whole machine is down? There are only S2 naked sentries left, no sentries to allow failover, even though there is R1 on the other machine, but failover is not performed.

A classic sentinel cluster looks like this:

The machine where M1 is located is down, and there are two sentries left. If two people see that he is down, then we can elect one to perform failover.

Warm men, small summary of the sentinel component’s main functions:

• Cluster monitoring: Monitors whether the Redis master and slave processes are working properly.

• Message notification: If a Redis instance fails, the sentinel sends a message as an alert to the administrator.

• Failover: If the master node fails, the slave node is automatically transferred to the master node.

• Configuration center: Notifies the client of the new master address if failover occurs.

I remember you also mentioned master-slave synchronization. Could you tell me how the data is synchronized between the master and slave?

Interviewer: Your memory is really a good one, I forgot you remember, I thank you, mentioned this, with my earlier mentioned data persistence RDB and AOF has a close relationship.

I first say why to use master and slave such architecture mode, mentioned in front of the QPS single-machine is limited, and Redis features is must support high read concurrent, that you a machine read and write, this who can withstand ah, not human ah! But if you let this master machine write, synchronize data to other slave machines, they all read, and distribute a lot of requests, isn’t that much better? And when you expand, you can easily expand horizontally.

Back to the point, how did they synchronize their data?

When you start a slave, it sends a psync command to the master, and if that slave connects to the master for the first time, it triggers a full replication. The master will start a thread, generate RDB snapshots, and cache all new write requests in memory. When the RDB file is generated, the master will send the RDB file to the slave, and the first thing the slave will do is write it to the local disk, and then load it into memory. And then the master will send all the new names that are cached in memory to the slave.

What if the Internet is disconnected or the server is down during the data transmission?

If there is any network problem during transmission, it will be reconnected automatically, and the missing data will be made up after connection.

Remember that when RDB snapshot data is generated, the cache must also start accepting new requests at the same time. Otherwise, your old data will be gone, and what will happen to your incremental data during synchronization? Isn’t it?

So with all that said, can you tell me about his memory elimination mechanism, and write the LRU code by hand?

Handwritten LRU? Do you want to just jump up and say: Are U F**k Kidding me?

This is a question I was asked when I was in Ant Financial. I don’t know if you have heard this question before.

Redis has two expiration policies: periodic deletion and lazy deletion.

Regular good understanding, the default 100ms to randomly select some set the expiration time of the key, to check whether expired, expired will be deleted.

Why not scan all keys with expiration time set?

If all keys in Redis have expired time, scan them again? That’s horrible, and we almost always have an expiration date online. A full scan is the same as a full table scan without a where condition. Redis is tired after 100ms.

If I don’t have a lot of random keys, there will be a lot of invalid keys.

Good question, inert delete, see name know meaning, inert, I do not take the initiative to delete, I am lazy, I wait for you to query I see you expired, expired deleted also do not give you back, did not expire how to how.

Finally is if if, regular did not delete, I also did not query, how can that do?

Memory obsolescence mechanism!

The official website to the memory elimination mechanism is the following:

• Noeviction: Returns error when memory limit is reached and the client tries to execute commands that will allow more memory to be used (most write instructions, but DEL and a few exceptions)

• Allkeys-lru: Attempts to reclaim the least-used key (LRU) so that there is room for newly added data.

• Volatile-lru: Attempts to reclaim the least used key (LRU), but only the ones in the expired set, so that there is room for newly added data.

• Allkeys-random: Reclaims random keys to make room for newly added data.

• Volatile -random: Reclaim random keys so that there is room for newly added data, but only for keys in the expired set.

• Volatile – TTL: Reclaim the keys in the expired set, and reclaim the keys with shorter TTL (TTL) first, so that new data can be stored.

  The policies volatile-lru, volatile-random and volatile-TTL are no different than noeviction if no key meets the reclaim prerequisite.

As for LRU, I will also briefly mention that handwriting is too long, you can go to Redis official website to see, I will show you the effect of myopia LUR

Tip: The reason Redis doesn’t use a real LRU implementation is that it requires too much memory. However, the approximate LRU algorithm should be equivalent for the application. Using the real LRU algorithm and the approximate algorithm can be compared by the following image.

You can see three kinds of dots in the picture, forming three kinds of bands.

• The light gray bands are objects that have been reclaimed.
• The gray bands are objects that have not been reclaimed.
• The green band is the object that was added.
• In the theory of an LRU implementation, the hope is that the first half of the old keys will expire. Redis’s LRU algorithm is an outdated key of probability.

As you can see, Redis 3.0 is better than Redis2.8 at all five samples. In Redis2.8, most objects remain between the last access. The approximation of Redis 3.0 using 10 sample sizes is very close to theoretical performance.

Note that the LRU is only a model that predicts how the keys will be accessed. In addition, if your data access pattern is very close to a power law, most of the access will be in a set of keys, and LRU’s approximation algorithm will handle it just fine.

In fact, in the familiar LinkedHashMap also implemented Lru algorithm, the implementation is as follows:

When the capacity exceeds 100, the LRU policy is executed: the least recently unused TimeoutInfoHolder object evICT is dropped.

Real interview will let you write LUR algorithm, you don’t do the original one, that really TM many, write not over, you either fight above this, or fight below this, find a data structure implementation of the Java version of LRU is relatively easy, know what principle is good.

The end of the interview

Young man, you do have something, HRBP will contact you, please keep your cell phone open, ok?

Ok, thank you, interviewer, very nice interviewer, I want to see you a few more times, poof.

I can’t help but like it for such a comprehensive and detailed answer

(hint point like, every time saw not point like, you want white fuck me? You guys are bad, but I love it)

conclusion

All right, let’s have fun, let’s have fun, let’s have fun, let’s not make fun of the interview, I’m writing this for the show, please take the interview seriously.

This issue is not as easy to understand as the previous one, right? I started it on my own server, and then went to the official website to see how to command a fool operation, and looked up some information. Here I recommend you a few classic Redis primer books and my reference materials.

• Redis Chinese official website

• Getting Started with Redis (2nd edition)

• Redis Actual Combat

• Redis Design and Implementation

• Technical Architecture of Large Websites — Li Zhihui

• Redis Design and Implementation by Huang Jianhong

• The Deep Adventures of Redis — Qian Wenpin

• “The Core Technology of Website Architecture with 100 million traffic — Zhang Kaitao”

• Huperia chinensis — Huperia chinensis

If all goes well, this is the penultimate issue of Redis, I don’t know what to write for the last issue, I have to think about it, plus it is not double 11 recently, I have to work overtime, you look at the beginning of me, how poor, why don’t you give me a like? Buy a server? Not sure how long the next issue, want to see the update early friends can go to the public number to urge more, the public number one to two days ahead of the update.