Let’s Go to Dafang Series – Cache Avalanche, breakdown, penetration

A little sigh with emotion

Could have been saved put the manuscript in the public, the shower night, thought for a moment, feel or turned on the computer to write something, with content that’s ok, it’s a bit of a personal feeling, don’t know how many friend saw the SKT VS G2 yesterday, and don’t know how many friend remember Faker hand shake the scene.

Don’t know what you see is feeling, when I saw his hand shaking, my heart is shaking the world “I support are LPL team, but I like li3 ge this person, that kind of clinging to the victory, so many years that insist on your insist, so many interests temptation just want victory, in front of such a man I love, I think a lot of people like that too.

Maybe as many netizens said, the hero is late, but I think he still has something, like a lot of people say that we programmers can only eat the same, but if you insist on their own insistence, do a belly of poetry and book gas from China, I think in the end will definitely get their own.

So I’m not going to get sentimental, so let’s get to technology.

The body of the

Last time we covered the basics of Redis, if you haven’t seen it yet, you can review it

The Slapping Interviewer series -Redis Basics

If you haven’t, you’ve heard of it. What’s the difference between the three, and what should we do to prevent them from happening? Let’s welcome our next victim.

The interview began

A paunchy, middle-aged man in a plaid shirt walks up to you with a scratched MAC, looks at his bald hair and thinks he must be a top architect. But our belly has poetry, books and qi from China, virtual is not virtual.

I see Redis on your resume, so let’s get straight to the point and answer some of the most common questions. Do you know about the Redis avalanche?

Hello, handsome and charming interviewer. As I know, at present, the home page and hot data of e-commerce will be cached. Generally, the cache is refreshed with scheduled tasks, or updated after it cannot be found.

Here’s a simple example: if all the keys on the home page are invalid for 12 hours and refreshed at 12 noon, I have a huge amount of activity at zero. Let’s say 6000 requests per second. The cache can handle 5000 requests per second, but all the keys in the cache are invalid. At this point, all 6000 requests in 1 second fall to the database, and the database is bound to fail to carry, it will report an alarm, and the real situation may be that even the DBA did not react directly hung up. At this point, if there is no special solution to handle the failure, the DBA is anxious and restarts the database, but the database is immediately killed by the new traffic. This is what I think of as a cache avalanche.

I deliberately looked at the next I have done the project feeling again hanging do not allow such a large QPS directly hit DB, but not slow SQL plus database, large table may also be able to top, but with the Redis gap is still very big

Widespread failure at the same time, the moment Redis like not, that the order of magnitude of other request directly to the database which is almost disastrous, imagine if you play hang is a user service of library, the other relies on almost all of his library interface will be an error, if you don’t do fusing strategy is basically hang a piece of the rhythm of the moment, By the time you can restart, the user will have gone to bed and lost faith in your product. What a piece of junk.

The interviewer touches his hair. That’s not bad. What about this? How do you deal with that?

It is easy to deal with cache avalanche. When storing data in batches to Redis, add a random value to the expiration time of each Key, so that the data will not fail in large areas at the same time. I believe that Redis can handle the traffic at this point.

SetRedis (Key, value, time + math.random () *10000);

Copy the code

If Redis is deployed in a cluster, the hotspot data can be evenly distributed in different Redis libraries to avoid total failure. However, when I operate the cluster in the production environment, each service corresponds to a single Redis fragment, which is for the convenience of data management, but it also has the disadvantage that it may fail. Random time of failure is a good strategy.

Or set the hot data will never expire, there is an update operation to update the cache is good (such as operation and maintenance updated the home page goods, then you brush the cache is done, do not set the expiration time), e-commerce home page data can also use this operation, insurance.

Do you know cache penetration and penetration? Can you tell me the difference between them and avalanche?

Ok, let me talk about cache penetration first. Cache penetration refers to the data that is not in the cache or the database, and the user keeps making requests, so the id of our database starts from 1 and increases from 1, such as the data that is initiated with id -1 or the data that is very large and does not exist. At this time the user is likely to be the attacker, the attack will lead to the database pressure is too large, serious will beat the database.

Postman can basically kill smaller stand-alone systems, like alibaba, which I bought myself

For example, if you do not check the parameters, the database ID is greater than 0, I always use less than 0 parameters to request you, every time I can bypass Redis directly to the database, the database can not be found, every time this happens, the concurrency peak is easy to crash.

As for cache breakdown, the cache is a bit like an avalanche, but have a bit different, cache the avalanche because the cache invalidation of large area, collapse, DB and breakdown of different cache is a cache breakdown is a Key is very hot, constantly carrying big concurrent, big concurrent concentrated on a visit to this point, when the Key in the instant of failure, Persistently high concurrency punctured the cache and directly requested the database, like a hole in an intact bucket.

The interviewer’s eyes were relieved. What did they do

Cache penetration I will add verification at the interface layer, such as user authentication verification, parameter verification, invalid parameter direct code Return, such as: ID for basic verification, id <=0 direct interception, etc.

I’d like to mention here is that we are in the development program should have a heart of “trust”, just don’t believe any caller, for example, if you provide the API interface to go out, do you have this a few parameters, that is I think as the caller, any possible parameters should be considered, do check, because you don’t believe in call you, You don’t know what parameters he’s going to pass to you.

For example, you have a paging query interface, but you do not limit the size of the paging parameters. If the caller checks integer. MAX_VALUE, it will take you a few seconds to make a request. Is the company colleagues call good big deal discovered change, but if it is a hacker or a competitor? I don’t need to tell you what will happen if you adjust this interface on the day of your Double 11. This is what the Leader told me before, and I think everyone should know about it.

From slow not to access data in a database also does not have to, then corresponding Key can also be the Value to write is null, position error, try again later asked so the Value of the concrete for what products, or look at specific scenarios, cache short effective time can be set point, such as 30 seconds (set too long can lead to normal can’t use).

This can prevent attacking users from repeatedly using the same ID for violent attacks, but we should know that normal users will not initiate so many requests in a single second. I also remember that there is a configuration item in Nginx of the gateway layer, which allows o&M to greatly block the IP addresses whose access times per second exceed the threshold.

What else can you do?

In addition, I remember that Redis also has an advanced usage of Bloom Filter, which is also very good to prevent the occurrence of cache penetration. Its principle is very simple, which is to use efficient data structures and algorithms to quickly determine whether your Key exists in the database. If it does not exist, you can return. You go to check DB refresh KV and return.

What if the hacker has many IP addresses to attack at the same time? I’m not sure it makes sense to me at all, but the average level hackers don’t have that many bugs, and the normal level Redis cluster can withstand this level of access, so I don’t think small companies will be interested. The high availability of the system, the cluster is still very can top.

In case of a cache breakdown, set the hotspot data to never expire. Or we could just add a mutex and I’m sure I’ll have the code ready for you as a warm guy

The end of the interview

It’s late today, so I’ll stop here for the interview. You can come back tomorrow for a second interview and I’ll continue to ask you questions about Redis cluster high availability, master-slave synchronization, sentry and other knowledge points.

I can’t believe there’s another interview! (Forcing the next stage of foisting haha) but in order to offer still have to lick, MMMM, good handsome interviewer.

I can’t help but like it for such a comprehensive and detailed answer

(hint point like, every time saw not point like, you want white fuck me? You guys are bad, but I like it)

conclusion

Let’s have fun, let’s have fun, let’s not joke about the interview.

This paper simply introduces the Redis avalanche, breakdown, penetrate, three is about the same, but there are some difference, actually it is asked to cache will be asked during an interview, you don’t put all mixed up, because the cache avalanches, penetration and breakdown, the biggest problem is to cache, either don’t appear, once appear, is a fatal problem, So the interviewer will definitely ask you.

You must understand how did it happen, and how to avoid, after how to rescue, you can not know very deep, but you can’t not to think about it, sometimes the interview is not necessarily the torture of knowledge, perhaps is torture of your attitude, if you think clearly, and then know how much that will be great, And you know how to prevent it so come to work.

Finally, I will continue to give you a little technical summary:

In general, to avoid the occurrence of the above situation, we will analyze from three time periods:

• Advance: Redis high availability, master-slave + sentinel, Redis cluster, avoid total crash.

• What happened: Local ehCache cache + Hystrix stream limiting + degradation to prevent MySQL** from being killed.

• After the event: Redis persists RDB+AOF, once restarted, automatically loads data from the disk, and quickly recovers cached data.

I will cover all of the above points in the Redis series this month, I think it will be possible to finish Redis, the flow limiting component, you can set the number of requests per second, how many can pass through the component, the rest of the failed requests, what to do? Go down! You can return some default value, or a reminder, or a blank value.

Benefits:

The database never dies, and the stream limiting component ensures that only a few requests per second pass through. As long as the database doesn’t die, that is, three out of five requests can be processed by the user. As long as 3/5 of the requests can be processed, your system is still alive. For the user, it may be a few clicks and the page does not appear, but a few more clicks and the page appears again.

This in the current mainstream of the Internet giant is one of the most common, are you curious, a star broke something, you find that you are blank screen how to brush to the weibo, but some people and directly into the, you come out to brush a few times more, now know, that is to do the downgrade, sacrifice part of the user experience for the safety of the service, ok?

Pay attention, don’t get lost

Ok everybody, that’s the end of this article, the people who can see here, are talents.

I will be updated every week after a few of the first line of Internet factory interview and commonly used technology stack related articles, thank you very much talent can see here, if this article is well written, feel “Aobin” I have something to ask for praise 👍 ask for attention ❤️ ask for sharing 👥 for warm men I really very useful!!

White whoring is not good, creation is not easy, your support and recognition, is the biggest motivation for my creation, we will see the next article!

AoBing | article “original”

If there are any mistakes in this blog, please comment and comment. Thank you very much!

---

The article continues to be updated every week. You can search “Three Prince Aobin” on wechat for the first time to read and hurry up (one or two articles earlier than the blog). This article has been included on GitHub github.com/JavaFamily, there is a pilot mind map of big factory face, and a lot of my documents have been sorted out. You can refer to the interview test point review, I hope we have something together.