directory

preface

At present, there are two kinds of distributed cache technologies used in the work, redis and memcached. The purpose of cache is to effectively reduce the pressure of DB in a high-concurrency system. However, some problems may be caused by improper design of cache structure when using it.

Serverside characteristics of two commonly used caching technologies

1. The Memcache server

The Memcache server does not have a cluster concept. All storage distribution is done by the MC client. Here I use Xmemcached.

The problem with this sharding method is that a large number of keys fail in a moment after adding or removing nodes, resulting in a cache avalanche that puts a lot of pressure on the DB. Therefore, our MC client enabled xmemcached consistent hash algorithm for data sharding:

XMemcachedClientBuilder builder = new XMemcachedClientBuilder(AddrUtil.getAddresses(servers));
builder.setOpTimeout(opTimeout);
builder.setConnectTimeout(connectTimeout);
builder.setTranscoder(transcoder);
builder.setConnectionPoolSize(connectPoolSize);
builder.setKeyProvider(keyProvider);
builder.setSessionLocator(new KetamaMemcachedSessionLocator()); // Enable the ketama consistent hash algorithm for data sharding
Copy the code

According to the characteristics of the consistent hash algorithm, the nodes adding or reducing MC will only affect a small part of the data. However, this mode also means uneven distribution, and the newly added nodes may not achieve the effect of data sharing in time. However, MC adopts the method of virtual nodes to optimize the original consistency hash algorithm (controlled and realized by ketama algorithm), realizing the ability of data sharing even after the newly added physical nodes.

Finally, MC server is multithreaded processing mode, the MC a value can only be stored greatest 1 m of data, all k – v will not automatically removed after expiration, but the next visit to do comparison with the current time, expiration time is less than the current time, are removed if a k – v has not visited again, then the data will always exist in memory, Until LRU is triggered.

2. Redis server

The Redis server has a cluster mode, and the key routing is processed by the Redis server. In addition, the Redis server has a master/slave configuration to achieve high service availability.

The Redis server is single-threaded, which means that if one instruction causes Redis to process too slowly, it will block the response of other instructions, so Redis prohibits heavy operations in production (such as keys, or slow transfers due to caching large values).

Redis server does not use consistent hash to do data sharding, but uses the concept of hash slot to do data sharding. A Redis cluster has 16384 hash slots. These hash slots are distributed on different nodes according to different numbered intervals, and then a key comes in. An internal hash algorithm (CRC16(key)) is used to calculate the slot position.

And the data stored in the corresponding hash groove corresponding space, redis in add or reduce node, is actually to redistribute the hash groove, new nodes, for example, new node mean that every node on the original hash slot interval is relatively narrow, cut the hash data in a tank, a corresponding node will be postponed to the following This process is coordinated by the Redis server as follows:

Migrate is a process in which keys in the tank are migrated in batches.

2. Cache structure selection

The MC provides simple K-V storage, where the value can store a maximum of 1M data. The multi-threaded processing mode prevents other requests from queuing because of one slow processing. It is suitable for storing text information of data.

Redis provides a rich data structure, and the server is a single-thread processing mode. Although the processing speed is very fast, if there is a bottleneck in a query, the subsequent operations will be blocked. Therefore, compared with k-V, which may lead to a bottleneck in network interaction due to large data, It is better suited for querying, sorting, paging and other operations on some data structures, which tend to be less complex and time consuming and are less likely to block redis processing.

Using these two caching services to build our cache data, we currently advocate that all data form their own information cache storage in terms of iconic fields (such as ID), which is generally stored by the K-V structure of MC.

Redis provides a lot of useful data structures. Generally, to construct structured cache data, redis data structure is used to save the basic structure of data, and then when assembling data, specific data is queried in MC according to the iconic fields cached in Redis, for example, the acquisition of a leaderboard interface:

In the figure above, REDis provides the structure storage of the leaderboard, in which id and score are stored. The IDS of all information in the structure can be obtained through REDis, and then detailed information can be found from MC by using the obtained ID. Redis is responsible for paging and sorting in this process, while MC is responsible for storing detailed information.

Above is more appropriate cache practices, it is recommended that each has a own basic cache data, it is convenient for management, rather than an interface of the huge structure completely cache to MC or in redis, this partition is too coarse, accumulate over a long period come down each interface or huge method has a cache, the key will be more and more, more and more complex.

Redis constructs the large index return source problem

If Redis is used for caching, there will always be an expiration time. If the expiration time is reached, the index built by Redis will disappear. At this time, the back to source method will be too slow if there is a large amount of data to build the Redis index.

Comment system consists of an ordered set as comments list index, storage is comment id, for sorting score threshold, according to the sort of dimensions, such as the release time and the number of thumb up, this also means that a list of resources under the comments according to different sort dimension there are multiple redis index list, and specific comment MC, The normal structure is as follows:

Above is normal triggered a resources in the comments section, each trigger read cache, are helping to extend a cache expiration time, so we can ensure that the content of warmer is not easily expired, but if a comments section, a long time nobody visited redis index will be expired, if a comment area there are tens of thousands of data, long time no one visit, Suddenly someone goes to the database and builds the redis index slowly in the back source. If there is no control, it can also cause the following cache penetration problem, which causes this heavyweight operation to be executed repeatedly by multiple threads, putting a lot of pressure on the DB.

For the above problem of slow index building back to source, the solution can be as follows:

This method of constructing redis indexes through message queues is more suitable than the direct implementation of the back source method. It first builds the index data for a single page or the first few pages, and then notifies the job (in this case, the consumer) to complete the index construction through the queue. Of course, this method is only suitable for scenarios where consistency is not high.

4. Consistency

In general, the data in the cache must be consistent with the source data of the database, which involves the active invalidation cache policy (popularly known as “clear cache”) after updating the DB. Most of the data will go through the following process:

Suppose there are two services, service A and service B. Now suppose that service A triggers A write operation, and service B is A read-only program, and the data is cached in A Cache service. Now suppose that service A updates the database once, then the following flow is combined with the figure above:

  1. Service A triggers the database update operation

  2. Delete the cache key corresponding to the data after the update

  3. The read-only service (service B) found a cache miss while reading the cache

  4. Service B reads the database source information

  5. Write to the cache and return the corresponding information

This process may seem fine at first glance, but often A multithreaded program can lead to unexpected results. Now imagine that services A and B are being run by multiple threads.

1. Consistency problems caused by concurrent read and write operations

  1. Thread 1 running service A first modifies the data and then deletes the cache

  2. When thread 3 running service B reads the cache, it finds a cache miss and starts to read the source data in the DB. It should be noted that the data read out this time is the one modified by thread 1

  3. At this point, thread 2, which is running service A, goes online and starts to modify the database, and again, deletes the cache. Note that this deletion is actually an empty cache, so it doesn’t make sense because thread 3 hasn’t done its job yet

  4. Thread 3 running service B writes the data read by thread 1 back to the Cache

The result is that the final data stored in the DATABASE is the data written by thread 2, while the data stored in the Cache is the data written by thread 1.

2. Consistency problems caused by master/slave synchronization delay

In this case, slightly modify the flow chart of the program, with an additional slave library:

At present, the read operation goes to the slave library. If the master/slave synchronization may be triggered later than the back-source process after the cache is deleted in the write operation of the master/slave library, the old data will still be read from the slave library during the back-source process.

3. Consistency problems caused by cache contamination

Each time for a new demand to update the original structure of the cache, or remove a few attributes, or add a few attributes, if new demand is to give a cache object O B a new attribute, if the new logic has been in the pretest or in gray, will appear in a production environment after back to the source of the cached data without B properties, while the pretest and gray level, the need to use a new logic properties, B This results in contamination of the production & pre-release cache. The process is roughly as follows:

How to deal with cache consistency?

The cache consistency problem is roughly divided into the following solutions, which are described one by one.

1. Binlog + message queue + consumer del cache

Each time the table changes, change messages are sent to the message queue through the binlog generated by mysql. Here, the service to monitor DB changes is provided by Canal, which can be simply understood as a slave library that implements mysql communication protocol. The mysql master/slave configuration is synchronized with the binlog, and it only receives the binlog. With this mechanism, it can naturally listen for changes in the database table data, and ensure that every change in the database is sent to the consumer in order to clear the corresponding cache key.

2. Slave binlog+ message queue + consumer del cache

The above procedure can ensure that the order of the cache is clear when writing to the library. It seems that there is no problem, but the production environment often has a master-slave separation, that is, if the source reads from the slave library, the above procedure still has a consistency problem:

How to solve the dirty read problem caused by the slave library delay?

It is only necessary to set the database monitored by Canal as the slave database to ensure that when the message is pushed by Canal, all the slave libraries are completely consistent with the master library. However, this only applies to the case of one master and one slave. If there are multiple slave libraries read by one master and multiple slave libraries read by the back source, There are some risks (a master with multiple slaves needs to subscribe to the binlog of each slave node, find the last one sent, and clear the cache to ensure that all slave nodes are consistent with the master node).

However, the synchronization from the binlog library is normally faster than from Canal, because Canal receives the binlog, assembles the data change entity (which has an extra cost), and then pushes it through the message queue to each consumer (which is also expensive). So even if the subscribed master library table changes, the probability of problems is very small.

3. The key is upgraded after the update

In view of the above consistency problem (cache contamination), modifying a cache structure may cause mutual contamination between the cache in the pre-sent or grayscale state and the cache in the actual production environment. In this case, it is recommended to upgrade the key each time the structure is updated (for example, adding _v2 suffix to the original key name).

Is ⚡⚡⚡binlog really accurate? ⚡ ⚡ ⚡

Not really, as in the case above:

  1. First thread 1 goes to service A, writes DB, and sends binlog to delete the cache

  2. Service B, which is running on thread 3, will cache miss, and then read the data from the DB source.

  3. At this point, thread 2 triggers service A to write to DB again and also sends binlog to delete the cache

  4. Finally, thread 3 writes the read data to the cache. As a result, the DATABASE stores the data written by thread 2, but the cache stores the data written by thread 1

This is difficult to trigger, because it is very rare for a write to the cache on thread 3 to be sent later than the second binlog, unless something else obstructs the return.

Therefore, according to the existing strategy, there is no perfect solution, only to ensure consistency as far as possible, but because of the actual production environment, in the multi-threaded concurrent read and write environment, even with binlog as the final guarantee, can not guarantee the sequence of the last back to the source method write cache. Unless the source is all done by the Binlog consumer, which is unlikely, then service B has no source method.

In view of this problem, the probability of occurrence is the most high probability of concurrent writing, this time accompanied by the hit ratio problem.

Six. Hit rate problem

The consistency problem has been solved by the previous process, regardless of the special factors, but another problem that comes with cleaning the cache is the hit ratio problem.

For example, if a data change is so frequent that too many binlog messages are generated, this time will trigger the clear cache operation of consumers. In this case, the cache hit ratio will drop instantly, causing most users to directly access DB.

In addition, such frequently changed data will increase the probability of the occurrence of problems (1). Therefore, for such frequently changed data, the cache key will not be deleted, but directly back to the source of the cache update from the binlog consumer. In this way, even if the table is frequently changed, the user will access the cache data updated by the consumer every time. It’s just that the consumer has to process the message strictly in order;

Otherwise there will be to write dirty danger, such as open two threads at the same time consumer binlog, thread 1 received first binlog data changes, and thread 2 received a second data changes binlog, then thread one reads data (old data), thread 2 read data (data) update cache, and then thread 1 update, The cache will get dirty again;

Therefore, in order to ensure the consumption order, it must be processed by a single thread. If you want to enable multi-thread sharing pressure, you can use identification fields such as key and ID to group tasks, so that binlog messages with the same ID will always be executed by the same thread.

7. Cache penetration

1. What is cache penetration?

Normally, when a user requests a data item, it will carry a marker parameter (such as ID), and our cache key will use these marker parameters to partition different cache values. Then we will check the cache based on these parameters and return, otherwise we will return to the source, and write to the cache service and return.

This process does not seem to be a problem, but in some cases, according to the parameters brought in, the corresponding information in the database can not be found, in this case, each request with this parameter will go back to the database source, this phenomenon is called cache penetration, typically occurs in the following cases:

  1. Malicious attacks or crawlers, carrying data that does not exist in the database as parameter back to the source

  2. When other business parties in the company call our interface, there are a lot of mistransmission of parameters due to improper communication or other reasons

  3. A large number of parameters are mistransmitted due to client bugs

2. How to solve the cache penetration problem?

At present, we advocate to Cache empty data directly when the information cannot be found back to the source (note: the expiration time of the empty data Cache should be as small as possible to prevent meaningless content from occupying too much Cache memory), so that even in the case of parameter mistransmission, malicious attacks, etc., the DB will not be called every time.

But now it still exists the risk of being attacked, if a malicious attack to carry a small amount of parameter well, so there is no empty data cache takes up only a small amount of memory, but if the attacker using a large number of penetrating attack, carrying parameters are various, this will generate a lot of meaningless empty object caching, make our cache server memory.

At this time, the server needs to carry out simple control: according to the estimation of the business, what is the approximate range of the reasonable ID. For example, the cache marked by the user ID can directly judge whether the user ID parameter passed exceeds a certain threshold before obtaining the cache, and return null directly if the value exceeds the threshold. (For example, the total number of users is only hundreds of thousands or millions, resulting in tens of millions or even hundreds of millions of user IDS are obviously unreasonable)

Cache breakdown

1. What is cache breakdown?

Cache breakdown refers to the problem that a large number of requests are sent back to the source method after a key failure.

This is not a problem for a small number of accesses. However, when a hot key fails, too much traffic will be poured into the DB when the source is back. As a result, the PRESSURE on the DB will increase sharply at this time.

2. How to solve cache breakdown?

  1. Add mutex to the back source method: this can avoid multiple back to the source, but in n instance group mode, there will still be instances and send back the source, this magnitude compared to the previous large number of calls, has been greatly reduced.

  2. Appending distributed locks to the source method: This completely avoids the case of multiple instances and sending back the source, but the disadvantage is that a new service is introduced, which means an increased risk of exceptions.

Cache avalanche

1. What is cache avalanche?

Cache avalanche refers to the situation that a large number of cache data failures occur at a certain moment, and all requests are sent to DB, resulting in a sudden increase in DB load in a short period of time. Generally speaking, cache avalanche is caused by the following situations:

  1. Cache service scaling: This is caused by the cache data sharding strategy. If the data is sharded using a simple modular operation, the server scaling will cause an avalanche.

  2. Cache service outage: at a certain moment, a large number of cache server outages occur, resulting in the cache service is not available, according to the existing implementation, is directly connected to the DB.

2. How to avoid avalanches?

  1. High availability configuration of cache server: the sharding strategy of MC and Redis has been mentioned above, so the avalanche probability caused by scaling is very small. Second, Redis service implements high availability configuration: cluster mode is enabled, one master, one slave configuration. Due to the optimization of the consistent hash algorithm, MC downtime, expansion and contraction have little impact on the overall, so the cache server server itself can ensure good availability, as far as possible to avoid the occurrence of avalanche (unless large-scale downtime, probability is very small).

  2. Data sharding policy adjustment: Adjust the sharding policy of the cache server, for example, enable the sharding policy of the consistent hash algorithm for THE MC as described in the first part above, to prevent the cache data from becoming unavailable after the cache server expands and shrinks.

  3. Source traffic limiting: If the cache service fails and all the requests are sent to the DB, the DB cannot handle them. In this case, you are advised to perform overall traffic limiting when the cache service returns to the source, and the limited requests are automatically degraded or an error is reported.

Hot key problem

1. What is a hot key problem?

Understand the cache server implementation, can know a certain key will always fall on a server, if a key in a production environment is a large number of access, leads to a caching service node flow, such as access to more than single node load, there may be a single point of failure, a single point of failure after the transfer of the key data to other nodes, If a single point of problem persists, it may continue to fail the nodes to which it is migrated, ultimately affecting the entire cache service cluster.

2. How to solve the hot key problem?

  1. Multiple cache copy: Indicates that hotspot access is detected in advancekey, generate multiple copieskeyIn this way, the hot key can be held by multiple cache servers, and then the source method will share one. When requesting, a copy of the key will be randomly accessed according to a certain algorithm.

  1. Local cache: a layer of short-lived local cache around the hotspot key to buffer the pressure of the hotspot server.

conclusion

If you think the article is good, click a like to share and forward, thanks for your support!!