Background:

Part of the project data such as page static data needs to be obtained from Redis to improve page loading speed. Page data modification is to modify mysql through the management screen, through the scheduled task or manually synchronize mysql data to Redis, for the page to obtain. In this case, data synchronization is not incremental synchronization, but synchronization after obtaining data sets based on conditional query. Some parameters in the gateway are also maintained in mysql, and then pushed periodically or manually to Redis, and notified the gateway to refresh the memory data. Sometimes adding a change of data will start multiple synchronous operations, such as adding an API, which should be displayed on the portal screen and pushed to the gateway. If you want to take effect in real time, you need two manual operations. This is very tedious, so design a real-time or quasi-real-time mysql to redis synchronization scheme.

Plan 1:

Update the database at the same time the implementation of synchronous REDis method, such as adding annotations in the method of deleting and modifying home data, using AOP to deal with the logic of data synchronization. This solution is essentially spring Cache. But instead of just synchronizing data to Redis, you also publish subscribe events, so you need to implement the AOP logic yourself.

Scheme 2:

Using CANAL, monitor mysql data changes for data synchronization and publish and subscribe events. This eliminates the need to modify the original code and decouples the logic. In addition, the synchronization operation will not be triggered if mysql modification fails.

To sum up, plan 2 is adopted.

Detailed scheme design:

Define a destination listening database on Canal, and call the corresponding service’s synchronous method through feign when there is a change in the data of interest (the synchronous method here reuses the existing manual call method, not new write). If no new data changes within a minute after a data change, the synchronization operation is triggered. In addition, the synchronization service is dynamically started and stopped by the Apollo configuration center.

Canal deployment:

Referring to the scheme on the official website, zooKeeper is used for service coordination to achieve high availability of Canal Server and Canal Client.

Because the test environments are divided into SIT, QA and UAT, each of which has its own database, Canal defines three destinations to listen to the corresponding database instances.

Future optimization direction:

  • Redis failed to get data fallback to database
  • Add a layer of memory cache
  • Perceiving whether data synchronization succeeds