Soul Overview of data synchronization interfaces

PluginDataSubscriber, PluginDataSubscriber, PluginDataSubscriber, PluginDataSubscriber, PluginDataSubscriber, PluginDataSubscriber MetaDataSubscriber AuthDataSubscriber. Let’s revisit the core logic of the Soul Gateway data synchronization module based on the following flow chart.

Event release

The Soul gateway provides a custom event type DataChangedEvent(core focus) based on the event mechanism provided by Spring. When a page or other interface accesses the gateway, it will start the event publishing, and the event publishing is divided into APP_AUTH. META_DATA, RULE, the SELECTOR, the PLUGIN several events, here need to pay attention to the definition of event types and unified by ConfigGroupEnum maintenance, and DataChangedEvent attribute also includes the class.

Admin side event processing

In spring-based ApplicationEventListener, DataChangedListener will be handed over to the specific subclass to implement. The current implementation methods include zooKeeper-based node monitoring communication, HTTP long polling, webSocket two-way communication, Configuration communication based on NACOS.

Each subclass has its own way of implementing data processing.

For example

  • Websocket sends corresponding information to the websocket client on the Bootstrap end for incremental synchronization.
  • Naocs maintains five types locally on the admin side (APP_AUTH…. as described above) In NacosDataChangedListener, no matter what type of data changes are made, the map on the Admin side is first modified, and then the full data is sent to the nacOS using the send configuration. It can also be assumed that nacOS-based data synchronization is full synchronization.
  • Zookeeper processes data on the admin side for each type of nodes stored on ZooKeeper (the three large nodes stored are in the format \soul\plugin….) And so on). (zk surveillance mechanism temporarily haven’t tested to monitor node, when the node under small nodes change for zk monitoring mechanism to monitor the data changes is the large node or a small nodes change data, here for the whole amount or incremental synchronization to make a question mark, it remains to be tested, the impression should be listening to the node changes)
  • HTTP long pollingData processing on the Admin side is done by combining five types (APP_AUTH… Etc.) in a large ConCurrentHashMap. During initialization, data will be synchronized to the Bootstrap end once, and changes to the data will be directlyFull search from the library, core point,Periodic thread pool periodically synchronizes data. Md5 checks whether data needs to be pushed, it can also be seen that the way isFull amount of synchronization The logic hasn’t been worked out and there’s something wrong with the logic.

Bootstrap event handling

In the data processing process of admin side, we can see that all data synchronization methods are used to make the Bootstrap side feel the data changes. Next, let’s see how the Bootstrap side deals with each synchronization type.

For example

  • Websocket When websocket is used for data synchronization, Bootstrap will create a client to connect to the Admin server and complete data synchronization through bidirectional communication. Core classes: WebsocketSyncDataService, WebsocketDataHandler. Focus on code: policy + simple factory + template implementation of data processing, Websocket heartbeat alive mechanism
  • When nacOS is used for data synchronization, bootstrap will listen on several configuration items of NACOS, and the data transfer is directly cancelled and then published, which further verifies the possibility that nacOS synchronization is full synchronization. Core classes: NacosSyncDataService, NacosCacheHandler.
  • When ZooKeeper is used for data synchronization, Bootstrap listens on zooKeeper’s Corresponding ZNodes. Note that only plugin, meta_data, and Auth are monitored. The selector and rule are associated with the child nodes under plugin, and subscribe to three big nodes during initialization. The watch mechanism based on ZooKeeper realizes the monitoring of data changes, and there is only one class ZookeeperSyncDataService, focusing on the monitoring mechanism of ZK.
  • HTTP long polling logic has not been carefully analyzed

The core points of bootstrap side processing

When creating the XXXSyncDataService class, you pass in the PluginDataSubscriber, MetaDataSubscriber, and AuthDataSubscriber instance data. So you might wonder what are these three types of data for?

Let’s take a look at them all

PluginDataSubscriber data subscription interface

Regardless of the way of data synchronization, targeted atPlugin, Rule, SELECTORData changes are processed byPluginDataSubscriberA subclass ofCommonPluginDataSubscriberTo achieve, forCommonPluginDataSubscriberClass, it not only completes the data changes in memory, but also is responsible for notifying the corresponding plug-in chain implementation of each data change, involving the classPluginDataHandler and its concrete implementation classes, the specific notification data type is shown in the figure below.No specific research has been carried out on the implementation method of the specific subclass as shown in the figure above.

MetaDataSubscriber metadata subscription interface

Similar to PluginDataSubscriber, it is targeted atMETA_DATAData processing, and finally are informedMetaDataSubscriber concrete implementation classThe specific implementation class here is shown in the figure below.

It can be seen from the above figure that the realization of subscription metadata processing is mainly RPC correlation and a special MetaDataAllSubscriber. Here, we put a question mark first, and then conduct in-depth analysis when tracing the specific plug-in chain in the later stage.

AuthDataSubscriber Authenticates data subscription

AuthDataSubscriber class implementation class only one SignAuthDataSubscriber, in this implementation class is mainly defined to the SignAuthDataCache cache data changes, too much after a design to the cache is mainly sign plug-in.

Summary of Data Synchronization

The process to comb

  • Data processing on the Admin side can be generally considered as how to notify the Bootstrap side when the corresponding data changes.
  • The logic on the bootstrap side basically means that when I learn about a data type change, I can obtain the changed data through implementation.

tips

  • Soul’s data synchronization module is a large template that provides several top-level interfaces for synchronizing data.
  • What is not mentioned above is how each method synchronizes the full amount of data during bootstrap initialization.
  • Can we extend other types of data synchronization based on the Soul large template pattern? Broadcast based on Redis, file, database, message queue