Introduction to the

This article explores how Zookeeper initializes and updates soul-admin data

An overview of

Zookeeper data synchronization in soul-admin configuration class location, and then explore the analysis

When no data is available, all data is read from the database and updated to Zookeeper. In the case of data, it will not be updated

Through the plug-in update processing function, through the call stack, a preliminary comb to get the general process of Zookeeper synchronous update data

  • 1. The background management interface (or Client registration, guess) sends a request to the HTTP interface, triggering data update
  • 2. The Admin background interface triggers and invokes the corresponding Service
  • 3. The event release center releases events
  • 4. The Zookeeper data synchronization component receives related events and processes them

For details about how to parse logs, see the source code Debug

Source code for the Debug

Looking for an entry point

First, according to Zhu Ming elder brother: Soul gateway source analysis -11 issues of the article positioning to the Admin module in the data synchronization core configuration class, the code is as follows:

@Configuration
public class DataSyncConfiguration {

    @Configuration
    @ConditionalOnProperty(prefix = "soul.sync.zookeeper", name = "url")
    @Import(ZookeeperConfiguration.class)
    static class ZookeeperListener {

        /** * Config event listener data changed listener. */
        @Bean
        @ConditionalOnMissingBean(ZookeeperDataChangedListener.class)
        public DataChangedListener zookeeperDataChangedListener(final ZkClient zkClient) {
            return new ZookeeperDataChangedListener(zkClient);
        }

        /** * Zookeeper data init zookeeper data init. */
        @Bean
        @ConditionalOnMissingBean(ZookeeperDataInit.class)
        public ZookeeperDataInit zookeeperDataInit(final ZkClient zkClient, final SyncDataService syncDataService) {
            return newZookeeperDataInit(zkClient, syncDataService); }}}Copy the code

Above we see the configuration of Websocket, HTTP, Zookeeper, and Nacos data synchronization (only Zookeeper is listed here). We can see that one is listening and one is initialized

Here’s an interlude: The Websocket configuration comments are as follows:

@Configuration
public class DataSyncConfiguration {

    @Configuration
    @ConditionalOnProperty(name = "soul.sync.websocket.enabled", havingValue = "true", matchIfMissing = true)
    @EnableConfigurationProperties(WebsocketSyncProperties.class)
    static class WebsocketListener {... }}Copy the code

In the previous article, we encountered a comment that commented out the Websocket synchronization configuration, but found that Admin enabled Websocket synchronization anyway. This is the reason. As you can see from the above comment, if there is no value for whether to enable Websocket synchronization, the default is true.

There are only two ways for Admin to disable Websocket synchronization:

  • 1. Set the configuration enabled to False
  • 2. Remove Websocket starter dependency (Admin module if available)

Data initialization

Let’s explore the initialization of Zookeeper data. The following class can be seen as an initialization class and is run at startup time

public class ZookeeperDataInit implements CommandLineRunner {

    @Override
    public void run(final String... args) {
        String pluginPath = ZkPathConstants.PLUGIN_PARENT;
        String authPath = ZkPathConstants.APP_AUTH_PARENT;
        String metaDataPath = ZkPathConstants.META_DATA;
        if(! zkClient.exists(pluginPath) && ! zkClient.exists(authPath) && ! zkClient.exists(metaDataPath)) {// Synchronize all datasyncDataService.syncAll(DataEventTypeEnum.REFRESH); }}}Copy the code

We set a breakpoint on the run function. Sure enough, the breakpoint is entered after Admin restarts, but we find that syncAll function cannot be entered during Debug

Zookeeper is initialized when it is empty

We removed the Soul node from Zookeeper to clean up all data

SyncAll = syncAll = syncAll = syncAll = syncAll = run

@Service("syncDataService")
public class SyncDataServiceImpl implements SyncDataService {

    // The basic logic is to read the data from the database and publish the event
    // The current five types of data are read and the event is published
    @Override
    public boolean syncAll(final DataEventTypeEnum type) {
        // type : refresh
        // Read the data and publish the event
        appAuthService.syncData();
        List<PluginData> pluginDataList = pluginService.listAll();
        eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.PLUGIN, type, pluginDataList));
        List<SelectorData> selectorDataList = selectorService.listAll();
        eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.SELECTOR, type, selectorDataList));
        List<RuleData> ruleDataList = ruleService.listAll();
        eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.RULE, type, ruleDataList));
        metaDataService.syncData();
        return true; }}Copy the code

In the above functions, pluginService is a cruD-related thing that we are all familiar with, but we won’t cover it here

As you can see, initialization is to read all the data from the database, and then publish the data update event

Process of plug-in data update

We look at another data listening class: ZookeeperDataChangedListener, find the very suspicious plugin updates the processing function, we on the breakpoint

Then we modify the status of the flow limiting plug-in in the Admin background management interface. Sure enough, we enter the breakpoint. After a little tracking, the final data will be written into the Zookeeper node. When Zookeeper data changes, it pushes data to Bootstrap, and the Bootstrap data is updated accordingly

public class ZookeeperDataChangedListener implements DataChangedListener {

    @Override
    public void onPluginChanged(final List<PluginData> changed, final DataEventTypeEnum eventType) {
        for (PluginData data : changed) {
            final String pluginPath = ZkPathConstants.buildPluginPath(data.getName());
            // delete
            if (eventType == DataEventTypeEnum.DELETE) {
                deleteZkPathRecursive(pluginPath);
                final String selectorParentPath = ZkPathConstants.buildSelectorParentPath(data.getName());
                deleteZkPathRecursive(selectorParentPath);
                final String ruleParentPath = ZkPathConstants.buildRuleParentPath(data.getName());
                deleteZkPathRecursive(ruleParentPath);
                continue;
            }
            //create or updateupsertZkNode(pluginPath, data); }}}Copy the code

Let’s move on to the call stack and see how it works

After receiving the event, it will distribute the event according to its GoupKey. The getSource should be the specific plug-in data, and the other is the event type, whether it is updated or deleted

public class DataChangedEventDispatcher implements ApplicationListener<DataChangedEvent>, InitializingBean {

    @Override
    @SuppressWarnings("unchecked")
    public void onApplicationEvent(final DataChangedEvent event) {
        for (DataChangedListener listener : listeners) {
            switch (event.getGroupKey()) {
                case APP_AUTH:
                    listener.onAppAuthChanged((List<AppAuthData>) event.getSource(), event.getEventType());
                    break;
                case PLUGIN:
                    listener.onPluginChanged((List<PluginData>) event.getSource(), event.getEventType());
                    break;
                case RULE:
                    listener.onRuleChanged((List<RuleData>) event.getSource(), event.getEventType());
                    break;
                case SELECTOR:
                    listener.onSelectorChanged((List<SelectorData>) event.getSource(), event.getEventType());
                    break;
                case META_DATA:
                    listener.onMetaDataChanged((List<MetaData>) event.getSource(), event.getEventType());
                    break;
                default:
                    throw new IllegalStateException("Unexpected value: "+ event.getGroupKey()); }}}}Copy the code

We continue to trace the call stack to the function below and find that it is a CRUD Service, where the processing logic is to perform database updates and then publish events

@Service("pluginService")
public class PluginServiceImpl implements PluginService {

    @Override
    @Transactional(rollbackFor = Exception.class)
    public String createOrUpdate(final PluginDTO pluginDTO) {
        final String msg = checkData(pluginDTO);
        if (StringUtils.isNoneBlank(msg)) {
            return msg;
        }
        PluginDO pluginDO = PluginDO.buildPluginDO(pluginDTO);
        DataEventTypeEnum eventType = DataEventTypeEnum.CREATE;
        if (StringUtils.isBlank(pluginDTO.getId())) {
            pluginMapper.insertSelective(pluginDO);
        } else {
            eventType = DataEventTypeEnum.UPDATE;
            pluginMapper.updateSelective(pluginDO);
        }

        // publish change event.
        eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.PLUGIN, eventType,
                Collections.singletonList(PluginTransfer.INSTANCE.mapToData(pluginDO))));
        returnStringUtils.EMPTY; }}Copy the code

Moving on to the Controller, the logic is simple enough to call the corresponding Service, which I won’t go into too much here. Here is the general process to go again

@RestController
@RequestMapping("/plugin")
public class PluginController {
    @PutMapping("/{id}")
    public SoulAdminResult updatePlugin(@PathVariable("id") final String id, @RequestBody final PluginDTO pluginDTO) {
        Objects.requireNonNull(pluginDTO);
        pluginDTO.setId(id);
        final String result = pluginService.createOrUpdate(pluginDTO);
        if (StringUtils.isNoneBlank(result)) {
            return SoulAdminResult.error(result);
        }
        returnSoulAdminResult.success(SoulResultMessage.UPDATE_SUCCESS); }}Copy the code

conclusion

This article preliminarily explores the processing flow of Zookeeper data synchronization on the Admin side, which can be roughly divided into initialization and data update (including deletion) processing flow

  • Data initialization:

    • If Zookeeper is not empty, the initialization operation is not performed
    • If Zookeeper is empty, the system initializes data, reads data from the database, and stores it to the database

  • Data processing process

    • HTTP interface call: can be the management background; It can also be a service registration Client
    • Service invocation: Updates the data in the database and invokes the publish event interface
    • Publish events: Publish events to data synchronization listeners
    • Data update: After receiving the event, update (Websocket push, Zookeeper write, HTTP update MD5, Nacos write)

Based on this analysis, the rough guess is as follows

It also accidentally resolved the previous Websocket data synchronization shutdown failure