Module loading mechanism

Basic overview

Module is a mechanism provided by Skywalking in OAP to manage features. Through the Module mechanism, you can easily define modules and provide multiple implementations. You can choose any implementation in the configuration file.

For details about modules, see Backend Setup and Configuration Vocabulary

The class diagram

Skywalking module management related functions in the org. Apache. Skywalking. Oap. Server. If the module package.

It can be seen from the class diagram that the Skywalking module mechanism is roughly divided into the following modules:

• Module configuration:ApplicationConfiguration 、 ModuleConfiguration 、 ProviderConfiguration
  • PS: Exactlyapplication.ymlThree-tier structure: module -> module implementation -> configuration of a module implementation.
• Module definition class:ModuleDefine
• The module provides classes:ModuleProvider
• Services:Service
• Management:ModuleManager
• Some helper classes
  • ModuleDefineHolder: Interface used by the module management class to search for modules
  • ModuleProviderHolder: a module defines the interface that the class needs to implement and provides the function of getting the service class of the module
  • ModuleServiceHolder: Module provides the interface that the class needs to realize, and provides the functions of registering service implementation and obtaining service objects
  • ModuleConfig: module configuration class, module definition class willProviderConfigurationMapped toModuleConfig
  • ApplicationConfigLoader:ApplicationConfigurationThe helper class willapplication.ymlConfiguration file loaded into memory, SettingsselectorThe correspondingProviderConfiguration information of

Class diagram source file: Skywalk-module.uml

The source code parsing

ModuleDefine

package org.apache.skywalking.oap.server.library.module;

import java.lang.reflect.Field;
import java.util.Enumeration;
import java.util.Properties;
import java.util.ServiceLoader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// Module definition
public abstract class ModuleDefine implements ModuleProviderHolder {
    private static final Logger LOGGER = LoggerFactory.getLogger(ModuleDefine.class);
    // Module actual
    private ModuleProvider loadedProvider = null;

    private final String name;

    public ModuleDefine(String name) {
        this.name = name;
    }

    / / module name
    public final String name(a) {
        return name;
    }

    // The implementation class can define the service classes provided by the module
    public abstract Class[] services();

    /**
     * Run the prepare stage for the module, including finding all potential providers, and asking them to prepare.
     *
     * @param moduleManager of this module
     * @param configuration of this module
     * @throws ProviderNotFoundException when even don't find a single one providers.
     */
    // In the preparation phase, locate the ModuleProvider object corresponding to the Configuration class and initialize it
    void prepare(/ / module management object ModuleManager ModuleManager, / class/module configuration ApplicationConfiguration ModuleConfiguration configuration, ServiceLoader
       
         moduleProviderLoader)
        throws ProviderNotFoundException, ServiceNotProvidedException, ModuleConfigException, ModuleStartException {
        // Find the ModuleProvider object corresponding to the Configuration class
        for (ModuleProvider provider : moduleProviderLoader) {
            if(! configuration.has(provider.name())) {continue;
            }
            if (provider.module().equals(getClass())) {
                if (loadedProvider == null) {
                    loadedProvider = provider;
                    loadedProvider.setManager(moduleManager);
                    loadedProvider.setModuleDefine(this);
                } else {
                    throw new DuplicateProviderException(this.name() + " module has one " + loadedProvider.name() + "[" + loadedProvider.getClass().getName() + "] provider already, " + provider.name() + "[" + provider.getClass().getName() + "] is defined as 2nd provider."); }}}if (loadedProvider == null) {
            throw new ProviderNotFoundException(this.name() + " module no provider found.");
        }

        // Copy the config file of the supplied class into the ModuleConfig object
        LOGGER.info("Prepare the {} provider in {} module.", loadedProvider.name(), this.name());
        try {
            copyProperties(
                loadedProvider.createConfigBeanIfAbsent(), 
                configuration.getProviderConfiguration(loadedProvider.name()), 
                this.name(), 
                loadedProvider.name()
            );
        } catch (IllegalAccessException e) {
            throw new ModuleConfigException(this.name() + " module config transport to config bean failure.", e);
        }
        // The module provides the object to be ready
        loadedProvider.prepare();
    }

    // Use reflection to copy properties
    private void copyProperties(ModuleConfig dest, Properties src, String moduleName, String providerName) throws IllegalAccessException {
        if (dest == null) {
            return; } Enumeration<? > propertyNames = src.propertyNames();while (propertyNames.hasMoreElements()) {
            String propertyName = (String) propertyNames.nextElement();
            Class<? extends ModuleConfig> destClass = dest.getClass();
            try {
                Field field = getDeclaredField(destClass, propertyName);
                field.setAccessible(true);
                field.set(dest, src.get(propertyName));
            } catch (NoSuchFieldException e) {
                LOGGER.warn(propertyName + " setting is not supported in " + providerName + " provider of " + moduleName + " module"); }}}private Field getDeclaredField(Class
        destClass, String fieldName) throws NoSuchFieldException {
        if(destClass ! =null) {
            Field[] fields = destClass.getDeclaredFields();
            for (Field field : fields) {
                if (field.getName().equals(fieldName)) {
                    returnfield; }}return getDeclaredField(destClass.getSuperclass(), fieldName);
        }
        throw new NoSuchFieldException();
    }

    // Get the Provider object corresponding to the module definition
    @Override
    public final ModuleProvider provider(a) throws DuplicateProviderException, ProviderNotFoundException {
        if (loadedProvider == null) {
            throw new ProviderNotFoundException("There is no module provider in " + this.name() + " module!");
        }
        returnloadedProvider; }}Copy the code

ModuleProviderHolder

package org.apache.skywalking.oap.server.library.module;

// The module provides a holding interface. Through this interface, you can obtain the Service holding interface corresponding to the module Provider object and obtain the Service object corresponding to the module Provider object
public interface ModuleProviderHolder {
    // Get the module supplied object
    ModuleServiceHolder provider(a) throws DuplicateProviderException, ProviderNotFoundException;
}
Copy the code

ModuleProvider

package org.apache.skywalking.oap.server.library.module;

import java.util.HashMap;
import java.util.Map;
import lombok.Setter;

// A module provides an abstract class from which all modules provide classes need to inherit
// A module definition can be configured to provide classes for multiple modules, by switching in application.yml
public abstract class ModuleProvider implements ModuleServiceHolder {
    // Module manager
    @Setter
    private ModuleManager manager;
    // The module defines the object
    @Setter
    private ModuleDefine moduleDefine;
    // The module provides the map of the corresponding service object
    private final Map<Class<? extends Service>, Service> services = new HashMap<>();

    public ModuleProvider(a) {}protected final ModuleManager getManager(a) {
        return manager;
    }

    // Get the name of the service provider implementation class
    public abstract String name(a);

    // Define the module definition class implemented by the module provider
    public abstract Class<? extends ModuleDefine> module(a);// Create the module definition configuration object
    public abstract ModuleConfig createConfigBeanIfAbsent(a);

    // Prepare (initialize things that are not related to other modules)
    public abstract void prepare(a) throws ServiceNotProvidedException, ModuleStartException;

    // Start phase (this phase can interoperate modules)
    public abstract void start(a) throws ServiceNotProvidedException, ModuleStartException;

    // Notification phase after completion (executed after all modules have successfully started)
    public abstract void notifyAfterCompleted(a) throws ServiceNotProvidedException, ModuleStartException;

    // The names of other modules on which the module depends
    public abstract String[] requiredModules();

    // Register the service implementation class
    @Override
    public final void registerServiceImplementation(Class<? extends Service> serviceType, Service service) throws ServiceNotProvidedException {
        if (serviceType.isInstance(service)) {
            this.services.put(serviceType, service);
        } else {
            throw new ServiceNotProvidedException(serviceType + " is not implemented by "+ service); }}// Ensure that all services are implemented
    void requiredCheck(Class<? extends Service>[] requiredServices) throws ServiceNotProvidedException {
        if (requiredServices == null)
            return;
        for (Class<? extends Service> service : requiredServices) {
            if(! services.containsKey(service)) {throw new ServiceNotProvidedException("Service:" + service.getName() + " not provided"); }}if(requiredServices.length ! = services.size()) {throw new ServiceNotProvidedException("The " + this.name() + " provider in " + moduleDefine.name() + " moduleDefine provide more service implementations than ModuleDefine requirements."); }}// Get the service implementation object
    @Override
    public @SuppressWarnings("unchecked")
    <T extends Service> T getService(Class<T> serviceType) throws ServiceNotProvidedException {
        Service serviceImpl = services.get(serviceType);
        if(serviceImpl ! =null) {
            return (T) serviceImpl;
        }
        throw new ServiceNotProvidedException("Service " + serviceType.getName() + " should not be provided, based on moduleDefine define.");
    }

    ModuleDefine getModule(a) {
        return moduleDefine;
    }

    String getModuleName(a) {
        returnmoduleDefine.name(); }}Copy the code

ModuleConfig

package org.apache.skywalking.oap.server.library.module;

// Module configuration class
public abstract class ModuleConfig {}Copy the code

ModuleServiceHolder

package org.apache.skywalking.oap.server.library.module;

// Module services hold interfaces
public interface ModuleServiceHolder {
    // Register the service implementation object
    void registerServiceImplementation(Class<? extends Service> serviceType, Service service) throws ServiceNotProvidedException;
    
    // Get the service implementation object
    <T extends Service> T getService(Class<T> serviceType) throws ServiceNotProvidedException;
}
Copy the code

Service

package org.apache.skywalking.oap.server.library.module;

// Service interface
public interface Service {}Copy the code

ModuleDefineHolder

package org.apache.skywalking.oap.server.library.module;

// The module defines the holding interface
public interface ModuleDefineHolder {
    // Check whether the module exists
    boolean has(String moduleName);
    
    // Get the module definition object by the module name
    ModuleProviderHolder find(String moduleName) throws ModuleNotFoundRuntimeException;
}
Copy the code

ModuleManager

package org.apache.skywalking.oap.server.library.module;

import java.util.Arrays;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
import java.util.ServiceLoader;

// Module management class, manage module life cycle
public class ModuleManager implements ModuleDefineHolder {
    // Whether all modules have passed the preparation stage
    private boolean isInPrepareStage = true;
    
    // All loaded modules define object maps
    private final Map<String, ModuleDefine> loadedModules = new HashMap<>();

    // Initialize all configured modules
    public void init(ApplicationConfiguration applicationConfiguration) throws ModuleNotFoundException, ProviderNotFoundException, ServiceNotProvidedException, CycleDependencyException, ModuleConfigException, ModuleStartException {
        // Get the module name in the configuration class
        String[] moduleNames = applicationConfiguration.moduleList();
        // SPI loads all module definition objects
        ServiceLoader<ModuleDefine> moduleServiceLoader = ServiceLoader.load(ModuleDefine.class);
        // SPI loads all module supplied objects
        ServiceLoader<ModuleProvider> moduleProviderLoader = ServiceLoader.load(ModuleProvider.class);
        // All modules defined in the configuration class are prepared
        LinkedList<String> moduleList = new LinkedList<>(Arrays.asList(moduleNames));
        for (ModuleDefine module : moduleServiceLoader) {
            for (String moduleName : moduleNames) {
                if (moduleName.equals(module.name())) {
                    module.prepare(this, applicationConfiguration.getModuleConfiguration(moduleName), moduleProviderLoader);
                    loadedModules.put(moduleName, module); moduleList.remove(moduleName); }}}// The preparation phase is over
        isInPrepareStage = false;

        if (moduleList.size() > 0) {
            throw new ModuleNotFoundException(moduleList.toString() + " missing.");
        }

        // Determine the initialization order of modules according to the requiredModules method in the module provision object (dependent modules are loaded first)
        BootstrapFlow bootstrapFlow = new BootstrapFlow(loadedModules);
        // All modules enter the startup phase
        bootstrapFlow.start(this);
        // All modules enter the completion notification phase
        bootstrapFlow.notifyAfterCompleted();
    }

    // Check whether the module exists
    @Override
    public boolean has(String moduleName) {
        returnloadedModules.get(moduleName) ! =null;
    }

    // Get the module definition object by the module name
    @Override
    public ModuleProviderHolder find(String moduleName) throws ModuleNotFoundRuntimeException {
        assertPreparedStage();
        ModuleDefine module = loadedModules.get(moduleName);
        if (module! =null)
            return module;
        throw new ModuleNotFoundRuntimeException(moduleName + " missing.");
    }

    // Assert whether the preparation phase is still underway, and if so, throw an exception
    private void assertPreparedStage(a) {
        if (isInPrepareStage) {
            throw new AssertionError("Still in preparing stage."); }}}Copy the code

BootstrapFlow

package org.apache.skywalking.oap.server.library.module;

import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// Determine the initialization order of modules according to the requiredModules method in the module provision object (dependent modules are loaded first)
class BootstrapFlow {
    private static final Logger LOGGER = LoggerFactory.getLogger(BootstrapFlow.class);

    private Map<String, ModuleDefine> loadedModules;
    // Modules that are sorted by dependency provide a list of objects
    private List<ModuleProvider> startupSequence;

    BootstrapFlow(Map<String, ModuleDefine> loadedModules) throws CycleDependencyException, ModuleNotFoundException {
        this.loadedModules = loadedModules;
        startupSequence = new LinkedList<>();

        // The dependent module is loaded first
        makeSequence();
    }

    @SuppressWarnings("unchecked")
    void start( ModuleManager moduleManager) throws ModuleNotFoundException, ServiceNotProvidedException, ModuleStartException {
        for (ModuleProvider provider : startupSequence) {
            LOGGER.info("start the provider {} in {} module.", provider.name(), provider.getModuleName()); provider.requiredCheck(provider.getModule().services()); provider.start(); }}void notifyAfterCompleted(a) throws ServiceNotProvidedException, ModuleStartException {
        for(ModuleProvider provider : startupSequence) { provider.notifyAfterCompleted(); }}private void makeSequence(a) throws CycleDependencyException, ModuleNotFoundException {
        List<ModuleProvider> allProviders = new ArrayList<>();
        // Check whether all dependent modules exist
        for (final ModuleDefine module : loadedModules.values()) {
            String[] requiredModules = module.provider().requiredModules();
            if(requiredModules ! =null) {
                for (String requiredModule : requiredModules) {
                    if(! loadedModules.containsKey(requiredModule)) {throw new ModuleNotFoundException(requiredModule + " module is required by " + module.provider().getModuleName() + "." + module.provider().name() + ", but not found.");
                    }
                }
            }

            allProviders.add(module.provider());
        }

        do {
            int numOfToBeSequenced = allProviders.size();
            for (int i = 0; i < allProviders.size(); i++) {
                ModuleProvider provider = allProviders.get(i);
                String[] requiredModules = provider.requiredModules();
                if (CollectionUtils.isNotEmpty(requiredModules)) {
                    // Whether all dependent modules are in startupSequence
                    boolean isAllRequiredModuleStarted = true;
                    for (String module : requiredModules) {
                        boolean exist = false;
                        for (ModuleProvider moduleProvider : startupSequence) {
                            if (moduleProvider.getModuleName().equals(module)) {
                                exist = true;
                                break; }}if(! exist) { isAllRequiredModuleStarted =false;
                            break; }}// If all dependent modules are in startupSequence, add objects provided by that module to startupSequence
                    if(isAllRequiredModuleStarted) { startupSequence.add(provider); allProviders.remove(i); i--; }}else {
                    // Add the startupSequence if the module provides objects that do not depend on any other modulesstartupSequence.add(provider); allProviders.remove(i); i--; }}// If no object is added to the startupSequence after a loop, then a loop dependency exists
            if (numOfToBeSequenced == allProviders.size()) {
                StringBuilder unSequencedProviders = new StringBuilder();
                allProviders.forEach(provider -> unSequencedProviders.append(provider.getModuleName()).append("[provider=").append(provider.getClass().getName()).append("]\n"));
                throw new CycleDependencyException("Exist cycle module dependencies in \n" + unSequencedProviders.substring(0, unSequencedProviders.length() - 1)); }}while(allProviders.size() ! =0); // If the list of supplied objects is not empty, the loop continues}}Copy the code

ApplicationConfiguration

package org.apache.skywalking.oap.server.library.module;

import java.util.HashMap;
import java.util.Properties;

// OAP application configuration class
public class ApplicationConfiguration {
    // The module defines the configuration map
    private HashMap<String, ModuleConfiguration> modules = new HashMap<>();

    // Module configuration list
    public String[] moduleList() {
        return modules.keySet().toArray(new String[0]);
    }

    // Add the module definition configuration
    public ModuleConfiguration addModule(String moduleName) {
        ModuleConfiguration newModule = new ModuleConfiguration();
        modules.put(moduleName, newModule);
        return newModule;
    }

    // Check whether the specified module name exists in the module definition configuration map
    public boolean has(String moduleName) {
        return modules.containsKey(moduleName);
    }

    // Get the module definition configuration
    public ModuleConfiguration getModuleConfiguration(String name) {
        return modules.get(name);
    }

    // The module defines the configuration class
    public static class ModuleConfiguration {
        // The module provides an object map
        private HashMap<String, ProviderConfiguration> providers = new HashMap<>();

        private ModuleConfiguration(a) {}// Get the configuration provided by the module
        public Properties getProviderConfiguration(String name) {
            return providers.get(name).getProperties();
        }

        // Whether there is a module to provide configuration
        public boolean has(String name) {
            return providers.containsKey(name);
        }

        // Add module provides configuration
        public ModuleConfiguration addProviderConfiguration(String name, Properties properties) {
            ProviderConfiguration newProvider = new ProviderConfiguration(properties);
            providers.put(name, newProvider);
            return this; }}// The module provides configuration classes
    public static class ProviderConfiguration {
        // The module provides attributes
        private Properties properties;

        ProviderConfiguration(Properties properties) {
            this.properties = properties;
        }

        private Properties getProperties(a) {
            returnproperties; }}}Copy the code

ApplicationConfigLoader

package org.apache.skywalking.oap.server.starter.config;

import java.io.FileNotFoundException;
import java.io.Reader;
import java.util.HashSet;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.apm.util.PropertyPlaceholderHelper;
import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
import org.apache.skywalking.oap.server.library.module.ProviderNotFoundException;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;
import org.apache.skywalking.oap.server.library.util.ResourceUtils;
import org.yaml.snakeyaml.Yaml;

// application.yml loads classes with three layers: module definition name. The module provides a name. property key
@Slf4j
public class ApplicationConfigLoader implements ConfigLoader<ApplicationConfiguration> {
    // Use "-" when no module provider is configured
    private static final String DISABLE_SELECTOR = "-";
    // This field selects the module provider
    private static final String SELECTOR = "selector";

    private final Yaml yaml = new Yaml();

    @Override
    public ApplicationConfiguration load(a) throws ConfigFileNotFoundException {
        ApplicationConfiguration configuration = new ApplicationConfiguration();
        this.loadConfig(configuration);
        this.overrideConfigBySystemEnv(configuration);
        return configuration;
    }

    @SuppressWarnings("unchecked")
    private void loadConfig(ApplicationConfiguration configuration) throws ConfigFileNotFoundException {
        try {
            Reader applicationReader = ResourceUtils.read("application.yml");
            Map<String, Map<String, Object>> moduleConfig = yaml.loadAs(applicationReader, Map.class);
            if (CollectionUtils.isNotEmpty(moduleConfig)) {
                selectConfig(moduleConfig);
                moduleConfig.forEach((moduleName, providerConfig) -> {
                    if (providerConfig.size() > 0) {
                        log.info("Get a module define from application.yml, module name: {}", moduleName);
                        ApplicationConfiguration.ModuleConfiguration moduleConfiguration = configuration.addModule(moduleName);
                        providerConfig.forEach((providerName, config) -> {
                            log.info("Get a provider define belong to {} module, provider name: {}", moduleName, providerName);
                            finalMap<String, ? > propertiesConfig = (Map<String, ? >) config;final Properties properties = new Properties();
                            if(propertiesConfig ! =null) {
                                propertiesConfig.forEach((propertyName, propertyValue) -> {
                                    if (propertyValue instanceof Map) {
                                        Properties subProperties = new Properties();
                                        ((Map) propertyValue).forEach((key, value) -> {
                                            subProperties.put(key, value);
                                            replacePropertyAndLog(key, value, subProperties, providerName);
                                        });
                                        properties.put(propertyName, subProperties);
                                    } else{ properties.put(propertyName, propertyValue); replacePropertyAndLog(propertyName, propertyValue, properties, providerName); }}); } moduleConfiguration.addProviderConfiguration(providerName, properties); }); }else {
                        log.warn("Get a module define from application.yml, but no provider define, use default, module name: {}", moduleName); }}); }}catch (FileNotFoundException e) {
            throw newConfigFileNotFoundException(e.getMessage(), e); }}private void replacePropertyAndLog(final Object propertyName, final Object propertyValue, final Properties target, final Object providerName) {
        final String valueString = PropertyPlaceholderHelper.INSTANCE.replacePlaceholders(propertyValue + "", target);
        if(valueString ! =null) {
            if (valueString.trim().length() == 0) {
                target.replace(propertyName, valueString);
                log.info("Provider={} config={} has been set as an empty string", providerName, propertyName);
            } else {
                // Use YAML to do data type conversion.
                final Object replaceValue = yaml.load(valueString);
                if(replaceValue ! =null) {
                    target.replace(propertyName, replaceValue);
                    log.info("Provider={} config={} has been set as {}", providerName, propertyName, replaceValue.toString()); }}}}private void overrideConfigBySystemEnv(ApplicationConfiguration configuration) {
        for(Map.Entry<Object, Object> prop : System.getProperties().entrySet()) { overrideModuleSettings(configuration, prop.getKey().toString(), prop.getValue().toString()); }}private void selectConfig(final Map<String, Map<String, Object>> moduleConfiguration) {
        final Set<String> modulesWithoutProvider = new HashSet<>();
        for (final Map.Entry<String, Map<String, Object>> entry : moduleConfiguration.entrySet()) {
            final String moduleName = entry.getKey();
            final Map<String, Object> providerConfig = entry.getValue();
            if(! providerConfig.containsKey(SELECTOR)) {continue;
            }
            final String selector = (String) providerConfig.get(SELECTOR);
            finalString resolvedSelector = PropertyPlaceholderHelper.INSTANCE.replacePlaceholders( selector, System.getProperties() ); providerConfig.entrySet().removeIf(e -> ! resolvedSelector.equals(e.getKey()));if(! providerConfig.isEmpty()) {continue;
            }

            if(! DISABLE_SELECTOR.equals(resolvedSelector)) {throw new ProviderNotFoundException("no provider found for module " + moduleName + "," + "if you're sure it's not required module and want to remove it, " + "set the selector to -");
            }

            // now the module can be safely removed
            modulesWithoutProvider.add(moduleName);
        }

        moduleConfiguration.entrySet().removeIf(e -> {
            final String module = e.getKey();
            final boolean shouldBeRemoved = modulesWithoutProvider.contains(module);
            if (shouldBeRemoved) {
                log.info("Remove module {} without any provider".module);
            }
            return shouldBeRemoved;
        });
    }

    private void overrideModuleSettings(ApplicationConfiguration configuration, String key, String value) {
        int moduleAndConfigSeparator = key.indexOf('. ');
        if (moduleAndConfigSeparator <= 0) {
            return;
        }
        String moduleName = key.substring(0, moduleAndConfigSeparator);
        String providerSettingSubKey = key.substring(moduleAndConfigSeparator + 1);
        ApplicationConfiguration.ModuleConfiguration moduleConfiguration = configuration.getModuleConfiguration(moduleName);
        if (moduleConfiguration == null) {
            return;
        }
        int providerAndConfigSeparator = providerSettingSubKey.indexOf('. ');
        if (providerAndConfigSeparator <= 0) {
            return;
        }
        String providerName = providerSettingSubKey.substring(0, providerAndConfigSeparator);
        String settingKey = providerSettingSubKey.substring(providerAndConfigSeparator + 1);
        if(! moduleConfiguration.has(providerName)) {return;
        }
        Properties providerSettings = moduleConfiguration.getProviderConfiguration(providerName);
        if(! providerSettings.containsKey(settingKey)) {return; } Object originValue = providerSettings.get(settingKey); Class<? > type = originValue.getClass();if (type.equals(int.class) || type.equals(Integer.class))
            providerSettings.put(settingKey, Integer.valueOf(value));
        else if (type.equals(String.class))
            providerSettings.put(settingKey, value);
        else if (type.equals(long.class) || type.equals(Long.class))
            providerSettings.put(settingKey, Long.valueOf(value));
        else if (type.equals(boolean.class) || type.equals(Boolean.class)) {
            providerSettings.put(settingKey, Boolean.valueOf(value));
        } else {
            return;
        }
        log.info("The setting has been override by key: {}, value: {}, in {} provider of {} module through {}", settingKey, value, providerName, moduleName, "System.properties"); }}Copy the code

ConfigLoader

package org.apache.skywalking.oap.server.starter.config;

// Configure the loading interface
public interface ConfigLoader<T> {
    T load(a) throws ConfigFileNotFoundException;
}
Copy the code

OAPServerBootstrap

package org.apache.skywalking.oap.server.starter;

import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.core.RunningMode;
import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader;
import org.apache.skywalking.oap.server.starter.config.ConfigLoader;
import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;

// OAP starts the class, loads the configuration file, initializes the module
@Slf4j
public class OAPServerBootstrap {
    public static void start(a) {
        String mode = System.getProperty("mode");
        // Start mode
        RunningMode.setMode(mode);

        ApplicationConfigLoader configLoader = new ApplicationConfigLoader();
        ModuleManager manager = new ModuleManager();
        try {
            // Load the configuration from the configuration file
            ApplicationConfiguration applicationConfiguration = configLoader.load();
            // Initialize the module
            manager.init(applicationConfiguration);

            // Send the boot time to Telemetry
            manager.find(TelemetryModule.NAME)
                   .provider()
                   .getService(MetricsCreator.class)
                   .createGauge("uptime"."oap server start up time", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE)
                   // Set uptime to second
                   .setValue(System.currentTimeMillis() / 1000d);

            if (RunningMode.isInitMode()) {
                log.info("OAP starts up in init mode successfully, exit now...");
                System.exit(0); }}catch (Throwable t) {
            log.error(t.getMessage(), t);
            System.exit(1); }}}Copy the code

OAPServerStartUp

package org.apache.skywalking.oap.server.starter;

// OAP start class
public class OAPServerStartUp {
    public static void main(String[] args) { OAPServerBootstrap.start(); }}Copy the code

In order toSkywalking OAPEnable the process analysis module loading mechanism

Sequence diagram

Source file: oapServerstartup. SDT

Case: Storage module loading analysis

The configuration file

From the application.yml configuration file, you can see that.

Modules are defined in a three-tier structure:

• The first layer: module definition name
• Layer 2: The module provides the name /selector
• Layer 3: modules provide configuration information /selectorThe selected module provides the configuration

storage:
  selector: ${SW_STORAGE:h2}
  elasticsearch:
    nameSpace: ${SW_NAMESPACE:""}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    # etc...
  elasticsearch7:
    nameSpace: ${SW_NAMESPACE:""}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    # etc...
  h2:
    driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
    url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db; DB_CLOSE_DELAY=-1}
    # etc...
  mysql:
    properties:
      jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:3306/swtest"}
      dataSource.user: ${SW_DATA_SOURCE_USER:root}
      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root@1234}
      # etc...
  tidb:
    properties:
      jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:4000/tidbswtest"}
      dataSource.user: ${SW_DATA_SOURCE_USER:root}
      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:""}
      # etc...
  influxdb:
    url: ${SW_STORAGE_INFLUXDB_URL:http://localhost:8086}
    user: ${SW_STORAGE_INFLUXDB_USER:root}
    password: ${SW_STORAGE_INFLUXDB_PASSWORD:}
    # etc...
Copy the code

Load the configuration

After org. Apache. Skywalking. Oap. Server. The starter. Config. ApplicationConfigLoader# load call, . Org. Apache. Skywalking oap. Server. The starter. OAPServerBootstrap# start to get to all need to be loaded modules, including storage module.

In org. Apache. Skywalking) oap) server. The starter. Config. ApplicationConfigLoader# selectConfig through storage. Also the selector = h2, The storage module only retains h2 configuration information:

storage:
    h2:
    driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
    url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db; DB_CLOSE_DELAY=-1}
    # etc...
Copy the code

Preparation stage

In the org. Apache. Skywalking) oap) server. If the module. The ModuleManager# init, through SPI loading the module definition object, storage module corresponding to the definition of the class is as follows:

PS: You can see that a large number of Service interfaces are defined

package org.apache.skywalking.oap.server.core.storage;

import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskLogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskQueryDAO;
import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
import org.apache.skywalking.oap.server.library.module.ModuleDefine;

/** * StorageModule provides the capabilities(services) to interact with the database. With different databases, this * module could have different providers, such as currently, H2, MySQL, ES, TiDB. */
public class StorageModule extends ModuleDefine {

    public static final String NAME = "storage";

    public StorageModule(a) {
        super(NAME);
    }

    @Override
    public Class[] services() {
        return newClass[]{ IBatchDAO.class, StorageDAO.class, IHistoryDeleteDAO.class, INetworkAddressAliasDAO.class, ITopologyQueryDAO.class, IMetricsQueryDAO.class, ITraceQueryDAO.class, IMetadataQueryDAO.class, IAggregationQueryDAO.class, IAlarmQueryDAO.class, ITopNRecordsQueryDAO.class, ILogQueryDAO.class, IProfileTaskQueryDAO.class, IProfileTaskLogQueryDAO.class, IProfileThreadSnapshotQueryDAO.class, UITemplateManagementDAO.class, IBrowserLogQueryDAO.class }; }}Copy the code

Also called org. Apache. Skywalking. Oap. Server. If the module. ModuleDefine# prepare into the preparation stage

        String[] moduleNames = applicationConfiguration.moduleList();
        ServiceLoader<ModuleDefine> moduleServiceLoader = ServiceLoader.load(ModuleDefine.class);
        ServiceLoader<ModuleProvider> moduleProviderLoader = ServiceLoader.load(ModuleProvider.class);

        LinkedList<String> moduleList = new LinkedList<>(Arrays.asList(moduleNames));
        for (ModuleDefine module : moduleServiceLoader) {
            for (String moduleName : moduleNames) {
                if (moduleName.equals(module.name())) {
                    module.prepare(this, applicationConfiguration.getModuleConfiguration(moduleName), moduleProviderLoader);
                    loadedModules.put(moduleName, module); moduleList.remove(moduleName); }}}Copy the code

In the org. Apache. Skywalking) oap) server. If the module. The ModuleDefine# will prepare by passing in the configuration of the matching in the configuration file only selected modules provide object.

        for (ModuleProvider provider : moduleProviderLoader) {
            if(! configuration.has(provider.name())) {continue;
            }

            if (provider.module().equals(getClass())) {
                if (loadedProvider == null) {
                    loadedProvider = provider;
                    loadedProvider.setManager(moduleManager);
                    loadedProvider.setModuleDefine(this);
                } else {
                    throw new DuplicateProviderException(this.name() + " module has one " + loadedProvider.name() + "[" + loadedProvider.getClass().getName() 
                                                         + "] provider already, " + provider.name() + "[" + provider.getClass().getName() + "] is defined as 2nd provider."); }}}if (loadedProvider == null) {
            throw new ProviderNotFoundException(this.name() + " module no provider found.");
        }

        LOGGER.info("Prepare the {} provider in {} module.", loadedProvider.name(), this.name());
        try {
            copyProperties(loadedProvider.createConfigBeanIfAbsent(), configuration.getProviderConfiguration(loadedProvider.name()), this.name(), loadedProvider.name());
        } catch (IllegalAccessException e) {
            throw new ModuleConfigException(this.name() + " module config transport to config bean failure.", e);
        }
        loadedProvider.prepare();
Copy the code

For example, the “profile” section, select the h2, is loaded with class is org. Apache. Skywalking. Oap. Server storage. The plugin. JDBC. H2. H2StorageProvider

The prepare method is as follows: All Service interfaces declared by the StorageModule are registered

    @Override
    public void prepare(a) throws ServiceNotProvidedException, ModuleStartException {
        Properties settings = new Properties();
        settings.setProperty("dataSourceClassName", config.getDriver());
        settings.setProperty("dataSource.url", config.getUrl());
        settings.setProperty("dataSource.user", config.getUser());
        settings.setProperty("dataSource.password", config.getPassword());
        h2Client = new JDBCHikariCPClient(settings);

        this.registerServiceImplementation(IBatchDAO.class, new H2BatchDAO(h2Client));
        this.registerServiceImplementation(
            StorageDAO.class,
            new H2StorageDAO(
                getManager(), h2Client, config.getMaxSizeOfArrayColumn(), config.getNumOfSearchableValuesPerTag())
        );

        this.registerServiceImplementation(
            INetworkAddressAliasDAO.class, new H2NetworkAddressAliasDAO(h2Client));

        this.registerServiceImplementation(ITopologyQueryDAO.class, new H2TopologyQueryDAO(h2Client));
        this.registerServiceImplementation(IMetricsQueryDAO.class, new H2MetricsQueryDAO(h2Client));
        this.registerServiceImplementation(
            ITraceQueryDAO.class, new H2TraceQueryDAO(
                getManager(),
                h2Client,
                config.getMaxSizeOfArrayColumn(),
                config.getNumOfSearchableValuesPerTag()
            ));
        this.registerServiceImplementation(IBrowserLogQueryDAO.class, new H2BrowserLogQueryDAO(h2Client));
        this.registerServiceImplementation(
            IMetadataQueryDAO.class, new H2MetadataQueryDAO(h2Client, config.getMetadataQueryMaxSize()));
        this.registerServiceImplementation(IAggregationQueryDAO.class, new H2AggregationQueryDAO(h2Client));
        this.registerServiceImplementation(IAlarmQueryDAO.class, new H2AlarmQueryDAO(h2Client));
        this.registerServiceImplementation(
            IHistoryDeleteDAO.class, new H2HistoryDeleteDAO(h2Client));
        this.registerServiceImplementation(ITopNRecordsQueryDAO.class, new H2TopNRecordsQueryDAO(h2Client));
        this.registerServiceImplementation(
            ILogQueryDAO.class,
            new H2LogQueryDAO(
                h2Client,
                getManager(),
                config.getMaxSizeOfArrayColumn(),
                config.getNumOfSearchableValuesPerTag()
            )
        );

        this.registerServiceImplementation(IProfileTaskQueryDAO.class, new H2ProfileTaskQueryDAO(h2Client));
        this.registerServiceImplementation(IProfileTaskLogQueryDAO.class, new H2ProfileTaskLogQueryDAO(h2Client));
        this.registerServiceImplementation(
            IProfileThreadSnapshotQueryDAO.class, new H2ProfileThreadSnapshotQueryDAO(h2Client));
        this.registerServiceImplementation(UITemplateManagementDAO.class, new H2UITemplateManagementDAO(h2Client));
    }
Copy the code

Startup phase

In the org. Apache. Skywalking) oap) server. If the module. The ModuleManager# init by calling Org. Apache. Skywalking) oap) server. If the module. The BootstrapFlow# start into the startup phase

    void start( ModuleManager moduleManager) throws ModuleNotFoundException, ServiceNotProvidedException, ModuleStartException {
        for (ModuleProvider provider : startupSequence) {
            LOGGER.info("start the provider {} in {} module.", provider.name(), provider.getModuleName()); provider.requiredCheck(provider.getModule().services()); provider.start(); }}Copy the code

For example, the “profile” section, select the h2, is loaded with class is org. Apache. Skywalking. Oap. Server storage. The plugin. JDBC. H2. H2StorageProvider

Its start method is as follows, as you can see: start H2Client and listen on ModelCreator

    @Override
    public void start(a) throws ServiceNotProvidedException, ModuleStartException {
        final ConfigService configService = getManager().find(CoreModule.NAME)
                                                        .provider()
                                                        .getService(ConfigService.class);
        final int numOfSearchableTracesTags = configService.getSearchableTracesTags().split(Const.COMMA).length;
        if (numOfSearchableTracesTags * config.getNumOfSearchableValuesPerTag() > config.getMaxSizeOfArrayColumn()) {
            throw new ModuleStartException("Size of searchableTracesTags[" + numOfSearchableTracesTags
                                               + "] * numOfSearchableValuesPerTag[" + config.getNumOfSearchableValuesPerTag()
                                               + "] > maxSizeOfArrayColumn[" + config.getMaxSizeOfArrayColumn()
                                               + "]. Potential out of bound in the runtime.");
        }
        final int numOfSearchableLogsTags = configService.getSearchableLogsTags().split(Const.COMMA).length;
        if (numOfSearchableLogsTags * config.getNumOfSearchableValuesPerTag() > config.getMaxSizeOfArrayColumn()) {
            throw new ModuleStartException("Size of searchableLogsTags[" + numOfSearchableLogsTags
                                               + "] * numOfSearchableValuesPerTag[" + config.getNumOfSearchableValuesPerTag()
                                               + "] > maxSizeOfArrayColumn[" + config.getMaxSizeOfArrayColumn()
                                               + "]. Potential out of bound in the runtime.");
        }

        MetricsCreator metricCreator = getManager().find(TelemetryModule.NAME)
                                                   .provider()
                                                   .getService(MetricsCreator.class);
        HealthCheckMetrics healthChecker = metricCreator.createHealthCheckerGauge(
            "storage_h2", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE);
        h2Client.registerChecker(healthChecker);
        try {
            h2Client.connect();

            H2TableInstaller installer = new H2TableInstaller(
                h2Client, getManager(), config.getMaxSizeOfArrayColumn(), config.getNumOfSearchableValuesPerTag());
            getManager().find(CoreModule.NAME).provider().getService(ModelCreator.class).addModelListener(installer);
        } catch (StorageException e) {
            throw newModuleStartException(e.getMessage(), e); }}Copy the code

Notification phase after completion

In the org. Apache. Skywalking) oap) server. If the module. The ModuleManager# init by calling Org. Apache. Skywalking) oap) server. If the module. The BootstrapFlow# notifyAfterCompleted enters the stage after the completion of the notice

    void notifyAfterCompleted(a) throws ServiceNotProvidedException, ModuleStartException {
        for(ModuleProvider provider : startupSequence) { provider.notifyAfterCompleted(); }}Copy the code

For example, the “profile” section, select the h2, is loaded with class is org. Apache. Skywalking. Oap. Server storage. The plugin. JDBC. H2. H2StorageProvider

Its notifyAfterCompleted method looks like this, and you can see: Nothing to do

    @Override
    public void notifyAfterCompleted(a) throws ServiceNotProvidedException, ModuleStartException {}Copy the code

H2StorageProviderComplete source code

package org.apache.skywalking.oap.server.storage.plugin.jdbc.h2;

// etc...

/** * H2 Storage provider is for demonstration and preview only. I will find that haven't implemented several interfaces, * because not necessary, and don't consider about performance very much. * 
 * If someone wants to implement SQL-style database as storage, please just refer the logic. */
@Slf4j
public class H2StorageProvider extends ModuleProvider {

    private H2StorageConfig config;
    private JDBCHikariCPClient h2Client;

    public H2StorageProvider(a) {
        config = new H2StorageConfig();
    }

    @Override
    public String name(a) {
        return "h2";
    }

    @Override
    public Class<? extends ModuleDefine> module() {
        return StorageModule.class;
    }

    @Override
    public ModuleConfig createConfigBeanIfAbsent(a) {
        return config;
    }

    @Override
    public void prepare(a) throws ServiceNotProvidedException, ModuleStartException {
        Properties settings = new Properties();
        settings.setProperty("dataSourceClassName", config.getDriver());
        settings.setProperty("dataSource.url", config.getUrl());
        settings.setProperty("dataSource.user", config.getUser());
        settings.setProperty("dataSource.password", config.getPassword());
        h2Client = new JDBCHikariCPClient(settings);

        this.registerServiceImplementation(IBatchDAO.class, new H2BatchDAO(h2Client));
        this.registerServiceImplementation(
            StorageDAO.class,
            new H2StorageDAO(
                getManager(), h2Client, config.getMaxSizeOfArrayColumn(), config.getNumOfSearchableValuesPerTag())
        );

        this.registerServiceImplementation(
            INetworkAddressAliasDAO.class, new H2NetworkAddressAliasDAO(h2Client));

        this.registerServiceImplementation(ITopologyQueryDAO.class, new H2TopologyQueryDAO(h2Client));
        this.registerServiceImplementation(IMetricsQueryDAO.class, new H2MetricsQueryDAO(h2Client));
        this.registerServiceImplementation(
            ITraceQueryDAO.class, new H2TraceQueryDAO(
                getManager(),
                h2Client,
                config.getMaxSizeOfArrayColumn(),
                config.getNumOfSearchableValuesPerTag()
            ));
        this.registerServiceImplementation(IBrowserLogQueryDAO.class, new H2BrowserLogQueryDAO(h2Client));
        this.registerServiceImplementation(
            IMetadataQueryDAO.class, new H2MetadataQueryDAO(h2Client, config.getMetadataQueryMaxSize()));
        this.registerServiceImplementation(IAggregationQueryDAO.class, new H2AggregationQueryDAO(h2Client));
        this.registerServiceImplementation(IAlarmQueryDAO.class, new H2AlarmQueryDAO(h2Client));
        this.registerServiceImplementation(
            IHistoryDeleteDAO.class, new H2HistoryDeleteDAO(h2Client));
        this.registerServiceImplementation(ITopNRecordsQueryDAO.class, new H2TopNRecordsQueryDAO(h2Client));
        this.registerServiceImplementation(
            ILogQueryDAO.class,
            new H2LogQueryDAO(
                h2Client,
                getManager(),
                config.getMaxSizeOfArrayColumn(),
                config.getNumOfSearchableValuesPerTag()
            )
        );

        this.registerServiceImplementation(IProfileTaskQueryDAO.class, new H2ProfileTaskQueryDAO(h2Client));
        this.registerServiceImplementation(IProfileTaskLogQueryDAO.class, new H2ProfileTaskLogQueryDAO(h2Client));
        this.registerServiceImplementation(
            IProfileThreadSnapshotQueryDAO.class, new H2ProfileThreadSnapshotQueryDAO(h2Client));
        this.registerServiceImplementation(UITemplateManagementDAO.class, new H2UITemplateManagementDAO(h2Client));
    }

    @Override
    public void start(a) throws ServiceNotProvidedException, ModuleStartException {
        final ConfigService configService = getManager().find(CoreModule.NAME)
                                                        .provider()
                                                        .getService(ConfigService.class);
        final int numOfSearchableTracesTags = configService.getSearchableTracesTags().split(Const.COMMA).length;
        if (numOfSearchableTracesTags * config.getNumOfSearchableValuesPerTag() > config.getMaxSizeOfArrayColumn()) {
            throw new ModuleStartException("Size of searchableTracesTags[" + numOfSearchableTracesTags
                                               + "] * numOfSearchableValuesPerTag[" + config.getNumOfSearchableValuesPerTag()
                                               + "] > maxSizeOfArrayColumn[" + config.getMaxSizeOfArrayColumn()
                                               + "]. Potential out of bound in the runtime.");
        }
        final int numOfSearchableLogsTags = configService.getSearchableLogsTags().split(Const.COMMA).length;
        if (numOfSearchableLogsTags * config.getNumOfSearchableValuesPerTag() > config.getMaxSizeOfArrayColumn()) {
            throw new ModuleStartException("Size of searchableLogsTags[" + numOfSearchableLogsTags
                                               + "] * numOfSearchableValuesPerTag[" + config.getNumOfSearchableValuesPerTag()
                                               + "] > maxSizeOfArrayColumn[" + config.getMaxSizeOfArrayColumn()
                                               + "]. Potential out of bound in the runtime.");
        }

        MetricsCreator metricCreator = getManager().find(TelemetryModule.NAME)
                                                   .provider()
                                                   .getService(MetricsCreator.class);
        HealthCheckMetrics healthChecker = metricCreator.createHealthCheckerGauge(
            "storage_h2", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE);
        h2Client.registerChecker(healthChecker);
        try {
            h2Client.connect();

            H2TableInstaller installer = new H2TableInstaller(
                h2Client, getManager(), config.getMaxSizeOfArrayColumn(), config.getNumOfSearchableValuesPerTag());
            getManager().find(CoreModule.NAME).provider().getService(ModelCreator.class).addModelListener(installer);
        } catch (StorageException e) {
            throw newModuleStartException(e.getMessage(), e); }}@Override
    public void notifyAfterCompleted(a) throws ServiceNotProvidedException, ModuleStartException {}@Override
    public String[] requiredModules() {
        return newString[] {CoreModule.NAME}; }}Copy the code

conclusion

The modular mechanism provided by Skywalking is a good design that can be borrowed from multiple N choose 1 scenarios at work.

Reference documentation

• Backend setup
• Configuration Vocabulary

Share and record what you learn and see