0 x00 the
SOFARegistry is ant Financial’s open source, production-grade, time-sensitive, and highly available service registry. This is the first article in a series that will take you through the implementation mechanism of MetaServer.
This series is a general reference to the official blog, see “0xFF Reference “for details. You can use the references as a general outline, and my series of articles as an addendum.
0x01 Service Registry
1.1 Introduction to the Service Registry
Under the microservice architecture, there are often a lot of intercalls between services behind the server of an Internet application. For example, service A depends on service B on the link. In this case, service A needs to know the address of service B before invoking the service. In a distributed architecture, each service is often deployed in a cluster and the machines in the cluster are constantly changing, so the address of service B is not fixed. In order to ensure the reliability of the business, the service caller needs to be aware of the address change of the called service.
Since thousands of service callers are aware of such changes, this awareness sinks into a fixed architectural pattern in microservices: service registries.
In a service registry, there are two important roles: service provider and service consumer. The service caller is the consumer and the service caller is the provider. For the same machine, it often plays both roles, being called by other services as well as calling other services. Service providers publish the service information provided by themselves to the service registry, and service consumers perceive whether the information of dependent services changes by subscribes.
In the scenario of service invocation, the service registry plays the role of “intermediary”. The Publisher publishes the service to the service registry, and the Subscriber can obtain the service information by visiting the service registry, so as to realize the invocation.
When the Subscriber invokes the service for the first time, it will find the IP address list of the corresponding service through Registry, and take a server of the service provider from the IP list to invoke the service through load balancing algorithm. At the same time, Subscriber will cache the service list data of Publisher locally for future use. When Subscriber calls Publisher later, the cached address list is preferentially used and Registry is not required.
+----------+ +---------+
|Subscriber| <--------+ +--------+ |Publisher|
+----------+ | | +---------+
| |
+----------+ | v
|Subscriber| <--------+ +----------++ +---------+
+----------+ <---+ Registry | <-------+Publisher|
| +----------++ +---------+
+----------+ | ^
|Subscriber| <--------+ |
+----------+ |
| +---------+
+----------+Publisher|
+---------+
Copy the codeThe primary capabilities of The service Registry are service registration and service discovery.
- The most important part of the service registration process is to store the information published by the service.
- The process of service discovery is the process of notifying subscribers of all changes (including node changes and service information changes) in a timely and accurate manner.
1.2 SOFARegistry overall architecture
1.2.1 layered
As a service registry, SOFARegistry is divided into four layers:
- The Client layer
The Client layer is the application server cluster. The Client layer is the application layer, where each application system programmatically uses the service publishing and service subscription capabilities of the service registry by relying on registrie-related Client JAR packages.
- The Session layer
The Session layer is the server cluster. As the name implies, the Session layer is the Session layer, which communicates with the application server of the Client layer through a long connection and is responsible for receiving the service publication and service subscription requests from the Client.
At the server side of the service registry, due to the limited amount of connection data for each storage node corresponding to the client side, a special partition must be made for the convergence of unlimited client connections, and then pass through the corresponding requests to the storage layer.
This layer only stores the publishing and subscription relationship of each service in memory. For specific service information, it only passes through and forwards between Client layer and Data layer. The Session layer is a proxy layer with no data status and can be expanded as the application scale of the Client layer grows.
Because SOFARegistry requires high timeliness in service discovery, it actively pushes changes to clients. Therefore, the main implementation of push is concentrated in the Session layer, and the internal push and pull combination is mainly to push changes to all Session nodes through the Data storage layer. Each Session node compares the data version information obtained by the first subscription with the stored subscription relationship, and finally decides to push the data to the service consumer clients.
- Data layer
Data server clusters. The Data layer stores the service registration Data of the application by means of fragment storage. Data is Hash sharded consistently based on the dataInfoId (unique identifier of each service data) and backed up with multiple copies to ensure high data availability. The Data layer can be smoothly expanded and shrunk as the Data scale grows without affecting services.
- Meta layer
Metadata server cluster. SOFARegistry acts as a service registry within the SOFARegistry architecture. SOFARegistry is a service registry that serves a wide range of application services. Meta clusters are Session and Data clusters that serve SOFARegistry. The Meta layer senses changes in Session and Data nodes and notifies other nodes in the cluster.
1.3 Why stratification
The reason for data layering within SOFARegistry is that system capacity is limited.
In the application scenario of SOFARegistry, there are two main types of data: Session data and service information data. The similarity between the two types of data is that the amount of data is constantly expanding, but the reason for expansion is different:
- Session is the connection corresponding to Client, and its data volume increases with the expansion of business machine scale.
- The growth of service information data is driven by Publisher releases.
So SOFARegistry uses a layered design to isolate the two types of data so that their expansion does not affect each other.
That’s why SOFARegistry has designed a three-tier model that uses the SessionServer to handle connections to clients and converges the number of connections per Client to one, so that as the number of clients grows, You only need to expand the SessionServer cluster. There are two main capabilities that SessionServer has to satisfy: fetching service information data from DataServer; And save the session with the Client.
0x02 MetaServer
2.1 introduction
MetaServer performs the role of cluster metadata management in SOFARegistry and maintains a list of cluster members. It can be thought of as a registry for the SOFARegistry registry.
As the metadata center of SOFARegistry, MetaServer’s core functions can be summarized as cluster member list management. Such as:
- Registration and storage of node lists
- Notification of changes to the node list
- Node health monitoring
When SessionServer and DataServer need to know the list of clusters and need to scale, MetaServer will provide the data.
Its internal architecture is shown in the figure below:
Based on Bolt, MetaServer provides external services in the form of TCP private protocol, including DataServer, SessionServer, etc., to process node registration, renewal, list query and other requests.
It also provides control interfaces based on THE Http protocol, for example, it can control whether the session node is enabled with change notification and health check interfaces.
The member list data is stored in Repository, which is wrapped by a consistency protocol layer and implemented as a state machine for SOFAJRaft. All operations on Repository are synchronized to other nodes, manipulating the storage layer via Rgistry.
MetaServer uses Raft protocol to ensure high availability and data consistency. It also maintains heartbeat with registered nodes and evicts nodes that have not been renewed due to heartbeat timeout to ensure data validity.
In terms of availability, the cluster can provide services as long as more than half of the nodes fail. Raft protocol cannot select master and log replication when more than half of the nodes fail. Therefore, the consistency and validity of the registered member data cannot be guaranteed. The whole cluster is unavailable without affecting the normal function of Data and Session nodes, but the node list changes cannot be sensed.
2.2 the problem
Just show the code. So let’s think about what MetaServer should do from a macro and micro perspective, and how to do it.
Think about:
- How MetaServer is implemented.
- How MetaServer achieves high availability.
- How MetaServer implements or applies the internal communication protocol.
- How MetaServer maintains the DataNode list and the SessionNode list.
- How MetaServer handles node list change push.
- How MetaServer handles node list queries.
- How MetaServer handles how MetaServer ensures data consistency.
- How to set up each cluster and achieve high availability.
Let’s analyze them one by one.
0x03 Code structure
We are on the sofa – registry – 5.4.2 / server/server/meta/SRC/main/Java/com/alipay/sofa/registry/server/meta directory and file structure.
We can get a rough idea of the functionality by following the catalog
- Metaapplication. Java: MetaServer program body, entry.
- Bootstrap: Start and configure MetaServer.
- Executor: responsible for various administrative tasks regularly, he started setting is in MetaServerBootstrap initRaft.
- Listener: SOFARegistry uses a handler-task & strategy-listener approach to deal with various scenarios and tasks in service registry. This processing model makes code and architecture as clean as possible.
- Node: Abstraction of service nodes, including DataNode, SessionNode, and MetaNode.
- Registry: Registry is used to manipulate the storage layer, and all operations on Repository are synchronized to other nodes.
- Remoting: Provides various external handlers.
- Repository: The cluster node list is stored in Repository and provides Bolt requests such as node registration, renewal, and list query via Raft strong consistency protocol to ensure that the data obtained by the cluster is consistent. Repository is wrapped by the consistency protocol layer and implemented as a state machine for SOFAJRaft.
- Resource: Interface of the HTTP Server to respond to control messages.
- Store: encapsulates storage node operations.
- Task: Encapsulates asynchronous execution logic, and uses TaskDispatcher and TaskExecutors to execute various defined asynchronous tasks.
The specific code structure is as follows:
. ├ ─ ─ MetaApplication. Java ├ ─ ─ the bootstrap ├ ─ ─ executor ├ ─ ─ the listener ├ ─ ─ node │ └ ─ ─ impl ├ ─ ─ registry ├ ─ ─ remoting │ ├ ─ ─ Connection │ ├─ hand ├─ Repository │ ├─ service ├─ resource ├─ store ├─ task ├─16 directories, 75 files
Copy the code0x04 Starts running
Start can refer to www.sofastack.tech/projects/so…
SOFARegistry supports two deployment modes, integrated and standalone.
4.1 Integration Deployment
4.1.1 Linux/Unix/Mac
Start command: sh bin/startup.sh
4.1.2 Windows
Double-click the startup.bat file in the bin directory.
4.1.3 Startup Information
We can see the startup information through the following log.
MetaApplication
[2020- 09 -12 20:23:05.463][INFO][main][MetaServerBootstrap] - Open meta server port 9612 success!
[2020- 09 -12 20:23: 08.198][INFO][main][MetaServerBootstrap] - Open http server port 9615 success!
[2020- 09 -12 20:23:10.298][INFO][main][MetaServerBootstrap] - Raft server port 9614start success! group RegistryGroup [2020- 09 -12 20:23:10.322][INFO][main][MetaServerInitializerConfiguration] - Started MetaServer
Copy the codeDataApplication
[2020- 09 -12 20:23:25.004][INFO][main][DataServerBootstrap] - Open http server port 9622 success!
[2020- 09 -12 20:23:26,084][INFO][main][DataServerBootstrap] - start server success
[2020- 09 -12 20:23:26,094][INFO][main][DataApplication] - Started DataApplication in 10.217 seconds (JVM running for 11.316)
Copy the codeSessionApplication
[2020- 09 -12 20:23:50.243][INFO][main][SessionServerBootstrap] - Open http server port 9603 success!
[2020- 09 -12 20:23:50.464][INFO][main][SessionServerInitializer] - Started SessionServer
[2020- 09 -12 20:23:50.526][INFO][main][SessionApplication] - Started SessionApplication in 12.516 seconds (JVM running for 13.783)
Copy the codeThe default ports of each Server are as follows:
meta.server.sessionServerPort=9610
meta.server.dataServerPort=9611
meta.server.metaServerPort=9612
meta.server.raftServerPort=9614
meta.server.httpServerPort=9615
Copy the codeAccess the health monitoring apis provided by the three roles, or view the log logs/ registrie-startup. log:
Health check interface for meta roles
$ curl http://localhost:9615/health/check
{"success":true."message":"... raftStatus:Leader"}
Health check interface for data role:
$ curl http://localhost:9622/health/check
{"success":true."message":"... status:WORKING"}
Health check interface for session role
$ curl http://localhost:9603/health/check
{"success":true."message":"..."}
Copy the code4.2 Independent Deployment
Here we can see how various clusters are built, and how to communicate within the cluster, distributed coordination.
Generally speaking, each cluster should rely on software such as ZooKeeper for its internal coordination, such as unified naming service, state synchronization service, cluster management, and distributed application configuration items. But we didn’t actually find that kind of use.
The configuration file shows that each machine has to set all metaServer hosts. This shows that Data Server and Session Server are strongly dependent on Meta Server.
In fact, MetaServer uses Raft protocol to ensure high availability and data consistency, but also maintains heartbeat with registered nodes and evicts nodes that are not renewed due to heartbeat timeout to ensure data validity. The Meta layer is aware of changes to Session and Data nodes and notifies the rest of the cluster.
This involves the failover mechanism of each role:
- MetaServer cluster deployment, internal Raft protocol based election and replication, as long as no more than 1⁄2 node down, external services.
- DataServer is deployed in a cluster. Based on consistent Hash, different data slices have multiple copies, including one primary copy and multiple standby copies. If DataServer fails, MetaServer notifies all DataServer and sessionServers that data fragments can be failover to other replicas, and that the DataServer migrates the fragmented data within the cluster.
- If a SessionServer is deployed in a cluster, when one SessionServer is down, the Client will automatically failover to another SessionServer and get the latest SessionServer list. The down SessionServer will not be connected later.
2 the deployment of meta
Each machine at deployment time need to modify the conf/application. The configuration of the properties:
# configure the IP or hostname of 3 meta machines below (the entered hostname will be internally resolved to the IP address)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
Copy the code4.2.2 the deployment of the data
Each machine at deployment time need to modify the conf/application. The configuration of the properties:
# configure the IP or hostname of 3 meta machines below (the entered hostname will be internally resolved to the IP address)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
data.server.numberOfReplicas=1000
Copy the codeHoldings deployment session
Each machine at deployment time need to modify the conf/application. The configuration of the properties:
# configure the IP or hostname of 3 meta machines below (the entered hostname will be internally resolved to the IP address)
nodes.metaNode=DefaultDataCenter:<meta_hostname1>,<meta_hostname2>,<meta_hostname3>
nodes.localDataCenter=DefaultDataCenter
nodes.localRegion=DefaultZone
Copy the code0x05 General logic
When MetaServer starts, it starts three Bolt Servers and registers the Processor Handler to process the request:
- DataServer: processes datanode-related requests.
- SessionServer: handles requests related to SessionNode.
- MetaServer: Handles metanode-related requests;
Then start HttpServer, which is used to process Admin requests, provide push switch, cluster data query and other Http interfaces.
Finally, start the Raft service, where each node acts as both a RaftClient and a Raft server for change and data synchronization between clusters.
5.1 Program Body
MetaServer is a SpringBootApplication that mainly functions as EnableMetaServer.
@EnableMetaServer
@SpringBootApplication
public class MetaApplication {
public static void main(String[] args) { SpringApplication.run(MetaApplication.class, args); }}Copy the codeSee the following figure for details
+-------------------+
| @EnableMetaServer |
| |
| MetaApplication |
+-------------------+
Copy the codeThe EnableMetaServer annotation introduces the MetaServerConfiguration
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(MetaServerConfiguration.class)
public @interface EnableMetaServer {
}
Copy the codeMetaServerConfiguration is various configurations, and introducing MetaServerInitializerConfiguration is responsible for starting.
@Configuration
@Import(MetaServerInitializerConfiguration.class)
@EnableConfigurationProperties
public class MetaServerConfiguration {}Copy the codeSo we know:
- MetaServerConfiguration: Responsible for configuration
- MetaServerInitializerConfiguration: responsible for startup
So the program summary structure evolves into the following figure:
(Init) +------------------------------------+ | MetaServerInitializerConfiguration | + -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + ^ + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | |@EnableMetaServer| | | | | | MetaApplication | | +-------------+-----+ | (Configuration) | +---------+---------------+ +--------------> | MetaServerConfiguration | +-------------------------+Copy the codeLet’s start importing the configuration.
5.2 configuration
The beans of MetaServer module are uniformly configured in JavaConfig, whose class is MetaServerConfiguration.
5.2.1 Configuring categories
MetaServerConfiguration includes the following types of configurations:
- Bootstrap, which is responsible for MetaServer startup configuration, is the core startup class.
- ServerConfig, responsible for MetaServer configuration items such as MetaServerConfig, NodeConfig, PropertySplitter.
- ServerServiceConfiguration, responsible for service related configuration, such as sessionNodeService storeServiceFactory, sessionStoreService.
- ServerRepositoryConfiguration, be responsible for the configuration Repository, such as dataRepositoryService sessionRepositoryService, etc.
- ServerRemotingConfiguration, responsible for network configuration, such as BoltExchange JerseyExchange, then key here.
- ResourceConfiguration, responsible for Resource configuration, such as jerseyResourceConfig, persistentDataResource.
- ServerTaskConfiguration, responsible for various task related configuration, such as dataNodeSingleTaskProcessor.
- ExecutorConfiguation, ExecutorManager configuration.
- MetaDBConfiguration: configures DB.
The specific abbreviated version code is as follows:
@Configuration
@Import(MetaServerInitializerConfiguration.class)
@EnableConfigurationProperties
public class MetaServerConfiguration {
@Bean
@ConditionalOnMissingBean
public MetaServerBootstrap metaServerBootstrap(a) {}@Configuration
protected static class MetaServerConfigBeanConfiguration {}@Configuration
public static class MetaServerServiceConfiguration {}@Configuration
public static class MetaServerRepositoryConfiguration {}@Configuration
public static class MetaServerRemotingConfiguration {}@Configuration
public static class ResourceConfiguration {}@Configuration
public static class MetaServerTaskConfiguration {}@Configuration
public static class ExecutorConfiguation {}@Configuration
public static class MetaDBConfiguration {}}Copy the codeOur graph evolved as follows:
(Init)
+------------------------------------+
| MetaServerInitializerConfiguration |
+-------------+----------------------+ +---------------------+
^ +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer | | | +---------------------------------+
| | | +--> |MetaServerConfigBeanConfiguration|
| MetaApplication | | | +---------------------------------+
+--------------+----+ | | +------------------------------+
| | +--> |MetaServerServiceConfiguration|
| | | +------------------------------+
| | | +---------------------------------+
| | +--> |MetaServerRepositoryConfiguration|
| | | +---------------------------------+
| | (Configuration) | +-------------------------------+
| +---------+---------------+ +--> |MetaServerRemotingConfiguration|
+--------------> | MetaServerConfiguration | +-----+ +-------------------------------+
+-------------------------+ | +----------------------+
+--> |ResourceConfiguration |
| +----------------------+
| +---------------------------+
+--> |MetaServerTaskConfiguration|
| +---------------------------+
| +---------------------+
+--> |ExecutorConfiguation |
| +---------------------+
| +--------------------+
+--> |MetaDBConfiguration |
+--------------------+
Copy the codeThe picture below is for mobile reading
5.2.2 Configuring handlers
Special mention is made here of handler configuration, as this is one of the subjects of subsequent analysis, for the three Bolt Server handlers.
@Configuration
public static class MetaServerRemotingConfiguration {
@Bean
public Exchange boltExchange(a) {
return new BoltExchange();
}
@Bean
public Exchange jerseyExchange(a) {
return new JerseyExchange();
}
@Bean(name = "sessionServerHandlers")
public Collection<AbstractServerHandler> sessionServerHandlers(a) {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(sessionConnectionHandler());
list.add(sessionNodeHandler());
list.add(renewNodesRequestHandler());
list.add(getNodesRequestHandler());
list.add(fetchProvideDataRequestHandler());
return list;
}
@Bean(name = "dataServerHandlers")
public Collection<AbstractServerHandler> dataServerHandlers(a) {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(dataConnectionHandler());
list.add(getNodesRequestHandler());
list.add(dataNodeHandler());
list.add(renewNodesRequestHandler());
list.add(fetchProvideDataRequestHandler());
return list;
}
@Bean(name = "metaServerHandlers")
public Collection<AbstractServerHandler> metaServerHandlers(a) {
Collection<AbstractServerHandler> list = new ArrayList<>();
list.add(metaConnectionHandler());
list.add(getNodesRequestHandler());
returnlist; }}Copy the codeSo our overall architectural evolution is detailed in the figure below
(Init)
+------------------------------------+
| MetaServerInitializerConfiguration |
+--------------+---------------------+
^
|
|
|
|
| +---------------------+
| +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer| | | +---------------------------------+ | | | +--> |MetaServerConfigBeanConfiguration| | MetaApplication | | | +---------------------------------+ +--------------+----+ | | +------------------------------+ +-----------------------+ | | +--> |MetaServerServiceConfiguration| +---> | sessionServerHandlers | | | | +------------------------------+ | +-----------------------+ | | | +---------------------------------+ | +--------------------+ | | +--> |MetaServerRepositoryConfiguration+-------> | dataServerHandlers | | | | +---------------------------------+ | +--------------------+ | | (Configuration) | +-------------------------------+ | +--------------------+ | +---------+---------------+ +--> |MetaServerRemotingConfiguration| +---> | metaServerHandlers | +-----------> | MetaServerConfiguration | +-----+ +-------------------------------+ +--------------------+ +-------------------------+ | +----------------------+ +--> |ResourceConfiguration | | +----------------------+ | +---------------------------+ +--> |MetaServerTaskConfiguration| | +---------------------------+ | +---------------------+ +--> |ExecutorConfiguation | | +---------------------+ | +--------------------+ +--> |MetaDBConfiguration | +--------------------+Copy the codeRead the following on your phone:
The handler configuration is further refined as shown below
(Init) +--> sessionConnectionHandler | | +--> sessionNodeHandler +-----------------------+ | +---> | sessionServerHandlers +--+ +---------------------+ | +-----------------------+ +--> renewNodesRequestHandler +--> | MetaServerBootstrap | | | | +---------------------+ | | | +---------------------------------+ | +--> getNodesRequestHandler +--> |MetaServerConfigBeanConfiguration| | | | +---------------------------------+ | | | +------------------------------+ | +--> fetchProvideDataRequestHandler +--> |MetaServerServiceConfiguration| | | +------------------------------+ | | +---------------------------------+ | +--> dataConnectionHandler +--> |MetaServerRepositoryConfiguration+---+ | | +---------------------------------+ | | | +-------------------------------+ | +--------------------+ +--> getNodesRequestHandler +-------------------------+ +--> |MetaServerRemotingConfiguration| +---> | dataServerHandlers +-----+ | MetaServerConfiguration | +-----+ +-------------------------------+ | +--------------------+ | +-------------------------+ | +----------------------+ | +--> dataNodeHandler +--> |ResourceConfiguration | | | | +----------------------+ | | | +---------------------------+ | +--> renewNodesRequestHandler +--> |MetaServerTaskConfiguration| | | | +---------------------------+ | | | +---------------------+ | +--> fetchProvideDataRequestHandler +--> |ExecutorConfiguation | | | +---------------------+ | | +--------------------+ | +--> |MetaDBConfiguration | | +---> metaConnectionHandler +--------------------+ | +--------------------+ | +----> | metaServerHandlers +-----+ +--------------------+ +---> getNodesRequestHandlerCopy the codePicture on your phone
This corresponds to the legend in reference:
When MetaServer starts, it starts three Bolt Servers and registers the Processor Handler to process the request:
- DataServer: processes datanode-related requests.
- SessionServer: handles requests related to SessionNode.
- MetaServer: Handles metanode-related requests;
0 x06 start
The system starts by controlling MetaServerBootstrap.
The beans of MetaServer module are uniformly configured in JavaConfig, whose class is MetaServerConfiguration.
Start the entry class for MetaServerInitializerConfiguration, this class is not JavaConfig management configuration, but the inherited the SmartLifecycle interface, at startup by the Spring framework to invoke the start method.
public class MetaServerInitializerConfiguration implements SmartLifecycle {
@Autowired
private MetaServerBootstrap metaServerBootstrap;
@Override
public void start(a) {
metaServerBootstrap.start();
MetaServerInitializerConfiguration.this.running.set(true);
}
@Override
public void stop(a) {
this.running.set(false); metaServerBootstrap.destroy(); }}Copy the codeBecause metaServerBootstrap is generated through configuration, the init process points to the configuration part.
(Init)
+------------------------------------+ start,stop
| MetaServerInitializerConfiguration +----------------+
+--------------+---------------------+ |
^ |
| |
| |
| |
| v
| +---------------------+
| +--> | MetaServerBootstrap |
+-------------------+ | | +---------------------+
| @EnableMetaServer| | | +---------------------------------+ | | | +--> |MetaServerConfigBeanConfiguration| | MetaApplication | | | +---------------------------------+ +--------------+----+ | | +------------------------------+ +-----------------------+ | | +--> |MetaServerServiceConfiguration| +---> | sessionServerHandlers | | | | +------------------------------+ | +-----------------------+ | | | +---------------------------------+ | +--------------------+ | | +--> |MetaServerRepositoryConfiguration+-------> | dataServerHandlers | | | | +---------------------------------+ | +--------------------+ | | (Configuration) | +-------------------------------+ | +--------------------+ | +---------+---------------+ +--> |MetaServerRemotingConfiguration| +---> | metaServerHandlers | +-----------> | MetaServerConfiguration | +-----+ +-------------------------------+ +--------------------+ +-------------------------+ | +----------------------+ +--> |ResourceConfiguration | | +----------------------+ | +---------------------------+ +--> |MetaServerTaskConfiguration| | +---------------------------+ | +---------------------+ +--> |ExecutorConfiguation | | +---------------------+ | +--------------------+ +--> |MetaDBConfiguration | +--------------------+Copy the codeHere’s what you see on your phone
6.1 architecture
MetaServerBootstrap is a core startup class that contains three main components: external node communication component, Raft service communication component, and timer component.
-
External node communication component: In this class there are several Server communication objects used to communicate with other external nodes. The httpServer mainly provides a series of HTTP interfaces for dashboard management and data query. SessionServer mainly processes some session-related services. DataServer is responsible for data-related services; MetaServer is responsible for registration of Meta Server;
-
Raft service: Used for change and data synchronization between clusters. RaftExchanger serves this role.
-
Timer components: such as periodic detection node information, periodic detection data version information; See ExecutorManager, which is a place to start various administrative threads. He started setting is in MetaServerBootstrap initRaft.
6.2 the class definition
MetaServerBootstrap is defined as follows:
public class MetaServerBootstrap {
@Autowired
private MetaServerConfig metaServerConfig;
@Autowired
private Exchange boltExchange;
@Autowired
private Exchange jerseyExchange;
@Autowired
private ExecutorManager executorManager;
@Resource(name = "sessionServerHandlers")
private Collection<AbstractServerHandler> sessionServerHandlers;
@Resource(name = "dataServerHandlers")
private Collection<AbstractServerHandler> dataServerHandlers;
@Resource(name = "metaServerHandlers")
private Collection<AbstractServerHandler> metaServerHandlers;
@Autowired
private ResourceConfig jerseyResourceConfig;
@Autowired
private ApplicationContext applicationContext;
@Autowired
private RaftExchanger raftExchanger;
private Server sessionServer;
private Server dataServer;
private Server metaServer;
private Server httpServer;
}
Copy the codeSee the following figure
+-----------------+ +----> | metaServerConfig| | +-----------------+ | +--------------+ +----> | boltExchange | | +--------------+ | +--------------+ +----> |jerseyExchange| | +--------------+ | +---------------+ +----> |executorManager| | +---------------+ | +---------------------+ +----> |sessionServerHandlers| | +---------------------+ | +-----------------+ +----> |dataServerHandler| (Init) | +-----------------+ +------------------------------------+ start,stop +---------------------+ | +------------------+ | MetaServerInitializerConfiguration +-------------> | MetaServerBootstrap | +--------> |metaServerHandlers| +------------------------------------+ +---------------------+ | +------------------+ | +-------------+ +----> |raftExchanger| | +-------------+ | +-------------+ +----> |sessionServer| | +-------------+ | +-----------+ +----> |dataServer | | +-----------+ | +-----------+ +----> |metaServer | | +-----------+ | +----------+ +----> |httpServer| | +----------+ | +---------------------+ +----> |jerseyResourceConfig | | +---------------------+ | +-------------------+ +----> |applicationContext | +-------------------+Copy the codeSee the picture below for mobile phone
6.3 communication Exchange
Because I had it in the previous code
@Autowired
private Exchange boltExchange;
@Autowired
private Exchange jerseyExchange;
Copy the codeA special note on Exchange.
Exchange acts as an abstraction of the Client/Server connection and is responsible for the connection between nodes. When establishing a connection, you can set up a series of channelHandlers for different tasks (called ChannelHandlers). These Channelhandlers either serve as listeners to handle connection events, or as processors to handle specific events. For example, service information and data changes, Subscriber registration and other events.
Figure – Each layer plays its own role to realize node communication
Each node starts up with a series of ChannelHandlers using Exchange, such as:
private void openDataRegisterServer(a) {
try {
if (dataStart.compareAndSet(false.true)) {
dataServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getDataServerPort()), dataServerHandlers
.toArray(newChannelHandler[dataServerHandlers.size()])); }}}Copy the code6.4 Startup Entrance
As mentioned earlier, start the entry class for MetaServerInitializerConfiguration, this class is not JavaConfig management configuration, but the inherited the SmartLifecycle interface, The Spring framework calls its start method at startup time.
This method calls the MetaServerBootstrap # start method, which is used to start a series of initialization services.
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.SmartLifecycle;
public class MetaServerInitializerConfiguration implements SmartLifecycle {
@Autowired
privateMetaServerBootstrap metaServerBootstrap; '@Override
public void start(a) {
metaServerBootstrap.start();
MetaServerInitializerConfiguration.this.running.set(true); }}Copy the code6.4.1 Starting the Service Server
MetaServerBootstrap starts three Bolt servers and registers the Processor Handler to process the request:
- DataServer: processes datanode-related requests.
- SessionServer: handles requests related to SessionNode.
- MetaServer: Handles metanode-related requests;
Then start HttpServer, which is used to process Admin requests, provide push switch, cluster data query and other Http interfaces.
Finally, start the Raft service, where each node acts as both a RaftClient and a Raft server for change and data synchronization between clusters. To support high availability, MetaServer stores SOFARegistry metadata and uses Raft protocol for election and replication to ensure consistency of MetaServer clusters.
See the specific code:
public void start(a) {
openSessionRegisterServer();
openDataRegisterServer();
openMetaRegisterServer();
openHttpServer();
initRaft();
}
private void openHttpServer(a) {
if (httpStart.compareAndSet(false.true)) {
bindResourceConfig();
httpServer = jerseyExchange.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getHttpServerPort()), newResourceConfig[] { jerseyResourceConfig }); }}private void initRaft(a) {
raftExchanger.startRaftServer(executorManager);
raftExchanger.startRaftClient();
raftExchanger.startCliService();
}
Copy the code6.4.1.1 BoltServer
Raft and Bolt are special implementations of SOFA, so we won’t cover the underlying mechanism for the moment. We’ll start with the three Bolt Servers when we have a chance.
private void openSessionRegisterServer(a) {
if (sessionStart.compareAndSet(false.true)) {
sessionServer = boltExchange
.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getSessionServerPort()), sessionServerHandlers
.toArray(newChannelHandler[sessionServerHandlers.size()])); }}private void openDataRegisterServer(a) {
if (dataStart.compareAndSet(false.true)) {
dataServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getDataServerPort()), dataServerHandlers
.toArray(newChannelHandler[dataServerHandlers.size()])); }}private void openMetaRegisterServer(a) {
if (metaStart.compareAndSet(false.true)) {
metaServer = boltExchange.open(new URL(NetUtil.getLocalAddress().getHostAddress(),
metaServerConfig.getMetaServerPort()), metaServerHandlers
.toArray(newChannelHandler[metaServerHandlers.size()])); }}Copy the codeThe handlers for these servers are the ones we configured earlier
@Resource(name = "sessionServerHandlers")
private Collection<AbstractServerHandler> sessionServerHandlers;
@Resource(name = "dataServerHandlers")
private Collection<AbstractServerHandler> dataServerHandlers;
@Resource(name = "metaServerHandlers")
private Collection<AbstractServerHandler> metaServerHandlers;
Copy the codeSee the following figure for details:
+------------------------+ +-----------------+ +---> |sessionConnectionHandler| +----> | metaServerConfig| | +------------------------+ | +-----------------+ | +-------------------+ | +--------------+ +---> |sessionNodeHandler | +----> | boltExchange | +-->+ +-------------------+ | +--------------+ | | +------------------------+ | +--------------+ | +---> |renewNodesRequestHandler| +----> |jerseyExchange| | | +------------------------+ | +--------------+ | | +----------------------+ | +---------------+ | +---> |getNodesRequestHandler| +----> |executorManager| | | +----------------------+ | +---------------+ +---------------------+ | | +------------------------------+ +-------------------------------------> |sessionServerHandlers+---+ +---> |fetchProvideDataRequestHandler| | +-------------+ +---------------------+ +------------------------------+ +----> |sessionServer| +------------------^ | +-------------+ +---------------------+ | ----> |dataConnectionHandler| | +------------------+ | +---------------------+ +---------------------+ +-------------------------------------> |dataServerHandlers+----------+ +----------------------+ | MetaServerBootstrap | +---+ +-----------+ +------------------+ +---> |getNodesRequestHandler| +---------------------+ +----> |dataServer +----------------------^ | +----------------------+ | +-----------+ | +---------------+ | +---> |dataNodeHandler| | +------------------+ | +---------------+ +------------------------------------> |metaServerHandlers+------+ | +------------------------+ | +-----------+ +------------------+ | +---> |renewNodesRequestHandler| +----> |metaServer +----------------------^ | | +------------------------+ | +-----------+ | | +------------------------------+ | | +---> |fetchProvideDataRequestHandler| | +-------------+ | +------------------------------+ +----> |raftExchanger| | | +-------------+ | | | | +----------+ | +----> |httpServer| | +---------------------+ | +----------+ | +--> |metaConnectionHandler| | +---------------------+ +----> +---------------------+ +----> |jerseyResourceConfig | | +---------------------+ | +---------------------+ +--> |getNodesRequestHandle| | +-------------------+ +---------------------+ +----> |applicationContext | +-------------------+Copy the codeOn the phone
In the case of SessionServer, call boltexchange.open during build.
private void openSessionRegisterServer(a) {
if (sessionStart.compareAndSet(false.true)) {
sessionServer = boltExchange
.open(
new URL(NetUtil.getLocalAddress().getHostAddress(), metaServerConfig
.getSessionServerPort()), sessionServerHandlers
.toArray(newChannelHandler[sessionServerHandlers.size()])); }}Copy the codeIn the BoltExchange
@Override
public Server open(URL url, ChannelHandler... channelHandlers) {
BoltServer server = createBoltServer(url, channelHandlers);
setServer(server, url);
server.startServer();
return server;
}
Copy the codeIn the BoltServer
public void startServer(a) {
if (isStarted.compareAndSet(false.true)) {
boltServer = new RpcServer(url.getPort(), true); initHandler(); boltServer.start(); }}private void initHandler(a) {
if (initHandler.compareAndSet(false.true)) {
boltServer.addConnectionEventProcessor(ConnectionEventType.CONNECT,
new ConnectionEventAdapter(ConnectionEventType.CONNECT,
getConnectionEventHandler(), this));
boltServer.addConnectionEventProcessor(ConnectionEventType.CLOSE,
new ConnectionEventAdapter(ConnectionEventType.CLOSE, getConnectionEventHandler(),
this));
boltServer.addConnectionEventProcessor(ConnectionEventType.EXCEPTION,
new ConnectionEventAdapter(ConnectionEventType.EXCEPTION,
getConnectionEventHandler(), this)); registerUserProcessorHandler(); }}Copy the codeThe last call, which sets up both synchronous and asynchronous handlers.
private void registerUserProcessorHandler(a) {
if(channelHandlers ! =null) {
for (ChannelHandler channelHandler : channelHandlers) {
if (HandlerType.PROCESSER.equals(channelHandler.getType())) {
if (InvokeType.SYNC.equals(channelHandler.getInvokeType())) {
boltServer.registerUserProcessor(new SyncUserProcessorAdapter(
channelHandler));
} else {
boltServer.registerUserProcessor(new AsyncUserProcessorAdapter(
channelHandler));
}
}
}
}
}
Copy the code6.4.1.2 HttpServer
Take the example of using Jetty’s openHttpServer
Start HttpServer, used to process Admin requests, provide push switch, cluster data query and other Http interfaces.
public class JerseyJettyServer implements Server {
public static org.eclipse.jetty.server.Server createServer(final URI uri,
final ResourceConfig resourceConfig,
final boolean start) {
JettyHttpContainer handler = ContainerFactory.createContainer(JettyHttpContainer.class,
resourceConfig);
int defaultPort = Container.DEFAULT_HTTP_PORT;
final int port = (uri.getPort() == -1)? defaultPort : uri.getPort();final org.eclipse.jetty.server.Server server = new org.eclipse.jetty.server.Server(
new JettyConnectorThreadPool());
final ServerConnector http = new ServerConnector(server, new HttpConnectionCustomFactory());
http.setPort(port);
server.setConnectors(new Connector[] { http });
if(handler ! =null) {
server.setHandler(handler);
}
if (start) {
try {
// Start the server.server.start(); }}returnserver; }}Copy the code6.4.1.3 @ RaftService
The following storage uses Raft to ensure data consistency, which is explained in detail later.
@RaftService(uniqueId = "sessionServer")
public class SessionVersionRepositoryService
@RaftService(uniqueId = "metaServer")
public class MetaRepositoryService
@RaftService(uniqueId = "dataServer")
public class DataRepositoryService
@RaftService(uniqueId = "sessionServer")
public class SessionRepositoryService
@RaftService(uniqueId = "dataServer")
public class DataConfirmStatusService
@RaftService(uniqueId = "sessionServer")
public class SessionConfirmStatusService
Copy the code6.4.2 ExecutorManager
Is a place to start various administrative threads, all of which are regular administrative tasks.
public class ExecutorManager {
private ScheduledExecutorService scheduler;
private ThreadPoolExecutor heartbeatCheckExecutor;
private ThreadPoolExecutor checkDataChangeExecutor;
private ThreadPoolExecutor getOtherDataCenterChangeExecutor;
private ThreadPoolExecutor connectMetaServerExecutor;
private ThreadPoolExecutor checkNodeListChangePushExecutor;
private ThreadPoolExecutor raftClientRefreshExecutor;
private MetaServerConfig metaServerConfig;
@Autowired
private Registry metaServerRegistry;
@Autowired
private MetaClientExchanger metaClientExchanger;
@Autowired
private RaftExchanger raftExchanger;
public void startScheduler(a) {
init();
scheduler.schedule(new TimedSupervisorTask("HeartbeatCheck", scheduler, heartbeatCheckExecutor,
metaServerConfig.getSchedulerHeartbeatTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerHeartbeatExpBackOffBound(), () -> metaServerRegistry.evict()),
metaServerConfig.getSchedulerHeartbeatFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("GetOtherDataCenterChange", scheduler, getOtherDataCenterChangeExecutor,
metaServerConfig.getSchedulerGetDataChangeTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerGetDataChangeExpBackOffBound(), () -> {
metaServerRegistry.getOtherDataCenterNodeAndUpdate(NodeType.DATA);
metaServerRegistry.getOtherDataCenterNodeAndUpdate(NodeType.META);
}), metaServerConfig.getSchedulerGetDataChangeFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(new TimedSupervisorTask("ConnectMetaServer", scheduler, connectMetaServerExecutor,
metaServerConfig.getSchedulerConnectMetaServerTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerConnectMetaServerExpBackOffBound(),
() -> metaClientExchanger.connectServer()), metaServerConfig.getSchedulerConnectMetaServerFirstDelay(),
TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("CheckSessionNodeListChangePush", scheduler, checkNodeListChangePushExecutor,
metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(),
() -> metaServerRegistry.pushNodeListChange(NodeType.SESSION)),
metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(
new TimedSupervisorTask("CheckDataNodeListChangePush", scheduler, checkNodeListChangePushExecutor,
metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS,
metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(),
() -> metaServerRegistry.pushNodeListChange(NodeType.DATA)),
metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS);
scheduler.schedule(new TimedSupervisorTask("RaftClientRefresh", scheduler, raftClientRefreshExecutor, metaServerConfig.getSchedulerCheckNodeListChangePushTimeout(), TimeUnit.SECONDS, metaServerConfig.getSchedulerCheckNodeListChangePushExpBackOffBound(), () -> raftExchanger.refreshRaftClient()), metaServerConfig.getSchedulerCheckNodeListChangePushFirstDelay(), TimeUnit.SECONDS); }}Copy the code6.4.2.1 start
ExecutorManager startup Settings is in MetaServerBootstrap initRaft, start RaftServer respectively, RaftClient, CliService.
private void initRaft(a) {
raftExchanger.startRaftServer(executorManager);
raftExchanger.startRaftClient();
raftExchanger.startCliService();
}
Copy the codeWhen Raft selects the Leader, it calls the ExecutorManager # startScheduler.
- Each ThreadPoolExecutor is generated first;
- And each TimedSupervisorTask run itself, it can call different handler, such as connectServer, getSchedulerHeartbeatFirstDelay, etc.
6.4.2.2 TimedSupervisorTask
The TimedSupervisorTask implements the TimerTask.
public class TimedSupervisorTask extends TimerTask {
private final ScheduledExecutorService scheduler;
private final ThreadPoolExecutor executor;
private final Runnable task;
@Override
public void run(a) {
Future future = null;
try {
future = executor.submit(task);
// block until done or timeout
future.get(timeoutMillis, TimeUnit.MILLISECONDS);
delay.set(timeoutMillis);
} catch{... }finally {
if(future ! =null) {
future.cancel(true);
}
scheduler.schedule(this, delay.get(), TimeUnit.MILLISECONDS); }}}Copy the code6.4.2.3 Managing Tasks
It can be seen that the management tasks are roughly as follows:
- HeartbeatCheck: indicates the HeartbeatCheck.
- GetOtherDataCenterChange: View changes in other data centers.
- ConnectMetaServer: Interacts with other MetaServer;
- The change of CheckSessionNodeListChangePush: look at the Session node;
- CheckDataNodeListChangePush: view the data node;
- RaftClientRefresh: Look at Raft service messages;
The Timedcontainer is designed to be executed regularly, Such as CheckDataNodeListChangePush this thread will be performed periodically metaServerRegistry. PushNodeListChange (NodeType. DATA)) to see if there is a change. In this case, the Confirm message is used when DataNode is registered.
@Override
public void pushNodeListChange(a) {
NodeOperator<DataNode> fireNode;
if((fireNode = dataConfirmStatusService.peekConfirmNode()) ! =null) {
NodeChangeResult nodeChangeResult = getNodeChangeResult();
Map<String, Map<String, DataNode>> map = nodeChangeResult.getNodes();
Map<String, DataNode> addNodes = map.get(nodeConfig.getLocalDataCenter());
if(addNodes ! =null) {
Map<String, DataNode> previousNodes = dataConfirmStatusService.putExpectNodes(
fireNode.getNode(), addNodes);
if(! previousNodes.isEmpty()) { firePushDataListTask(fireNode, nodeChangeResult, previousNodes,true); } } firePushSessionListTask(nodeChangeResult, fireNode.getNodeOperate().toString()); }}Copy the codeAnother example is to periodically remove expired nodes:
public class MetaServerRegistry implements Registry<Node> {
@Override
public void evict(a) {
for (NodeType nodeType : NodeType.values()) {
StoreService storeService = ServiceFactory.getStoreService(nodeType);
if(storeService ! =null) {
Collection<Node> expiredNodes = storeService.getExpired();
if(expiredNodes ! =null && !expiredNodes.isEmpty()) {
storeService.removeNodes(expiredNodes);
}
}
}
}
}
Copy the code6.4.3 ServiceFactory
ServiceFactory requires a special description. It provides a set of services required by the system. The exception is that ServiceFactory is not started by MetaServerBootstrap, but by Spring. Because ServiceFactory inherits ApplicationContextAware, it is generated at startup.
In Web applications, the Spring container is typically configured declaratively: Developers simply configure a Listener in web.xml that initializes the Spring container. MVC framework can call beans directly from the Spring container without accessing the Spring container itself. In this case, the beans in the container are container-managed and do not need to actively access the container, only to receive dependency injection from the container.
However, in special cases where a Bean needs to implement a function that must be implemented with the help of the Spring container, the Bean must first obtain the Spring container and then implement the function with the help of the Spring container. To have a Bean get its Spring container, you can have the Bean implement the ApplicationContextAware interface.
As you can see from the following code, a series of services are started.
public class ServiceFactory implements ApplicationContextAware {
private static Map<NodeType, StoreService> storeServiceMap = new HashMap<>();
private static Map<NodeType, NodeConnectManager> connectManagerMap = new HashMap<>();
private static Map<NodeType, NodeService> nodeServiceMap = new HashMap<>();
}
storeServiceMap = {HashMap@5107} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionStoreService@5526}
{Node$NodeType@4815} "DATA" -> {DataStoreService@5527}
{Node$NodeType@5528} "META" -> {MetaStoreService@5529}
connectManagerMap = {HashMap@5532} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionConnectionHandler@5548}
{Node$NodeType@4815} "DATA" -> {DataConnectionHandler@5549}
{Node$NodeType@5528} "META" -> {MetaConnectionHandler@5550}
nodeServiceMap = {HashMap@5533} size = 3
{Node$NodeType@5525} "SESSION" -> {SessionNodeServiceImpl@5540}
{Node$NodeType@4815} "DATA" -> {DataNodeServiceImpl@5541}
{Node$NodeType@5528} "META" -> {MetaNodeServiceImpl@5542}
Copy the codeThis completes the architecture and startup of MetaServer, and we’ll cover basic features such as registration, storage, renewal, and so on.
0xEE Personal information
★★★★ Thoughts on life and technology ★★★★★
Wechat official account: Rosie’s Thoughts
If you want to get a timely news feed of personal articles, or want to see the technical information of personal recommendations, please pay attention.
0 XFF reference
Due to the word limit, at the end of the series together issued, forgive me.