The opening

This article is based on the SOFA Meetup Hefei station sharing summary, mainly aimed at the location and function introduction of the registry, through the analysis of the history of the ant registry, lead you to understand how the ant registry has evolved into the current scale and characteristics step by step.

For more in-depth technical details, please join the SOFA and SOFARegistry community to find out.

What is a registry

Service discovery & Service registration

Registries are simply designed to solve the problem of mutual discovery between services in distributed scenarios.

As shown in the figure below, when service A wants to call service B, it needs to know where the address of SERVICE B is. How to solve this problem?

In general, there are two points:

  1. Service discovery can have a centralized component, or store, that hosts the addresses of all services and provides a query and subscription capability. Consumers of services can interact with this centralized store to obtain a list of addresses of service providers.
  2. Service registration: The same centralized component as above, however, the service information at this time can take two measures
    1. Services connect to the registry and report their services and metadata (also the focus of today’s article)
    2. There is a centralized control plane that writes the mapping of user-defined services and IP to the registry, such as AWS CloudMap

Calling process

As shown in the figure above, SOFARegistry and Nacos are one of the most popular registries.

  1. Service A and service B report their service information to the registry through THE SDK or REST
  2. When service A needs to invoke service B, it makes A request to the registry to pull the list of service IP addresses and information associated with service B
  3. After obtaining the list of service B, service B can be accessed through a self-defined load balancing algorithm

The heartbeat

Heartbeat is the default way for the registry to pull out services when they are unavailable, as shown in the following figure

  1. A node of service B is disconnected or “hung”, causing a heartbeat timeout. Or downtime, broken chain directly caused heartbeat failure
  2. The registry pulls the problem node out of its own storage (depending on the implementation: either directly deleted or marked as unhealthy)
  3. Service A receives A notification from the registry to get the latest list of Service B

The DUBBO Registry

Using the DUBBO example, let’s look at how the registry is used and the process

First of all, the configuration of DUBBO in 2.7 and 3.0 is slightly different, but it is simple and easy to understand

DUBBO – 2.7 –

DUBBO – 3.0 –

On the RPC client, you only need to configure a registry address, which contains the basic three elements

  1. Protocol Protocol type For example, ZooKeeper
  2. host
  3. port

Based on this, the registration process for Dubbo is shown in the figure below

  1. The service producer initiates a register to the Registry through the DUBBO client.
  2. Consumers of the service subscribe via the DUBBO client
  3. The registry delivers a list of services to service consumers by notification

The nature of the registry

From the previous explanation, along with a specific example of the DUBBO component, we can summarize the nature of the registry

Storage + O&M

  1. On the one hand, registries need the ability to store information about services, such as lists of applications
  2. On the other hand, the registry, in practice, needs to provide the necessary means of operation and maintenance, such as shutting down traffic to a service

Chronicle of the Ant Registry

Prehistoric times

The prehistoric ant is a fairly ancient architecture, when all services were deployed on the same physical machine or JVM, and there was no cross-machine invocation between services, which is omitted here

Hard load Age

Later, in order to solve the difficulties in deployment and operation and maintenance caused by the coupling between applications, we split the services. After splitting the services, we encountered a problem, that is, how to deal with the invocation relationship between services. At this time, ants used two hard loads, F5 or LVS.

Through the simple 4-layer proxy, we can deploy the service behind the proxy, and the services can access each other through the proxy, achieving the purpose of cross-machine invocation

First generation registries – Hardload to soft load evolution

On the one hand, the hard load access solves the problem of mutual invocation between services, and the deployment architecture is easy to understand. On the other hand, after the rapid growth of the business, it has brought certain problems:

  1. Single point problem (if all calls go to F5, once F5 fails, many services will be unavailable)
  2. Capacity issues (the F5 carries too much traffic, which itself creates a performance bottleneck)

At this time, Ant introduced a product of Alibaba Group called ConfigServer, which is used as a registry. The architecture of this registry is very similar to that mentioned at the beginning. Services can be directly accessed through IP, which reduces the strong dependence on load balancing products and reduces the single point of risk.

The second generation registry — ScaleUp? ScaleOut? It’s a problem

However, the problem persists, and that is that the registry itself is a single point, so it continues to encounter the two problems mentioned above

  1. Single point of risk (the registry itself is a stand-alone application)
  2. Capacity bottlenecks (the number of connections and capacity to store data in a single registry is limited)

There are two ways to do this

  1. Scale-up (Taobao) : By increasing the configuration of the machine, to enhance the capacity and the ability to carry links; At the same time, availability is guaranteed through the active-standby architecture
  2. Scale-out (ant) : Through the sharding mechanism, data and links are evenly distributed on multiple nodes to achieve horizontal expansion. High availability is achieved through the backup after sharding

The ants and Taobao have taken two different paths, pushing the evolution of a separate ecosystem behind the ants

Ant evolution architecture is as follows, resulting in two different application nodes

  1. The session node, which is specially used for anti-link use, is stateless and can be rapidly expanded. The single node has very little resource consumption
  2. Data nodes are used to store data. The storage capacity of a single node is reduced by fragmentation to control resource usage

Birth of Meta Node, the fifth generation of registry

The above architecture has been very consistent with the mainstream distributed architecture, but in the process of operation and maintenance, a series of problems, such as

  1. All data is distributed, and service discovery between data needs to be decoupled from standard o&M by being given a configuration file at startup
  2. Before logging in or out of data nodes, modify the configuration file in a timely manner; otherwise, cluster restart may be affected
  3. Distributed storage consistency issues. The PAAS platform needs to be locked in each iteration to prevent inconsistency caused by node changes

For all these problems, we found that we could introduce a Meta node to solve the problem of data and session management. Data and Session can access the Meta through layer 4 load or layer 7 load.

Compared with industry solutions, there are similar models, such as HDFS Name Node and Kafka that rely on ZK, Oceanbase that rely on RootServer, or configuration center Apollo that rely on Euraka.

The emergence of Meta node alleviates the bottleneck of manual operation and maintenance registry, but still does not fundamentally solve the problem, so what is the problem? See the analysis below.

The sixth generation registry – the registry for operation and maintenance

As mentioned above, the emergence of Meta node takes over the problem of service discovery between Data and Session. However, there are still many problems that can not be solved without cluster cloud testing, such as

  1. The release of Data nodes is still a pain point under the premise of large Data volume
  2. A new node added to the Session node may have no traffic for a long time

For these problems, on the basis of Sofaregistry5.x, we quickly iterated version 6.0, mainly for operations and maintenance registry.

In the final analysis, the difficulty of releasing Data nodes is a problem of impact scope. How to control the release of a single Data node or suspend the impact on Data is the source of solving the problem. Here we adopt two measures

  1. Improve the data store algorithm (consistent-hash -> hash-slot)
  2. Application-level service discovery

Evolution of storage algorithms

We used the consistent hash algorithm before. As shown in the figure below, each node carries a part of the data. The hash value of the stored content is calculated through the storage, and then the hash value falls into the storage interval responsible for the data to store the data.

When a data node breaks down or restarts, the next data node receives data from the broken node and supports data access.

In this way, the granularity of data migration can only be measured by the data stored on a single data node. In the case of a large amount of data (8G for a single node), data reconstruction may be affected to some extent. In addition, in the case of continuous data breakdown, data loss or inconsistency may occur.

In the improved algorithm, we refer to the algorithm mechanism of Redis Cluster and use hash slot for data fragmentation

In this way, in the process of data publishing, data migration can be controlled in the unit of slot (multiple slots for a single data node, configurable).

At the same time, in order to solve the scenario of inconsistent data writing during migration or downtime, we introduce the compensation mechanism of data playback. After promotion is the master of the slot, data will actively complete a data comparison/verification with all sessions and incremental synchronization of new data

Application-level service discovery

Application level service discovery is designed to solve the problem of large data storage, which is omitted here due to space

Open source

SOFARegistry started the open source process early on in the project. Here’s how it compares to the current mainstream registries

In our opinion, the first thing the registry needs to solve is the problem of availability. Therefore, for the issue of distributed consistency, we choose the AP model, which is also consistent with the mainstream registries, such as Euraka and Nacos.

Secondly, in terms of performance, SOFARegistry based on Long connection has a shorter push delay than Nacos1.0 (Nacos1.0 based on Long Polling model, Nacos2.0 also uses Long connection model).

On the protocol side, SOFARegistry uses the Ant open source protocol stack: BOLT Protocol (similar to HTTP2.0) streaming protocol, which is more lightweight, while the full-duplex mode of the protocol itself: no blocking, which greatly improves resource utilization.

Feature Consul Zookeeper Etcd Eureka Nacos SOFARegistry
Service Health Check Regular healthcheck (HTTP/TCP/script/docker) Session + TTL for periodic heartbeat retention Regular refresh (HTTP) + TTL Periodic heartbeat +TTL; Support custom healthCheck Link heartbeat + disconnect regularly Regular connection heartbeat + disconnected sensitivity
Multi-data center support – – – support support
Kv Storage Service support support support – support support
consistency raft ZAB raft Final consistency Final Consistency (Registry) Raft (Configuration Center) Final consistency
cap cp cp cp ap ap+cp ap
Using interfaces (multilingual capability) Supports HTTP and DNS The client http/grpc The client/HTTP Client (multilingual) HTTP Client (Java)
Watch support Full/support long polling support Support long polling Not supported (client periodic fetch) support Support (server push)
security acl/https acl HTTPS support – https acl
Spring cloud integration support support support support support support

Compared to Nacos as we know it, we have a huge advantage in financial grade and distribution (on the storage scale), and are still playing catch-up in terms of ease of use and cloud native

Welcome to join us

One person can go fast, but a group of people can go further

SOFARegistry is an open source project and an important part of the SOFA open source community. We hope to use the power of the community to push SOFARegistry forward, rather than just ant engineers. We also launched two projects this year to support more developers to get involved:

  1. Trun-key Project (out of the box) : github.com/sofastack/s…
  2. Deep-dive Project: github.com/sofastack/s…

The project is still in the early stages, welcome to join us, you can help us solve an issue, or write a document, can better help the community, help yourself to grow.

Recommended Reading of the Week

  • RFC8998+BabaSSL– Let the state secret to the farther star sea

  • Still struggling with multi-cluster management? OCM come!

  • MOSN subproject Layotto: Opens a new chapter of service Grid + Application runtime

  • Open a new chapter of cloud native MOSN – merging the Envoy and GoLang ecology

For more articles, please scan the code to follow the “financial level distributed Architecture” public account