The author | xiao soil Alibaba senior engineer

A Map of Essential Knowledge for Distributed System Design in the Cloud Native Era

Abstract: This paper tries to introduce the distributed knowledge system outline based on MSA (Microservice Architecture) from the distributed basic theory, architecture design pattern, engineering application, deployment operation and maintenance, industry solutions, so as to have a three-dimensional understanding of the evolution from SOA to MSA. Take a closer look at the distributed nature of microservices conceptually and in the application of tools, and experience the process of building a complete microservice architecture.

Follow the “Alibaba Cloud Native” public account, reply”distribution“, you can download the distributed system and its knowledge system clear big picture!

With the development of mobile Internet and the popularity of intelligent terminal, computer systems from a single early collaboration work independently transition to more machine, the cluster according to the theory of distributed construct complex application service, on the basis of distributed is in a cloud native technology revolution, break the traditional mode of development thoroughly, liberated the productivity of a new generation.

Distributed system knowledge system map

Follow the “Alibaba Cloud Native” public account, reply”distribution“, you can download the distributed system and its knowledge system clear big picture!

The basic theory

Evolution of SOA to MSA

SOA Service Oriented Architecture

As business develops to a certain extent, services need to be decoupled, and then a single large system is logically divided into different subsystems that communicate through service interfaces. The service-oriented design pattern, which ultimately requires bus integration of services and, most of the time, a shared database, leads to a single point of failure at the bus level, which can further overwhelm the database, leading to more independent design solutions.

MSA microservices architecture

Micro service is truly independent service, from the service entrance to the data persistence layer, logic is isolated independently, without the service bus to access, but also increased the whole distributed system construction and management difficulty, need to layout and management of service, so the rise of the service, the service of ecological technology stacks of a complete set of also need seamless access, To support the governance concept of microservices.

Nodes and Networks

node

The traditional node is also a single physical machine, all services are kneaded into the service and database; With the development of virtualization, a single physical machine can be divided into multiple virtual machines to maximize resource utilization. The concept of node has become a single virtual machine service. As container technology has matured in recent years, services have become completely containerized, meaning that nodes are just lightweight container services. In general, a node is a collection of logical computing resources that can provide a unit of service.

network

The foundation of distributed architecture is the network, whether it is a LAN or a public network. Without the network, computers cannot work together, but the network also brings a series of problems. Network messages are transmitted successively, and message loss and delay are common occurrences. We define three network working modes:

• Synchronous network
  • Synchronous node execution
  • Limited message delay
  • Efficient global locking
• Semi-synchronous network
  • Lock range relaxation
• Asynchronous network
  • Node independent execution
  • There is no upper limit on message delay
  • There is no global lock
  • Some of the algorithms are not feasible

The characteristics of two common network transport layer protocols are introduced:

• TCP protocol
  • First, TCP transmits reliably, although other protocols can transmit faster
  • TCP solves duplicate and out-of-order problems
• UDP protocol.
  • Constant data stream
  • Packet loss is not fatal

Time and Order

time

In slow physical space-time, time flows alone, and for serial transactions, it is simply to follow the steps of time, first come, first come. And then we invented the clock to describe what happened in the past, and the clock kept the world in order. But in a distributed world, dealing with time can be a pain.

Distributed in the world, we should coordinate the first relationship between the different nodes, time and independent of different nodes themselves admit, so we created the network time protocol (NTP) try to solve the standard time between different nodes, but the NTP itself performance is not satisfactory, so we construct the logical clock, Finally, vector clock was improved:

• Some disadvantages of NTP cannot fully meet the coordination problem of concurrent tasks in distributed system
  • Time is not synchronized between nodes
  • Hardware clock drift
  • Threads may sleep
  • Operating System Hibernation
  • The hardware of dormancy

• The logical clock
  • Define events to come first
  • t’ = max(t, t_msg + 1)

• Vector clock
  • t_i’ = max(t_i, t_msg_i)
• Atomic clocks

The order

With a tool for measuring time, it is natural to solve the order problem. Since the whole theory of distribution is based on how to negotiate the consistency of different nodes, and order is the basic concept of consistency theory, we need to spend the time above to introduce scales and tools for measuring time.

Consistency theory

To talk about consistency theory, we must look at a comparison of the effects of consistency on system construction:

The graph compares the balance of transactions, performance, errors, and delays for different consistency algorithms.

Strong consistent ACID

In single-machine environments, we have harsh requirements for traditional relational databases. ACID is the principle of ensuring transactions due to network latency and message loss. The four principles are familiar to us without even needing to explain them:

• Atomicity: All operations in a transaction are either completed or not completed, and do not end up somewhere in the middle;
• Consistency: Consistency that the integrity of the database is not compromised before and after a transaction begins
• Isolation: Isolation: the database allows multiple concurrent transactions to read, write, and modify its data at the same time. Isolation prevents data inconsistency due to cross-execution when multiple transactions are concurrently executed.
• Durabilit: After the transaction is completed, changes to the data are permanent and are not lost even if the system fails.

Distributed Consistency CAP

In the distributed environment, we cannot guarantee the normal connection of the network and the transmission of information, so we developed the three important theories of CAP/FLP/DLS:

• CAP: Distributed computing systems cannot ensure Consistency, availability, and Partition tolerance at the same time;
• FLP: In an asynchronous environment, if there is no upper limit on network latency between nodes, as long as a malicious node exists, no algorithm can reach a consensus in a limited time;
• DLS:
  • A protocol running on a partially synchronous network model (i.e., network latency is bounded but we don’t know where) can tolerate a third of arbitrary (in other words, Byzantine) errors;
  • Deterministic protocols (with no network latency upper limit) in an asynchronous model are not fault-tolerant (although this paper does not mention that randomised algorithms can tolerate 1/3 errors);
  • The protocol in the synchronization model, where the network latency is guaranteed to be less than the known d time, can be surprisingly 100% fault-tolerant, although there is a limit on how much error can occur at half the nodes.

Weakly consistent BASE

In most cases, we do not necessarily require strong consistency. Some businesses can tolerate a certain degree of delayed consistency. Therefore, in order to take into account efficiency, the final consistency theory BASE was developed. BASE is Basically Available, Soft State, Eventual Consistency:

• Basically Available: Basically Available means that the distributed system is allowed to lose part of its availability in case of failure, that is, the core is guaranteed to be Available;
• Soft State: The Soft State allows the system to have an intermediate State without affecting the overall availability of the system. In distributed storage, a copy of data usually has at least three copies. The delay of synchronizing copies between different nodes is a reflection of the soft state.
• Eventual Consistency: Eventual Consistency means that all copies of data in a system can reach the same state after a certain amount of time. Weak consistency is the opposite of strong consistency, and final consistency is a special case of weak consistency.

Consistency algorithm

The core of distributed architecture lies in the realization and compromise of consistency, so how to design a set of algorithms to ensure the communication and data between different nodes to achieve infinite consistency is very important. It is very difficult to ensure that different nodes can achieve the consistency of the same copy in the uncertain network environment, and a lot of research has been done on this topic.

First we need to understand the “CALM” premise: CALM principle is full name of Consistency and Logical support.It mainly describes the relationship between monotonous logic and Consistency in a distributed system. It is described as follows:

• In distributed systems, the monotonous logic can guarantee the “final consistency”, which does not need to rely on the central node scheduling.
• Any distributed system, if all non-monotone logic has a central node scheduling, then the distributed system can achieve the ultimate “consistency”.

Then we look at the Data structure CRDT(Conflict-free Replicated Data Types) of distributed systems: After we understand some principles of distributed, we need to consider how to implement solutions. The premise of consistent algorithm is data structure, or all algorithms are based on data structure. Well-designed data structure and exquisite algorithm can effectively solve real problems. After continuous exploration, we know that the distributed system is widely used in the data structure CRDT. See Talk about CRDT,A comprehensive study of Convergent and Commutative Replicated Data Types

• State-based: That is, the CRDT data between each node can be merged directly, so that all nodes can be merged into the same state, and the order of data merging will not affect the final result.
• Operation-based: Each operation on data is notified to other nodes. As long as nodes are aware of all operations on the data (which can be received in any order), they can be merged into the same state.

After understanding the data structure, we need to focus on some important distributed system protocols ***HATs(Highly Available Transactions), ZAB(Zookeeper Atomic Broadcast) : Refer to high Availability Transactions, ZAB Protocol Analysis

Finally, I need to learn the mainstream consistency algorithm in the industry: to tell you the truth, I have not fully understood the specific algorithm. Consistency algorithm is the core essence of distributed system. The development of this part will also affect the innovation of architecture, and the application of different scenarios will also give rise to different algorithms.

• Paxos: The Elegant Paxos Algorithm
• Raft: The Raft Consistency Algorithm
• Gossip: Gossip Visualization

In this section, we have finished the core theoretical basis of distributed system and how to achieve data consistency between different nodes. Next, we will talk about the mainstream distributed system at present.

Scene classification

The file system

There is always an upper limit on the storage of a single computer. With the emergence of network, the scheme of multiple computers collaborating to store files has been put forward successively. The earliest distributed file systems were actually called network file systems, and the first file servers were developed in the 1970s. In 1976 diddo corporation designed The File Access Listener (FAL), and The modern distributed File System came out of The famous Google paper, The Google File System, which laid The foundation for The distributed File System. Modern mainstream distributed file systems Refer to Distributed File System Comparison. The following are some commonly used file systems:

• HDFS
• FastDFS
• Ceph
• mooseFS

The database

Databases, of course, are also file systems, and master data adds advanced features such as transactions, retrieval, and erasure, so complexity increases. Data consistency needs to be considered as well as sufficient performance. In order to take into account the transaction and performance characteristics of traditional relational database, the development of distribution is limited. Non-relational database gets rid of the strong transaction consistency constraint, and achieves the final consistency effect, so there is a leap of development, NoSql(Not Only Sql) has also produced multiple architecture database types. Including KV, column storage, document type, etc.

• Column storage: Hbase
• Document storage: Elasticsearch, MongoDB
• KV type: Redis
• Relationship: Spanner

To calculate

The distributed computing system is built on the basis of distributed storage, which gives full play to the features of data redundancy and disaster recovery and efficient data acquisition from multiple copies of the distributed system, and then performs parallel computing. It divides the tasks that originally need a long time of computing into multiple tasks for parallel processing, thus improving the computing efficiency. The distributed computing system is divided into offline computing, real-time computing and streaming computing.

• Offline: Hadoop
• Real-time: the Spark
• Streaming: Storm, Flink/Blink

The cache

Caching is a powerful tool for improving performance everywhere, from small CPU cache architectures to large distributed application storage. Distributed cache system provides random access mechanism of hotspot data, greatly improving access time, but the problem is how to ensure the consistency of data, the introduction of distributed lock to solve this problem, the mainstream distributed storage system is basically Redis.

• Persistence: Redis
• Nonpersistent: Memcache

The message

Distributed message queue system is a great tool to eliminate a series of complex steps brought by asynchronism. In the scenario of high concurrency of multiple threads, we often need to carefully design business code to ensure that there is no deadlock problem caused by resource competition in the case of concurrent threads. Message queues, on the other hand, store asynchronous tasks in a delayed consumption pattern and then digest them one by one.

• Kafka
• RabbitMQ
• RocketMQ
• ActiveMQ

monitoring

With the development of distributed system from single machine to cluster, the complexity is also greatly improved, so the monitoring of the whole system is also essential.

• Zookeeper

application

The core module of distributed system is to apply how to deal with business logic, and the application directly calls rely on specific protocols for communication, which are based on RPC protocol, and also based on general HTTP protocol.

• HSF
• Dubbo

The log

Error responses to distributed systems are common, and we need to design systems to consider fault tolerance as a universal phenomenon. So when there is a failure, quick recovery and troubleshooting is very important. Distributed log collection, storage and retrieval can provide us with a powerful tool to locate problems in the request link.

• Log collection: Flume
• Log storage: ElasticSearch/Solr, SLS
• Log location: Zipkin

books

Previously we mentioned the so-called distributed system, is forced by the performance of the single machine is limited, and the heap hardware can not increase endlessly, the heap hardware will eventually encounter performance growth curve bottleneck. So we use multiple computers to do the same job, but such a distributed system always needs a central node to monitor or schedule the system’s resources, even though the central node may be composed of multiple nodes. Blockchain is a real region-centered distributed system, in which there are only P2P network protocols to communicate with each other, and no real central nodes. They coordinate the generation of new blocks according to the computing power and rights and other mechanisms of blockchain nodes.

• The currency
• The etheric fang

Design patterns

Last we list the different scenarios of distributed system architecture and implementation of the role of function, in this section, we further generalize distributed system design is how to consider structure design, different design directly, difference and emphasis of different scenarios need to choose design patterns of cooperation, to reduce the cost of trial and error, The following issues need to be considered when designing distributed systems.

availability

Availability is the percentage of time a system is up and running, usually measured as a percentage of uptime. It can be affected by system errors, infrastructure issues, malicious attacks, and system load. Distributed systems typically provide service level agreements (SLAs) for users, so applications must be designed to maximize availability.

• Health check: The system implements full-link function check, and external tools access the system periodically through public endpoints
• Load balancing: Use queues as a peaking buffer between requests and services to smooth intermittent heavy loads
• Throttling: Limits the range of resources consumed by application levels, tenants, or entire services

Data management

Data management is a key element of distributed systems and affects most quality attributes. Data is often hosted in different locations and on multiple servers for performance, scalability, or availability reasons, which can present a number of challenges. For example, data consistency must be maintained, and it is often necessary to synchronize data across different locations.

• Cache: Load data from the data store layer into the cache as needed
• Command Query Responsibility Segregation (CQRS) : Separation of Command Query responsibilities
• Event traceability: Records only a complete series of events in the domain in append mode
• Index tables: Create indexes on fields that are frequently referenced by queries
• Materialized view: Generates one or more data prepopulated views
• Split: Split data into horizontal partitions or shards

Design and implementation

Good design includes factors such as consistency of component design and deployment, maintainability that simplifies administration and development, and reusability that allows components and subsystems to be used by other applications and other solutions. Decisions made during the design and implementation phases have a huge impact on distributed systems and quality of service and total cost of ownership.

• Proxy: reverse proxy
• Adapters: Implement the adapter layer between modern applications and legacy systems
• Front-end separation: Back-end services provide interfaces for front-end applications to invoke
• Computing resource consolidation: Consolidation of related tasks or operations into a single cell
• Configuration separation: Move configuration information from the application deployment package to the configuration center
• Gateway aggregation: Multiple individual requests are aggregated into a single request using a gateway
• Gateway uninstallation: Uninstalls shared or dedicated service functions to the gateway proxy
• Gateway routing: Requests are routed to multiple services using a single endpoint
• Leadership election: Coordinate the cloud of a distributed system by selecting one instance as the administrator responsible for managing the other instances
• Pipes and filters: Break down complex tasks into a series of individual components that can be reused
• Sidecar: Deploy an application’s monitoring components into a separate process or container to provide isolation and encapsulation
• Static content hosting: Deploy static content to a CDN to speed up access efficiency

The message

Distributed systems require a messaging middleware that connects components and services, ideally in a loosely coupled manner, to maximize scalability. Asynchronous messaging is widely used and provides many benefits, but it also presents challenges such as message ordering, idempotentiality, and so on

• Competing consumers: Multi-threaded concurrent consumption
• Priority queue: Message queues have different priorities. The message queues with higher priorities are consumed first

Management and monitoring

Distributed systems run in remote data centers and do not have full control over the infrastructure, which makes management and monitoring more difficult than stand-alone deployments. Applications must expose runtime information that administrators can use to manage and monitor systems, as well as to support changing business requirements and customizations, without stopping or redeploying applications.

Performance and scalability

Performance represents the responsiveness of a system to perform any operation within a given time interval, while scalability is the ability of a system to handle increased load without compromising performance or easily increasing available resources. Distributed systems typically experience varying loads and peaks of activity, especially in multi-tenant scenarios, that are nearly impossible to predict. Instead, the application should be able to scale within limits to meet peak demand, and scale when demand decreases. Scalability involves not only compute instances, but also other elements such as data stores, message queues, and so on.

The elastic

Resilience is the ability of a system to gracefully handle and recover from a failure. Distributed systems are often multi-tenant, use shared platform services, compete for resources and bandwidth, communicate over the Internet, and run on commercial hardware, meaning an increased likelihood of transient and more permanent failures. To maintain resilience, faults must be detected and recovered quickly and efficiently.

• Isolation: Isolate the elements of an application into a pool so that if one fails, the others continue to run
• Circuit breakers: Handle faults that may take different times to repair when connected to a remote service or resource
• Compensation transaction: Undo the work performed by a series of steps that together define the final consistent action
• Health check: The system implements full-link function check, and external tools access the system periodically through public endpoints
• Retry: Transparently retry previously failed operations to allow applications to handle expected temporary failures when trying to connect to a service or network resource

security

Security is the ability of a system to prevent malicious or accidental behavior outside of its intended use, and to prevent disclosure or loss of information. Distributed systems operate on the Internet outside trusted local boundaries, are usually open to the public, and can serve untrusted users. You must protect your application from malicious attacks, restrict access to only approved users, and protect sensitive data.

• Federated identity: Delegate authentication to an external identity provider
• Gatekeeper: Protects applications and services by using a dedicated host instance that acts as a proxy between clients and applications or services, validating and cleaning requests, and passing requests and data between them
• Valet key: Use a token or key that provides a client with restricted direct access to a specific resource or service

The engineering application

Above we introduced a distributed system is the core of the theory, facing some of the challenges and the compromise way of thinking to solve the problem, list the classification of the existing mainstream distributed system, and sums up the construction of distributed system methodology, then we will introduce from engineering point of view to resort to build a distributed system contains the contents and steps.

Resource scheduling

All of our software systems are built on the basis of hardware servers. From the direct deployment of software systems on physical machines at the beginning, to the application of virtual machines, and finally to the cloud container of resources, the use of hardware resources has also begun the intensive management. This section compares the responsibilities of traditional o&M roles. In a DevOPS environment, development, operation and maintenance are integrated, and resources are used flexibly and efficiently.

Elastic scaling

In the past, if the software system needs to increase machine resources as the number of users increases, the traditional way is to apply for machines for operation and maintenance, and then deploy the software service access cluster. The whole process relies on the human experience of operation and maintenance personnel, which is inefficient and error-prone. Microservice distribution does not need human flesh to add physical machines. With the support of containerization technology, we only need to apply for cloud resources and then execute container scripts.

• Application capacity expansion: When users surge, services need to be expanded automatically, including automatic capacity expansion and automatic capacity reduction after the peak
• Machine offline: For obsolete applications, the application is offline, and the cloud platform reclaims the container hosting resources
• Machine replacement: For faulty machines, container host resources can be replaced, services start automatically, seamless switchover

Network management

With computing resources, the other most important thing is network resources. In the current cloud context, we hardly have direct access to physical bandwidth resources, but directly manage bandwidth resources by the cloud platform. What we need is the maximum application and effective management of network resources.

• Domain name application: Application application is a set of domain name mapping rules
• Domain name change: Unified platform management of domain name change
• Load management: set access policies for multi-server applications
• Secure communication: Basic access authentication to intercept illegal requests
• Unified access: Provides a unified access permission application platform and unified login management

Fault snapshot

In the case of system failure, our first priority is to restore the system. Meanwhile, it is also very important to retain the crime scene. The resource scheduling platform needs to have a unified mechanism to preserve the fault scene.

• Onsite retention: Saves resources such as memory distribution and thread count, such as JavaDump hook access
• Debugging access: Bytecode technology is adopted, and service code intrusion is not required. It can be used for on-site log debugging in production environment

Traffic scheduling

After we build a distributed system, the first to be tested mark is the gateway, and then we need to focus on system of traffic situation, that is how the management of the traffic, we pursue is within the system can accommodate traffic limit, leave the resource with the highest quality traffic to use, the illegal malicious traffic block at the door, This saves costs while ensuring that the system does not crash.

Load balancing

Load balancing is our general design for how services digest traffic. It is usually divided into hard load balancing of the physical layer and soft load balancing of the software layer. Load balancing solutions are mature in the industry. They are usually optimized for specific services in different environments. The following load balancing solutions are commonly used

• switches
• F5
• LVS/ALI-LVS
• Nginx/Tengine
• VIPServer/ConfigServer

The gateway design

The gateway is the first place for load balancing, because the gateway is the first place for centralized cluster traffic. If the gateway fails to handle the pressure, the whole system will become unavailable.

• High performance: The first consideration in gateway design is high performance traffic forwarding. A single node of a gateway can normally handle millions of concurrent traffic
• Distributed: For traffic pressure sharing and disaster recovery, the gateway design also needs to be distributed
• Service filtering: The gateway is designed with simple rules to exclude most malicious traffic

Traffic management

• Request verification: How many illegal requests can be blocked and cleaned by request authentication
• Data caching: Most stateless requests have data hotspots, so using A CDN can consume a significant portion of the traffic

Flow control control

For the rest of the real traffic, we use different algorithms to split the requests.

• Flow distribution

  • counter
  • The queue
  • funnel
  • The token bucket
  • Dynamic flow control

• Traffic restrictions at the time of traffic surges, usually we need limited flow measures to prevent the system from an avalanche, you will need to estimate maximum flow of the system, and then set the upper limit, but flow increased to a certain threshold, the extra traffic will not enter the system, by sacrificing some flow to preserve system availability.

  • Current limiting strategy
  • QPS granularity
  • Thread number granularity
  • RT threshold
  • Flow limiting tool – Sentinel

The service dispatch

The so-called iron needs its own hard, traffic scheduling management, the rest of the service is its own robustness. The failure of distributed system services is very common, even we need to consider the failure itself as part of the distributed service.

The registry

Our network Management section describes gateways, where traffic is distributed, and regiregies, where services are based.

• Status type: first, the status of the application service, which can be detected by the registry
• Life cycle: The different states of the application service constitute the life cycle of the application

Version management

• Cluster version: A cluster does not need to have its own version. A cluster consisting of different services also needs to have a large version
• Version rollback: If the deployment is abnormal, the rollback management can be performed according to the large cluster version

Service choreography

Service choreography is defined as controlling the interaction of various parts of the resources through the sequence of messages. The resources involved in the interaction are all peer-to-peer, with no centralized control. In a microservice environment with many services, we need an overall coordinator to protocol dependencies and invocation relationships between services, and K8s is our best choice.

• K8s
• Spring Cloud
  • HSF
  • ZK+Dubbo

Service control

So far we have solved the problem of network robustness and efficiency. This section shows how to make our service more robust.

• In the section of Resource Discovery management, we introduced that after applying for container-hosted resources from the cloud platform, the application services can be started through automated scripts. After the service is started, the service needs to discover the registry and register its service information with the service gateway, which is gateway access. The registry monitors the different states of services, performs health checks, and labels services that are not available.

  • The gateway access
  • Health check

• Downgrade: When users surge, the first thing we do is fix the traffic side, which is to limit the traffic. When we found that the system response was slower after limiting the flow, which could lead to more problems, we also needed to do something about the service itself. Service downgrading means turning off functionality that is not currently core, or extending accuracy that is not critical, and doing some manual remediation later.

  • Reduced consistency constraints
  • Disable non-core services
  • Simplify the function

• Circuit breaker: After we have done the above operations, we still feel uneasy, so we need to further worry. The circuit breaker is a kind of self-protection against overload, just like we switch trip. For example, when our service is constantly querying the database, if business problems cause query problems, the database itself needs to be circuit breaker to ensure that it will not be dragged down by the application, and access friendly information to tell the service not to blindly call.

  • closed
  • ajar
  • Off state
  • Fuse breaker – Hystrix

• Idempotent: We know that the characteristic of an idempotent operation is that any number of executions have the same effect as a single execution. Therefore, we need to assign a global ID to the single operation, so that after multiple requests, we can determine that the request comes from the same client, so as to avoid dirty data.

  • Global Consistency ID
  • Snowflake

Data scheduling

The biggest challenge of data storage is the management of data redundancy. With more redundancy, the efficiency is low and consumes resources. With fewer copies, disaster recovery cannot be performed.

State transition

Separating the state to global storage and the request to stateless traffic, for example, we usually cache the login information to global Redis middleware, without the need to remove redundant user login data across multiple applications.

Depots table

Data scales horizontally.

Shard partition

Multiple copies are redundant.

Automatic operation and maintenance

We started with the trend of DevOPS when it came to resource application management, and really integrating development and operation requires different middleware.

Configuration center

The global configuration center is differentiated by environment and managed in a unified manner, reducing the confusion of multiple configurations.

• switch
• diamend

Deployment strategy

Distributed deployment of microservices is a common occurrence. To make our services better support business development, we need to consider robust deployment strategy first. The following deployment strategy is suitable for different businesses and different stages.

• Stop the deployment
• Scroll to deploy
• Blue green deployment
• Gray scale deployment
• A/B testing

Job scheduling

Task scheduling is an essential part of the system. The traditional way is to configure the Crond scheduled task on the Linux machine or to complete the scheduling business directly in the business code. Now it is a mature middleware to replace it.

• SchedulerX
• Spring Scheduled Tasks

Application management

A large part of the o&M work involves restarting, logging in and out of the application, and clearing logs.

• Restart the application
• Use offline
• Log cleaning

Fault-tolerant processing

Now that we know that distributed system failures are common, solutions to them are essential. Usually we have active and passive ways to deal with:

• Initiative is when mistakes occur, we try to try a few more times, maybe it will be successful, successful then can avoid the mistake
• The passive way is that the wrong thing has happened, in order to recover, we just do the time to deal with it, to minimize the negative impact

Retry design

The key to retry design is to design the number of retries and the number of retries. If you go beyond the number of retries, or a period of time, then retries are meaningless. The open source project Spring-Retry is a good way to implement our retry plan.

Transaction compensation

Transaction compensation is consistent with our idea of ultimate consistency. Compensation transactions do not necessarily return the data in the system to the state in which it was at the beginning of the original operation. Instead, it compensates for the work performed by the successfully completed steps before the operation fails. The order of steps in a compensated transaction is not necessarily the exact opposite of the order of steps in the original operation. For example, one datastore may be more sensitive to inconsistencies than another datastore, so steps to undo changes to this datastore in a compensation transaction should occur first. Placing a short-term timeout-based lock on each resource needed to complete the operation and acquiring those resources in advance helps increase the likelihood of success for the overall activity. Work should be performed only after all resources have been obtained. All operations must be completed before the lock expires.

The whole stack monitoring

Due to the distributed system is made up of many machine work together, and also no guarantee that the network is completely available, so we need to build a system for each link can be monitored, so that we can monitor from the bottom to all levels of business, unexpected can repair the fault in time, avoid more problems.

Base layer

The basic layer is the monitoring of container resources, including the load of various hardware metrics

• CPU, I/O, memory, thread, throughput

The middleware

Distributed systems are connected to a large number of middleware platforms, and the health of the middleware itself needs to be monitored.

The application layer

• Performance monitoring: At the application layer, real-time indicators (QPS, RT) and upstream and downstream dependency of each application are monitored
• Business monitoring: In addition to the monitoring degree of the application itself, business monitoring is also a link to ensure the normal system. By designing reasonable business rules, alarm Settings are made for abnormal situations

Monitor the link

• zipkin/eagleeye
• sls
• goc
• Alimonitor

Fault recovery

When a fault has occurred, the first thing to do is to rectify the fault immediately and ensure that the system services are available normally. In this case, a rollback operation is usually performed.

Application of the rollback

Before rolling back an application, save the fault site for troubleshooting.

Baseline back

After the application service is rolled back, the code baseline also needs to revert to the previous version.

Version rollback

The overall rollback requires service orchestration, and the cluster is rolled back based on the larger version number.

Performance tuning

Performance optimization is a big topic for distributed systems, and it covers so many areas that it could have been a series on its own, rather than being covered in this section. The process of service governance is also a process of performance optimization. Refer to the Highly Concurrent Programming Knowledge System

A distributed lock

Caching is a great way to solve performance problems. Ideally, it’s fastest to get results immediately on each request without any additional computation. From the three-level cache of the CPU to the distributed cache, the cache is everywhere. What the distributed cache needs to solve is the consistency of data. At this time, we introduce the concept of distributed lock.

High concurrency

Multithreaded programming improves system throughput, but also brings business complexity.

asynchronous

Event-driven asynchronous programming is a new programming mode, which eliminates the complex business processing problems of multithreading and improves the response efficiency of the system.

conclusion

In conclusion, if possible, try to use a single-node approach rather than a distributed system. Distributed systems are accompanied by some failed operations, and to handle catastrophic failures, we use backups; To improve reliability, we introduce redundancy.

A distributed system is essentially a bunch of machines working together, and all we have to do is figure out how to make the machines work as expected. Such a complex system needs to understand the access of each link and each middleware, which is a very large project. Fortunately, in the context of microservices, much of the basic work has already been done. The distributed architecture described above can be basically built by using a distributed three-piece suite (Docker+K8S+Srping Cloud) when the project is implemented.

The distribution diagram of core technologies of distributed architecture is as follows:

Source: dzone.com/articles/de…

The distributed technology stack uses middleware:

Source: dzone.com/articles/de…

“Alibaba Cloud Native wechat Official account (ID: Alicloudnative) focuses on micro Service, Serverless, container, Service Mesh and other technical fields, focuses on cloudnative popular technology trends, cloudnative large-scale practice, and becomes the technical official account of cloudnative developers who understand cloudnative best.”