Distributed architecture Systems Review

1) Distributed system concept

A distributed system is one in which hardware or software components are distributed across different network computers, communicating and coordinating with each other only through messaging

In the system.

A distributed system, is a business split into multiple child business, distribution in different server nodes, constitute the system called cloth type systems, the same server nodes in a distributed system can be randomly distributed in the space deployment, the server may be put into different cabinet, may also be in a different room, even distribution in different cities.

The difference between distributed and cluster:

Cluster: multiple people doing the same thing together. Distributed: multiple people doing different things together.Copy the code

Characteristics of distributed systems:

(1) distribution (2) reciprocity (3) concurrency (4) Lack of a global clock (5) Failures always occurCopy the code

2) Development of distributed systems

Alibaba’s “de-IOE” campaign (referring to IBM minicomputers, Oracle databases, EMC’s high-end storage). Alibaba in 2009 “to IOE” strategy technology director revealed that as of May 17, 2013 alibaba’s last IBM minicomputer in Alipay offline.

Why go to IOE?

  1. The cost performance of upgrading single machine processing capacity is getting lower and lower
  2. There is a bottleneck in the single-node processing capability
  3. Stability and availability are two metrics that are hard to achieve

3) Evolution of distributed architecture

Phase 1: Single application architecture

Phase 2: The application server is separated from the database server

Phase 3: Application server cluster

Phase 4: Load balancing on application servers

Phase 5: Database read/write separation

Stage 6: Add a search engine to relieve the stress of reading libraries

Phase 7: Add a caching mechanism to relieve pressure on the database

Phase 8: Database horizontal/vertical split

Phase 9: Application split

Stage 10: Servitization

Problems faced by distributed systems

1) Abnormal communication

Network itself not reliability, so every network communication with the network, not with the risk of (optical fiber, routing, hardware, such as DNS, or system is not available), can lead to eventually distributed a network communication system cannot smoothly carries on, in addition, even if the distributed system network communication between each node can be normal operation, the delay will be greater than the single machine operation, There is a huge delay difference, which also affects the process of sending and receiving messages, so message loss and message delay become very common.

2) Network partition

Appeared no network connectivity between network, but each network’s internal network is normal, resulting in the network environment of the whole system been split into several isolated area, a distributed system can appear local small clusters, in extreme cases, these small cluster can achieve that need to be can not complete the entire distributed system functions, including data of the transaction, This presents a very big challenge for distributed consistency.

3) The node is faulty

Node failure is another common problem in distributed systems. It refers to the failure or “zombie” phenomenon of the server nodes that make up a distributed system. As a rule of thumb, every node can fail and it happens frequently.

4) three states

Each request and response in distributed system has a unique concept of “three states”, that is, success, failure and timeout.

In distributed system, because the network is not reliable, although in most cases, the network communication can receive the response of success or failure, but when the network is abnormal, there will be timeout phenomenon, usually in the following two cases:

    1. Due to network problems, the request is not successfully sent to the receiver, but is lost during the sending process.
    1. The successful recipient receives the request and processes it. However, message loss occurs when the response is sent back to the sender.

Distributed theory: consistency

What is distributed consistency

Distributed data consistency refers to the consistency of data stored in multiple copies.

Copy consistency

In distributed systems, there are often multiple copies of data. If a single database handles all data requests, then the four ACID Principles basically guarantee data consistency. Multiple copies guarantee multiple copies of the data. This creates synchronization problems because there is little guarantee that we can update all the machines at the same time including backing up all the data. Network latency. Even if I send data update requests to all machines at the same time, there is no guarantee that these requests will be responded to at the same time. There will be data inconsistency between some machines.

In general, we cannot find a distributed consistent solution that satisfies all the system attributes of distributed systems. Therefore, how to ensure data consistency without affecting system performance is the key consideration and tradeoff of every distributed system. Thus, consistency levels are born:

Consistent classification

Strong consistency

This level of consistency is the most intuitive, requiring the system to write and read what is good user experience, but implementation often has a large impact on system performance. But strong consistency is hard to achieve.

Weak consistency

This consistency level constrains that the system does not guarantee that the written value can be read immediately after the data is written, nor does it guarantee how long it will take for the data to reach the consistency state. However, the system tries to ensure that the data can reach the consistency state after a certain time level (such as second level).

Read and write consistency

Users read the consistency of their own writing results, ensuring that users can always see their updated content in the first time. For example, if we send a circle of friends, it is not important whether the content of the circle of friends is seen by friends for the first time, but it must be displayed in their own list. Solution: Solution 1: One solution is to go to the main library every time for certain content. Scheme 2: We set an update time window. During the period of update, we will read from the master library by default. After this window, we will select the slave library that has been updated recently for reading. We directly record the updated timestamp of the user and carry it with us when requesting it. Any slave library with a last updated time less than this timestamp will not respond.Copy the code

Monotonic read consistency

The data read this time cannot be older than the data read last time. Due to the time of master-slave node to update data inconsistencies, lead to users constantly refreshed, sometimes can brush, refresh again later, you can find data missing, refresh and likely to brush out again, as if meet the paranormal solution: is based on the user ID to calculate a hash value, again through the hash value is mapped to the machine. The same user will only be mapped to the same machine no matter how much it is refreshed. This ensures that you don't read content from other libraries and have a bad user experience.Copy the code

Causal consistency

If node A notifies node B after updating some data, node B's subsequent access to and modification of the data is based on the updated value of NODE A. Meanwhile, data access of node C, which has no causal relationship with node A, has no such restriction.Copy the code

Final consistency

The final consistency is the weakest of all distributed consistency models. It can be thought of as the "weakest" consistency without any optimization, which means THAT I don't take into account the effects of all intermediate states, but only guarantee that after a certain period of time, when there are no new updates, the data in all copies of the system will be correct. It guarantees the concurrency capability of the system to the greatest extent, and therefore, it is the most widely used consistency model in high concurrency scenariosCopy the code

Distributed theory: CAP theorem

The CAP theorem

In July 2000, The CAP conjecture was proposed by Eric Brewer, a professor at the University of California, Berkeley, and two years later, by MIT

Seth Gilbert and Nancy Lynch proved the possibility of conjecture theoretically. Since then, CAP theorem has officially become an accepted theorem in the field of distributed computing in academic circles. And deeply influenced the development of distributed computing.

CAP theory means that A distributed system cannot simultaneously satisfy C:Consistency, A: Availability and P: Partition tolerance, but only two of them at most.

options describe
C consistency Consistency in distributed systems refers to the consistency of data on all nodes, or on all copies
A usability Reads and writes always succeed. This means that the system is always available and the service is always normal
P partition fault tolerance The system can provide consistent and availability services even when some nodes or network partitions fail

C – Consistency Indicates that the data read operation after the value write operation can read the latest data status. If the data is distributed on multiple nodes, the data read from any node is the latest. To meet the consistency requirements, the following objectives must be achieved: 1. If the commodity and service are successfully written to the main database, the data to be queried from the database is also successfully. 2. If writing goods and services to the master database fails, the query to the slave database also fails. After writing data to the master database, synchronize data to the slave database. 2. After writing data to the master database, lock the slave database during data synchronization. The characteristics of distributed consistency are as follows: 1. The response of write operations is delayed due to database synchronization 2. To ensure data consistency, resources are temporarily locked and released after data synchronization is complete. 3. If the node fails to synchronize data, it will return an error message and must not return old data

A-availability means that any operation can receive response results without response timeout or error. Commodity information read and write meets availability requirements to achieve the following objectives: 1. After receiving a query request from the database, it immediately responds to the data query result 2. How to achieve availability by not allowing response timeouts or errors from the database? 1. After writing data to the primary database, synchronize data to the secondary data. 2. Even if the data has not been synchronized, query data, even old data, is returned from the database, but errors and timeouts cannot be returned.

P-partition tolerance Each node of a distributed system is deployed in different subnets, and it is inevitable that communication between nodes will fail due to network problems. In this case, services can still be provided to the outside world. This is p-partition fault tolerance. The read and write of product information must meet the following requirements: 1. The primary database fails to synchronize data from the secondary database. 2. The failure of one node does not affect the external services provided by the other node. How to achieve fault tolerance of a zone? 1. Replace synchronous operation with asynchronous operation as far as possible, for example, synchronize data from the master database to the slave database in asynchronous mode, so that the loose coupling between nodes can be effectively realized; 2. Add database nodes. If one secondary node fails, other secondary nodes provide services

CAP can only choose 2 from 3

So that’s all we have to say about CAP, and then we’re going to try to prove why CAP can’t all work.

Suppose you have a system that looks like this: 

A user sent a request to N1 to change the data, to update the database from V0 to V1. Because the network is disconnected, N2 database is still V0, what if N2 has a request to N2, but N2 has no way to directly give the latest result V1? At this time, there are no two ways, one is to go ahead and return the wrong V0 data to the user. The second is blocking wait, where the network communication is restored and the data in N2 is updated before being returned to the user. Obviously the former sacrifices consistency and the latter usability. This example is simple, but the point is important. In distributed system, we cannot satisfy the three characteristics of CAP at the same time, so we must abandon one of them. There are obviously three possible permutations without one of them.Copy the code

1. Discard A(availability) and retain CP(consistency and partition fault tolerance)

One system guarantees consistency and fault tolerance of partitions at the expense of availability. In other words, in extreme cases, the system is allowed to be inaccessible. At this time, the user experience is often sacrificed and the user waits until the system data is consistent before resuming the service. Like on 12306, just let you try againCopy the code

2. Discard C(consistency) and retain AP(availability and partition fault tolerance)

This is the design of most distributed systems, guaranteeing high availability and fault tolerance of partitions, but sacrificing consistency. For example, when you update your status, you wait a certain amount of time before everyone can receive itCopy the code

3. Discard P(partition fault tolerance) and retain CA(consistency and availability)

If P is to be discarded, then the distributed system is to be discarded, and CAP is out of the question. It can be said that P is a prerequisite for distributed systems, so this case does not exist.Copy the code

Distributed theory: BASE theory

What is BASE theory

BASE:

  • Basically Available
  • Soft state
  • Eventually consistent three

Short for the phrase, from the architects at ebay.

BASE is the result of tradeoff between consistency and availability in CAP. The core idea of BASE theory is

Even if strong consistency cannot be achieved, each application can adopt appropriate methods to achieve final consistency according to its own service characteristics.

1-Basically Available(Basically Available)

Basic availability is the ability of a distributed system to allow a partial loss of availability in the event of unforeseen failures

Note, however, that this is by no means equivalent to the system being unavailable. Here are two examples of “basic availability”

  • Loss in response time: Normally, an online search engine needs to return the search results to users within 0.5 seconds. However, due to faults (such as power failure or network disconnection in some equipment rooms of the system), the response time of the search results increases to 1 to 2 seconds.

  • Loss of functionality: Under normal circumstances, when shopping on an e-commerce site such as Taobao, consumers can complete almost every order without any problems. However, when some festivals promote shopping peak (such as Singles’ Day and Singles’ Day), due to the surge of consumers’ shopping behavior, in order to protect

For system stability (or to ensure consistency), some consumers may be directed to a degraded page, as shown below

2-Soft state

What is a soft state? As opposed to consistency, requiring that copies of data across multiple nodes be consistent is a “hard state.”

Soft state refers to:

Data in the system is allowed to exist in an intermediate state, which is considered not to affect the overall availability of the system, that is, the system is allowed to have a delay in the process of data synchronization between multiple copies of data on different nodes.

3-Eventually consistent

Final consistency emphasizes that all copies of data in the system can eventually reach a consistent state after a period of synchronization. Therefore, the essence of final consistency is that the system needs to ensure the consistency of the final data, but does not need to ensure the strong consistency of the system data in real time.

Distributed transaction

Database transaction Review

Basic transaction characteristics:

We know that transactions have four very important properties, commonly known as (ACID).

  • Atomicity

    The transaction is an indivisible whole and all operations must be done or not be done at all. If one of the operations in the transaction fails and is rolled back to the state before the transaction starts, all operations that have been performed before are invalid and should be rolled back to the state before the transaction starts.

  • Consistency

    It means that before and after transaction execution, data must be consistent from one state to another state. For example, A transfers money to B (the total amount of A and B is A consistent state). It is impossible for A to deduct money while B does not receive it.

  • Isolation

    Multiple concurrent transactions are isolated from each other and cannot interfere with each other. The isolation of transactions may not be well understood, but a concurrent transaction is a case where two transactions operate on the same data; For the isolation of the same data in concurrent transactions, it is required that there should be no dirty read or phantom read, that is, transaction A cannot read the data that has not been submitted by transaction B, or transaction B is not allowed to update the data first when transaction A reads and updates the data. And in order to solve this problem, the common means is to lock, for the database is through the relevant database lock mechanism to ensure.

  • Changes to the database are permanent after the transaction has completed.

What are distributed transactions

In fact, distributed transaction is essentially the same as the concept of database transaction. Since it is a transaction, it needs to meet the basic characteristics of transaction (ACID). However, distributed transaction has a very different form of expression compared with local transaction

Distributed theory: Consistency protocol 2PC

What are the 2 PC

2PC (two-Phase Commit) divides the entire transaction process into Two phases: Prepare Phase and Commit Phase. 2 refers to Two phases, P refers to the preparation Phase, and C refers to the Commit Phase.

In the computer, some relational databases such as Oracle and MySQL support the two-phase commit protocol.

Two-stage process:

    1. The preparation phase:
    • The transaction manager sends a Prepare message to each participant
    • Each database participant performs transactions locally and writes a local Undo/Redo log
    • The transaction is not committed at this time.

    Undo logs record data before modification and are used for database rollback. Redo logs record data after modification and are used for writing data files after transaction submission

    1. Commit Phase
    • If the transaction manager receives an execution failure or timeout message from each participant, it sends a Rollback message to each participant.
    • Otherwise, send a Commit message; According to the instructions of the transaction manager, participants perform commit or rollback operations and release lock resources used during transaction processing.

    Note: Lock resources must be released at the last stage.

Protocol Description: As the name implies, two-phase commit is to divide the transaction commit process into two phases for processing. The process is as follows:

2PC Execution process

Successfully executed the transaction commit process 

Phase 1:1. Transaction ask the coordinator to send transaction content to all participants, ask if the transaction commit operation can be performed, and wait for the response of each participant. 2. Execute transaction (write local Undo/Redo log) 3. Summarize the response of each participant to the transaction query to the coordinator: Each participant votes whether to allow the transaction to proceedCopy the code

Tip: What is Ack

An ACK acknowledgment character, a transmission control character sent from a receiving station to a transmitting station in data communication. Indicates that the received data is confirmed.Copy the code
Phase 2:1. Send a commit request: The coordinator issues a COMMIT request to all participants. 2. Transaction commit: After receiving the COMMIT request, the participant officially performs the transaction commit and releases the transaction resources occupied during the entire transaction execution. 3. Feedback transaction commit results: Participants send Ack messages to the coordinator after completing the transaction commit. 4. Completion of transaction: The coordinator completes the transaction after receiving Ack information from all participantsCopy the code

The interrupt transaction steps are as follows:If any participant gives a No response to the coordinator, or if the coordinator is unable to receive feedback from all participants after the wait times out, the transaction is interrupted

Phase 1:

1. Transaction query The coordinator sends the transaction content to all participants, asks if the transaction commit operation can be performed, and waits for the response from each participant. 2. Execute transaction (write local Undo/Redo log) 3. Summarize the response of each participant to the transaction query to the coordinator: Each participant votes whether to allow the transaction to proceed.Copy the code

Stage 2:

1. Send a Rollback request: The coordinator issues a Rollback request to all participants. 2. Transaction Rollback: After receiving the Rollback request, participants use the Undo information recorded in phase 1 to perform the transaction Rollback and release the resources occupied during the entire transaction execution after the Rollback. 3. Feedback transaction rollback results: The participant sends an Ack message to the coordinator after completing the transaction rollback. 4. Interrupt transaction: After receiving Ack information from all participants, the coordinator completes the interrupt of the transaction. From the logic above, a two-phase commit does two things: vote and execute.Copy the code

2PC Advantages and disadvantages

advantages

The principle is simple and the implementation is convenient

disadvantages

Synchronization blocking, single point of problem, inconsistent data, too conservative

Synchronous blocking:

The most obvious and biggest problem of the two-phase commit protocol is synchronous blocking. During the execution of the two-phase commit, all the logic involved in the transaction operation is blocked. In other words, each participant cannot perform other operations while waiting for the response of other participants. This synchronous blocking greatly limits the performance of distributed systems.

Single point problem:

The coordinator is important throughout the two-phase commit process. If the coordinator fails during the commit phase, the process will not work and, more importantly, the other participants will be locked in transaction resources and unable to complete the transaction.

Inconsistent data:

Suppose that after the coordinator sends commit requests to all participants, a local network exception occurs or the coordinator crashes itself before all commit requests are sent, resulting in only some participants receiving commit requests. This leads to serious data inconsistencies.

Too conservative:

If in the submission query stage of two-stage submission, the coordinator cannot obtain the response information of all participants due to the failure of the participants, then the coordinator can only rely on its own timeout mechanism to determine whether the transaction needs to be interrupted. Obviously, this strategy is too conservative. In other words, there is no well-designed fault tolerance mechanism for the two-stage committee-delivery protocol, and the failure of any node will lead to the failure of the whole transaction.

Distributed theory: Consistency protocol 3PC

What is three-phase commit

3PC, full name of “Three Phase Commit”, is an improved version of 2PC, which divides the process of “commit transaction request” of 2PC into two parts, forming a transaction processing protocol consisting of CanCommit, PreCommit and doCommit phases.

Phase one: CanCommit

  • 1- Transaction inquiry

    The coordinator sends a canCommit request containing the transaction content to all participants, asks if the transaction commit operation can be performed, and waits for the response from each participant.

  • 2- Each participant feedback the response to the transaction query to the coordinator

    After receiving a canCommit request from the coordinator containing the transaction content, the participant normally responds with Yes and enters the preparatory state if it thinks the transaction can be successfully executed, and No if it does not

Phase 2: PreCommit

After receiving responses from all participants, the coordinator has two options to perform the action based on the results: perform the transaction pre-commit, or interrupt the transaction

If all participants in the feedback are Yes, the transaction pre-commit will be performed.

**(1) The transaction pre-commit is divided into 3 steps **

1. Send a preCommit request: The coordinator issues a preCommit request to all the participant nodes and enters the Prepared phase. 2. Transaction preCommit: After receiving a preCommit request, a participant performs a transaction and records Undo and Redo information to the transaction log. 3. Each participant reports back to the coordinator the results of the transaction execution: if the participant successfully executed the transaction, it reports back an AckCopy the code

If any participant gives a No response, or if the coordinator is unable to receive feedback from all participants after the wait times out, the transaction is interrupted

(2) Interrupt transaction is also divided into two steps:

1. Send interrupt requests: The coordinator issues abort requests to all participants. 2. Interrupt a transaction: Participants interrupt a transaction either after receiving an ABORT request from the coordinator or while waiting for a timeout during the coordinator's requestCopy the code

Phase 3: Do Commit

This phase does either the actual transaction commit or the completion transaction rollback, so two situations occur:

Perform transaction commit

1- Send commit request: To enter this phase, assuming that the coordinator is in a working state and has received Ack responses from all participants, he will transition from pre-commit state to commit state and send doCommit requests to all participants. 2- Transaction commit: After receiving the doCommit request, the participant officially performs the transaction commit operation and releases the transaction resources occupied during the entire transaction execution. 3- Feedback transaction commit result: The participant sends an Ack response to the coordinator after completing the transaction commit. 4- Completion of transaction: The coordinator completes the transaction after receiving Ack messages from all participants.Copy the code

Interrupt the transaction

1- Send interrupt request: The coordinator sends abort requests to all participant nodes. 2- Transaction rollback: After the ABORT request is received, the participant performs the transaction rollback based on the recorded Undo information and releases the resources occupied during the entire transaction execution once the rollback is completed. 3- Feedback transaction rollback results: The participant sends an Ack message to the coordinator after completing the transaction rollback. 4- Interrupt transaction: The coordinator interrupts the transaction after receiving Ack messages from all participants.Copy the code

Note: Once phase three is entered, two failures may occur:

  • The coordinator has a problem
  • Network failure between coordinator and participant

In either case, participants eventually fail to receive doCommit requests or abort requests, in which case they wait for a timeout before continuing to commit the transaction

2 PC compared to 3 PCS

  1. First for the coordinator and participants set the timeout mechanism (in 2 PC, only the coordinator has a timeout mechanism, namely, if not received the news of the participants in the agreed upon period of time the default failure), main is to avoid the participants in a long time with the coordinator node communication (coordinator to hang off), unable to release resources problem, Because participants have their own timeout mechanism, they automatically commit locally to release resources after a timeout. This mechanism also reduces the blocking time and scope of the overall transaction.

  2. Through the design of CanCommit, PreCommit and DoCommit, compared with 2PC, an additional buffer stage is set to ensure that the states of all participating nodes are consistent before the final submission stage.

  3. PreCommit is a buffer that ensures that the states of participating nodes are consistent until the final commit phase.

Problem: The 3PC protocol does not completely solve the data inconsistency problem.