This is the 11th day of my participation in the August More Text Challenge. For details, see:August is more challenging
- 📢 welcome to like: 👍 collect ⭐ message 📝 If there is any mistake please correct it, send roses, hand left fragrance!
- 📢 This article is written by Webmote.
- 📢 author’s motto: life lies in tossing about, when you do not toss about life, life will start tossing you, let us work together! 💪 💪 💪
Preface 🎏
In the past, when choosing an architecture, it is enough to know what it does and what the advantages and disadvantages are, because most Internet enterprises are light on documents and heavy on fast iteration, and often do not struggle with too much choice of scheme.
As long as the plan is right, do it.
At that time, the waterfall development mode was popular, and the importance of the scheme and requirements was placed at the top position. Often, the scheme needed to write dozens or hundreds of pages, and the key technical options also needed to write the key technical options. Only after several rounds of review could the final requirements, summary, detail and summary be achieved. Of course, the review of the leadership of various perspectives, often let a person at a loss.
After years of “relaxation”, the level of technical documentation has become worse and worse, and even a lot of middleware is just scratching the surface, not looking at the underlying principles, but focusing on what can be done with the middleware.
Because of the lack of theoretical guidance, many things that had been taken for granted in the past had to be carefully elaborated and agreed by the leaders, but they encountered various problems that could not be explained clearly by themselves.
How to quickly persuade the leadership, met great resistance. Also let me see clearly their own problem: lack of theoretical basis. Of course, whether a big leader needs to get deeply involved in technical issues is not discussed here, because the only biggest thing in life is to cultivate oneself.
Therefore, this article to myself, training in a little.
🎏 01. What does ETCD do?
Etcd is a highly consistent distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It handles leadership elections elegantly within the network and can tolerate machine failures without affecting high consistency even if the leader fails.
ETCD, one of the three best consistency solutions (along with ZooKeeper and Consul), is characterized by its high consistency.
🎏 02. The concept of consistency
Consistency is a concept proposed in distributed systems, because for single-machine systems, there are almost no issues of consistency.
In order to solve the single point of failure of the single-node storage system, the system is upgraded from the architecture to the active/standby system. The backup system uses various solutions to back up the data on the primary system so as to keep the data synchronization with the primary system.
Once the data is replicated, consistency issues arise.
Consistency: for a given client, a read operation is guaranteed to return the latest result of a write operation.
🎏 2.1 CAP theorem
As the most basic theorem in the distributed system, CAP is a conjecture first proposed by Brewer of the University of California, and finally proved, which makes it a recognized theorem in the field of distributed computing, so it is also called Brewer’s theorem.
The CAP theorem states that in a distributed system (a collection of interconnected and shared data nodes), only two of the three guarantees are Consistency, Avaliability, and Partition Tolerance when it comes to read and write operations. The other must be sacrificed.
Availability refers to the non-failed node returning a reasonable response (without errors or timeouts) within a reasonable amount of time. Partition fault-tolerant: The system can continue to perform its duties when a network partition is present. In other words, after some nodes fail, the properly connected nodes can still use the services provided by the system.
Do not understand search, partition fault tolerance this concept is not easy to understand, so here quote zhihu’s answer.
In a distributed system, the network of nodes is supposed to be connected. However, maybe because of some faults, some nodes are disconnected from each other, and the whole network is divided into several areas. The data is scattered across these disconnected regions. This is called partitioning.
When you have a data item stored in only one node, when a partition appears, the data cannot be accessed by the part that is not connected to the node. Partition is intolerable.
One way to improve partition tolerance is to copy a data item to multiple nodes, so that once a partition is present, the data item can be distributed across the partitions. Tolerance increases.
However, when data is copied to multiple nodes, consistency issues arise, and data on multiple nodes may not be consistent. To be consistent, every write operation has to wait for all nodes to be written successfully, and this waiting creates usability problems.
In general, the more nodes where data exists, the higher the partition tolerance, but the more updated data to replicate, the more difficult it is to ensure consistency. To ensure consistency, the longer it takes to update data on all nodes, the less available it will be.
For distributed systems, it is theoretically impossible to abandon P, so the alternatives are often CP or AP architecture. Of course, abandoning does not mean doing nothing. When partitioning occurs, you can do something, and wait until the partition recovers to reach the CA state again.
🎏 2.2 Consistency Classification
According to the replication policy, consistency can be divided into four types:
- Strict consistency: Any write is immediately synchronized to all nodes. In a distributed system, synchronization of data takes time, so it is conceivable that this model cannot be achieved.
- Linear consistency: Any read can read the data of the last write of a data. All processes in the system, see the order of operation, and the global clock is the same order.
- Sequential consistency: No matter how the system is run, the result is as if all operations on all nodes are run in some sequential order, but operations from the same node remain in the order they were specified within the nodes.
- Final consistency (weak consistency)
It is not guaranteed that the same data on any node is the same at any time, but with the migration of time, the same data on different nodes always changes in the direction of convergence. Simply put, the data between nodes will eventually reach a consistent state after a period of time.
🎏 2.3 consensus
Consensus is the most important and fundamental problem in distributed computing, and it wants to do something: get all the nodes to agree on something. In the consensus problem, all nodes must reach a consensus. Since the ultimate goal is that all nodes must reach a consensus, there is no difference between strong and weak consensus.
For example, Paxos is a Consensus algorithm rather than a Consistency protocol. Consensus algorithms do not distinguish between consistency levels.
🎏2.4 Two-phase submission
Two-phase commit is an algorithm that achieves atomic commit across multiple nodes, ensuring that all nodes commit or all nodes abort.
ETCD writing of data synchronization is also used by the two phase commit, in phase 1 leader after receiving the write data change by written notice, if more than half of the node to write the log, launched a second phase of the formal writing data, and inform the individual nodes, if half writing successfully submit data, otherwise don’t roll back data, but the follow-up can be covered.
Two-phase key agreement, it through two stages to the unreliable transaction commit failure rate reduced to the minimum, the first stage generally occupies the entire transaction, most of the time and really commit the transaction in the second stage is almost instantaneous, therefore the probability of the second phase error is reduced greatly, so this is the clever of the two stages.
In reality, we rarely use the two-phase commit protocol to ensure transactionality. Why?
- In real world scenarios, the most commonly used is the final consistency based on BASE theory
- The commit protocol requires locking resources and has a performance penalty, which is not appropriate in high-concurrency scenarios.
- The commit protocol introduces a transaction manager (TM), resulting in complex system implementations that few people are willing to do.
- The highest level of distributed transactions is no distributed transactions!
🎏 03. RAFT algorithm for ETCD
Summarize as follows, detailed can search all kinds of articles.
Select leader, each node can have three roles: leader, follower, and candidate (candidate). Cluster nodes use heartbeat to check the online status of each node. If the leader is found kneeling, the followers can promote themselves to be candidates and broadcast to all nodes. The election starts, and if more than half of them agree, the leader can be elected.
Of course, in order to prevent the node with incorrect data from being elected as the leader, the node must bring the latest data sequence saved by itself during the election for comparison by other nodes. Other nodes can only elect you as leader if they check that your data is correct or updated.
Ooop, there is no canvassing or cheating here.
Of course, there is no special requirement for the candidates. If all the conditions are the same, the candidates will wait in line in chronological order.
After the leader is selected, data is written to the leader first, and then the leader broadcasts the data to all nodes. In a two-phase commit mode, data is written to the entire cluster.
🎏 04. Communication protocol
ETCD uses ProtoBuf encoding and GRPC protocol to read and write data. Of course, ETCD also provides higher level HTTP to read and write data. .net Core has a github package library, but did not find the official maintenance SDK.
🎏 05. Summary
The study of theory is still boring, thanks to your company, it is not a kind of happiness to share!
Routine summary, rational view!
Knot is what, knot is I want you to like but can not get the loneliness. 😳 😳 😳
👓 all see this, still care to point a like?
👓 all points like, still care about a collection?
Do you care about a comment when you have collected 👓?