Master slave replication requires that all write requests go through the master node, and any slave node can only accept read-only queries. For read-intensive loads, it is a good choice to create multiple slave nodes and distribute requests to them to lighten the load on the primary node. In this system, the service throughput of read requests can be improved by adding more slave nodes, but this method can only use asynchronous replication in fact, because in synchronous replication, the failure of a single node will lead to the failure of the entire system, and the more slave nodes, the higher the probability of failure. One problem with asynchronous replication is that there may be inconsistencies between the slave node and the master node. There is no upper limit for such inconsistencies. If the system is approaching the upper limit or there is a problem with the network, the lag time may be as long as seconds or even minutes

When the lag time is too long, the resulting inconsistency can be a real problem. This article will focus on three possible problems of replication lag and introduce corresponding solutions

Read your own writing

Reading your own write is a very common application scenario, characterized by reading immediately after writing, such as when a user posts a comment, it is immediately displayed in the comment list. However, for asynchronous replication, due to the possible replication lag, the user may not be able to find the data from the node immediately after the write, and the effect will appear as if the data is lost, which is a serious problem

For this, we need “write read consistency”, also known as read-write consistency. This mechanism ensures that a user can see his or her most recent submission when he or she reloads the page (but makes no promises to other users, who may not see his or her submission until later). How does a system based on master/slave replication achieve read/write consistency? There are several possible solutions:

  • If the content accessed by the usage is mutable, the master node reads it, whereas the slave node reads it. The premise here is to know that the content is modifiable. For example, only the owner can edit the user information of a website, and others have no right to edit it, so the owner reads it from the master node, and other users read it from the slave node
  • If most of the content of the application can be modified, the above scenario results in most of the reads being from the master node and thus losing the scalability of the read operation. Other schemes are needed to determine where to read from. For example, you can track the time of the last update and, if it is less than a minute after the update, read from the primary node
  • The client can remember the most recently updated timestamp and attach it to the request, so the information system can ensure that the read service to the user should contain at least the updated timestamp. If it is not new enough, either hand it over to another slave node or wait for the last update to be received from that node
  • If replicas are distributed across multiple data centers, the situation becomes more complicated and requests need to be routed to the data center where the primary node resides

The situation is even more complicated when you consider accessing data across devices, such as a desktop Web browser and a mobile app, requiring the user to enter information on one device and then see what they just entered on the other. There are a few things to consider in this case:

  • Remember that metadata such as a user’s last update timestamp needs to be shared globally and accessible across devices
  • If multiple DCS exist, different devices cannot be connected to the same DC

Boring to read

The user does not write, reads only, and still has problems in the case of lagging replication.

Assuming that the user has made multiple reads from different replicas, as shown in the figure above, each time the user refreshes the page, the request may be randomly sent to a node, resulting in different results for different accesses. Here we need to achieve monotone read consistency, which is weaker than strong consistency and stronger than final consistency.

Monotonic consistency is achieved by ensuring that each user always reads from a fixed node, such that copies can be selected based on user ID hashing rather than random selection. But if the copy fails, the user’s query must be updated and routed to another copy

Prefix consistent reading

Replication lag will lead to causal anomaly. For example, in normal circumstances, there is question first and then answer, but the existence of replication lag may lead to a third party observer seeing answer first and then answer, resulting in causal dislocation and logical confusion. Preventing such exceptions requires the introduction of a guarantee, namely prefix consistency. The guarantee is that a series of write requests that occur in a certain order will be read in the order in which they were written. This is a particular problem with partitioned databases, which we will cover later when we introduce partitioning. In general, if the database always writes in the same order, reads always see the known sequence and this exception does not occur. But in many distributed databases, the different partitions run independently, so there is no global write order, which results in a user reading data from the database seeing some old values and some new values

The solution is to ensure that any causal writes are assigned to a partition, but this can lead to inefficient implementation. There are some new algorithms that can display trace event causality, which we will cover in more detail later

Replication lag solution

When designing our systems, we need to think about what happens to the application layer if the replication delay increases to a few minutes or hours. If the answer is no, then it doesn’t matter. But if there is a problem, then greater consistency needs to be considered at design time

Transactions are a way for databases to provide stronger assurance of correctness. Transactions on a single node are well established, but many systems have chosen to abandon support for transactions when they move to distributed (replication and partitioning supported). There is much more to discuss and explore about transactions, which we will cover later