instructions

Rocketmq4.3 now supports transactions!! , took a look over the weekend, and the following is a summary of RocketMQ transactions.

Note: Today’s article is just an introduction, and does not involve a lot of details, the general process of the first to clarify, and then specific details for the opening.

The theme

  • Introduces distributed transaction related content.
  • RocketMQ transaction messages.
  • How RocketMQ transaction messages are used.
  • How the RocketMQ transaction message is implemented.
  • Why transaction messages are needed to check the mechanism.
  • How RocketMQ does transaction message checking.
  • What are the limitations of RocketMQ for distributed transaction resolution? And instructions.

Introduces distributed transaction related content

Here is mainly to explain what is the background of this kind of problem.

First of all, let’s talk about transaction. When talking about transaction, the first thing that reminds me is that when I was in college, my teacher often gave an example of transferring money. The example is as follows:

Bank transfer! SQL > transfer $100 to bill’s account

  • Subtract $100 from John’s account
  • Add 100 yuan to Li Si’s account

If the program is interrupted after the first SQL statement is successfully executed, but before the second SQL statement is executed (perhaps because of an exception thrown, or for some other reason), then $100 is not added to Li’s account, but $100 is subtracted from Zhang’s account. That’s not going to work!

You probably know what a transaction is by now. Multiple operations in a transaction, either complete success or complete failure! There is no such thing as half success! In other words, if subtracting $100 from Joe’s account succeeded, then adding $100 to Joe’s account must also be successful; Otherwise, it will fail to subtract 100 yuan from Zhang SAN and add 100 yuan to Li Si!

If we use Spring within a single service, we solve this problem with a single annotation (the @Transactional annotation solves this and basically nothing else).

As services become more complex and data volumes increase, companies are slowly introducing technologies such as microservices and subtables, which are useful but create problems in resolving transactions.

Note: distributed transactions, we generally emphasize final consistency, not strong consistency!!

It can be divided into three categories:

  • Based on a single JVM, databases are stored and tabulated (across multiple databases).
  • Based on multiple JVMS, services are split (not across databases).
  • Based on multiple JVMS, services are split and databases are divided into repositories and tables.

It is in order to solve the above three types of problems that distributed transaction related technologies are introduced to solve these problems.

There are many distributed transaction solutions, and the RocketMQ transaction messages described below are just one of them.

RocketMQ transaction messages

Here is a reference to the official RocketMQ documentation.

http://rocketmq.apache.org/rocketmq/the-design-of-transactional-message/

Half(Prepare) Message

A temporarily undeliverable message is a message that has been successfully sent to the MQ server by the sender but has not received a second acknowledgement from the producer. The message is marked as “temporarily undeliverable” and is referred to as a half-message.

Message back to check

The secondary confirmation of a transaction message is lost due to intermittent network disconnection or producer application restart. When the MQ server detects that a message has been half-message for a long time, it proactively asks the message producer for the final status of the message (Commit or Rollback). This process is called message Rollback.

Execution flow chart

  1. Send Sends a message to the MQ server.
  2. After the MQ Server persists the message successfully, it confirms to the sender ACK that the message has been successfully sent, and the message is a half-message.
  3. The sender starts executing the local transaction logic.
  4. Based on the result of the local transaction, the sender submits a secondary confirmation (Commit or Rollback) to the MQ Server. Upon receipt of the Commit status, the MQ Server marks the half-message as deliverable and the subscriber will finally receive the message. MQ Server receives the Rollback status and deletes the half-message, which will not be accepted by the subscriber.
  5. In the case of network disconnection or application restart, the MQ Server checks the second confirmation message submitted in Step 4 after a fixed period.
  6. After receiving the message, the sender needs to check the final result of the local transaction execution of the corresponding message.
  7. The sender submits a second acknowledgement based on the final status of the local transaction checked, and MQ Server continues with step 4 for the half-message.

Transaction message sending corresponds to steps 1, 2, 3, and 4, and transaction message lookup corresponds to steps 5, 6, and 7.

How are RocketMQ transaction messages used

1. Introduce RocketMQ-client

Not yet, wait a few days will have, or download the source code for packaging to their own private warehouse is the same, after the introduction of 4.2.0, is changed to 4.3.0.

< the dependency > < groupId > org. Apache. Rocketmq < / groupId > < artifactId > rocketmq - client < / artifactId > < version > 4.3.0 < / version > </dependency>Copy the code

2. Writing Producer

Refer to RocketMQ4.3.0 for example, address:

https://github.com/apache/rocketmq/tree/release-4.3.0/example/src/main/java/org/apache/rocketmq/example/transaction

Note: Instead of sending DefaultMQProducer, we use TransactionMQProducer.

Key points: Implement the TransactionListener interface yourself, and implement the executeLocalTransaction method ( The checkLocalTransaction method is used to provide the broker with a local transaction message, and stores the result of the local transaction in redis or DB to prepare the data for the local transaction.

Take again: subtract $100 from John’s account and add $100 to John’s account. At this time, we send is equivalent to doing the operation of subtracting 100 yuan from Zhang SAN’s account, and the operation of DB is successful. The following is the operation of adding 100 yuan to Li Si’s account, this time is equivalent to defining the topic that just sent transaction messages, and there is no difference between the consumer end and our ordinary consumer end.

Consider: what if this time we consumer end failure to do? (This issue will be discussed later, there are two kinds of consumption failure, the first kind is timeout, we try again, the second kind is really failed? What should I do?

How are RocketMQ transaction messages implemented

At first glance, this is very similar to timed messaging, but more complex than timed messaging. The timing message content is covered in RocketMQ(IX) : Message Sending (continued).

The topic of a scheduled message is changed to SCHEDULE_TOPIC_XXXX. RMQ_SYS_TRANS_HALF_TOPIC = RMQ_SYS_TRANS_HALF_TOPIC = RMQ_SYS_TRANS_HALF_TOPIC = RMQ_SYS_TRANS_HALF_TOPIC = RMQ_SYS_TRANS_HALF_TOPIC If rollback is needed, delete the temporary topic content.

Topic breakpoint for Half(Prepare) Message

This article is just an introduction, and the details will be analyzed in subsequent chapters.

Consider: if you combine sending normal messages with local execution logic in one transaction, sending normal messages if the transaction succeeds, and rolling back if the transaction fails, what are the advantages or benefits of this transaction message?? Think about.

Why do you need transaction message lookup mechanisms

In RocketMQ transaction messages, the second confirmation of a transaction message is lost due to network outages and producer application restarts. The MQ server scans and detects that a transaction message has been “half message” for a long time. You need to proactively ask the message producer for the final status of the message (Commit or Rollback), a process known as message back lookup. So what does RocketMQ do? I’ll talk about that in a minute.

How does RocketMQ do transaction message checking

The Half(Prepare) Message is checked every 60 seconds for messages whose topic topic is RMQ_SYS_TRANS_HALF_TOPIC.

The broker calls its own checkLocalTransaction method that implements the TransactionListener interface.

Note: RocketMQ has already implemented this mechanism. Today’s post is just an introduction and does not cover a lot of details.

What are the limitations of RocketMQ for distributed transaction resolution? As well as the description

In how RocketMQ transaction messages are used, we mentioned ** what if the consumption fails? ** consumption failed in two ways, the first is timeout, we try again, the second is really failed? If you think about it, this is pretty complicated. What if you need 7-8 business modules, and the sixth one fails? ** This retry several times still failed, what should we do?? Does ** roll back the previous 5 operations? So complicated, so complicated, why doesn’t RocketMQ provide automatic rollback? I hope you think about it, if the failure rate is relatively large, then it is a system problem that needs to optimize the code business…

As far as Zero knows, a lot of this is through manual intervention and T+1 reconciliation and compensation mechanism.

conclusion

My level is limited, it is inevitable that there will be some misunderstanding, if found, you are welcome to point out, thank you!!

If you feel that you have gained something after reading it, please click “like”, “follow” and add the official account “Ingenuity Zero” to check out more wonderful history!!