Found the problem

Generally, the number of fans to be pushed is maintained between 3W-7W. In order to ensure the security of no error, it is imported into the twitter followers table in accordance with the amount of 1W each time, and then to implement a sending interface.

For a long time, this interface has been used in this way, and there have been no problems with text push. However, recently I need to perform a 150W image push task, so I will try to send 2W in small steps. Something unexpected happened.

According to the log statistics, when about 1.3w logs are pushed, the log is broken and the interface’s direct response times out. The steps to implement the functionality inside this interface were described in the previous article. There’s something in there that says after the tweet is sent, the processing status is written back to the list. Go to the database and look up the table. Found that the 2w data has not been updated, the status is still “not sent [0]”.

In this case, it is not possible to batch send 2W, just the 2W from the end of the log, depending on the order, also can not find out.

According to the quantity of 1W batch operation, then send 150W, can you go home tonight?

To analyze problems

After repeatedly confirming the function of the interface method, there is really no problem, after all, 1W is normal to send out. , in the process of batch send log has been printed out WeChat interface process to be obtained, but in fact this time I want to import as soon as possible because the next batch of 1 w data, will go to refresh the data, but found that the data is not updated every time, even [1] “in the” send status update, only after a period of time, find logs don’t go, When you refresh the database, you will find that the status changes to “sent [2]”. That’s a mystery.

Supposedly, you go through the middle first. When you think about it this way, it looks like a mechanism. Yes, the operation was added to the transaction.

To solve the problem

No control of the transaction is found internally in the method from the call to the external interface to the data update layer. There are no transactions added to the method. So it’s natural to look up the class, and sure enough, this class has @Transaction added to it.

With the transaction control removed, it is important to note that the JPA method has been used to save the save method, but now the JPA method has not been used to save the save method. Therefore, the JPA method should be changed to saveAndFlush, otherwise the data update will not take effect.

Re-import 3W quantity in batches. When push starts, go to the database to refresh immediately, and you will immediately find that the data is constantly updated.

It used to take 8-10min to complete a batch of 1w tweets, but it will take several hours to complete the total task. Now it is observed that the second batch of 1W data can be imported in 1-2min, because the subsequent wechat interface is called by asynchronous threads, which does not hinder the data import. Next, try 2W, 4W batch data, no problem, finally the remaining 8W import and send out. The total waiting time is less than half an hour.

The problem’s analyse

All other data processing interfaces are additive transactions, just as this one was written before. When optimizing, eliminate transactions. This is of course handled in conjunction with the purpose of the interface. The data involved in the interface itself is only used for reading and will not be used as business data. There is no need to add transaction control. In addition, the matter of sending tweets, few data are not updated, and more data are updated, it is not a very serious thing, there is no need to add transaction control.

The transaction mechanism should be used in conjunction with business requirements, rather than seeing data operations and adding transactions.

note

For background, please go to this article: juejin.cn/post/697033…