1

The time goes back to one night in December 2018. I was preparing to go home after Posting a demand online. As soon as I pressed the publish button, an alarm sounded. Only one or two errors were reported, so we concluded that it was only an accident.

After sending out the rest of the machine, a few more of the same mistakes appeared. As a good programmer, this kind of problem must be tracked down.

2

Deftly querying the error log

org.apache.ibatis.exceptions.PersistenceException: ### Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed

Looking at the abnormal information, I was lost in thought

  • The ostensible error was the use of a closed data source
  • When does the data source close? Only when the process is killed
  • Is the application closing not smooth enough? Release will first remove traffic ah, should not be ah

It was getting late, and I was dragging my log aimlessly, wearily searching for new clues, when I noticed a word in the error log: RocketMQ

There is also a dedicated RocketMQ consumer in the application that has been consuming messages. When the application was shut down, external traffic was removed, but no one notified RocketMQ Consumer, so it threw an exception.

3

Due to my shallow and somewhat superficial understanding of RocketMQ, its consumption is ack, if an error is reported, the message will be retried later, no messages will be lost, and if the consumption code is idemidemous, there will be no business exceptions, but it doesn’t matter because I didn’t write it either.

Taking a look at the consumer’s code (which I won’t post here, because you wouldn’t read it anyway), the consumer registers a ShutdownHook, where the consumer gracefully exits by executing shutdown. It also sets the highest priority for the shutdownThread, which is not useful in practice.

ShutdownHook Principle article also know that ShutdownHook is executed concurrently, spring container closure is also a ShutdownHook, they did not order before.

After understanding the reason, I immediately thought of a solution similar to dubbo pick flow. I wrote an interface to gracefully shut down RocketMQ cosnumer, and called the interface before applying the kill script to shut down the rocketMQ cosnumer. I solved the problem perfectly, and went home from work before I died.

4

I fall asleep at night dreaming that my boss wants me to change all the systems, which scares me out of my wits.

I thought about it again the next day, thinking that implementing an interface in an application and calling it in a stop script was a very inelegant thing to do, and more importantly, it couldn’t be replicated in other projects.

The problem is that the beans are destroyed in the order in which the Spring container is closed. No sooner said than done.

At first I encountered dependencies like this:

Hand-in-hand matching the Depend -on relationship in each bean of the XML also seems to work.

But when I opened the second project, the dependencies between its beans looked something like this:

Boy, 26 letters is almost not enough, my mood was like this

So I think at the current rate, it could be 9102 before we finish all the projects.

5

Some time later, I found the official rocketMQ implementation spring-boot-Starter on Github and clicked on it. Oh, boy, call 666 when you’re done.

The official Starter implements Spring’s SmartLifecycle interface. Its start method can be called after all beans have been initialized, and its stop method can be called before the beans have been destroyed, making it perfect for RocketMQ Consumer.

By the way, could you also review the spring container closed, code in AbstractApplicationContext doClose method, here I summarize into a picture:

As you can see in the figure above, lifecycle bean is closed and ContextClosedEvent is sent before destroying the bean. The official starter has chosen to implement the Lifecycle interface.

6

It’s time for me to report to my boss that the rocketMQ Consumer launch was not smooth because of our use posture. Although it has no impact on the business, it is not elegant. There are two solutions, boss you can choose:

  • All the official starter, relying on spring-boot, official maintenance, renovation costs are high,
  • Listening for ContextClosedEvent to gracefully close, this can be encapsulated and let the business side introduce the dependency

Now that you’ve seen this, don’t you want to pay attention? Wechat search public account “Bug Catching Master”