Condition Description:

The latest version of the project will die soon. A condition is a state of suspended animation. Index queries are slow, almost impossible to do anything, slowly stuck. When we release the next version, we can only replace or add class files based on the war package we made earlier.

Comparison and analysis:

Because the code has been cleaned up before, and a lot of codes have been submitted, it is difficult to solve the problem by rolling back the version. One is that the achievements of rectification should not be lost in vain. Second, so many documents, rely on manual comparison search, more difficult.

Solution 1:

Previously, I was skeptical about the way the current project was packaged, so when things went wrong this time, the first people I suspected were Jenkins and the Tomcat server. When I connected to production from a Fortnite, I found that starting the application with Jenkins started two or two Tomcat processes.

Then, it seemed to strengthen my opinion, I immediately found the operation and maintenance, but after confirmation, there was no problem. One is started as root user, and the other is started as Tomcat user. One daemon, one application process.

Solution 2:

With the server out of the way, we started thinking about the program head on. Resend the faulty version of the project, Dump the logs, and quickly roll back to observe. The dump log of a single machine has 5 G:

Through Memory Analyzer analysis, in the Leak Supects Report view, there are the following analysis results:

As shown in the figure above, there are three types of problems: A, B and C; There are some others, of type D.

Let’s start with the first question (which turns out to be the same question).

Click on Details to see:

The figure above shows an obvious problem thread: the address is 0x7C8FF3DF0 and the name is Pool-16-thread-1. The Accumulated Objects in Dominator Tree view shows that in this thread there is a large List object that houses a large number of mysql JDBC Objects.

We want to see what data is stacked inside the JDBC object. Next we open the view open Dominator Tree for Entire Heap:

Find the thread named 0x7C8FF3DF0 pool-16-thread-1. As you can see, this thread is taking up a lot of space. Layer by layer to open the objects inside:

This data right here is data from one of our user tables. So it follows that a thread, a list contains a large number of user objects retrieved from the database! Thinking of this, we went to see the problem description of B and C, which is also the same problem. Presumably at various points in time, some of it has been recovered through gc.

Let’s look at a’s stackTrace stack.

The problem is obvious, as shown in the figure above. In line 112 of the Service, the findByCustomerID method is called to scan the entire table. After analysis, the corresponding position is found, and the corresponding code is:

customerID = StringUtils.isNotBlank(customerID)? customer.getCustomer().getCustomerID():null; Customer oldCutsomer = customerService.findByCustomerID(customerID);Copy the code

It is obvious that process here, customerID forever is empty, so the customerService. FindByCustomerID (customerID) method, scanning will perform a full table operations. Because of the large amount of data in this table, every index query that the developer thinks the user executes actually becomes a slow query and needs to return the full table data. A large number of threads occupy a large number of database connections, resulting in insufficient database connections; Each thread takes so much time to process that the project is in a state of suspended animation.

Summary and analysis:

1. Code management must be standardized at work in order to improve development efficiency and reduce the risk of enterprise losses;

2. Through this practice, I found that the current development authority is limited, resulting in low cross-department coordination efficiency. In the following development, the project development process should be established at the beginning of project approval to improve the development efficiency;

3, the method to solve the problem encountered bottleneck, try the second method, multi-angle multi-level to break through the problem.

Author: Liu Zhengquan, Creditease Institute of Technology