Cause analysis of deadlock problems caused by ReentrantLock and JPA (Spring.jpa. open-in-view).
The problem
During the pressure test, the service does not respond after a period of time and cannot be recovered automatically.
Analysis of the
From the above representation of the problem, it is assumed that the service is deadlocked, causing all threads to wait for the lock and thus unable to respond to all subsequent requests.
Next, a large number of container threads are waiting to connect to the database through jStack output stack information
"XNIO-1 task-251" #375 prio=5 os_prio=0 tid=0x00007fec640cf800 nid=0x53ea waiting on condition [0x00007febf64c5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000081565b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at com.alibaba.druid.pool.DruidDataSource.takeLast(DruidDataSource.java:1899) at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1460) at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1255) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1235) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1225) at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProvid erImpl.java:122) at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl .java:106) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.jav a:136) at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:542)Copy the codeIf you look at the DruidDataSource source code, you can see that there are no database connections currently available, so the thread is waiting.
DruidConnectionHolder takeLast(a) throws InterruptedException, SQLException {
try {
while (poolingCount == 0) {
emptySignal(); // send signal to CreateThread create connection
if (failFast && failContinuous.get()) {
throw new DataSourceNotAvailableException(createError);
}
notEmptyWaitThreadCount++;
if (notEmptyWaitThreadCount > notEmptyWaitThreadPeak) {
notEmptyWaitThreadPeak = notEmptyWaitThreadCount;
}
try {
// Wake up when a new connection is created or another thread releases the connection
notEmpty.await(); // signal by recycle or creator
} finally {
notEmptyWaitThreadCount--;
}
notEmptyWaitCount++;
if(! enable) { connectErrorCountUpdater.incrementAndGet(this);
throw newDataSourceDisableException(); }}}catch (InterruptedException ie) {
notEmpty.signal(); // propagate to non-interrupted thread
notEmptySignalCount++;
throw ie;
}
decrementPoolingCount();
DruidConnectionHolder last = connections[poolingCount];
connections[poolingCount] = null;
return last;
}
Copy the codeThere are 8 threads waiting for the lock 0x000000008437E2C8. This lock is ReentrantLock, indicating that the ReentrantLock has been held by another thread.
The analysis may be due to the fact that the eight threads did not release the database connection at some point, so that the other threads could not acquire the database connection (why eight, because the database connection pool is configured by default and the maximum number of connections is 8 by default).
The thread holding the ReentrantLock is holding the ReentrantLock lock, but is waiting to connect to the database. An exception caused the lock not to be released after the last acquisition (not in the finally block), but simply not releasing the ReentrantLock does not cause the entire application to deadlock, because the thread can normally acquire the ReentrantLock the next time it executes a request. Then, if the ReentrantLock is released normally, everything goes back to normal. The root cause of the problem is not that ReentrantLock did not release the lock, but that the database connection was not released correctly.
According to the above analysis, one thread holds the ReentrantLock lock, but is waiting for the database connection, while the other eight threads hold the database connection, but is waiting for the ReentrantLock lock, resulting in deadlock.
But normally, when the database operation completes, the thread should release the database connection, which it did not. Since we use JPA, we guess it may be a problem with JPA.
The JPA warning log is found in the SpringBoot boot log, as shown below
spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
Copy the codeThinking this might be due to this configuration problem, I started looking for Spring Data JPA documentation. Finding this configuration will result in the database connection still being held after the request completes the database operation. Because for JPA (which is a Hibernate implementation by default), the ToOne relationship is loaded immediately by default and the ToMany relationship is lazily loaded by default. When we pass the JPA query to an object, may go to call ToMany relations corresponding entities of the get method, obtain corresponding entity set, if not Hibernate Session will be submitted to lazyinitializationexceptions, So by default SpringBoot adds an interceptor that holds a Session when the request starts and closes the Session when the request completes, releasing the database connection.
Check out the interceptor source code for Spring.jpa. open-in-view
public class OpenEntityManagerInViewInterceptor extends EntityManagerFactoryAccessor implements AsyncWebRequestInterceptor {
@Override
public void preHandle(WebRequest request) throws DataAccessException {
EntityManagerFactory emf = obtainEntityManagerFactory();
if (TransactionSynchronizationManager.hasResource(emf)) {
// ...
}
else {
logger.debug("Opening JPA EntityManager in OpenEntityManagerInViewInterceptor");
try {
// Create an EntityManager and bind it to the current thread
EntityManager em = createEntityManager();
EntityManagerHolder emHolder = new EntityManagerHolder(em);
TransactionSynchronizationManager.bindResource(emf, emHolder);
AsyncRequestInterceptor interceptor = new AsyncRequestInterceptor(emf, emHolder);
asyncManager.registerCallableInterceptor(key, interceptor);
asyncManager.registerDeferredResultInterceptor(key, interceptor);
}
catch (PersistenceException ex) {
throw new DataAccessResourceFailureException("Could not create JPA EntityManager", ex); }}}@Override
public void afterCompletion(WebRequest request, @Nullable Exception ex) throws DataAccessException {
/ / close the EntityManager
if(! decrementParticipateCount(request)) { EntityManagerHolder emHolder = (EntityManagerHolder) TransactionSynchronizationManager.unbindResource(obtainEntityManagerFactory()); logger.debug("Closing JPA EntityManager in OpenEntityManagerInViewInterceptor"); EntityManagerFactoryUtils.closeEntityManager(emHolder.getEntityManager()); }}@Override
public void afterConcurrentHandlingStarted(WebRequest request) {
// Unbind
if(! decrementParticipateCount(request)) { TransactionSynchronizationManager.unbindResource(obtainEntityManagerFactory()); }}}Copy the codeconclusion
Because spring.jpa.open-in-view is not configured (default true), the database connection is not released after the jPA method completes (will not be released until the request completes), and the ReentrantLock lock is not released correctly due to an exception. The ReentrantLock lock cannot be acquired by any thread that has already acquired a connection to the database, and the ReentrantLock lock cannot be acquired by any other thread (including the thread that holds the ReentrantLock lock), resulting in a deadlock. The fix is as simple as releasing the lock in the finally code block and optionally turning off the spring.jpa.open-in-view configuration.
For the configuration of Spring.jpa. open-in-view, there are roughly two views. One thinks that this configuration is necessary to improve the development efficiency, and the other thinks that this configuration will affect the performance (the Controller method will release the connection after completion), resulting in a waste of resources. However, if the connection is released after the database operation, there is no way to get the entity set corresponding to the ToMany relationship (or to get it manually, which is obviously not appropriate).
In fact, there is no right or wrong between these two views, but you need to choose according to the actual situation of the business. For this reason, when the user does not actively configure Spring.jpa. open-in-view, a warning log is printed during the startup process to inform the user to pay attention to this configuration and make an active choice.