C3p0 throws "ThreadPoolAsynchronousRunner DeadlockDetector APPARENT DEADLOCK!" when using Hibernate


#1

Hi,
I am seeing this error : com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@512bc420 – APPARENT DEADLOCK!!! and system shows high cpu usage for 2 hours and then normalizes. Post investigation the application doesn respond as it run out of DB connection.

I have these setting for Oracle Database

login ="xxxx"
password="xxxxx"
url = "jdbc:oracle:thin:@localhost:1521:orcl"	
driver = "oracle.jdbc.driver.OracleDriver"

Default Configuration :

c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, 
acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, 
checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> 
com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb, 
debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> oracle.jdbc.driver.OracleDriver, 
factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb, 
idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:oracle:thin:@10.0.2.164:1521:TFIMPROD1, maxAdministrativeTaskTime -> 0, 
maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 0, 
minPoolSize -> 3, numHelperThreads -> 3, 
preferredTestQuery -> SELECT 1, properties -> {user=******, password=******}, 
propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> true, testConnectionOnCheckout -> false, 
unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ]

I am getting the following error occasionally and can’t replicate this issue. This cause further no more database connection and gets lock for approx 2 hours and suddenly its back to normal. please can some look at exception provide hint what is wrong

com.mchange.v2.async.ThreadPoolAsynchronousRunner:com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@512bc420 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2018-12-18  12:11:56,909-HikariPool-1 housekeeper--WARN -com.zaxxer.hikari.pool.HikariPool:HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=45s839ms133µs57ns).
2018-12-18  12:12:53,089-C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-AdminTaskTimer--WARN -com.mchange.v2.async.ThreadPoolAsynchronousRunner:com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@512bc420 -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 1
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@20c2adcd
			on thread: C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-HelperThread-#2
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4d207e88
Pool thread stack traces:
	Thread[C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-HelperThread-#0,5,main]
		java.lang.Object.wait(Native Method)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:635)
	Thread[C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-HelperThread-#1,5,main]
		java.lang.Object.wait(Native Method)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:635)
	Thread[C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-HelperThread-#2,5,main]
		java.lang.ClassLoader.findBootstrapClass(Native Method)
		java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1061)
		java.lang.ClassLoader.loadClass(ClassLoader.java:412)
		java.lang.ClassLoader.loadClass(ClassLoader.java:410)
		sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
		java.lang.ClassLoader.loadClass(ClassLoader.java:356)
		java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2565)
		java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
		java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
		java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
		java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
		java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
		java.util.ResourceBundle.getBundle(ResourceBundle.java:724)
		oracle.net.ns.Message11.getMessage(Message11.java:64)
		oracle.net.ns.NetException.getMessage(NetException.java:222)
		oracle.jdbc.driver.DatabaseError.createSqlException(DatabaseError.java:447)
		oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:418)
		oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:536)
		oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:228)
		oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
		oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
	com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
	com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
	com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
	com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
	com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
	com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
	com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
	com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)

#2

So, you are using both HikariCP:

and c3p0:

2018-12-18 12:12:53,089-C3P0PooledConnectionPoolManager[identityToken->2rvy5g9z1gwdhfx1x0nm6l|20ec0cdb]-AdminTaskTimer–WARN -com.mchange.v2.async.ThreadPoolAsynchronousRunner:com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@512bc420 – APPARENT DEADLOCK!!! Complete Status:

That’s wrong. Also, this is not a question about Hibernate, so you should ask it either on the c3p0 user forum or StackOverflow.