[SOLVED] JDBC connection exhaust

Technical Discussion
1

After long time running. My database connection exhausted. From the MBeans I can see many closed connections, and the in_pool connections is equals (db_allowed_max_conns - mark_as_closed_conns).
And I found some exceptions in my moqui.log file.

--- 2021-08-14 06:53:37.625 [oquiScheduled-2] WARN                  bitronix.tm.resource.common.XAPool []
 error closing a JdbcPooledConnection from datasource transactional_DS in state CLOSED with usage count 0 wrapping org.postgresql.xa.PGXAConnection@2287aa1
java.util.ConcurrentModificationException: null
        at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) ~[?:1.8.0_265]
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752) ~[?:1.8.0_265]
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750) ~[?:1.8.0_265]
        at bitronix.tm.resource.jdbc.LruStatementCache.clear(LruStatementCache.java:171) ~[moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.close(JdbcPooledConnection.java:150) ~[moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.common.XAPool.expireStatefulHolder(XAPool.java:534) [moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:320) [moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:181) [moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:150) [moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:277) [moqui_temp3453359234312533122WEB-INF_lib_btm-3.0.0-SNAPSHOT.jar.:3.0.0-SNAPSHOT]
        at org.moqui.impl.entity.EntityFacadeImpl.getConnection(EntityFacadeImpl.groovy:2031) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.entity.EntityQueryBuilder.makeConnection(EntityQueryBuilder.java:66) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.entity.EntityFindImpl.oneExtended(EntityFindImpl.java:78) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.entity.EntityFindBase.oneInternal(EntityFindBase.groovy:941) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.entity.EntityFindBase.one(EntityFindBase.groovy:720) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.service.ScheduledJobRunner.runInternal(ScheduledJobRunner.groovy:136) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at org.moqui.impl.service.ScheduledJobRunner.run(ScheduledJobRunner.groovy:74) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_265]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_265]
        at org.moqui.impl.context.ContextJavaUtil$CustomScheduledTask.run(ContextJavaUtil.java:693) [moqui_temp7464524058079123453WEB-INF_lib_moqui-framework-3.0.0-rc8.jar.:3.0.0-rc8]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]

Don’t known what cause java.util.ConcurrentModificationException.

2

Because the get, put, clear methods in LruStatementCache all protected by synchronized, I think the concurrent modification is from same thread. I suspect the LruEvictionListener called PreparedStatement.close(), which called Connection.close(), and reenter the cache.clear() method.

3

All of this is caused by there’s a buggy btm-3.0.0-SNAPSHOT.jar provided by the maven mirror site I am using. Now I deleted all my maven mirror configuration.

1 Like
4

Shoud we change the customized btm’s version?

5

I’m not sure why a problem with a buggy maven repository package would require a change with the version of the package. Is there a problem with btm v3.0.0? because it sounds like it was just a problem with the mirror site’s package that you were using.

If I’m wrong let me know, but I guess I’m confused about how this problem would affect any other installation of moqui.

6

Moqui using customized btm, It is not same as official v3.0.0-SNAPSHOT.

1 Like
7

Oh wow that’s actually a little worrying. Do we have access to the modifications to understand the alterations? Maybe we should fork bitronix to publish these changes and then maybe maintain our own version since bitronix is dead anyways.

1 Like
8

On changing the version maybe so, use something like btm-3.0.0-moqui.jar (instead of -SNAPSHOT.jar)? I’m guessing the reason for that is to make it easier to make sure it is not coming from another maven repo (always from framework/lib).

For what it’s worth, the jar file in the moqui-framework repo is a build from a fork I did a long time ago, and it is 4 commits behind the upstream repo now:

Chunlin, are you saying maybe one of these more recent commits in bitronix/btm:master is causing this problem, or perhaps the jar file from a different maven repo had other custom changes?

9

Not the recent commits in bitronix/btm:master. I configed a maven mirror repo, the btm from that repo causing this problem. So give a different version can prevent same problem in future.