On fetching search results - HikariCP apparent connection leak detection triggered issue

Hi,
We are sometimes facing this warning when fetching the search results. Currently using version 6.0.0.Beta7. Attaching the relevant stacktrace as well:

java.lang.Exception: Apparent connection leak detected
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128) ~[HikariCP-3.4.1.jar!/:?]
	at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2104) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2034) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.entity.DynamicBatchingEntityLoaderBuilder$DynamicEntityLoader.doTheLoad(DynamicBatchingEntityLoaderBuilder.java:564) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.entity.DynamicBatchingEntityLoaderBuilder$DynamicEntityLoader.doEntityBatchFetch(DynamicBatchingEntityLoaderBuilder.java:530) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.entity.DynamicBatchingEntityLoaderBuilder.performOrderedBatchLoad(DynamicBatchingEntityLoaderBuilder.java:200) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.entity.DynamicBatchingEntityLoaderBuilder.performOrderedMultiLoad(DynamicBatchingEntityLoaderBuilder.java:160) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.loader.entity.DynamicBatchingEntityLoaderBuilder.multiLoad(DynamicBatchingEntityLoaderBuilder.java:64) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.multiLoad(AbstractEntityPersister.java:4411) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.internal.SessionImpl$MultiIdentifierLoadAccessImpl.lambda$multiLoad$1(SessionImpl.java:2950) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.internal.SessionImpl$MultiIdentifierLoadAccessImpl.perform(SessionImpl.java:2931) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.internal.SessionImpl$MultiIdentifierLoadAccessImpl.multiLoad(SessionImpl.java:2950) ~[hibernate-core-5.4.8.Final.jar!/:5.4.8.Final]
	at org.hibernate.search.mapper.orm.search.loading.impl.HibernateOrmByIdEntityLoader.loadEntities(HibernateOrmByIdEntityLoader.java:121) ~[hibernate-search-mapper-orm-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.mapper.orm.search.loading.impl.HibernateOrmByIdEntityLoader.loadBlocking(HibernateOrmByIdEntityLoader.java:53) ~[hibernate-search-mapper-orm-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.engine.search.loading.spi.DefaultProjectionHitMapper.loadBlocking(DefaultProjectionHitMapper.java:42) ~[hibernate-search-engine-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.backend.elasticsearch.search.query.impl.ElasticsearchLoadableSearchResult.loadBlocking(ElasticsearchLoadableSearchResult.java:60) ~[hibernate-search-backend-elasticsearch-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.backend.elasticsearch.search.query.impl.ElasticsearchSearchQueryImpl.fetch(ElasticsearchSearchQueryImpl.java:127) ~[hibernate-search-backend-elasticsearch-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.backend.elasticsearch.search.query.impl.ElasticsearchSearchQueryImpl.fetch(ElasticsearchSearchQueryImpl.java:41) ~[hibernate-search-backend-elasticsearch-6.0.0.Beta7.jar!/:6.0.0.Beta7]
	at org.hibernate.search.engine.search.query.dsl.spi.AbstractSearchQueryOptionsStep.fetch(AbstractSearchQueryOptionsStep.java:168) ~[hibernate-search-engine-6.0.0.Beta7.jar!/:6.0.0.Beta7]

What are the possible scenarios for this to trigger?

The most likely explanation of your problem is incomplete configuration of HikariCP. I have absolutely no experience with that, but this stackoverflow answer suggests that HikariCP detects leaks by monitoring how long connections stay open, so if you set it to a low number (2s) and you use connections for longer than that (for example loading thousands of entities) you’ll probably get this exception.

By the way, if you’re trying to fetch all entities at once: you shouldn’t. Always use fetch with a limit, e.g. fetch(20), and avoid fetch(null) or fetchAll()/fetchAllHits() which will trigger the loading of all matching entities, of which there could be thousands: this will perform very badly.
If you really need to fetch all matching hits, use projections to fetch directly from Lucene/Elasticsearch and avoid the database roundtrip, or rely on pagination combined with session.clear() to fetch entities in smaller chunks. At some point you will also have the option of scrolling but that’s not implemented yet in Hibernate Search 6.0.0.Beta8.

You should also check that you properly close all Hibernate ORM sessions/entitymanagers that you opened yourself The ones injected by your framework (Quarkus/WildFly/Spring/…) should not require explicit closing, but the ones you created through a call to EntityManagerFactory.createEntityManager/SessionFactory.openSession (if any) probably do.

You should also make sure you’re using up-to-date versions of Hibernate Search, ORM and HikariCP. I see your version of Hibernate ORM (5.4.8.Final) is relatively old, so you should try upgrading: maybe it’s a bug that has been solved in recent versions.

You can also try upgrading to the latest version of Hibernate Search, 6.0.0.Beta8, released today. It changes the way entities are retrieved from Hibernate ORM: instead of using Session.byId it simply uses queries. I’d be surprised if it changed anything, but in the (unlikely) event there is a bug in Session.byId that triggers a connection leak, this may solve your problem.

@yrodiere Yes we always fetch with a limit like 10 or 20, or 0 when we only need aggregations.
We have already configured the HikariCP leak detection threshold to a very high value like 30 minutes or so.
Will try upgrading to latest versions of the dependency to see if it makes any difference.