Beta8: getAllClassMetadata causes CancellationException during startup

Hibernate Search Team, please let me know how I can help troubleshoot this issue. Thank you!

I am working with the latest Hibernate Search 6 Beta8 release, and am having trouble getting our application to start up after migration from Hibernate Search 5. The stack trace goes no deeper than the CancellationException, but I can see that the exception is triggered by our call to getAllClassMetadata on SessionFactory, which ends in a RollbackException, caused by the CancellationException on an Executor thread. In the Executor thread, Hibernate Search is at line 135 in HibernateSearchEventListener.java:


	/**
	 * Make sure the indexes are updated right after the hibernate flush,
	 * avoiding entity loading during a flush. Not needed during transactions.
	 */
	@Override
	public void onFlush(FlushEvent event) {
		HibernateOrmListenerContextProvider contextProvider = state.getContextProvider();  <---135
		EventSource session = event.getSession();

Stack trace:

Stack Dump = javax.transaction.RollbackException
	at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(TransactionImpl.java:980)
	at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:778)
	at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:711)
	at com.ibm.tx.jta.impl.TranManagerImpl.commit(TranManagerImpl.java:165)
	at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:113)
	at com.ibm.ejs.csi.TranStrategy.commit(TranStrategy.java:864)
	at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:186)
	at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:481)
	at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:3686)
	at net.paradisesoftware.config.slb.EJSLocalNSLConfigBean_160ca7c0.getAllClassMetadata(EJSLocalNSLConfigBean_160ca7c0.java)
	at net.paradisesoftware.base.service.BaseBootstrapService.initialize(BaseBootstrapService.java:88)
	at flex.messaging.config.MessagingConfiguration.createServices(MessagingConfiguration.java:381)
	at flex.messaging.config.MessagingConfiguration.configureBroker(MessagingConfiguration.java:128)
	at flex.messaging.MessageBrokerServlet.init(MessageBrokerServlet.java:133)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:291)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1373)
	at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:1181)
	at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:1149)
	at com.ibm.ws.webcontainer.webapp.WebApp.initialize(WebApp.java:1047)
	at com.ibm.ws.webcontainer.webapp.WebApp.initialize(WebApp.java:6647)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost.startWebApp(DynamicVirtualHost.java:467)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost.startWebApplication(DynamicVirtualHost.java:462)
	at com.ibm.ws.webcontainer.osgi.WebContainer.startWebApplication(WebContainer.java:1150)
	at com.ibm.ws.webcontainer.osgi.WebContainer.startModule(WebContainer.java:947)
	at com.ibm.ws.app.manager.module.internal.ModuleHandlerBase.deployModule(ModuleHandlerBase.java:99)
	at com.ibm.ws.app.manager.module.internal.DeployedModuleInfoImpl.installModule(DeployedModuleInfoImpl.java:49)
	at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.deployModules(SimpleDeployedAppInfoBase.java:597)
	at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.installApp(SimpleDeployedAppInfoBase.java:511)
	at com.ibm.ws.app.manager.module.internal.DeployedAppInfoBase.deployApp(DeployedAppInfoBase.java:347)
	at com.ibm.ws.app.manager.ear.internal.EARApplicationHandlerImpl.install(EARApplicationHandlerImpl.java:76)
	at com.ibm.ws.app.manager.internal.statemachine.StartAction.execute(StartAction.java:144)
	at com.ibm.ws.app.manager.internal.statemachine.ApplicationStateMachineImpl.enterState(ApplicationStateMachineImpl.java:1295)
	at com.ibm.ws.app.manager.internal.statemachine.ApplicationStateMachineImpl.run(ApplicationStateMachineImpl.java:886)
	at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.CancellationException
	at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
	at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.lambda$orchestrateBootAndShutdown$3(HibernateOrmIntegrationBooterImpl.java:178)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire$$$capture(CompletableFuture.java:783)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.lambda$orchestrateBootAndShutdown$1(HibernateOrmIntegrationBooterImpl.java:140)
	at org.hibernate.search.mapper.orm.bootstrap.impl.ExtendedBeanManagerSynchronizer.beanManagerInitialized(ExtendedBeanManagerSynchronizer.java:36)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.ibm.ws.jpa.container.v21.cdi.internal.IBMHibernateExtendedBeanManager$1.run(IBMHibernateExtendedBeanManager.java:82)
	at com.ibm.ws.jpa.container.v21.cdi.internal.IBMHibernateExtendedBeanManager$1.run(IBMHibernateExtendedBeanManager.java:76)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at com.ibm.ws.jpa.container.v21.cdi.internal.IBMHibernateExtendedBeanManager.notifyHibernateAfterBeanDiscovery(IBMHibernateExtendedBeanManager.java:76)
	at com.ibm.ws.jpa.container.v21.cdi.internal.CDIJPAEMFPropertyProviderImpl.notifyHibernateAfterBeanDiscovery(CDIJPAEMFPropertyProviderImpl.java:118)
	at com.ibm.ws.jpa.container.v21.cdi.internal.JPAContainerCDIExtension.afterDeploymentValidation(JPAContainerCDIExtension.java:66)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95)
	at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:187)
	at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330)
	at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:123)
	at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308)
	at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286)
	at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124)
	at org.jboss.weld.util.Observers.notify(Observers.java:166)
	at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285)
	at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273)
	at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177)
	at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:171)
	at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53)
	at org.jboss.weld.bootstrap.events.AbstractDeploymentContainerEvent.fire(AbstractDeploymentContainerEvent.java:35)
	at org.jboss.weld.bootstrap.events.AfterDeploymentValidationImpl.fire(AfterDeploymentValidationImpl.java:28)
	at org.jboss.weld.bootstrap.WeldStartup.validateBeans(WeldStartup.java:505)
	at org.jboss.weld.bootstrap.WeldBootstrap.validateBeans(WeldBootstrap.java:93)
	at com.ibm.ws.cdi.impl.CDIContainerImpl.startInitialization(CDIContainerImpl.java:157)
	at com.ibm.ws.cdi.liberty.CDIRuntimeImpl.applicationStarting(CDIRuntimeImpl.java:453)
	at com.ibm.ws.container.service.state.internal.ApplicationStateManager.fireStarting(ApplicationStateManager.java:51)
	at com.ibm.ws.container.service.state.internal.StateChangeServiceImpl.fireApplicationStarting(StateChangeServiceImpl.java:50)
	at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.preDeployApp(SimpleDeployedAppInfoBase.java:547)
	at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.installApp(SimpleDeployedAppInfoBase.java:508)
	... 9 more

Hello,

The error message is not very clear, I must admit. I’ll work on that.

But the actual problem is there is a bug in org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl#orchestrateBootAndShutdown: the destroy event is sent on startup, and vice-versa.
It wasn’t caught earlier because the “environment synchronizer” is only used in application servers, which haven’t been tested until now.

I opened HSEARCH-3938. This will be fixed in the next release.

In the meantime, a workaround may be for you to create a class named HibernateOrmIntegrationBooterImpl in package org.hibernate.search.mapper.orm.bootstrap.impl, copy the code from Hibernate Search and fix the method orchestrateBootAndShutdown. Assuming your application server uses a flat classloader, this may override the code from Hibernate Search and allow you to test further.

EDIT: Fixes are inbound: https://github.com/hibernate/hibernate-search/pull/2299

Yoann, you are awesome, your service level is impeccable. Thank you!

1 Like

Thank you for reporting this!

FYI this was fixed in Hibernate Search 6.0.0.Beta9: https://in.relation.to/2020/08/03/hibernate-search-6-0-0-Beta9/

Sorry for the delay… I hope you were able to rely on snapshots or the hack I mentioned in the meantime :slight_smile: