HS 6.0 Exception While Scrolling

Hibernate Search
1

I’m encountering an exception trying to execute the following in HS 6.0. It makes a number of passes through the for loop before excepting out on line 512 (marked by comment). I don’t see a major delta between this and the scroll example in the documentation. If anyone can spot the issue or if this is a known HS6 bug I’d appreciate the heads up. Note. Using pagination on these these same classes works without issue. Thanks, Keith

Code snippet:

      List<PatentDocSearchResultDO> docResults = new ArrayList<>();
      PatentDocSearchResultListDO resultList = new PatentDocSearchResultListDO();
      try (
         SearchScroll<PatentDocSearchResultDO> scroll = Search.session(entityManager)
            .search(PatentDocDO.class).extension(LuceneExtension.get())
            .select(f -> f.composite(PatentDocSearchResultDO::new, f.score(), f.entity()))
            .where(f -> f.bool((b -> {
               addSearchClausesFromCriteria(f, b,criteria);
            })))
           .scroll(CHUNK_SIZE)) {        
            for (SearchScrollResult<PatentDocSearchResultDO> chunk = scroll.next();
                  chunk.hasHits(); chunk = scroll.next() ) {  //This is line 512 in the exception
               for ( PatentDocSearchResultDO hit : chunk.hits() ) { 
                  docResults.add(hit);
               }
               resultList.setEstResultCount((int) chunk.total().hitCount());
            }
            resultList.setDocResults(docResults);
            highlightTitleDuringSearch(criteria, resultList.getDocResults());
            return resultList;
      }

Exception

[dispatcherServlet]                ERROR  2021-02-04 17:31:44,995  [http-nio-8080-exec-5]
 org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]
   Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.InvalidDataAccessApiUsageException: org.hibernate.QueryException: Unrecognized parameter label : ids_84; nested exception is java.lang.IllegalArgumentException: org.hibernate.QueryException: Unrecognized parameter label : ids_84] with root cause
org.hibernate.QueryException: Unrecognized parameter label : ids_84
        at org.hibernate.loader.hql.QueryLoader.getNamedParameterLocs(QueryLoader.java:619)
        at org.hibernate.loader.Loader.buildNamedParameterLocMap(Loader.java:1130)
        at org.hibernate.loader.Loader.createSubselects(Loader.java:1092)
        at org.hibernate.loader.Loader.processResultSet(Loader.java:1009)
        at org.hibernate.loader.Loader.doQuery(Loader.java:959)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
        at org.hibernate.loader.Loader.doList(Loader.java:2850)
        at org.hibernate.loader.Loader.doList(Loader.java:2832)
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664)
        at org.hibernate.loader.Loader.list(Loader.java:2659)
        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
        at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
        at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
        at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
        at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
        at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
        at org.hibernate.query.Query.getResultList(Query.java:165)
        at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:76)
        at org.hibernate.search.mapper.orm.search.loading.impl.HibernateOrmEntityIdEntityLoader.doLoadEntities(HibernateOrmEntityIdEntityLoader.java:144)
        at org.hibernate.search.mapper.orm.search.loading.impl.HibernateOrmEntityIdEntityLoader.loadBlocking(HibernateOrmEntityIdEntityLoader.java:69)
        at org.hibernate.search.engine.search.loading.spi.DefaultProjectionHitMapper.loadBlocking(DefaultProjectionHitMapper.java:49)
        at org.hibernate.search.backend.lucene.search.query.impl.LuceneLoadableSearchResult.loadBlocking(LuceneLoadableSearchResult.java:73)
        at org.hibernate.search.backend.lucene.search.query.impl.LuceneSearchScrollImpl.doNext(LuceneSearchScrollImpl.java:127)
        at org.hibernate.search.backend.lucene.search.query.impl.LuceneSearchScrollImpl.next(LuceneSearchScrollImpl.java:83)
        at org.hibernate.search.backend.lucene.search.query.impl.LuceneSearchScrollImpl.next(LuceneSearchScrollImpl.java:28)
        at com.propat.pas.db.core.PatentDocRepositoryImpl.findUsingCriteria(PatentDocRepositoryImpl.java:512)
        at com.propat.pas.db.core.PatentDocRepositoryImpl$$FastClassBySpringCGLIB$$d6bb6b84.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.propat.pas.db.core.PatentDocRepositoryImpl$$EnhancerBySpringCGLIB$$6db2add3.findUsingCriteria(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:289)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:524)
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:285)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:531)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:156)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy217.findUsingCriteria(Unknown Source)
        at com.propat.pas.svc.core.PatentDocService.searchPatentDocs(PatentDocService.java:111)
        at com.propat.pas.svc.core.PatentDocService.serviceSearchPatentDocs(PatentDocService.java:190)
        at com.propat.pas.svc.core.PatentDocService$$FastClassBySpringCGLIB$$ed98a44f.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
        at com.propat.pas.svc.core.PatentDocService$$EnhancerBySpringCGLIB$$f6a2af14.serviceSearchPatentDocs(<generated>)
        at com.propat.pas.mvc.search.PatentDocController.searchScoredSubmit(PatentDocController.java:584)
        at com.propat.pas.mvc.search.PatentDocController$$FastClassBySpringCGLIB$$6ef63c51.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.propat.pas.mvc.search.PatentDocController$$EnhancerBySpringCGLIB$$1158624e.searchScoredSubmit(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1061)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:289)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:289)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:115)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)
2

First, I see you’re essentially retrieving the whole result list and storing everything in the same list. Scrolling won’t help you much here: it’s supposed to help you process data in chunks, releasing memory after each chunk, which you’re obviously not doing here.
I’d suggest using .fetchAllHits() instead of scroll() if your intent really is to fetch the whole result list. Whatever approach you choose, be warned that you may hit an OutOfMemoryError if your dataset is too large.

Now about the exception… it definitely looks like a bug. I’m investigating and will update you.

3

I confirm it is a bug: [HSEARCH-4150] Entity loading fails when using FetchMode.SUBSELECT - Hibernate JIRA

I’ll try to come up with a fix ASAP.

4

I have a fix, I’ll try to release it soon.

In the meantime, as a workaround, you can set the fetch size to at least CHUNK_SIZE, e.g.:

      List<PatentDocSearchResultDO> docResults = new ArrayList<>();
      PatentDocSearchResultListDO resultList = new PatentDocSearchResultListDO();
      try (
         SearchScroll<PatentDocSearchResultDO> scroll = Search.session(entityManager)
            .search(PatentDocDO.class).extension(LuceneExtension.get())
            .select(f -> f.composite(PatentDocSearchResultDO::new, f.score(), f.entity()))
            .where(f -> f.bool((b -> {
               addSearchClausesFromCriteria(f, b,criteria);
            })))
           .loading(o -> o.fetchSize(CHUNK_SIZE)) // ADD THIS
           .scroll(CHUNK_SIZE)) {
...

See also this section of the documentation.

5

Thanks for the workaround. Yeah, what you have in the code I submitted is an incomplete example of what I am porting. I have cases in which some of our services may need to operate on upwards of 25K docs at a time. In HS 5.5 I found that fetching that many docs in one shot to be extremely slow (this is all likelihood an artifact of Hibernate and not HS). So, instead I have a method implemented that returns the whole list, but pages in a chunk at a time internally to build the result set (limited to 25K). That sped things up dramatically. Once I saw HS 6 had this scrolling support, it occurred to me I can do that same thing without the downsides of paging, so I started implementing that when I ran into this issue. In the end, the final implementation will limit the max documents I scroll through. Having said all of this I should go back and test the performance of the one shot query again to see if I am still seeing the same behavior. Thanks again.

6

Definitely: the “fetch size” I was referring to does exactly what you want, i.e. it breaks down ORM entity loading into multiple query executions, by default 100 elements at a time. Even for “normal” (non-scrolling) fetches.

Though of course that won’t work in your case until [HSEARCH-4150] - Hibernate JIRA gets fixed :slight_smile:

7

For anyone else having this problem: this was fixed in 6.0.1.Final.

8

Excellent. I will upgrade tomorrow. Thanks for the update!