Hibernate search and elastics search are on a boat

Hibernate Search
1

But this boat don’t navigate together very well…

Hi all, i have see this video https://www.youtube.com/watch?v=pQJY-Mu-yGA

In an old application we have a lot of function on postgresql to make some statistics and this take very long time.
With this video we can take the powerfull of hibernate search (spring data + hibernate + jpa repository) to index directly some data in elastics search -> that’s works very well ! thank a lot !

But we need to retrieve some statistics (search, agregation, filter) to display some graph, i will use the ElasticsSearch REST HighLevel Http client for that

https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client

But i have a conflict in maven dependencies. A old version (ES HTTP client: 5.6.7 include elastic search 2.4.5, Lucenne 5.5.4 -> java.lang.NoClassDefFoundError: org/apache/lucene/document/Field$TermVector) of ElasticSearch Http Client is used for hibernate Search with an old lucene version. Can you update the dependencies in the next version for be able to use the power of hibernate search and elastics search (query to retrieve entity with your’s project and be able to extract data statistics with elastics search REST api).

Thank’s,

I don’t have writte a message in english since a long time ago, sorry if my english is not perfect.

2

Hi @alexis-puska ,

So, the big issue is that the Elasticsearch high level rest client depends on Elasticsearch itself. This is unfortunate.

The other unfortunate thing is that we won’t be able to upgrade the Lucene dependency of Search for now. We are working on Search 6 but it’s still in very early stages.

That being said, I wonder if you could make it work by excluding the Lucene dependency brought by the Elasticsearch client (not the one coming from Hibernate Search, of course). From what I see, the client is just using the Elasticsearch dependency to easily deserialize the JSON in Java objects so it might not use Lucene at all.

Try that and come back to us, I hope it will help.

3

I will try this on the next week, i keep you in touch.

4

I have had some time to make this test. I have exclude of the elastic package the dependencies to Lucene. But not works.

org.hibernate hibernate-search-orm 5.8.1.Final org.hibernate hibernate-search-elasticsearch 5.8.1.Final 
	<dependency>
	    <groupId>org.elasticsearch</groupId>
	    <artifactId>elasticsearch</artifactId>
	    <version>5.6.7</version>
	    <exclusions>
			<exclusion>
			    <groupId>org.apache.lucene</groupId>
		    	<artifactId>lucene-core</artifactId>
		    </exclusion>
		    <exclusion>
				<groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-analyzers-common</artifactId>
			</exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-backward-codecs</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-grouping</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-highlighter</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-join</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-memory</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-misc</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-queries</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-queryparser</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-sandbox</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-spatial</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-spatial-extras</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
			    <artifactId>lucene-spatial3d</artifactId>
		    </exclusion>
		    <exclusion>
			    <groupId>org.apache.lucene</groupId>
				<artifactId>lucene-suggest</artifactId>
		    </exclusion>
		</exclusions>
	</dependency>
	<dependency>
	    <groupId>org.elasticsearch.client</groupId>
	    <artifactId>elasticsearch-rest-high-level-client</artifactId>
	    <version>5.6.7</version>
	</dependency>

Caused by: java.lang.NoSuchFieldError: LUCENE_6_0_0

The only solution for me that to reimplement all your work make in the dependencies hibernate search elastic search. (rebuild index, document indexation on JPA event (insert/delete/update), create index on startup if don’t exists, get id for full text search… or wait for the update of Lucene inside hibernateSearch.

One way hibernate search with elastic search, but the only way to use aggregation is the low level reste api
The second way, implement your works and use high rest client api to request for aggregation.

Or see with the team of elastic search for an Rest api without any Lucene dependencies.

5

I have found a very ugly workaround, exclude all dependencies of Lucene is not enough. I have found an old version of the class Version or.apache.lucene.Version

I had a function missing and some static version number in this class definition, i have had this and finally the webApp works, finally the webApp works. HighRestClient object can be instantiate!

In waiting for other solution i think this can be able to continue to develop quickly.

I have send a message to the elasticsearch team to fix this high relation with lucene in this class. This library doesn’t need Lucene to communicate with elasticsearch for me…

Thank’s for the start of solution :slight_smile:

6

@alexis-puska could you try to share a project based on https://github.com/hibernate/hibernate-test-case-templates/tree/master/search/hibernate-search-elasticsearch/hibernate-search-elasticsearch-5 where we can see the issue?

I think your use case is quite common and it would be nice if we could find a proper solution.

Thanks for sharing.

7

I couldn’t find @alexis-puska 's message to the Elasticsearch team regarding the dependency problem, so reported it on their GitHub issue tracker: https://github.com/elastic/elasticsearch/issues/29184
Let’s see how it goes.
cc @gsmet

8

To anyone ending up here because they are affected by this problem: Hibernate Search 6 more cleanly splits the dependencies, and as a result you no longer need to hack you way into using the high level REST Client: there won’t be any dependency conflict anymore.

9

I see that, i made a simple project with pring boot 2.2, hibernate search 6.0 beta and the rest high level api from elasticsearch 7, no conflicts, the data are automaticly indexed in ES and we can request with the rest high level api to create aggregation with search aprameter. I wait now to jhipster for server side update, currently based on spring boot 1.X…)

A great job have been made since this ticket in the both side. Hibernate search for include communication and indexing at every jpa event, elastic search for refactoring server and delete dependencies with lucene inside the rest high api.

10

Are you sure about that one? I thought it was still a pending issue?

11

As far as I know the problem wasn’t solved on the Elasticsearch side. It just happens that Hibernate Search 6 no longer depends on Lucene when you only use the Elasticsearch integration, so the Elasticsearch client is free to depend on whichever version of Lucene it wants.

12

Hi @yrodiere , Iam getting the below exception when iam trying to use elasticsearch high level rest client , instead of hibernate search

Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/document/Field$TermVector
at org.hibernate.search.engine.impl.AnnotationProcessingHelper.getTermVector(AnnotationProcessingHelper.java:123)

i tried shading also and all other things but nothing is working , can someone provide me the quicker solution

13

Hibernate Search 5 and Elasticsearch’s High Level Rest client depend on different versions of Lucene. That’s the reason for this exception. I do not know a quick solution to that problem.

Ideally Elasticsearch would remove the dependency of their client to Lucene, because they don’t really need it in theory, but in practice it would be quite hard.

The only reasonable solution, as far as I know, is to upgrade to Hibernate Search 6. Which, indeed, represents a fair amount of work (though there are some nice benefits too). You can find a migration guide here.

You may be able to downgrade the Elasticsearch High Level client to some version that uses the same version of Lucene as Hibernate Search 5, but that would mean downgrading your ES cluster as well, which is definitely not a good idea.