MassIndexer says indexing completed but it's not completed

Hibernate Search
1

I am trying to index large data(13.5 million records with association to 7-8 tables) in elastic search using MassIndexer. It shown message that it indexed all the records but before that line it printed the progress as 39.08%. I am getting the same issue in local and production, the percentage varies for every execution.

my code:

fullTextEntityManager
				.createIndexer(XYZ.class)
				.batchSizeToLoadObjects(500).cacheMode(CacheMode.IGNORE).threadsToLoadObjects(2).idFetchSize(Integer.MIN_VALUE)
				.startAndWait();

console:

23:05:25,338 (Hibernate Search: Elasticsearch transport thread-2)  INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.105591 documents/second; progress: 39.08%
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2)  INFO SimpleIndexingProgressMonitor:87 - HSEARCH000030: 5322450 documents indexed in 4904960 ms
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2)  INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.115845 documents/second; progress: 39.08%
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2)  INFO SimpleIndexingProgressMonitor:87 - HSEARCH000030: 5322500 documents indexed in 4904961 ms
23:05:25,339 (Hibernate Search: Elasticsearch transport thread-2)  INFO SimpleIndexingProgressMonitor:90 - HSEARCH000031: Indexing speed: 1085.125854 documents/second; progress: 39.08%
23:05:36,103 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_forcemerge' with query parameters {} in 16734ms. Response had status 200 'OK'.
23:05:37,666 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_flush' with query parameters {} in 1562ms. Response had status 200 'OK'.
23:05:37,668 (Hibernate Search: Elasticsearch transport thread-3) DEBUG request:194 - HSEARCH400082: Executed Elasticsearch HTTP POST request to path '/xyz/_refresh' with query parameters {} in 1ms. Response had status 200 'OK'.
23:05:37,668 (main)  INFO SimpleIndexingProgressMonitor:78 - HSEARCH000028: Reindexed 13618954 entities

Versions:
Hibernate ORM:5.4.1.Final
Hibernate Search: 5.11.1.Final

2

This looks a lot like HSEARCH-3462, which was fixed in 6.0.0.Alpha2 but wasn’t backported to 5.11.

Long story short: it’s a logging problem, not an indexing problem. The last line that states that everything was reindexed is the one you should believe.

I will see if we can easily backport the fix to 5.10/5.11, but it might take some time until we release these branches again.

3

Seems like this was crossposted to Stackoverflow: https://stackoverflow.com/questions/56678524/why-hibernate-massindexer-says-indexing-completed-but-really-it-isnt-completed/56680120#56680120

Closing this, let’s continue the discussion there if necessary.