MassIndexer consistently stops at 684,000

Hibernate Search
1

When using the MassIndexer to index approximately 2.5 million “incidents”, it consistently stops at exactly 684,000. But, the indexer reports success. The index only contains 684,000 entries. Threads and Heap are not a problem. There are 300 connections available in the dB pool. I am using JPA and here is my code:

@TransactionAttribute(TransactionAttributeType.NEVER)
	public void index() {
		MassIndexerProgressMonitor monitor = new IndexingProgressMonitor();
		FullTextEntityManager fullTextEntityManager = getFullTxtEntityManager(em);
				
		try {
			MassIndexer massIndexer = fullTextEntityManager.createIndexer(Incident.class);
			
			massIndexer.purgeAllOnStart(true)    //default
					.optimizeAfterPurge(true)    //default
					.optimizeOnFinish(true)      //default
					.threadsToLoadObjects(10)
					.batchSizeToLoadObjects(100)
					.idFetchSize(1000)
					.progressMonitor(monitor)
					.cacheMode(CacheMode.IGNORE) //default
					.startAndWait();
		} catch (Exception e) {
			log.log(Level.SEVERE, "MassIndexer failed. See log for details.", e);
		} finally {
			fullTextEntityManager.flushToIndexes();
			fullTextEntityManager.close();
		}
	}

I will appreciate any suggestions.

2

There are many possible causes for mass indexing failures, so the most pressing matter here is to get information about what went wrong.

Did you check your logs? It’s possible that one of the (many) threads failed and died with an error message in the logs.

Also:

  • Please check that you are using the default error handler and that logs are not silenced for the logger named org.hibernate.search.exception.impl.LogErrorHandler. For example run something like this just before indexing, and check that the message appears in your logs:
    org.jboss.logging.Logger.getMessageLogger(
    org.hibernate.search.util.logging.impl.Log.class,
    org.hibernate.search.exception.impl.LogErrorHandler.class.getName()
).exceptionOccurred("THIS IS A TEST", new RuntimeException());
  • Can you reproduce the issue with a different database vendor? E.g. did you try h2, or some other database that is simple enough to set up?

If none of the above shows the source of the problem… well, we’ll need more information. Please at least provide:

  • The versions of Search and ORM you are using
  • A list of the various Hibernate Search integrations you are using (ES integration or Lucene? JGroups/JMS backend or not? …)
  • The logs of indexing (preferably set at the DEBUG level for org.hibernate and TRACE for org.hibernate.search, but lower it as necessary if it’s spitting out gigabytes of logs)
  • The vendor and version of the database you’re using
  • The framework you’re using (JavaEE, I assume? WildFly, maybe?)
  • The source code of your Incident class and any class that is @IndexedEmbedded from Incident, if possible.
3

Thank you for you quick response and advice. I will give your suggestions a try.

4

Using the logging as suggested, showed that I was running into a data problem. To be honest, I didn’t even know that Hibernate had its own logging. I sincerely appreciate the time and attention given to solve my problem.

5

No problem. Glad I could help!