Hibernate MassIndexer on Unstable DB Server

Hibernate Search
2

There are multiple questions here. I’ll start with the easiest one:

Detecting loading failures during searches

during the search functionality of our application, while running search on the index, the server disconnected and the results can not be retrieved

So from what I understand, the database disconnects while Hibernate Search is retrieving the entities from the database. Hibernate Search is not designed to recover from such failures. Depending on what fails exactly, it will throw an exception. Can’t you just catch this exception and return empty results accordingly?

If there’s, somehow, a problem with that strategy, another option is to ask Hibernate Search to only retrieve entity IDs (from the index), then do the loading yourself. Then you’ll be able to handle loading errors as you see fit.
To that end, just use projections: Hibernate Search 5.11.12.Final: Reference Guide

FullTextQuery query = fullTextEntityManager.createFullTextQuery(luceneQuery, MyEntity.class);
query.setProjection(ProjectionConstants.ID);
List<Object[]> idProjections = query.list();

// Then load the corresponding entities
List<Serializable> ids = new ArrayList<>( idProjections.size() );
for ( Object[] projection : idProjections ) {
    ids.add( (Serializable) projection[0] );
}
try {
    return entityManager.unwrap( Session.class ).byMultipleIds( clazz )
        .with( CacheMode.<pick a cache mode> ) // May be ommitted
        .withBatchSize( <pick a batch size> ) // May be ommitted
        .multiLoad( ids );
}
catch (RuntimeException e) {
   // Something went wrong when loading, handle this here.
}

I’m not sure why you would need that, though, since Hibernate Search should also throw an exception when loading fails.

Merging results from multiple searches

for (DatabaseSearchHandler handler : searchHandlers)

I see you’re running multiple queries on multiple servers and then trying to merge the results back into a single list. Be warned: if you want to use paging (and you likely will at some point), merging the results back into a single list will not be an easy task. What you’re doing will only be easy if you just want to retrieve the top results.

Detecting MassIndexing failure

We would like to find a way to identify that the index that we are using completed correctly and not broken

You could use an ErrorHandler to detect any error happening during indexing. The default one just logs errors, but you can plug in your own behavior.

Implement the dedicated interface:

package com.mycompany;

public class MyErrorHandler implements ErrorHandler {
	public static final LongAdder ERROR_COUNT = new LongAdder();

	public static void resetErrorCount() {
		ERROR_COUNT.reset();
	}

	public static void getErrorCount() {
		ERROR_COUNT.longValue();
	}

	@Override
	public void handle(ErrorContext context) {
		// Handle indexing errors, typically I/O errors while writing to the index.
		
		// Don't forget to log with your logging framework. There's more information in the context if you want a better message.
		logger.error( "Error while indexing", context.getThrowable() );

		// Increment the error count: 
		errorCount.increment();
	}

	@Override
	public void handleException(String errorMsg, Throwable exception) {
		// Handle other errors, typically errors while getting information from the database.
		
		// Don't forget to log with your logging framework.
		logger.error( errorMsg, context.getThrowable() );

		// Increment the error count: 
		ERROR_COUNT.increment();
	}
}

Then reference your implementation in the configuration:

hibernate.search.error_handler com.mycompany.MyErrorHandler

Then in your code:

FullTextEntityManager fullTextEntityManagerL = Search.getFullTextEntityManager(em);
MyErrorHandler.resetErrorCount();
fullTextEntityManagerL.createIndexer(classes)
        .batchSizeToLoadObjects(50)
        .threadsToLoadObjects(5)
        .typesToIndexInParallel(2)
        .idFetchSize(15)
        //.progressMonitor(monitor)
        .transactionTimeout(1800)
        .startAndWait();
if ( MyErrorHandler.getErrorCount() > 0 ) {
    // Something went wrong
}

Recovering from MassIndexing failure

In case it is broken we would like to use previous successfully completed index and restart the task to try again in 1 hour.

That’s… tough. The MassIndexer only works on one copy of the index, so it will completely purge the existing index on starts. So if it fails, the index is gone.
Your only solution would be to run the mass indexer in a separate application with a separate index location, and copy the index directory to your main application if indexing succeeds.

Before you copy the index directory, if you’re using the Lucene integration, don’t forget to tell Hibernate Search to release all locks:

EntityManagerFactory entityManagerFactory = ...; // this can be either injected with @PersistenceUnit, or retrieved from an entity manager using .getEntityManagerFactory
SearchIntegrator searchIntegrator = SearchIntegratorHelper.extractFromEntityManagerFactory( entityManagerFactory );
for ( EntityIndexBinding value : searchIntegrator.getIndexBindings().values() ) {
    for ( IndexManager indexManager : value.getIndexManagerSelector().all() ) {
        // Flush pending index changes and release lock
        indexManager.flushAndReleaseResources();
    }
}

Be aware that this code uses SPIs, meaning you might experience incompatible changes in minor releases of Hibernate Search. If you’re ready to update your code when upgrading Hibernate Search, that should not be a problem.

If you run into index corruption, read this thread, which is fairly close from your use case and explains a few things: Lucene index corruption