We have started POC to implement Hibernate Search in our current transaction banking project.
Main entity which is used for data persistence is quite heavy with multiple associations.
Find on this entity generates a select query with 828 fields in select clause, 30 tables in from clause.
This table contains 8.5 million records.
Now, when I tried to create initial indexing using Mass Indexer, it results in Out of Memory exception with 2GB of heap space allocation to application.
Its not even able to index 5000 records.
.batchSizeToLoadObjects( 5 )
//.cacheMode( CacheMode.NORMAL )
.threadsToLoadObjects( 5 )
.idFetchSize( 15 )
.transactionTimeout( 1800 )
//.progressMonitor( monitor ) //a MassIndexerProgressMonitor implementation
It seems that its not flusing documents to the lucene index directly until complete process is not completed.
How to create initial indexing with this much of data?