MassIndexer parallel run with working indexes

sandersru · October 9, 2023, 1:47pm

Hi everyone!
I start to use hibernate search and have case what need to solve.

I have indexed database with 1TB of indexes in open search and need to rebuild indexes and use new only when build finished. Any possibility to do this over hibernate search MassIndexer or something else?

Yoann,
maybe your can help

yrodiere · October 9, 2023, 2:07pm

Hi,

Yes it’s possible within certain limits (no large, incompatible changes in your mapping in particular).

I’m pretty sure this question has already been asked (and answered) here or on stackoverflow, but anyway there’s a tip about that in the documentation (though it’s a bit hidden): Hibernate Search 6.2.2.Final: Reference Documentation

We do plan to provide this as a builtin feature in a future version (don’t ask me when): [HSEARCH-4954] - Hibernate JIRA

sandersru · October 9, 2023, 2:18pm

Thanks for quick answer!

We do plan to provide this as a builtin feature in a future version (don’t ask me when)

As usual with mulitask work

yrodiere · November 6, 2023, 10:20am

FWIW I implemented something in one of my applications to handle no-downtime mass indexing on the user side with Elasticsearch/OpenSearch:

github.com

quarkusio/search.quarkus.io/blob/main/src/main/java/io/quarkus/search/app/indexing/Rollover.java

package io.quarkus.search.app.indexing;

import java.io.Closeable;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.function.Function;
import java.util.stream.Stream;

import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;

This file has been truncated. show original

You can see it used here:

github.com

quarkusio/search.quarkus.io/blob/main/src/main/java/io/quarkus/search/app/indexing/IndexingService.java

package io.quarkus.search.app.indexing;

import java.io.IOException;
import java.util.Iterator;
import java.util.concurrent.atomic.AtomicBoolean;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.context.control.ActivateRequestContext;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import org.elasticsearch.client.Request;
import org.elasticsearch.client.RestClient;

import org.hibernate.Session;
import org.hibernate.search.backend.elasticsearch.ElasticsearchBackend;
import org.hibernate.search.mapper.orm.mapping.SearchMapping;
import org.hibernate.search.util.common.impl.Closer;

import io.quarkus.logging.Log;

This file has been truncated. show original

sandersru · November 9, 2023, 7:55am

Hi Yoann,
thanks. Will check soon in my production and return back.

sandersru · November 24, 2023, 9:51pm

Hi Yoann!
You idea stolen and now working fine in production. Many-many-many thanks for help!
See you in Quarkus git and chat as before few times

Topic		Replies	Views
Re-indexing on PRODUCTION site Hibernate Search	5	964	March 1, 2019
Index creation (Hibernate Search 5.11) (mass indexer) taking a long time Hibernate Search	4	748	February 14, 2022
Issues indexing large dataset with mass indexer Hibernate Search	5	996	January 7, 2021
Hibernate MassIndexer on Unstable DB Server Hibernate Search	4	968	August 28, 2019
Writing Tests that Exercise Hibernate Search Hibernate Search	4	596	November 22, 2022

MassIndexer parallel run with working indexes

Related Topics