Dynamic indexes per multiple entity managers

Hibernate Search
#1

My application uses multiple datasources (all with the same structure) for different clients. In Hibernate Search 5 I used custom abstract implementation of ShardIdentifierProviderTemplate with override shared identifier for each client, then set “hibernate.search.default.sharding_strategy” with client implementation for each entity manager. In elastic search it index each indicate with my client suffix like: entity-client1, entity-client2.

In Hibernate Search 6 I tried to do something similar using implementation of IndexLayoutStrategy for each client and set “hibernate.search.default.sharding_strategy” in entity manager, but when it doesn’t work properly I found that IndexLayoutStrategy works per backend.

So finally I would like to dynamically (depends on entity manager) use customized index suffix.
Is such case possible in Hibernate Search 6?

#2

Yoann is on vacation for the Christmas break, he will happily answer when he’s back!

#3

Hello,

If you have one entity manager factory per tenant, then you can configure a different IndexLayoutStrategy for each entity manager factory and all is well.

If you have one entity manager factory common to all tenants and are leveraging Hibernate ORM’s multi-tenancy features, then the best you can do at the moment is to rely on discriminator-based multi-tenancy: store the data of all tenants in the same index, but have Hibernate Search filter it automatically using a discriminator field when searching to only include results from the current tenant.

If you absolutely need a separate index per tenant, then you might want to get involved in HSEARCH-3683, which will allow just that, but hasn’t been implemented yet. Note however that you will need to know the list of tenants in advance (at boot time). Dynamically adding a new tenant while the application is running would require deeper changes.

#4

So looks that something doesn’t work properly. The result is that I get in Elasticsearch only indexes for first Client like: enity1-client1, entity2-client1. This is when I use hibernate.search.schema_management.strategy = create. When Use hibernate.search.schema_management.strategy = drop-and-create then I get indexes for the last Client: enity1-client2, entity2-client2.
Below I past some of my code. In the code you I use Clone instead of Client, but treat it the same:

public abstract class CloneLayoutStrategy implements IndexLayoutStrategy {

    private static final Pattern UNIQUE_KEY_PATTERN =
            Pattern.compile( "(.*)-\\d+-\\d+-\\d+" );

    protected abstract Clone getClone();

    @Override
    public String createInitialElasticsearchIndexName(String hibernateSearchIndexName) {
        return hibernateSearchIndexName + "-" + getClone().name();
    }

    @Override
    public String createWriteAlias(String hibernateSearchIndexName) {
        return hibernateSearchIndexName + "-write";
    }

    @Override
    public String createReadAlias(String hibernateSearchIndexName) {
        return hibernateSearchIndexName;
    }

    @Override
    public String extractUniqueKeyFromHibernateSearchIndexName(
            String hibernateSearchIndexName) {
        return hibernateSearchIndexName;
    }

    @Override
    public String extractUniqueKeyFromElasticsearchIndexName(
            String elasticsearchIndexName) {
        Matcher matcher = UNIQUE_KEY_PATTERN.matcher( elasticsearchIndexName );
        if ( !matcher.matches() ) {
            throw new IllegalArgumentException(
                    "Unrecognized index name: " + elasticsearchIndexName
            );
        }
        return matcher.group( 1 );
    }

}

public class LayoutStrategyClone1 extends CloneLayoutStrategy {

    @Override
    protected Clone getClone() {
        return Clone.CLONE1;
    }

}

public class LayoutStrategyClone2 extends CloneLayoutStrategy {

    @Override
    protected Clone getClone() {
        return Clone.CLONE2;
    }

}


public abstract class CommonEntityManager {

    @Bean
    @ConfigurationProperties(prefix = "spring.jpa.properties.hibernate")
    public Properties hibernateConfig() {
        return new Properties();
    }

    public LocalContainerEntityManagerFactoryBean getEntityManager() {
        final LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
        em.setPackagesToScan(ENTITY_PACKAGE);

        final HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
        em.setJpaVendorAdapter(vendorAdapter);
        final HashMap<String, Object> properties = new HashMap<>();
        properties.put("hibernate.hbm2ddl.auto", this.hibernateDDL);
        properties.put("hibernate.dialect", this.hibernatePlatform);
        properties.put("hibernate.physical_naming_strategy",
                "org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy");
        properties.put("hibernate.implicit_naming_strategy",
                "org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy");

        for (Entry<Object, Object> entry : this.hibernateConfig().entrySet()) {
            properties.put("hibernate." + entry.getKey(), entry.getValue());
        }

        properties.put("hibernate.search.backend.layout.strategy",
                ENTITY_MANAGER_PACKAGE + ".EntityManager" + StringUtils.capitalize(getClone().name().toLowerCase(Locale.ENGLISH)));


        em.setJpaPropertyMap(properties);
        return em;
    }
...
}

@Configuration
public class EntityManagerClone1 extends CommonEntityManager {

    @Primary
    @Bean
    public LocalContainerEntityManagerFactoryBean clone1EntityManager() {
        final LocalContainerEntityManagerFactoryBean em = getEntityManager();
        em.setDataSource(clone1DataSource());
        return em;
    }
...
}

@Configuration
public class EntityManagerClone2 extends CommonEntityManager {

    @Bean
    public LocalContainerEntityManagerFactoryBean clone2EntityManager() {
        final LocalContainerEntityManagerFactoryBean em = getEntityManager();
        em.setDataSource(clone2DataSource());
        return em;
    }
...
}
#5

Your layout strategy implementation looks wrong: it will assign identical aliases to indexes from different clients.

Make sure to include the tenant ID (I suppose that’s what the “clone name” is?) in every value returned by the layout strategy.

Also, the unique key pattern is incorrect, since it will not match the index name. Anyway, you don’t need to implement this part unless you change the type naming strategy, so you can probably ignore this.

Try something like this:

public abstract class CloneLayoutStrategy implements IndexLayoutStrategy {

    protected abstract Clone getClone();

    @Override
    public String createInitialElasticsearchIndexName(String hibernateSearchIndexName) {
        return hibernateSearchIndexName + "-" + getClone().name() + "-000001";
    }

    @Override
    public String createWriteAlias(String hibernateSearchIndexName) {
        return hibernateSearchIndexName + "-" + getClone().name() + "-write";
    }

    @Override
    public String createReadAlias(String hibernateSearchIndexName) {
        return hibernateSearchIndexName + "-" + getClone().name() + "-read";
    }

}

If you access the indexes externally (e.g. through your own HTTP client), and absolutely need the name to be enity1-client1 without a suffix, then I’d recommend dropping the -read suffix of the read alias.

#6

Great thanks, works as expected, each Client (Clone) has own index.