HSEARCH700088: Invalid indexing request: the add and update operations require a non-null entity

Hibernate Search
1

Hi all!

I have used Hibernate for quite a while, and now I just started adding Hibernate Search 6.1.7.Final functionality on my Hibernate 5.6.8.Final application. The backend is OpenSearch 1.2.4 running locally in Docker in HTTP mode (not HTTPS)

I got the following error that I can’t seem to find on Google or in this forum anywhere

HSEARCH000058: Exception occurred org.hibernate.search.util.common.SearchException: HSEARCH700088: Invalid indexing request: the add and update operations require a non-null entity.
Failing operation:
Indexing instance of entity 'Item' during mass indexing


org.hibernate.search.util.common.SearchException: HSEARCH700088: Invalid indexing request: the add and update operations require a non-null entity.
	at org.hibernate.search.mapper.pojo.work.impl.PojoIndexerImpl.add(PojoIndexerImpl.java:44)
	at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingEntityLoadingRunnable$IndexingBatch.startIndexing(PojoMassIndexingEntityLoadingRunnable.java:208)
	at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingEntityLoadingRunnable$IndexingBatch.startIndexingList(PojoMassIndexingEntityLoadingRunnable.java:155)
	at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingEntityLoadingRunnable$LoadingContext$1.accept(PojoMassIndexingEntityLoadingRunnable.java:125)
	at org.hibernate.search.mapper.orm.loading.impl.HibernateOrmMassEntityLoader.load(HibernateOrmMassEntityLoader.java:49)
	at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingEntityLoadingRunnable.runWithFailureHandler(PojoMassIndexingEntityLoadingRunnable.java:60)
	at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingFailureHandledRunnable.run(PojoMassIndexingFailureHandledRunnable.java:32)
	at org.hibernate.search.util.common.impl.CancellableExecutionCompletableFuture$CompletingRunnable.run(CancellableExecutionCompletableFuture.java:70)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: org.hibernate.search.util.common.AssertionFailure: Processing a non-indexed type in the MassIndexer: null -- this may indicate a bug or a missing test in Hibernate Search. Please report it: https://hibernate.org/community/
		at org.hibernate.search.mapper.pojo.massindexing.impl.PojoMassIndexingIndexedTypeGroup.lambda$extractReference$0(PojoMassIndexingIndexedTypeGroup.java:119)

And here I am… reporting to the community :wave::wave:

Configuration

Here’s my configuration file application-local.properties

spring.jpa.properties.hibernate.search.backend.hosts=localhost:9200
spring.jpa.properties.hibernate.search.backend.protocol=http
spring.jpa.properties.hibernate.search.backend.username=admin
spring.jpa.properties.hibernate.search.backend.password=admin

MassIndexer

I have also added a MassIndex.java to mass-index the existing DB

@RequiredArgsConstructor
@Slf4j
@Component
public class MassIndex {
    private final EntityManager em;

    @EventListener(ContextRefreshedEvent.class)
    @Transactional(readOnly = true)
    public void massIndex() {
        try {
            var searchSession = Search.session(em);
            var indexer = searchSession.massIndexer(Item.class);
            indexer.startAndWait();
        } catch (Exception exception) {
            log.error("INDEXING FAIL ", exception);
        }
    }
}

#Entity
The entity Item being indexed is as follows

@Entity
@Table(name = "item")
@Getter
@Setter
@Indexed
public class Item {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ITEM_SEQ")
    @SequenceGenerator(sequenceName = "_item_id_seq", allocationSize = 1, name = "ITEM_SEQ")
    @ToString.Include
    private Long id;

    @FullTextField
    @Column(name = "name", length = 100)
    private String name;

There are other fields, but I only annotated name for hibernate search. Note that all Items have a non-null name

2

Hey,

Thanks for reporting. It would seem there are null entities moving around in your mass indexer. That’s certainly unexpected. The only way I can see this happening is if loading an entity by ID resulted in a null entity…

Is it possible that your application deletes entities concurrently to the mass indexing? Depending on the isolation level of your transactions, I suppose it could lead to such problem, though I’d be hard-pressed to reproduce it…

Do you think you would be able to reproduce the problem in a simple project based on our test case templates? Then I would be able to find out what’s going on and, hopefully, fix it.

3

Is it possible that your application deletes entities concurrently to the mass indexing?

Not that I know of, it doesn’t do any DB writes at startup, only when specific endpoints were hit.

I tried migrating some of the components to the test case templates, and it magically works, so at this point I’m super confused hahah

Not sure how useful it’s gonna be but I posted it here GitHub - ptrkhh/orm-elasticsearch-poc

Maybe I’ll try to migrate the features one by one until I found the culprit

Out of curiosity, if I didn’t mark the variable as @FullTextField or sth like that, it wouldn’t matter at all, right?

4

Seems like I found the problem. So I dived deeper into the full app, and there is this thing inside the Item:

@JoinColumn(name = "category_id", nullable = false)
@ManyToOne(optional = false)
@NotFound(action = NotFoundAction.IGNORE) // to support operation with default category (id = 0)
private Category category;

When I copied it to the POC (Github link above), then the application immediately goes bananas. So I guess this is the culprit. I have pushed it to the GitHub repo

5

Thanks, I’ll have a look at the reproducer.

So the relationship is mandatory, but sometimes the target may not be found? I’m surprised Hibernate ORM even accepts that… You may want to change that when you can.

6

After consulting with the Hibernate ORM team, this mapping should indeed be invalid and fail, you’re just lucky it doesn’t: it doesn’t make sense to use NotFoundAction.IGNORE (which will lead to null values) on an optional association (which forbids null values).

EDIT: One suggestion to fix your mapping was to create that row with id = 0 in your database, and remove @NotFound. I’ll let you decide if that makes sense for you.

I will still have a look at the reproducer, because the issue in Hibernate Search itself might not be specific to that invalid mapping (maybe it can happen with a valid mapping as well).

7

I debugged your reproducer and it’s really a problem in Hibernate ORM: with your mapping, Hibernate ORM will return null for entities with a “not found” association, instead of returning an entity whose association was set to null. Consequently Hibernate Search gets nulls where it’s not supposed to, and it eventually fails.

But again, IMO the mapping is at fault here as it’s contradictory: the association is marked as non-optional and also marked with @NotFound(IGNORE) which implies the association might be null in some cases. So, I filed a ticket to reject that kind of mapping in the future: [HHH-15657] - Hibernate JIRA

It’s strange that you only noticed this through Hibernate Search, though, as any session.get/session.find will fail for such entities… this will affect much more than just Hibernate Search. Is this mapping really in use in production? Aren’t you affected by the problem when doing session.get/session.find, somehow?

8

It somehow didn’t create problems in the staging environment that we used. But either way, I managed to fix it after I saw your post, literally few hours before it goes production, many thanks!!!

Since then, Hibernate Search has been working great till now after that rough start ahahah. Now its just a matter of tuning the parameters. Amazing work by your team!

2 Likes