Large data leads to OutOfMemory

Hibernate Search
1

I’m back to the topic.
I have a model that has several oneToMany relations. Some of them have thousands of records. I want to display a list from my master object but in the list i want to display a column that will contain the sum of all the debit of the detail, totalDebit, with the 1000 records.
Also the detail has another relationship oneToMany, detail2, with another 100 records each and I need to bring in the master list an “exist” column if there is a record in detail2.

i have used the following so that i can then filter or sort with column debit and column exist in the master list.

@Indexed
public class Master {
    ....
    private List<Detail1> detail1 = new ArrayList<>();

    @Transient
	@IndexingDependency(derivedFrom = {
			@ObjectPath({
					@PropertyValue(propertyName = "detail1"),
					@PropertyValue(propertyName = "debit")})
	}, reindexOnUpdate = ReindexOnUpdate.SHALLOW)
	public BigDecimal getTotalDebit () {
		BigDecimal debit = BigDecimal.ZERO;
		if (CollectionUtils.isNotNullOrEmpty(detail1)) {
			for (final Detail1 detail: detail1) {
				debit = debit .add(detail.getDebit() != null ? detail.getDebit() : BigDecimal.ZERO);
			}
		}
		return null;
	}

    @Transient
	@IndexingDependency(derivedFrom = {
			@ObjectPath({
					@PropertyValue(propertyName = "detail1"),
                    @PropertyValue(propertyName = "detail2"),
					@PropertyValue(propertyName = "id")})
	}, reindexOnUpdate = ReindexOnUpdate.SHALLOW)
	public int getExist () {
		if (CollectionUtils.isNotNullOrEmpty(detail1)) {
			for (final Detail detail: detail1) {
				if (CollectionUtils.isNotNullOrEmpty(detail1.getDetail2())) {
					return 1;
				}
			}
		}
        return 0;
	}
}

@Indexed
public class Detail1 {
    ....
    @IndexedEmbedded(structure = ObjectStructure.NESTED, includePaths = {"id"})
	@IndexingDependency(reindexOnUpdate = ReindexOnUpdate.SHALLOW)
    private List<Detail2> detail2 = new ArrayList<>();
}

In practice this means that whether I do mass indexing or update the master it will select the details due to the indexingDependency.
In mass action to save the master, about 500k master objects, leads to outOfMemory.

Thank you !

2

hey @Tony

You could try doing these indexing operations in smaller chunks and clear the session. But the real problem is the structure of your data and which parts of it you want to index.

I’d suggest considering denormalising things a bit. Instead of recalculating all these derived attributes:

  • create real DB columns for them
  • prepopulate them using existing data with a DB migration script
  • update the Search mapping to use these new attributes
  • do a full index with a mass indexer
  • any time a new record is added or updated in one of the lists - update the derived value in the “main” entity.

This way you’d have the precomputed values for all your derived attributes, and there will be no need to fetch all these lists on each index operation. And since you’d be updating the “main” entity attributes when the lists are modified – that’ll trigger the index operation (because, well, the entity itself was modified)

1 Like
3

@Tony Also, please stop opening a new topic every few days. Let’s stick to this one from now on, please.

1 Like
4

Thank you for your answer ! :slight_smile:

5

ok, i will not open again new topic ! sorry !