Create indexes for transient fields

Hibernate Search
1

Hello i use hibernate search 6 and i would like to reduce loop of the list. list size is over 700k records.

  1. Can i add the below fields in indexes without to loop seperate them ?

  2. Is it good practise for performance to use multiple PropertyValue for specific field or is the same to use only the “list” ?

    @Transient
	@IndexingDependency(derivedFrom = @ObjectPath({
			@PropertyValue(propertyName = "list"),
			@PropertyValue(propertyName = "debit")
	}))
	@ScaledNumberField(decimalScale = 2, sortable = Sortable.YES)
	private BigDecimal totalDebit;
	
	public BigDecimal getTotalDebit() {
		BigDecimal debit = BigDecimal.ZERO;
		if (CollectionUtils.isNotNullOrEmpty(list)) {
			for (final Object object : list) {
				if (object.getDebit() != null) {
					debit = debit .add(object.getDebit());
				}
			}
		}
		
		return debit ;
	}
	
	@Transient
	@IndexingDependency(derivedFrom = @ObjectPath({
			@PropertyValue(propertyName = "list"),
			@PropertyValue(propertyName = "credit")
	}))
	@ScaledNumberField(decimalScale = 2, sortable = Sortable.YES)
	private BigDecimal totalTraderstranCredit;
	
	public BigDecimal getTotalCredit() {
		BigDecimal credit = BigDecimal.ZERO;
		if (CollectionUtils.isNotNullOrEmpty(list)) {
			for (final Object object : list) {
				if (object.getCredit() != null) {
					credit = credit.add(object.getCredit());
				}
			}
		}
		
		return credit;
	}
	
	@Transient
	@IndexingDependency(derivedFrom = @ObjectPath({
			@PropertyValue(propertyName = "list"),
			@PropertyValue(propertyName = "totalamount")
	}))
	@ScaledNumberField(decimalScale = 2, sortable = Sortable.YES)
	private BigDecimal totalTotalamount;
	
	public BigDecimal getTotalTotalamount() {
		BigDecimal totalamount = BigDecimal.ZERO;
		if (CollectionUtils.isNotNullOrEmpty(list)) {
			for (final Object object : list) {
				if (object.getTotalamount() != null) {
					totalamount = totalamount .add(object.getTotalamount());
				}
			}
		}
		return totalamount;
	}

Thank you.

2

Fetching 700k records from the database and iterating over them for indexing purposes (or any other for that matter) is going to cause problems, be it one iteration to calculate multiple attributes at the same time or to calculate one and do multiple iterations.

I’d suggest looking at the data model and trying to figure out how to reduce the number of records needed in that list. If it is some sort of transactional log record, maybe it is possible to keep the records from now to some small period of time in the past within this list and have the more “historically older” records already grouped and required values computed for them… But that is more of a data modelling question than a Search one.

now if you must iterate through the list you could try:

  1. Creating DB functions that perform the computations and calling them through a hibernate’s ORM @Formula this would just move the computation from your app to your DB.
  2. Adding a @PostLoad callback and run that loop within it to calculate all your properties and set them to those transient fields.

But no matter which route, 700k records is a lot and it won’t perform well :slightly_frowning_face:

3

now i notice that i wrote 700k records for the list.
all table has 700k records but the list that i use have almost 1000 records !

is it better to use

    @Transient
	@Formula("(SELECT COALESCE(SUM(object.debit), 0) FROM object WHERE objectB.object.id= id)") // where id the id of my object
	@ScaledNumberField(decimalScale = 2, sortable = Sortable.YES)
	private BigDecimal totalDebit;

instead of

    @Transient
	@IndexingDependency(derivedFrom = @ObjectPath({
			@PropertyValue(propertyName = "list"),
			@PropertyValue(propertyName = "debit")
	}))
	@ScaledNumberField(decimalScale = 2, sortable = Sortable.YES)
	private BigDecimal totalDebit;
	
	public BigDecimal getTotalDebit() {
		BigDecimal debit = BigDecimal.ZERO;
		if (CollectionUtils.isNotNullOrEmpty(list)) {
			for (final Object object : list) {
				if (object.getDebit() != null) {
					debit = debit .add(object.getDebit());
				}
			}
		}
		
		return debit ;
	}

Will @Formula update the indexes or it needs IndexingDependency?

4

it would need an indexing dependency, yes.

if the list is loaded for some other reason than these computations, then it would only put more load on your DB to do additional computations.
If the list is removed… then there won’t be what to reference in the indexing dependency.

You could try running the single loop and do all computations within it and then run multiple sequential loops (one pe property) and record the execution time to see if the difference is of any significance. If it is then maybe try to do one loop in the @PostLoad. (run the loops just in some service method or test or anywhere it is convenient/easy)

1 Like
5

Thank you for your help ! :slight_smile: