Hibernate Search 6 + Elasticsearch(7.16.3) , sorting by not existing property in a single index

Hibernate Search
1

Hello,

I’m wondering why it’s not possible to sort by properties that doesn’t exist in a single index?

HS6 adds “unmapped_type” to search query when create request for multiple indexes but for single index doesn’t.

I checked direct request to Elasticsearch (7.16.3), and when to generated request (from HS) for single index I add “unmapped_type” to request it just work fine.

So what is the reason of such restriction? I just need to search in one index and sort by property that can but don’t have to exists, so it force me to search in multiple indexes, one target index and the second that doesn’t match with fields. It’s a big workaround that I would like to avoid.

2

Hello,

It is possible, as far as I know?

If you’re having a problem, please post your code and the stacktrace.

EDIT: Okay, I didn’t understand what you meant. Adding another answer below.

3

It’s possible when you’re searching multiple indexes and at least one of them has that property, because then Hibernate Search knows what to put inside unmapped_type: the type of the property in the index that actually has that property.

It’s not possible when you’re searching a single index because then Hibernate Search has no idea what type that property should have, and thus it cannot automatically fill unmapped_type.

Also, I’d argue it doesn’t make much sense to sort on a property that doesn’t exist… ? That will never apply any sort! What’s your use case exactly?

4

Let’s say i have a products for different regions.
Products have a list of properties for example (property.A, property.B, property.C)
In products index it could look like this (JSON view from Kibana)
“property”: {
“A”: “A”,
“B”: “B”,
“C”: “C”,
},

I want to have different sorting for regions:
If(region.equals(“EU”) then sorty by prop.A
If(region.equals(“NA”) then sorty by prop.B

Properties are reloaded and then reindexed every week, so this week products for EU can have property.A but in the next week doesn’t have to.
And property will be mapped like this, empty list
“property”: {}

Part of sorting request:
“sort”:[
“_score”,
{
“property.A”:{
“missing”:"_last"
}
}
]

And then when sorting by property.A exception will be thrown:
“No mapping found for [property.A] in order to sort on”.
But if I add unmapped_type for example

“sort”:[
“_score”,
{
“property.A”:{
“missing”:"_last",
“unmapped_type”:“keyword”
}
}
]

then it works

5

Well, sure. But how exactly is Hibernate Search supposed to guess that unmapped_type should be set to keyword and not, say, integer? The answer is, it cannot, because Hibernate Search doesn’t know anything about property.A.

So, let’s see why you end up in this situation, because there’s something strange here.

This is the part I don’t get:

Why would the mapping change depending on the data? You could have a mapping with all possible properties, and index documents that happen not to use those properties at any point.

Evidently, you need these properties in the mapping at search time, because you use them, so it’s not like they are useless in your mapping. So, why are you using an empty mapping?

I suspect you’re using dynamic mapping to some extent, but I’m not sure how you’re using it exactly. And this the crux of the problem, because Hibernate Search 6 is able to handle dynamic mappings, but it requires specific care.

What is your mapping on the Hibernate Search side (annotations, bridges)?

6

Yes, i create these properties dynamically, with template, these properties are strings:

IndexSchemaObjectField propertyField = context.indexSchemaElement().objectField(“property”);
propertyField.fieldTemplate(“propertyTemplate”, context.typeFactory().asString()
.sortable(Sortable.YES)
)

and then in bridge if product has any property

DocumentElement propertyFieldElement = target.addObject(fieldReference);
document.addValue(fieldName, propertyValue); //addValue(“A”, “A”), addValue(“B”, “B”)

in product search function I have
provider.addHibernateSearchSortField(Product.PROPERTY_A) //sorty by property.A

and sort function:
.sort(f → {
CompositeSortComponentsStep<?> composite = f.score().then().composite();

for (String sortField : sortFields) {
composite.add(f.field(sortField).missing().last());
}
return composite;
})

Mapping doesn’t change, only data that are indexed change every week. So sometimes product doesn’t have these properties required to sort.

When HS generate request to ES (sorting part in ElasticsearchStandardFieldSort.java)
if ( indexNames().size() > 1 ) { // is that restriction necessary?
unmappedType = builder.field.type().elasticsearchTypeAsJson(); //can be for example“keyword”
}
else {
unmappedType = null; // for single index always null
}

and then

if ( unmappedType != null && !DataTypes.SCALED_FLOAT.equals( unmappedType.getAsString() ) ) {
// There are multiple target indexes; some of them may not declare the field.
// Instruct ES to behave as if the field had no value in that case.
UNMAPPED_TYPE.set( innerObject, unmappedType );
}

So if HS knows that properties are string type then why cannot he just set proper unmapped_type for single index search? Or why cannot I choose to add unmapped_type or not?

Thank you for really fast reply.

7

Alright, if you’re using dynamic field templates then Hibernate Search is indeed able to add the unmapped type. And what you’re asking definitely makes sense.

I would have expected Elasticsearch to not require the unmapped type in this case (since it knows about the type, through dynamic templates), but if it does… yes, we can definitely consider solving this in Hibernate Search. I’ll create a ticket.

1 Like
8

Created [HSEARCH-4531] - Hibernate JIRA

1 Like
9

This was fixed in Hibernate Search 6.1.4.Final. Thanks for reporting the problem.

1 Like