HS 6: Fetch field which is a part of index but not of Hibernate entity

Hibernate Search
1

Hi,

I’m using Hibernate search 6.0.0.Beta9 in my project.

Hibernate Entities(removed all other irrelevant fields and annotations):

@Entity(name = "listing")
@Indexed(index = "listing")
data class Listing(
        @OneToMany(fetch = FetchType.EAGER, mappedBy = "listing", cascade = [PERSIST, MERGE, REFRESH], orphanRemoval = true)
        @IndexedEmbedded(structure = NESTED)
        var products: MutableList<Product> = mutableListOf()
)

@Entity(name = "product")
@Indexed(index = "product")
data class Product(
        // This field is not a part of Entity, but it is present in the listing index
        @IndexedEmbedded(structure = NESTED)
        var offers: HashSet<Offer>? = null
)

@Embeddable
data class Offer(
        @GenericField(projectable = Projectable.YES, sortable = Sortable.YES)
        val id: String? = null,

        @GenericField(projectable = Projectable.YES, sortable = Sortable.YES)
        val expiryTime: Long? = null
)

{
        "_index": "listing",
        "_type": "_doc",
        "_id": "3",
        "_score": 1.0,
        "_source": {
          "id": "LISTING_003",
          "products": {
            ....
            "offers": [
              {
                "expiryTime": 2547012568000,
                "id": "OFFER_002"
              }
            ]
          },
          "_entity_type": "listing"
        }
}

My use case is to load the offers from listing index(we’re updating offers field manually) but it’s coming as null since it’s not a part of listing entity. Is there any to do it?

2

Hi. What’s the code you’re using to retrieve the offers?

3 
val session = Search.session(entityManager)
        val listings = session.search<Listing>(Listing::class.java)
                .where { f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }.sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<Listing>
4

First, I’ll warn you that this design, where you have a primary datasource (the database), a secondary datasource (Elasticsearch) derived from the first one, and you update the secondary datasource based on a third datasource, is fragile. I’m quite sure it will give you headaches when Hibernate Search unexpectedly reindexes an entity and the offers are gone all of a sudden. To be clear: I would not do this and would look for other solutions, for example loading the offers from the third datasource when you reindex.

Now, if you want to do this anyway…

By default Hibernate Search retrieves data from the database. Since your data is not in the database, it just won’t be retrieved.

You can retrieve data from the index, however. This is what projections are for. There are ways to retrieve fields, and even to combine projections on multiple fields to create composite objects. However, at the moment, there is no way to preserve the structure, i.e. you would get a list of all offer IDs, but you won’t be able to know which product each offer corresponds to.

So, the only solution right now would be to retrieve the raw JSON, and optionally transform it afterwards.

For example:

val session = Search.session(entityManager)
        val listings = session.search<Listing>(Listing::class.java)
                // Add the two lines below
                .extension(ElasticsearchExtension.get())
                .select(f -> f.source())
                .where {f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }.sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<JsonObject>

You will get results as JsonObject instances, and will be able to extract the data from there. The Gson library offers various ways to convert a JsonObject to a type of your choosing, but I’d refrain from converting it to Listing, as the resulting object will not be managed by Hibernate ORM, so it will be very hard to use it for write operations on the database. Prefer declaring a different class, e.g. ListingDTO, that includes a ProductDTO and an OfferDTO for example.

EDIT: You may also be able to retrieve data from both the database and the Elasticsearch index, then combine them to restore the offers in your entity. For example:

(I don’t know Kotlin that well, so I’ll use Java syntax for my additions…)

val session = Search.session(entityManager)
        val listings = session.search<Listing>(Listing::class.java)
                // Add the two lines below
                .extension(ElasticsearchExtension.get())
                .select(f -> f.composite((listing, jsonObject) -> {
                    // Recontruct all offers from the json object:
                    Map<String, Set<Offer>> offersByProductId;
                    jsonObject.get( ... ) ... // You'll have to code this :)
                    // Re-attach offers to products:
                    for (Product p : listing.products) {
                      p.offers = offersByProductId.get(p.getId());
                    }
                    return listing;
                }, f.entity(), f.source()))
                .where {f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }.sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<JsonObject>
5

I agree that there are loopholes with the current design but this is only experimental to fulfil the search requirements for now. I’ll be changing this to more robust system(depending on the requirements). I also have multiple scenarios where I need to merge multiple entities into one and also need to add fields after computation.

I’m facing a compilation error in the line below.

.extension(ElasticsearchExtension.get())

Acc to the documentation -> the parameter will automatically take the appropriate value when calling, but getting

Type inference failed: Not enough information to infer parameter H in 
fun <H : Any!, R : Any!, E : Any!, LOS : Any!> get( ): ElasticsearchExtension<H!, R!, E!, LOS!>!
Please specify it explicitly.

I’ve tried few things, but couldn’t resolve it.

EDIT:

session.search<Listing>(Listing::class.java)
                .extension(ElasticsearchExtension.get<T, EntityReference, Listing, SearchLoadingOptionsStep>())
                .select { f ->
                    f.composite(BiFunction<Listing, JsonObject, Listing> { a, b ->
                        for (p in a.products) {
                            p.offers.add(OfferDto("1", 1))
                        }
                        a
                    }, f.entity(), f.source())
                }
                .where { f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }
                .sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<Listing>

This is also not working

6

That looks like a Kotlin problem, since it works with Java… I don’t know enough about Kotlin to debug this, unfortunately.

Maybe this monstrosity will compile:

session.search<Listing>(Listing::class.java)
                .extension(ElasticsearchExtension.get<Object, EntityReference, Listing, SearchLoadingOptionsStep>())
                .select { f ->
                    f.composite(BiFunction<Listing, JsonObject, Listing> { a, b ->
                        for (p in a.products) {
                            p.offers.add(OfferDto("1", 1))
                        }
                        a
                    }, f.entity(), f.source())
                }
                .where { f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }
                .sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<Listing>

Alternatively, you can try this:

session.search<Listing>(Listing::class.java)
                .select { f ->
                    f.composite(BiFunction<Listing, JsonObject, Listing> { a, b ->
                        for (p in a.products) {
                            p.offers.add(OfferDto("1", 1))
                        }
                        a
                    }, f.entity(), f.extension(ElasticsearchExtension.get()).source())
                }
                .where { f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }
                .sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<Listing>
7

@yrodiere Thanks a lot. This worked, but still facing an issue in debug mode. Yes, issue is related to Kotlin only.

session.search<Listing>(Listing::class.java)
                .extension(ElasticsearchExtension.get<T, EntityReference, Listing, SearchLoadingOptionsStep>())
                .select { f ->
                    f.composite(BiFunction<Listing, JsonObject, Listing> { a, b ->
                        for (p in a.products) {
                            p.offers.add(OfferDto("1", 1))
                        }
                        a
                    }, f.entity(), f.source())
                }
                .where { f ->
                    validateAndGetPredicateClauses(andFilters, orFilters, f)
                }
                .sort { f -> f.field("created") }
                .fetch(request.pageNo, request.pageSize) as SearchResult<Listing>
8

I’m sorry, I don’t get what issue your are facing if this worked?

9

This expression compiled and ran successfully. But while evaluating it in the debug mode, I was getting “cannot access private field products” even after making the field public. I also tried it after deleting all the class files, but getting the same issue again and again. I still have to resolve this.

10

Ok, that’s definitely related to your debugger or compiler, not to Hibernate Search. Good luck :slight_smile: