How to make lazy loading truly lazy (V5.4)

Kai_Hackemesser · January 8, 2019, 11:48pm

Hi,

Since the begin of my project I notice difficulties with my ER model that hibernate is querying ToOne relationships even if I explicity mark them lazy. Here is a bit of code how it is currently done:

	@ManyToOne(fetch = LAZY, optional = false)
	@LazyToOne(NO_PROXY)
	@JoinColumn(name = "audit_tran_id")
	public AuditTransaction getAuditTransaction() {
		return auditTransaction;
	}

the classes are bytecode-enhanced by the maven plugin:

            <plugin>
                <groupId>org.hibernate.orm.tooling</groupId>
                <artifactId>hibernate-enhance-maven-plugin</artifactId>
                <version>${hibernate.version}</version>
                <executions>
                    <execution>
                        <configuration>
                            <enableLazyInitialization>true</enableLazyInitialization>
                        </configuration>
                        <goals>
                            <goal>enhance</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

and when my SpringBoot app is started it logs that, too
HHH000157: Lazy property fetching available for: [...]

Still, I notice unnecessary queries during the JPA load, in this case because the reference was broken, I got a nice stack trace:

javax.persistence.EntityNotFoundException: Unable to find mypackage.AuditTransaction with id 3133020786
	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$JpaEntityNotFoundDelegate.handleEntityNotFound(EntityManagerFactoryBuilderImpl.java:162) 
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:211) 
	at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:246) 
	at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:105) 
	at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:73) 
	at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1260) 
	at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1143) 
	at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:687) 
	at org.hibernate.type.EntityType.resolve(EntityType.java:464) 
	at org.hibernate.type.ManyToOneType.resolve(ManyToOneType.java:239) 
	at org.hibernate.type.EntityType.resolve(EntityType.java:457) 
	at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:272) 
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyPropertiesFromDatastore(AbstractEntityPersister.java:1210) 
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyProperty(AbstractEntityPersister.java:1121) 
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor$1.doWork(LazyAttributeLoadingInterceptor.java:105) 
	at org.hibernate.bytecode.enhance.spi.interceptor.Helper.performWork(Helper.java:97) 
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:76) 
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:72) 
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.intercept(LazyAttributeLoadingInterceptor.java:61) 
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.readObject(LazyAttributeLoadingInterceptor.java:296) 
	at mypackage.Achieved.$$_hibernate_read_version(Achieved.java) ~[classes!/:1-SNAPSHOT]
	at mypackage.Achieved.getVersion(Achieved.java:272) ~[classes!/:1-SNAPSHOT]
	at mypackage.Service.lambda$checkForDuplicateAchievement$2(Service.java:97) ~[classes!/:1-SNAPSHOT]

That brings up my question, did I oversee anything to get this a true lazy relation (because the audit information is only used when inserting),

And secondly, is there a JPA way to handle the situation gently when there is a referenced entity missing?

vlad · January 9, 2019, 5:03am

The @ManyToOne associations don’t need bytecode enhancement to be loaded lazily.

Only the parent-side @OneToOne associations need that.

Kai_Hackemesser · January 9, 2019, 6:09am

Yes, that was my knowledge, too, but for reasons I do not fully comprehend, Hibernate now seems to see the difference between lazy and eager to be loaded with a join immediately or with a secondary query, before returning from the entity manager query. Nothing of my code uses the getter function, and I’d expect no query to be executed before my code needs it. But from the stack trace it is clear that Hibernate does so.
Why?

vlad · January 9, 2019, 6:30am

Hibernate now seems to see the difference between lazy and eager to be loaded with a join immediately or with a secondary query, before returning from the entity manager query.

Nope. THat’s EAGER fetching. Lazy means that the association will not be fetched unless you navigate it explicitly.

Why?

The only way to figure it out is if you write a replicating test case that demonstrates the issue.

Kai_Hackemesser · January 9, 2019, 9:16am

EAGER: -> left join
LAZY -> query on demand.
At least from the documentation, but if it would be that easy, I would not have spent two weeks in December learning all about the maven-plugin to enhance the classes, to get things less database intense. And I still see those rogue queries for things not required by my logic. I cannot share most of my current code because it is closed source, but trust me, Hibernate is querying the entities related by ManyToOne even if I set the annotation to be lazy, even if I tell it to use the NO_PROXY approach and enhance the classes.
The problem could be the same as in https://hibernate.atlassian.net/browse/HHH-13134?jql=project%20%3D%20HHH%20AND%20component%20%3D%20bytecode-enhancement
But I started all that enhancing just because of the unwanted queries of manytoone relationships.
Is there a way to produce useful log files explaining why some lazy relations are loaded before anything is returned from the entity manager?

vlad · January 9, 2019, 9:47am

And I still see those rogue queries for things not required by my logic.

If you are using LAZY and Hibernate fetches associations eagerly even if you use BYtecode Enhancement, then it’s a regression.

Since you already mentioned HHH-13134, the behavior might be caused by that issue.

I noticed there’s a test case provided for that issue too. Maybe Luis Barreiro will take a look at it. I pinged him on Jira since he knows the Bytecode Enhancement part better than anyone else in the team.

Kai_Hackemesser · February 4, 2019, 11:49pm

In https://stackoverflow.com/questions/27799455/hibernate-creating-unnecessary-queries-for-manytoone-annotated-property there is a programmer with a similar issue, the question is from 2015. This might be used as an example for a unit test, if one tells me how to do query counts. I still have secondary queries in my JPA named query executions.

vlad · February 5, 2019, 6:02am

That question is about the FetchType.EAGER used by default by @ManyToOne or @OneToOne associations.

In fact, that’s what the JPA 1.0 spec demanded although in Hibernate 3 all associations were lazy. So, although a terrible default, it’s actually the way JPA was supposed to work.

Nevertheless, since EAGER fetching is very bad for performance, you should just set all associations to FetchType.LAZY and, in case you need to initialize a lazy association, override the fetching strategy at query time.

Kai_Hackemesser · February 6, 2019, 12:58am

So what is now about my original problem then. How can I make hibernate absolutely not query for related entities in a ManyToOne relationship until my code asks for it? When Fetchtype.LAZY didn’t do the task, and enhancing didn’t do it?

vlad · February 6, 2019, 4:58am

As long as ypu ise LAZY, you should be fine. Only for parent-side one-to-one associations, you need bytecode enhancement, but if you shared the PK, you don’t need the parent dide at all.

Kai_Hackemesser · February 6, 2019, 5:02am

I used LAZY, found this not working (i.e. its not eager/joins, but still secondary queries immediately), then tried to add the enhancement.

vlad · February 6, 2019, 7:57am

Unless you replicate it with a test case, I have no idea what you are talking about.

Kai_Hackemesser · February 7, 2019, 5:15am

I am trying that now. In a simple entity model derived from the maven spring-boot-data-jpa archetype, extra queries do not happen if i set ManyToOne to fetch lazy. I updated the hibernate version I use. So the problem must involve more than that. Maybe the database driver? Its Sybase in my problem.
Entity A is extending an Entity B (joined tables) that extends a mapped superclass, that has the ManyToOne property that should be lazy loaded (i.e. not at all in this scenario). I still work on a replication case from my original sources.

Kai_Hackemesser · February 8, 2019, 1:23am

i am not able to reproduce it in a separate model. But what I found in my original model, when I turned off the bytecode enhancement, a lot more extra secondary queries and left joins happen on properties still marked as
@ManyToOne(fetch = LAZY)
@LazyToOne(NO_PROXY)

so the enhancement has very much so an effect on ManyToOne.

Kai_Hackemesser · February 10, 2019, 9:42pm

Vlad,

Can you point me to what class/method in Hibernate ORM would be responsible for its decision to load a lazy relation during a named query execution? I can debug a bit there then

Kai_Hackemesser · February 10, 2019, 11:45pm

I am debugging through my query and am now hanging in this stack (i turn it upside down so I can comment with my findings):

getOwnedOnes:384, Owner (mycorp.model)
$$_hibernate_read_ownedOnes:-1, Owner (mycorp.model)
//the next few lines are from the bytecode enhancer, noticing a proxy access:
readObject:296, LazyAttributeLoadingInterceptor (org.hibernate.bytecode.enhance.spi.interceptor)
intercept:61, LazyAttributeLoadingInterceptor (org.hibernate.bytecode.enhance.spi.interceptor)
fetchAttribute:72, LazyAttributeLoadingInterceptor (org.hibernate.bytecode.enhance.spi.interceptor)
loadAttribute:76, LazyAttributeLoadingInterceptor (org.hibernate.bytecode.enhance.spi.interceptor)
performWork:97, Helper (org.hibernate.bytecode.enhance.spi.interceptor)
doWork:105, LazyAttributeLoadingInterceptor$1 (org.hibernate.bytecode.enhance.spi.interceptor)
initializeLazyProperty:1065, AbstractEntityPersister (org.hibernate.persister.entity)
// here the OwnedOne related entities are loaded
initializeCollection:2259, SessionImpl (org.hibernate.internal)
onInitializeCollection:75, DefaultInitializeCollectionEventListener (org.hibernate.event.internal)
initialize:691, AbstractCollectionPersister (org.hibernate.persister.collection)
initialize:87, AbstractLoadPlanBasedCollectionInitializer (org.hibernate.loader.collection.plan)
executeLoad:86, AbstractLoadPlanBasedLoader (org.hibernate.loader.plan.exec.internal)
executeLoad:122, AbstractLoadPlanBasedLoader (org.hibernate.loader.plan.exec.internal)
extractResults:133, ResultSetProcessorImpl (org.hibernate.loader.plan.exec.process.internal)
finishUp:209, AbstractRowReader (org.hibernate.loader.plan.exec.process.internal)
// here the code decides to look into properties that are related entities, to decide to load them or set up a lazy proxy for them
performTwoPhaseLoad:241, AbstractRowReader (org.hibernate.loader.plan.exec.process.internal)
initializeEntity:129, TwoPhaseLoad (org.hibernate.engine.internal)
doInitializeEntity:172, TwoPhaseLoad (org.hibernate.engine.internal)
// the OwnedOne entity has a ManyToOne relationship to another entity:
resolve:239, ManyToOneType (org.hibernate.type)
resolve:464, EntityType (org.hibernate.type)
resolveIdentifier:687, EntityType (org.hibernate.type)
// the ID and the type of the related entity are known. as the relationship is marked LAZY I'd expect this to return a proxy, but no, it returns an entity freshly looked up:
internalLoad:1143, SessionImpl (org.hibernate.internal)
fireLoad:1257, SessionImpl (org.hibernate.internal)

Here is what entityType looks like at the begin of the resolveIdentifier method:

this = {ManyToOneType@12672} "org.hibernate.type.ManyToOneType(mycorp.model.AnotherEntity)"
 propertyName = "linkedProperty"
 ignoreNotFound = false
 isLogicalOneToOne = false
 scope = {TypeFactory$lambda@12754} 
 associatedEntityName = "mycorp.model.AnotherEntity"
 uniqueKeyPropertyName = null
 eager = false
 unwrapProxy = false
 referenceToPrimaryKey = true
 associatedIdentifierType = {LongType@12755} 
 associatedEntityPersister = {SingleTableEntityPersister@12740} "SingleTableEntityPersister(mycorp.model.AnotherEntity)"
 returnedClass = null

and these are

The annotations on the entity that I call “AnotherEntity” here:

	@ManyToOne(fetch = LAZY, optional = false)
	@JoinColumn(name = "another_entity_id")

this comes from AbstractEntityPersister:

	public boolean hasProxy() {
		// skip proxy instantiation if entity is bytecode enhanced
		return entityMetamodel.isLazy() && !entityMetamodel.getBytecodeEnhancementMetadata().isEnhancedForLazyLoading();
	}

I do not understand why proxy instantiation is disabled here if the classes are bytecode enhanced. but as this method returns false when the bytecode enhancement is active I try next with enhancement disabled.

vlad · February 11, 2019, 5:43am

In debug mode, the IDE will trigger the proxy initialization, so debugging is very hard if you want to check this mechanism.

Better add a breakpoint in BasicBinder which is called when a statement is executed. This way, you can navigate the stack to find why the proxy was initialized.

Kai_Hackemesser · February 11, 2019, 8:44pm

Without bytecode enhancement plugin I actually see the hasProxy method returning true and skip the secondary lookup query - the entity property becomes a proxy as wanted.
But I am still wondering why this affects the ManyToOne relation proxy forming?

Adnan_Ghazanfar · July 15, 2020, 10:29pm

Any success on this ? I am also getting the same issue with this plug in “lazy loading for many to one associations” does not work as expected.

Abhishek_Sharma · September 22, 2020, 9:28am

I got it working with the following annotations in the entity being queried and the associated entities

@ManyToOne(fetch = FetchType.LAZY)
@LazyToOne(LazyToOneOption.NO_PROXY)

And BytecodeEnhancement with InstrumentationLoadTimeWeaver bean

And following properties for hibernate

hibernate.enhancer.enableLazyInitialization=true
hibernate.bytecode.allow_enhancement_as_proxy=true
hibernate.enhancer.enableDirtyTracking=true

Topic		Replies	Views
ManyToOne - always EAGER loading Hibernate ORM	9	9760	April 7, 2022
Hibernate 5.2.18 cannot get a lazy OneToOne anymore Hibernate ORM	7	938	December 30, 2020
@ManyToOne Lazy fetch works on select, but is treated as Eager for update Hibernate ORM	1	1498	May 31, 2021
Lazy loading of ManyToOne does not work when hibernate.jpa.compliance.proxy is set and L2 cache is enabled Hibernate ORM	0	1871	October 16, 2019
Bytecode enhancement Hibernate ORM	2	1598	July 13, 2020

How to make lazy loading truly lazy (V5.4)

Related topics