With a large insert I can end up with 1gb+ of objects in the actionQueue/afterTransactionProcesses queue. With the 2nd level cache turned off the same process can complete with a couple 100 mb. Is there anything that can be done besides utilizing smaller transactions? I tried periodic flushing and it only slowed the problem down a bit.
The ActionQueue is for the 1st level cache, not the 2nd.
Using smaller transactions which clear the Session periodically is the best way to go. For more details, check out this article.
Thanks for pointing me in the right direction.
It looks like the main difference is that Entity Update Actions would be cleared during auto flush periodically. Without the cache the actions have to be saved till the end of the transaction (based on a call to needsAfterTransactionCompletion)
It looks like the way my code building the entities I’m ending up with a large number of Entity Update Actions all for the same entity (one for each item added to a many-to-many)
The best way to deal with batch processing tasks is to split the data set that need to be processed into individual subsets that you can process in parallel using multiple threads, each one splitting its subset into more successive batches.
Check out this article for some tips about batch processing in general.