I am looking for a benchmark to try a tool that I have migrated. The idea is that I should observe how a default configuration of Hibernate, an “optimized” configuration and a configuration with my tool enabled performs. I want to see for example how many queries that were executed and how much time these executions took. I have read some research papers and they usually mention two benchmarks: TORPEDO and 007
However, searching for these on google gave me close to nothing. Do you know any good way to do some performance analysis? I know that hibernate has its built-in statistics, and I am going to use this in my evaluation when running against a target project, but I would like to have some more ways to evaluate the performance.
Benchmarking properly can be very tricky. We have an entire team dedicated to it, unfortunately they focus mostly on established enterprise grade benchmarks which are not open source, such as the ones from SPEC.
JMH is absolutely awesome to run micro-benchmarks, so it’s the right tool when you want - for example - to compare which of two method implementations is better.
When it comes to measure larger applications it becomes harder though; JMH is still somewhat helpful but you need to keep in mind things like the overall warmup time of the JVM will no longer be seconds (could be 30 minutes or more! check compiler logs), and it becomes very important to be able to faithfully simulate “production load” as much as possible.
Running some hot loops as fast as you can is actually not realistic and - I’ve learned the bad way - will lead you into wrong conclusions or even fool you into making the wrong optimisations.
Be sure to read some more on the topic and always be very critical of your results. Especially don’t stop at being satisfied with “good numbers” but try to reason on why you get certain results.
Also, make sure to set goals or it will become a never ending journey I started with such things ten years ago, and here I am still working on making Hibernate faster day by day…
Thanks Sanne, it certainly seems to be a rather difficult topic. Luckily for me my task is not that complex, but rather to just check how many queries that are being executed during a certain execution of my program with two different configurations, and then compare the results. Maybe I don’t even need a proper benchmark-framework, but can just use the built-in statistics. We will see, thanks for the insight!
I started with such things ten years ago, and here I am still working on making Hibernate faster day by day…