(I’d normally file this in JIRA, but my Atlassian account created several days ago still cannot create issues in HHH despite waiting as suggested in this thread - so reporting here.
Happy to move it to JIRA once my account works, or if a maintainer prefers to file it.)
HibernateMetrics registers the failed-transactions counter as a difference of two independently updated counters: s.getTransactionCount() - s.getSuccessfulTransactionCount().
StatisticsImpl.endTransaction() increments the two LongAdders without atomicity, so a scrape that reads them while a transaction completes can observe a negative value — exactly -1 in
apps where transactions virtually always succeed.
Up to Micrometer 1.12 this was silently exposed, but Micrometer 1.13+/1.14+ (new Prometheus Java client) rejects negative counters, so one racy read fails the entire
/actuator/prometheus scrape with HTTP 500: java.lang.IllegalArgumentException: -1.0: counters cannot have a negative value.
Observed in production on Spring Boot 3.4.12 / hibernate-micrometer 6.6.36.Final: several unrelated high-traffic services fail a scrape about once a week, always exactly -1.0. On
services still running the old simpleclient we even found stored Prometheus samples where the counter dips to -1 for a single scrape interval. A local reproducer (writer thread
calling endTransaction(true) in a loop while reading the difference) observes negatives within ~2,000 reads on hibernate-core 6.6.36.Final.
Suggested fix - clamp the derived value: s → Math.max( 0d, s.getTransactionCount() - s.getSuccessfulTransactionCount() ). I’m happy to submit the PR. Workaround for affected users: a
MeterFilter.deny on hibernate.transactions with tag result=failure.
The suggested fix sounds ok to me, but maybe it’s better to just change the order of operations to ensure this can’t happen by doing -s.getSuccessfulTransactionCount() + s.getTransactionCount()?
Right, that’s cleaner than clamping - since endTransaction() increments transactionCount before committedTransactionCount, reading the successful count first (which -s.getSuccessfulTransactionCount() + s.getTransactionCount() does, given left-to-right evaluation) guarantees every transaction visible in the second operand is also visible in the first, so the result can never go negative. The residual race only transiently overcounts failures by in-flight transactions, which passes validation and self-corrects on the next scrape. I’m happy to submit a PR with that change - should I wait for my JIRA account to get activated to file the ticket (jongsu@toss.im), or would you prefer to create one?
We heard a couple of times from users that it takes a bit of time until newly created accounts are fully activated, so maybe try again the next day.
I’m also open to introducing a new AtomicLong counter for the failed transactions in case you prefer that, given that failed transaction counter going up could lead to some notifications in certain monitoring tools, which would be inconvenient if the errors are just transient concurrency related artifacts.
Good news: my Jira account finally got activated - filed as [HHH-20783] with a test case based on the official test-case templates (reproduces ~31k negative reads in 10s against 6.6.55.Final).
A dedicated counter sounds best semantically - it removes the derivation entirely, so the value is exact, monotonic, and alert-safe. I’d be happy to implement that (presumably a new Statistics#getFailedTransactionCount with a derived default implementation for compatibility, and a real counter in StatisticsImpl).
One question: we’re hitting this in production on 6.6.x - would the new-counter change be backportable there, or would you prefer the evaluation-order fix for 6.6 as a lower-risk backport, with the dedicated counter going to main?
Thanks for creating the ticket and opening a PR. Unfortunately, the Hibernate ORM 6.6 branch is in limited maintenance mode and we only backport fixes to that branch if they are requested by Red Hat or IBM customers through official support channels.