Intermittent "Could not interpret path expression" on the first HQL parse per JVM (7.4.5.Final, after HHH-19758)

[dingko]

Hi,

We are seeing a rare HQL parse failure that looks like the same class of defect as HHH-19758 (HQL parse failure with SLL can lead to wrong parse), but on a version that already contains that fix. I could not reproduce it in isolation, so I am posting here rather than filing a Jira issue.

Environment

• Hibernate ORM 7.4.5.Final (Spring Boot 4.1.0 manages 7.4.1.Final; we override to 7.4.5.Final because of HHH-20467)
• Spring Boot 4.1.0, Spring Data JPA 4.1.0
• ANTLR runtime 4.13.2
• JDK 25, MySQL
• Kubernetes, several pods behind a load balancer

Symptom

Spring Data JPA derived query methods occasionally fail with SemanticException: Could not interpret path expression, surfaced by Spring as BadJpqlGrammarException. The JPQL printed in the very same message is valid, and the identical query succeeds on every later call.

Two shapes appeared in production over 7 days, 11 occurrences in total:

Could not interpret path expression 'S.chipId';
Bad SELECT a FROM AdCta a WHERE a.chipId IN :chipIds ORDER BY a.seq asc grammar [JPQL]

Could not interpret path expression 'a';
Bad SELECT a FROM AdChip a WHERE a.groupId = :groupId AND a.active = TRUE ORDER BY a.displayOrder asc grammar [JPQL]

Why this looks like a broken parse tree rather than a bad query

1. In the second case the unresolvable path is 'a' — the alias itself, declared right there in FROM AdChip a. The alias appears not to have been registered while the WHERE clause was being resolved.
2. In the first case the path is 'S.chipId', but there is no alias S anywhere in the query. S is the first character of SELECT, which is what CommonToken.getText() returns when a token’s start and stop indices have both collapsed to 0.

Both fragments in that message come from the same String in Spring Data’s PartTreeJpaQuery, so under normal operation they cannot disagree.

It happens exactly once per JVM

Aggregating the occurrences by pod shows exactly one per pod, with distinct ReplicaSet hashes, and never a second one on the same pod. That is consistent with QueryInterpretationCache being per-JVM: the failed parse is not cached, the next call re-parses successfully, and the pod is healthy from then on. The visible impact is one failed request per pod, and the user succeeds on retry.

It also lines up with the parser’s prediction caches being JVM-wide static, while parser and lexer instances are created per query by HqlParseTreeBuilder:

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache;

public HqlParser(TokenStream input) {
    super(input);
    _interp = new ParserATNSimulator(this, _ATN, _decisionToDFA, _sharedContextCache);
}

So the “caches are still cold” window exists only once per JVM, which matches the once-per-pod pattern. Our guess is that several threads parsing for the first time concurrently can yield a wrong parse from the SLL pass, but we have not been able to prove that.

Relation to HHH-19758

The fix for HHH-19758 (Don't reset lexer on SLL parse error, PR #10894, backported in #11226 and #11227) is already present in 7.4.5.Final — StandardHqlTranslator.parseHql() carries the commented-out call together with its warning:

// Resetting the lexer causes it to hand out tokens again from the start, which will then
// append to the CommonTokenStream and cause a wrong parse
// hqlLexer.reset();

So what we are seeing would be a remaining path that can still produce a wrong parse, possibly in the SLL to LL fallback in that same method.

What I tried (no reproducer)

• Executing the same derived query methods against the real runtime classpath produces byte-identical JPQL that parses and runs correctly.
• A parser stress test (64 threads, millions of iterations, with plan cache size 1 and 2048) did not reproduce it.
• The limitation is that those runs used an already warmed-up JVM. The production condition appears to be “cold JVM, several threads parsing for the first time concurrently”, and the window closes once the DFA stabilises after the first few parses.
• Our staging environment never reproduces it, which is consistent with its lower traffic never creating concurrent first parses.

Questions

1. Is this a known remaining path after HHH-19758, or is it worth a separate Jira issue?
2. Is there a supported way to force the LL prediction mode, or to pre-warm the parser, that we could use to confirm the hypothesis?
3. If it helps, I can run instrumentation or a patched build in production, since we can observe this roughly once per pod start.

Thanks!

[beikov]

@dingko

1. Is this a known remaining path after HHH-19758, or is it worth a separate Jira issue?

Not sure what you mean by “known remaining path”, but whatever problem you are encountering here, it may be that HHH-19758 was not fully fixed. We will need more details to understand the cause for this problem though.

@dingko

2. Is there a supported way to force the LL prediction mode, or to pre-warm the parser, that we could use to confirm the hypothesis?

No. You can build a custom version of Hibernate ORM that changes org.hibernate.query.hql.internal.StandardHqlTranslator#parseHql to never use SLL, and if you don’t see the error after that change anymore, dig deeper into how the SLL handling could be the reason for your issue.

@dingko

3. If it helps, I can run instrumentation or a patched build in production, since we can observe this roughly once per pod start.

For a start, it would be helpful to see the full stack trace and the involved entity classes.

[dingko]

Thanks for the quick reply!

Before I go build a patched Hibernate — is replacing the translator via hibernate.query.hql.translator (QuerySettings.SEMANTIC_QUERY_PRODUCER) an equivalent experiment? HqlTranslator is a single-method interface, so I could ship a copy of StandardHqlTranslator#parseHql that skips the SLL pass and goes straight to PredictionMode.LL with the DefaultErrorStrategy. That would let me run it in production, where we reliably see the error about once per pod start, instead of only in a test environment where I have never reproduced it.

If that is not equivalent (e.g. the custom translator is not used on every path, or something else about the setup differs), I will build the patched jar instead — just let me know which you prefer.

Everything you asked for is below.

Full stack trace

BadJpqlGrammarException — AdCta case

org.springframework.data.jpa.repository.query.BadJpqlGrammarException: org.hibernate.query.SemanticException: Could not interpret path expression ‘S.chipId’; Bad SELECT a FROM AdCta a WHERE a.chipId IN :chipIds ORDER BY a.seq asc grammar [JPQL]
at org.springframework.data.jpa.repository.query.PartTreeJpaQuery$QueryPreparer.createQuery(PartTreeJpaQuery.java:242)
Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
*__checkpoint ⇢ Handler com.example.app.SomeResource#someHandler(…) [DispatcherHandler]
Original Stack Trace:
at org.springframework.data.jpa.repository.query.PartTreeJpaQuery$QueryPreparer.createQuery(PartTreeJpaQuery.java:242)
at org.springframework.data.jpa.repository.query.PartTreeJpaQuery.doCreateQuery(PartTreeJpaQuery.java:125)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.createQuery(AbstractJpaQuery.java:266)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:141)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:100)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:166)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:156)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:169)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:167)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:146)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:133)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:130)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:135)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.util.NullnessMethodInvocationValidator.invoke(NullnessMethodInvocationValidator.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222)
at jdk.proxy3/jdk.proxy3.$Proxy246.findAllByChipIdInOrderBySeqAsc(Unknown Source)
at com.example.app.ChipProvider$getChips$2.invokeSuspend(ChipProvider.kt:49)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
Caused by: java.lang.IllegalArgumentException: org.hibernate.query.SemanticException: Could not interpret path expression ‘S.chipId’
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:134)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:173)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:179)
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:1102)
at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:158)
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:1011)
at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:158)
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:154)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:360)
at jdk.proxy3/jdk.proxy3.$Proxy215.createQuery(Unknown Source)
… (Spring proxy frames omitted) …
at org.springframework.data.jpa.repository.query.PartTreeJpaQuery$QueryPreparer.createQuery(PartTreeJpaQuery.java:240)
… 50 common frames omitted
Caused by: org.hibernate.query.SemanticException: Could not interpret path expression ‘S.chipId’
at org.hibernate.query.hql.internal.BasicDotIdentifierConsumer$BaseLocalSequencePart.resolveTerminal(BasicDotIdentifierConsumer.java:180)
at org.hibernate.query.hql.internal.BasicDotIdentifierConsumer$BaseLocalSequencePart.resolvePathPart(BasicDotIdentifierConsumer.java:153)
at org.hibernate.query.hql.internal.BasicDotIdentifierConsumer.consumeIdentifier(BasicDotIdentifierConsumer.java:90)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSimplePath(SemanticQueryBuilder.java:5790)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitGeneralPathFragment(SemanticQueryBuilder.java:5645)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitGeneralPathExpression(SemanticQueryBuilder.java:1815)
at org.hibernate.grammars.hql.HqlParser$GeneralPathExpressionContext.accept(HqlParser.java:7847)
at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:46)
at org.hibernate.grammars.hql.HqlParserBaseVisitor.visitBarePrimaryExpression(HqlParserBaseVisitor.java:761)
at org.hibernate.grammars.hql.HqlParser$BarePrimaryExpressionContext.accept(HqlParser.java:7296)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitInPredicate(SemanticQueryBuilder.java:3267)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitInPredicate(SemanticQueryBuilder.java:247)
at org.hibernate.grammars.hql.HqlParser$InPredicateContext.accept(HqlParser.java:6392)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitWhereClause(SemanticQueryBuilder.java:2330)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitWhereClause(SemanticQueryBuilder.java:247)
at org.hibernate.grammars.hql.HqlParser$WhereClauseContext.accept(HqlParser.java:6146)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuery(SemanticQueryBuilder.java:1163)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuerySpecExpression(SemanticQueryBuilder.java:956)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQuerySpecExpression(SemanticQueryBuilder.java:247)
at org.hibernate.grammars.hql.HqlParser$QuerySpecExpressionContext.accept(HqlParser.java:2113)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQueryExpression(SemanticQueryBuilder.java:985)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitQueryExpression(SemanticQueryBuilder.java:247)
at org.hibernate.grammars.hql.HqlParser$QueryExpressionContext.accept(HqlParser.java:2003)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitSelectStatement(SemanticQueryBuilder.java:472)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.visitStatement(SemanticQueryBuilder.java:431)
at org.hibernate.query.hql.internal.SemanticQueryBuilder.buildSemanticModel(SemanticQueryBuilder.java:292)
at org.hibernate.query.hql.internal.StandardHqlTranslator.translate(StandardHqlTranslator.java:61)
at org.hibernate.query.internal.QueryInterpretationCacheStandardImpl.createHqlInterpretation(QueryInterpretationCacheStandardImpl.java:165)
at org.hibernate.query.internal.QueryInterpretationCacheStandardImpl.resolveHqlInterpretation(QueryInterpretationCacheStandardImpl.java:147)
at org.hibernate.query.spi.QueryEngine.interpretHql(QueryEngine.java:61)
at org.hibernate.internal.AbstractSharedSessionContract.interpretHql(AbstractSharedSessionContract.java:1043)
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:1094)
… 50 common frames omitted

Note the failure comes from the SLL/LL path in StandardHqlTranslator.translate → SemanticQueryBuilder.buildSemanticModel, reached through QueryInterpretationCacheStandardImpl.createHqlInterpretation (i.e. a cache miss).

Entities

Kotlin, but plain JPA annotations — no Hibernate-specific mappings, no associations between these two entities (AdCta.chipId is a plain Long, not a @ManyToOne).

@MappedSuperclass
@EntityListeners(AuditingEntityListener::class)
abstract class BaseEntity {
@CreatedDate
@Column(name = "created_at", nullable = false, updatable = false)
lateinit var createdAt: LocalDateTime

  @LastModifiedDate
  @Column(name = "modified_at")
  lateinit var modifiedAt: LocalDateTime

}

@EntityListeners(AuditingEntityListener::class)
@Entity
@Table(name = "ad_cta")
class AdCta(
@Id @GeneratedValue(strategy = GenerationType.IDENTITY)
val id: Long? = null,

  @Column(name = "chip_id", nullable = false)
  val chipId: Long,

  @Column(name = "seq", nullable = false)
  var seq: Int,

  @Column(name = "label", length = 50, nullable = false)

@Column(name = "label", length = 50, nullable = false)
var label: String,

@Enumerated(EnumType.STRING)
@Column(name = "link_type", length = 16, nullable = false)
var linkType: AdCtaLinkType,

@Column(name = "link_value", length = 1024, nullable = false)
var linkValue: String,

) : BaseEntity()

@EntityListeners(AuditingEntityListener::class)
@Entity
@Table(name = "ad_chip")
class AdChip(
@Id @GeneratedValue(strategy = GenerationType.IDENTITY)
val id: Long? = null,

@Column(name = "group_id", nullable = false)
val groupId: Long,

@Column(name = "display_order", nullable = false)
var displayOrder: Int,

@Column(name = "active", nullable = false, columnDefinition = "TINYINT(1)")
var active: Boolean = true,

@Column(name = "list_label", length = 8, nullable = false)
var listLabel: String,

@Enumerated(EnumType.STRING)
@Column(name = "material_type", length = 16, nullable = false)
var materialType: AdMaterialType,

// ... remaining columns are plain String?/Int? fields

) : BaseEntity()

Repositories (both are plain derived queries)

interface AdCtaRepository : JpaRepository<AdCta, Long> {
fun findAllByChipIdOrderBySeqAsc(chipId: Long): List
fun findAllByChipIdInOrderBySeqAsc(chipIds: List): List   // ← fails
fun deleteAllByChipId(chipId: Long)
}

interface AdChipRepository : JpaRepository<AdChip, Long> {
fun findAllByGroupIdOrderByDisplayOrderAsc(groupId: Long): List
fun findAllByGroupIdAndActiveIsTrueOrderByDisplayOrderAsc(groupId: Long): List  // ← also fails
// ...
}

Both are called from the same method, one right after the other:

val chips = chipRepository.findAllByGroupIdAndActiveIsTrueOrderByDisplayOrderAsc(group.id!!)
if (chips.isEmpty()) return@withContext emptyList()

val ctasByChipId = ctaRepository
.findAllByChipIdInOrderBySeqAsc(chips.map { it.id!! })
.groupBy { it.chipId }

This runs inside withContext(Dispatchers.IO), so both queries execute on a shared JDBC thread pool — several requests can be parsing HQL concurrently right after startup.

Relevant Hibernate settings

hibernate.hbm2ddl.auto: validate
hibernate.jdbc.time_zone: Asia/Seoul
hibernate.default_batch_fetch_size: 100
hibernate.cache.use_second_level_cache: false
hibernate.cache.use_query_cache: false
hibernate.order_inserts: true
hibernate.order_updates: true
hibernate.jdbc.batch_size: 50
hibernate.jdbc.batch_versioned_data: true

No custom HqlTranslator, no custom dialect beyond the MySQL one, query plan cache left at defaults.

[beikov]

@dingko

Before I go build a patched Hibernate — is replacing the translator via hibernate.query.hql.translator (QuerySettings.SEMANTIC_QUERY_PRODUCER) an equivalent experiment?

Yes, you can also do that.

@dingko

HqlTranslator is a single-method interface, so I could ship a copy of StandardHqlTranslator#parseHql that skips the SLL pass and goes straight to PredictionMode.LL with the DefaultErrorStrategy. That would let me run it in production, where we reliably see the error about once per pod start, instead of only in a test environment where I have never reproduced it.

Correct.

[beikov]

The error message says that it can’t interpret the path S.chipId for the query SELECT a FROM AdCta a WHERE a.chipId IN :chipIds ORDER BY a.seq asc, which is correct, since there is no alias S defined. I don’t know where this S comes from, but maybe also try to specify the HQL/JPQL query directly on the repository method to rule out Spring Data being the culprit.

[dingko]

Thanks — I tried the @Query route first since it was the cheaper test.

Result: the error no longer reproduces.

I replaced only the two failing derived queries with explicit @Query methods,
keeping the method signatures intact:

@Query("""
    SELECT c FROM AdChip c
    WHERE c.groupId = :groupId AND c.active = true
    ORDER BY c.displayOrder ASC
""")
fun findAllByGroupIdAndActiveIsTrueOrderByDisplayOrderAsc(groupId: Long): List<AdChip>

@Query("""
    SELECT c FROM AdCta c
    WHERE c.chipId IN :chipIds
    ORDER BY c.seq ASC
""")
fun findAllByChipIdInOrderBySeqAsc(chipIds: List<Long>): List<AdCta>

Since the deploy it has not appeared once, where before it showed up on
essentially every pod startup. The contrast is unambiguous, though I’d note the
observation window is still short.

One caveat about how conclusive this is. The HQL string is not byte-identical
to what Spring Data generates. Mine differs in three ways:

	Spring Data derived	my @Query
alias	a (this matches the unresolvable alias 'a' from my first post)	c
parameters	positional ?1	named :groupId
keyword case	lowercase	uppercase

So this rules out “the derived-query path as a whole”, but it does not yet
separate generation path from string shape. If that distinction matters,
I can capture the exact HQL Spring Data emits and feed it verbatim into @Query —
that would leave the generation path as the only variable.

One more data point that may be relevant. The same repository already had a
hand-written @Query method before any of this:

@Query("""
    SELECT c.groupId AS groupId, COUNT(c.id) AS activeCount
    FROM AdChip c
    WHERE c.groupId IN :groupIds AND c.active = true
    GROUP BY c.groupId
""")
fun countActiveByGroupIdIn(groupIds: Collection<Long>): List<ActiveChipCount>

It uses the same IN :param form and has never appeared in the failures. It sits
on a different call path (admin list view) though, so it may simply never have
been parsed during the concurrent startup window.

Environment, for completeness: Spring Boot 4.1.0, Spring Data JPA 4.1.0,
Hibernate 7.4.5.Final (overriding Boot’s managed 7.4.1.Final).

Given that the derived-query path is implicated, should I take this to the Spring
Data side, or is there something on the Hibernate parser side you’d still want
tested first? I’m happy to run the LL-prediction-mode HqlTranslator as well if
that would produce more useful information.

[beikov]

I don’t know what to tell you. The error does not make any sense to me, so I suspect some concurrency related problem, but I can’t imagine how this can go wrong. I’ll dig a bit into it, but OTOH, you should be able to reproduce the same problem also with the static query. From what I can tell, Spring Data JPA also uses a “static” query, so it shouldn’t make a difference what approach you use. Maybe it changes the timing a bit and causes it to be less likely.

[beikov]

What architecture are you running on in your Kubernetes cluster? Are you using e.g. AWS Graviton? I might have found something that could explain the problem you’re having.

[dingko]

Yes — AWS Graviton, and it is the majority of our fleet.

Our production Kubernetes nodes are roughly 84% aarch64 (Graviton) and 16% x86_64 (127 vs 25 nodes). The JVM runs on whichever node the pod lands on, so most of our pods are on ARM.

Our staging cluster has almost the same mix (~82% aarch64), and we have never seen the error there — but I want to be careful not to present that as evidence against an ARM explanation, because staging gets nowhere near enough traffic to create the window. Over the last 7 days, the endpoint that triggers this query received:

• production: 23,838 requests
• staging: 125 requests

That is ~190x, or about 18 requests per day in staging. The chance of several of them landing concurrently on a freshly started pod is essentially zero, so staging would stay silent regardless of architecture.

I tried to check retroactively whether the failing pods specifically landed on aarch64 nodes, but I could not:

• the nodes are recycled frequently, so the hosts from past occurrences are no longer in the host inventory
• the log lines themselves do not carry a CPU-architecture tag

What I could confirm again is the distribution: over the last 30 days there were 29 occurrences in production across 29 distinct hosts — exactly one per host, never a second one on the same host.

If it helps, I can add os.arch to the log line (or as a Datadog tag) and report back. The error shows up reliably about once per pod start, so a few days after the next deploy should give a clean answer on whether it only happens on aarch64 — with ~16% of our nodes being x86_64, a strong skew either way should be visible quickly.

Out of curiosity, what did you find? If there is a known ARM-related issue in this area I would be happy to test a candidate fix in production.

[beikov]

See [#4936] Ensure DFAState#edges are accessed in a thread safe manner by beikov · Pull Request #4978 · antlr/antlr4 · GitHub