Elasticsearch tracing

Hibernate Search
1

Hello Hibernate Team,

I was recently developing a microservice via quarkus with the new hibernate search extension. Hibernate search is working great thanks to all the contributors.

I wanted to include tracing functionality with opentracing-contrib/java-elasticsearch-client yet i could not find a way to initialize the elasticsearch RestClient in hibernate search.

I explicitly created the client as the client and was able to observe the trace to elasticsearch cluster such as;

// Build RestClient adding TracingHttpClientConfigCallback
RestClient restClient = RestClient.builder(
new HttpHost(…))
.setHttpClientConfigCallback(new TracingHttpClientConfigCallback(tracer))
.build();

Is there a way to initialize the RestClient in hibernate search as such? Could this be added as a configurable component?

2

There is a way to configure the underlying HTTP client. Be aware this is SPI, to it’s intended for integrators and may change in backward-incompatible ways, more often than APIs (e.g. in minor versions).

You will need to:

  • Implement org.hibernate.search.backend.elasticsearch.client.spi.ElasticsearchHttpClientConfigurer
  • Implement BeanConfigurer like this:
    public class MyBeanConfigurer implements BeanConfigurer {
	@Override
	public void configure(BeanConfigurationContext context) {
		context.assignRole(
				ElasticsearchHttpClientConfigurer.class,
				BeanReference.of( MyClientConfigurer.class )
		);
	}
}
  • Register your bean configurer by adding a file named META-INF/services/org.hibernate.search.engine.environment.bean.spi.BeanConfigurer to your resources, containing a single line with the fully-qualified class name of MyBeanConfigurer (e.g. com.mycompany.MyBeanConfigurer).

You can have a look at the code of the AWS integration for an example: it needs to change the HTTP client configuration as well.

3

Hello @yrodiere,

Thanks for your kind reply.

I understand better now how to create custom beans. However, I am still having some difficulties.

I tried the following and made MyBeanConfigurer class and added it into resources/META-INF/services/org.hibernate.search.engine.environment.bean.spi.BeanConfigurer


public class MyElasticsearchHttpClientConfigurer implements ElasticsearchHttpClientConfigurer {

    @Override
    public void configure(HttpAsyncClientBuilder httpAsyncClientBuilder, ConfigurationPropertySource configurationPropertySource) {

        RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback = new TracingHttpClientConfigCallback(new QuarkusJaegerTracer());
        httpClientConfigCallback.customizeHttpClient(httpAsyncClientBuilder);

    }
}

However still was not able to get the traces.

Also, I am not completely sure this is the correct way to add a
RestClientBuilder.HttpClientConfigCallback to the httpAsyncClientBuilder. Do you think I am doing this incorrect?

4

Not totally related to your question but I think we probably need some additional work to make this work in native.

The service file won’t be included if we don’t say so.

At least we need to be sure this is resolved at static init and not later.

1 Like
5

First step would be to put a breakpoint in your new class, and launch the application in debug mode to check that it’s correctly being detected. If it’s not, your new resource file may be missing from your JAR?

The snippet of code you showed looks good.

That being said, I don’t know where the tracer gets its information from, but if it relies on thread locals to store then fetch context, then it definitely won’t work. Elasticsearch requests are processed and sent from dedicated thread pools, not from application threads. That might explain why you’re not getting any trace?

1 Like
6

Hi @gsmet are you saying the service file won’t be added for native specifically or in JVM mode as well.

@yrodiere I added a breakpoint and launched the application yet still was not able to detect MyElasticsearchHttpClientConfigurer implementation. I checked the jar file and all the following where included

META-INF/services/org.hibernate.search.engine.environment.bean.spi.BeanConfigurer
com/acme/client/MyElasticsearchHttpClientConfigurer.class
com/acme/config/bean/MyBeanConfigurer.class

That being said, I don’t know where the tracer gets its information from, but if it relies on thread locals to store then fetch context, then it definitely won’t work. Elasticsearch requests are processed and send from dedicated thread pools, not from application threads. That might explain why you’re not getting any trace?

I am not completely sure to be honest. I would have to research.

7

Did you put a breakpoint in MyBeanConfigurer as well? Also, did you start the application with mvnw clean quarkus:dev -Dsuspend? The code is executed right on startup, so if you connect your debugger one second after the startup, it won’t catch anything.

I just tried the solution I described on the Quarkus quickstart, and the HTTP client configurer is successfully executed, at least in JVM mode: Add a custom ElasticsearchHttpClientConfigurer · yrodiere/quarkus-quickstarts@edc9a44 · GitHub

Now whether tracing will actually do something is another question…

8

You are correct the -Dsuspend command worked. I was able to detect the MyBeanConfigurer and MyElasticsearchHttpClientConfigurer.

I assume you say this because of Elasticsearch search requests are processed and sent from dedicated thread pools?

I did however added a self created elasticsearch RestClient and was able to register the tracer to jaeger as such.

    @GET
    @Path("test")
    @Produces(MediaType.APPLICATION_JSON)
    public void test() throws IOException {
        RestClient elasticClient = RestClient.builder(new HttpHost("localhost",9200))
                .setHttpClientConfigCallback(new TracingHttpClientConfigCallback(new QuarkusJaegerTracer()))
                .build();
        Request request = new Request(
                "GET",
                "/");
        Response response = elasticClient.performRequest(request);
    }

Trace:

Screenshot from 2020-03-12 15-03-43
Screenshot from 2020-03-12 15-03-431920Ă—1080 114 KB

However, this approach is clearly not in a dedicated thread pools.

Any suggestions?

9

Not really. Passing the tracing context from the application threads to the thread pools that actually send the Elasticsearch requests would require changes to Hibernate Search; I don’t think there’s anything you can do as a user with the current version of Hibernate Search. But the next versions could support it :slight_smile:

If you’re interested in such modifications, you can open a ticket and describe how you think this should be configured from the user’s point of view. Ideally, if you have any idea, you could also explain how to get the context the tracer needs from the application thread, and how to pass the context to the tracer… or even propose a solution :slight_smile:

We’ll want this to be implemented in a technologically agnostic way, i.e. without a direct dependency from pre-existing Hibernate Search modules to Jaeger. That can be achieved by defining an SPI and one implementation per technology (Jaeger, …), or by relying on a “standard” API such as OpenTracing.

10

@yrodiere thanks for you kind replies :slightly_smiling_face:

I will think of a solution and create a ticket.

I would love to have a crack at implementing this :slight_smile:

Thanks again

1 Like