AWS Elasticsearch Direct Access via RestClient

Hibernate Search
1

OK, I’m going to cross my fingers on this one as I’m not 100% certain this belongs on this forum. I’m attempting to submit an _msearch query (otherwise I would need to submit a couple hundred individual queries) against the Elasticsearch 6.3 instance running on AWS that Hibernate Search (5.10.5.Final) is running against.

I’m doing this through the direct access method mentioned here:

https://docs.jboss.org/hibernate/search/5.10/reference/en-US/html_single/#elasticsearch-client-access

This works locally (of course) when connecting to my local running instance of ES. In my QA environment against the AWS instance, I’m getting a 400 Bad Request response.

Has anyone run into this? Is this a known issue? I’ve seen some mentions of similar issues in other forums but no clear resolutions. Any assistance would be most appreciated!

Below is the exception be thrown:

Caused by: org.elasticsearch.client.ResponseException: method [POST], host [https://vpc-park-qa-jrg3sp11z2rg4abrr7k63zfo2i.us-west-2.es.amazonaws.com], URI [_msearch], status line [HTTP/1.1
400 Bad Request]
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
</body>
</html>

Cheers!

2

Since you didn’t mention it, I’m going to take the risk of stating the obvious: did you configure the AWS access key and secret key? See this section of the documentation.

If you did configure it, do simple requests (e.g. a simple search through the Hibernate Search DSL) work correctly?

If they do not work correctly, I’d advise to double check your credentials.

If they work correctly but your _msearch query does not, it’s likely a problem with how you use the rest client, and we’ll need to know more about what you’re doing exactly, and how you’re doing it.

Please note that the Elasticsearch Service on AWS does not support all operations. You may want to check this page.

3

Thanks for the rapid response!!

I think I’m covered on your question regarding AWS key mgmt. Here’s a scrubbed version of the appropriate section of my apps yml file below. And I’m able to perform DSL queries without any issues - so I think that ok from an access/security stand point:

aws:
  signing:
    enabled: true
  access_key: ${INJECTED_AWS_ACCESS_KEY_ID:placeholder}
  secret_key: ${INJECTED_AWS_SECRET_ACCESS_KEY:placeholder}
  region: ${AWS_REGION:us-west-2}

Here’s the code I"m attempting to implement (the code is slightly modified for reabability and such). The short of it is I’m adding the index info (it’s the same for each query) prior to each query:

String PERSON_INDEX = "{\"index\" : \"com.mycompany.domain.person\"}\n";

And then I’m actually unwrapping the Query that the hibernate search DSL is generating (and this works when I don’t call it through the _msearch method(ie, just using the DSL method) as follows:

    Request request = new Request( "POST", "_msearch" );
    StringBuilder json = new StringBuilder();
    List<Group> dynamicGroups = getDynamicGroups( companyId );
    // order is important as the response matches the input order
    dynamicGroups.stream().forEachOrdered( group ->
    {
      json.append( buildJsonQuery( companyId, personId, group ) );
    } );
    request.setJsonEntity( json.toString() );

    Response response = buildRestClient().performRequest( request );
    GroupSearchView groupSearchView = objectMapper.readValue( response.getEntity().getContent(), GroupSearchView.class );


  private String buildJsonQuery( UUID companyId, UUID personId, Group group )
  {
    StringBuilder sb = new StringBuilder( PERSON_INDEX );
    QueryBuilder queryBuilder = getFullTextEntityManager().getSearchFactory().buildQueryBuilder().forEntity( Person.class ).get();
    BooleanJunction<BooleanJunction> booleanJunctions = buildGroupCriteria( queryBuilder, companyId, group );
    // Add the Person ID as a required criteria
    booleanJunctions.must( queryBuilder.keyword().onField( "id" ).matching( personId ).createQuery() );
    // NOTE: Elasticsearch is really picky about having a new-line character as it uses it as a
    // delimiter
    return sb.append( unwrapQuery( booleanJunctions.createQuery() ) + "\n" ).toString();
  }

  // Yep, this is some good old fashioned string parsing..yucko
  private String unwrapQuery( Query query )
  {
    String unwrapped = getFullTextEntityManager().createFullTextQuery( query, Person.class ).toString();
    int idxStart = "FullTextQueryImpl(".length();
    int idxEnd = unwrapped.lastIndexOf( ")" );
    return unwrapped.substring( idxStart, idxEnd );
  }

Appreciate you looking at this, really!

Cheers!

4

I’ve never used _msearch myself, but your syntax looks good… and you told me it works locally, so I suppose it’s fine as long as you’re using the same ES version.

What does your buildRestClient() method do? Could you try just sending a GET request to / and check that it works correctly? It should return JSON containing information about the Elasticsearch instance.

As a side note, you really should use fullTextQuery.getQueryString() rather than parsing the result of the toString() method.

Also… are you absolutely sure that just running a single query wouldn’t work? If it’s only about applying different criteria to each person, you can do it very simply by having multiple levels of boolean junctions. In essence you would have the following structure:

booleanJunction1
- should
   - booleanJunction1.1
      - must( id = personId1 )
      - must( some company-related criterion )
      - must( some group-related criterion )
- should
   - booleanJunction1.2
      - must( id = personId2 )
      - must( some company-related criterion )
      - must( some group-related criterion )
- and so on for each "dynamicGroup"

Of course, that may or may not work depending on what your GroupSearchView is. If a person can appear in multiple groups and you need to know which group matched, then yes, you’re probably stuck with _msearch at the moment.

5

Wrote up some of the answers to your questions below, but I’m now thinking this is not an issue with hibernate search (so feel free to not read the rest!). I did try performing a GET to with a URI of / from the rest client and I was able to obtain the ES info. And in hindsight, I really should have been tipped off by the response from my _msearch POST. That response was HTML, which I don’t believe ES even deals with(it responds in JSON only AFAIK). So that tells me something else in the network is generating that response and am following up on that. Will probably close this issue in the next day or two pending resolution.

As for using fullTextQuery.getQueryString() - I don’t see that in my IDE as an auto-complete option. It’s not present in the interface but I do see now it’s available in the concrete impl org.hibernate.search.query.hibernate.impl.FullTextQueryImpl. Which I confirmed produces identical results. :thumbsup:

Below is the construction of the RestClient:

  private RestClient buildRestClient()
  {
    SearchFactory searchFactory = getFullTextEntityManager().getSearchFactory();
    IndexFamily indexFamily = searchFactory.getIndexFamily( ElasticsearchIndexFamilyType.get() );
    ElasticsearchIndexFamily elasticsearchIndexFamily = indexFamily.unwrap( ElasticsearchIndexFamily.class );
    return elasticsearchIndexFamily.getClient( RestClient.class );
  }

Lastly, on your question regarding the query - you are correct in that a Person can belong to 0-n groups and I must evaluate each group to return those that a Person is a member of. So I am kinda stuck with the _msearch method or the even less efficient DSL method of making a call-per-group. :frowning:

1 Like
6

Right on…found the issue. This worked fine when I prefix the URI with a forward slash:

Request request = new Request( "POST", "/_msearch" );

Still don’t understand why/how this worked on my local development machine…but I’m past caring at this point. Thanks again for taking a look - sorry for the wild-goose chase!

Cheers!