# High performance autocomplete optimization

**URL:** <https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803>\
**Category:** Hibernate Search\
**Created:** [June 25, 2024, 6:00pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803 "2024-06-25T18:00:29Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![fivearise](https://avatars.discourse-cdn.com/v4/letter/f/838e76/32.png) [@fivearise](https://discourse.hibernate.org/u/fivearise)\
**Post date:** [June 25, 2024, 6:00pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/1 "2024-06-25T18:00:30Z")

</div>

I have a database with roughly 20k entries. I want to add an autocomplete for **one field only**.

Here’s the rough version of the autocomplete right now:

```auto
entity.keyword().fuzzy().withEditDistanceUpTo(1).onField("title").matching(userInput)

```

Is there a better way to do this?

I want to optimize the autocomplete so that only the first two words in the field are returned. I know I could trim off any remaining words manually, but I’m hoping there’s a better built-in solution.

To elaborate, I want the autocomplete to **match up to two words**.

---

<div class="post-metadata">

**Author:** ![mbekhta](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/mbekhta/32/2044_2.png) [@mbekhta](https://discourse.hibernate.org/u/mbekhta)\
**Post date:** [June 26, 2024, 6:53am UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/2 "2024-06-26T06:53:16Z")

</div>

Usually, for auto-complete kind of scenarios [edge ngram](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenfilter.html) or [ngram](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenfilter.html) filters are used at index time.

As to this part:

> [@fivearise](#):
>
> I want to optimize the autocomplete so that only the first two words in the field are returned. I know I could trim off any remaining words manually, but I’m hoping there’s a better built-in solution.
> 
> To elaborate, I want the autocomplete to **match up to two words**.

I’m not sure I understand what you are trying to achieve, but if you want the search to be performed only on the first two words of the title, e.g., for a title `High performance autocomplete optimization,` only autocomplete on `High performance`. If that’s so - you can add a [limit token count token filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-limit-token-count-tokenfilter.html). Just make sure that the limit filter is added before the ngram filter.

And just in case, here’s how to configure the analyzers with Hibernate Search:

- [Analysis for the Lucene backend](https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#backend-lucene-analysis)
- [Analysis for the Elasticsearch backend](https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#backend-elasticsearch-analysis)

---

<div class="post-metadata">

**Author:** ![fivearise](https://avatars.discourse-cdn.com/v4/letter/f/838e76/32.png) [@fivearise](https://discourse.hibernate.org/u/fivearise)\
**Post date:** [June 26, 2024, 3:30pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/3 "2024-06-26T15:30:38Z")

</div>

That looks good! You know what I’m looking for 🙂

Do you have the relevant documentation of ngram and the token limit for Lucene? The closest thing I could find it `@KeywordField`.

---

<div class="post-metadata">

**Author:** ![mbekhta](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/mbekhta/32/2044_2.png) [@mbekhta](https://discourse.hibernate.org/u/mbekhta)\
**Post date:** [June 26, 2024, 4:10pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/4 "2024-06-26T16:10:48Z")

</div>

ohh ok so for the Lucene you’d want to implement `LuceneAnalysisConfigurer` see this [example](https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#backend-lucene-analysis-analyzers) to get started.

As for the filters in the Lucene case you just have to look through the Lucene packages, in your case these are the ones you were looking for:

- `org.apache.lucene.analysis.ngram.NGramFilterFactory` : `nGram`
  - parameters: `minGramSize` / `maxGramSize`

- `org.apache.lucene.analysis.ngram.EdgeNGramFilterFactory` : `edgeNGram`
  - parameters: `minGramSize` / `maxGramSize`

- `org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilterFactory` : `limitTokenCount`
  - parameters: `maxTokenCount` / `consumeAllTokens`

and it’ll look something like the example below (`hibernate.search.backend.analysis.configurer`):

Create your configurer (don’t forget to pass it to Search, see the link above):

```java
public class YourAnalysisConfigurer implements LuceneAnalysisConfigurer {
	@Override
	public void configure(LuceneAnalysisConfigurationContext context) {
		context.analyzer( "someAnalyzerName" ).custom()
				.tokenizer( WhitespaceTokenizerFactory.class )
				// add some filters to clean up your text:
				.tokenFilter( StopFilterFactory.class )
				.tokenFilter( LowerCaseFilterFactory.class )
				// after these ^ filters you are expecting to have the words you want to auto-complete on
				// now add the limit filter to only keep two words
				.tokenFilter( LimitTokenCountFilterFactory.class )
				.param( "maxTokenCount", "2" )
				// add the ngram / or edgengram filter to generate the tokens
				.tokenFilter( EdgeNGramFilterFactory.class )
				.param( "minGramSize", "2" )
				.param( "maxGramSize", "15" );

		// same as above config but without the ngram filter, you'd use this one to be applied as a search analyzer
		context.analyzer( "someAnalyzerNameSearch" ).custom()
				.tokenizer( WhitespaceTokenizerFactory.class )
				// add some filters to clean up your text:
				.tokenFilter( StopFilterFactory.class )
				.tokenFilter( LowerCaseFilterFactory.class );
	}
}

```

and use apply these analyzers to your entity:

```java
@Entity
@Indexed
public class MyEntity {

	// ... other things

	@FullTextField(analyzer = "someAnalyzerName", searchAnalyzer = "someAnalyzerNameSearch")
	private String myText;

}

```

---

<div class="post-metadata">

**Author:** ![fivearise](https://avatars.discourse-cdn.com/v4/letter/f/838e76/32.png) [@fivearise](https://discourse.hibernate.org/u/fivearise)\
**Post date:** [June 26, 2024, 4:20pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/5 "2024-06-26T16:20:01Z")

</div>

This looks perfect! How would I perform the search? Is it as simple as `entity.match().field("myText").matching(userInput)`? (Sorry this is all very new to me 😛 )

---

<div class="post-metadata">

**Author:** ![fivearise](https://avatars.discourse-cdn.com/v4/letter/f/838e76/32.png) [@fivearise](https://discourse.hibernate.org/u/fivearise)\
**Post date:** [June 26, 2024, 5:53pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/6 "2024-06-26T17:53:59Z")

</div>

OK, I got the autocomplete to work thanks to you, ty! I have another question though: is it possible to conditionally apply the analyzer on certain queries?

For instance, I want the autocomplete to use the analyzer, but I want the actual search to use the default `FullTextField` analyzer with fuzzy search. Should I create two columns with the same value, and apply the analyzer to one of them?

---

<div class="post-metadata">

**Author:** ![mbekhta](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/mbekhta/32/2044_2.png) [@mbekhta](https://discourse.hibernate.org/u/mbekhta)\
**Post date:** [June 27, 2024, 7:12am UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/7 "2024-06-27T07:12:23Z")

</div>

> [@fivearise](#):
>
> Should I create two columns with the same value, and apply the analyzer to one of them?

It is common to have multiple index fields with different analysis configuration derived from the same entity field. In other words, you can just add multiple full-text annotations to it like:

```java
@Entity
@Indexed
public class MyEntity {

	// ... other things

	// Use this one for autocomplete query 
	// (make sure it has a name specified so that each field has a unique one).
	@FullTextField(name = "myTextAutocomplete", analyzer = "someAnalyzerName", searchAnalyzer = "someAnalyzerNameSearch")
	// Use this field for a search queries where autocomplete is not needed.
	@FullTextField(analyzer = "someOtherAnalyzerNameOrNoneIfDefaultIsGood")
	private String myText;

}

```

---

<div class="post-metadata">

**Author:** ![horvoje](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/horvoje/32/779_2.png) [@horvoje](https://discourse.hibernate.org/u/horvoje)\
**Post date:** [July 4, 2024, 11:00pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/8 "2024-07-04T23:00:27Z")

</div>

What about autocomplete on multiple fields (product name, manufacturer name, provider name, provider city, provider country)?

Right now I have a String property “autocomplete” inside product entity and I populate it during entity persisting process.  
Later I use MySQL stored procedure which uses LIKE operator.

So if you have a product “Almond Milk” from manufacturer “Alpro” sold in a store “Aldi” in Killorglin, Ireland - you can get this product using first letters of any of these data (“alp”, “ire” and so on).  
When you type another word, stored procedure adds AND operator and another LIKE.

Is there a way to get the same behaviour by Lucene?

---

<div class="post-metadata">

**Author:** ![mbekhta](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/mbekhta/32/2044_2.png) [@mbekhta](https://discourse.hibernate.org/u/mbekhta)\
**Post date:** [July 8, 2024, 2:58pm UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/9 "2024-07-08T14:58:57Z")

</div>

Hey @horvoje

you could have these fields you are combining annotated with a `@FullTextField` and use `ngram` filter in the analyzer for them, e.g. something like:

```java
@FullTextField(name = "productAutocomplete", analyzer = "someAnalyzerName", searchAnalyzer = "someAnalyzerNameSearch")
String product;
@FullTextField(name = "manufacturerAutocomplete", analyzer = "someAnalyzerName", searchAnalyzer = "someAnalyzerNameSearch")
String manufacturer;
.... other fields

```

And then use aquery string predicate targeting multiple fields ([Hibernate Search 7.1.1.Final: Reference Documentation](https://docs.jboss.org/hibernate/search/7.1/reference/en-US/html_single/#search-dsl-predicate-common-multiple-fields)):

```java
List<Product> hits = searchSession.search( Product.class )
        .where( f -> f.simpleQueryString()
                .field( "product" ).field( "manufacturer" )
                .matching( "alp" ) )
        .fetchHits( 20 );

```

for this part:

> stored procedure adds AND operator and another LIKE

you’d just use the `AND` operator option on a simple query string predicate ([Hibernate Search 7.2.0.Alpha2: Reference Documentation](https://docs.jboss.org/hibernate/search/7.2/reference/en-US/html_single/#search-dsl-predicate-simple-query-string-boolean-operators-default)):

```java
List<Product> hits = searchSession.search( Product.class )
        .where( f -> f.simpleQueryString()
                .field( "product" ).field( "manufacturer" )
                .matching( "alp" )
                .defaultOperator( BooleanOperator.AND ) )
        .fetchHits( 20 );

```

---

<div class="post-metadata">

**Author:** ![dinaadams](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/dinaadams/32/3342_2.png) [@dinaadams](https://discourse.hibernate.org/u/dinaadams)\
**Post date:** [July 10, 2024, 9:27am UTC](https://discourse.hibernate.org/t/high-performance-autocomplete-optimization/9803/10 "2024-07-10T09:27:35Z")

</div>

Hello,  
Thanks for sharing, this is very helpful for me.