# Hibernate Search on special characters

**URL:** https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768
**Category:** Hibernate Search
**Created:** [February 20, 2020, 2:55pm UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768 "2020-02-20T14:55:08Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![kamokaizen](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/kamokaizen/32/959_2.png) [@kamokaizen](https://discourse.hibernate.org/u/kamokaizen)
#### Post date: [February 20, 2020, 2:55pm UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/1 "2020-02-20T14:55:08Z")

</div>

Hi guys, I’m using the hibernate full-text search on my application. My problem is that I could not search if the search string has some special characters. For example, the search strings are like that “Honda CR-V” or “Peugeot 206+”, hibernate search returns empty response.

I use the below configuration on the index. Also, I try with WhitespaceTokenizerFactory with changing Tokenizer, but it did not work.

@AnalyzerDef(name = “searchTextAnalyzer”,  
tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),  
filters = {  
@TokenFilterDef(factory = LowerCaseFilterFactory.class),  
@TokenFilterDef(factory = StandardFilterFactory.class)  
})

My hibernate version is 5.4.5.Final and hibernate search version is 5.11.3.Final.  
Could you help me to solve the problem. Thanks for any help.

---

<div class="post-metadata">

### Author: ![yrodiere](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/yrodiere/32/1760_2.png) [@yrodiere](https://discourse.hibernate.org/u/yrodiere)
#### Post date: [February 20, 2020, 3:04pm UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/2 "2020-02-20T15:04:23Z")

</div>

Hello,

I’ll need more information. In particular:

- The code that creates and executes the search query.
- The code of the the relevant entities, including in particular `@Indexed`/`@Field` annotations.
- Do you use the experimental Elasticsearch integration, or just Lucene (the default)?
- An example of a search string that doesn’t work **with the text that it should match**. E.g. you’re looking for “Peugeot 206+”, but what is the name of the car in your database? It it “Peugeot 206+” too, or something slightly different such as “peugeot 206+”?

---

<div class="post-metadata">

### Author: ![kamokaizen](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/kamokaizen/32/959_2.png) [@kamokaizen](https://discourse.hibernate.org/u/kamokaizen)
#### Post date: [February 21, 2020, 6:37am UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/3 "2020-02-21T06:37:06Z")

</div>

Hi yrodiere,

1- The generic query method is like the below code; it uses the entity fields and projection fields.

Fields are used to full text search “vehicleId”, “category.categoryId”, “category.name”, “modelDescription”, “longModelDescription”, “brand”, “equipmentType”, “modelType”, “body”, “startYear”, “endYear”

Projection Fields; “vehicleId”, “category.categoryId”, “brand”, “modelDescription”, “longModelDescription”, “startYear”, “endYear”, “body”, “hp”

```auto
protected Analyzer analyzer = new WhitespaceAnalyzer();

    @Override
    public List<T> searchProjection(String searchText, String[] fields, String[] projectionFields, BasicTransformerAdapter resultTransformer, Sort sort, int firstResult, int maxResult, Class<T> entityClass) {
        if (Strings.isNullOrEmpty(searchText)) {
            return new ArrayList<>();
        }

        List<String> keywords = this.tokenizeString(this.analyzer, searchText);

        try {
            FullTextSession fullTextSession = this.getFullTextSession();
            QueryBuilder qb = this.getQueryBuilder(fullTextSession, entityClass);

            BooleanJunction<BooleanJunction> booleanJunction = qb.bool();
            booleanJunction.must(qb.keyword().onField("deleted").matching(false).createQuery());

            for (String keyword : keywords) {
                booleanJunction.must(qb.keyword().wildcard().onFields(fields).matching(keyword + "*").createQuery());
            }
            FullTextQuery fullTextQuery = fullTextSession.createFullTextQuery(booleanJunction.createQuery(), entityClass);
            if (projectionFields != null && projectionFields.length > 0) {
                fullTextQuery.setProjection(projectionFields);
                fullTextQuery.setResultTransformer(resultTransformer);
            }
            fullTextQuery.setFirstResult(firstResult); //start from the firstResult element
            fullTextQuery.setMaxResults(maxResult); //return max elements
            if (sort != null) {
                fullTextQuery.setSort(sort);
            }
            return fullTextQuery.getResultList();
        } catch (EmptyQueryException ex) {
            logger.error("Something went wrong while searching: {}, exception:{}", searchText, ex.getLocalizedMessage());
            return new ArrayList<>();
        } catch (Exception ex) {
            logger.error("Something went wrong while searching: {}, exception:{}", searchText, ex.getLocalizedMessage());
        }
        return null;
    }

    /**
     * Validate input against the tokenizer and return a list of terms.
     *
     * @param analyzer
     * @param string
     * @return
     */
    public List<String> tokenizeString(Analyzer analyzer, String string) {
        List<String> result = new ArrayList<>();
        try {
            TokenStream stream = analyzer.tokenStream(null, new StringReader(string));
            stream.reset();
            while (stream.incrementToken()) {
                result.add(stream.getAttribute(CharTermAttribute.class).toString().toLowerCase());
            }
            stream.close();
        } catch (IOException e) {
            throw new RuntimeException(e);
        } catch (Exception ex) {
            ex.printStackTrace();
        }
        return result;
    }

```

2- Entity model is like below;

```auto
@Data
@Entity
@Indexed
@Table(name = "vehicle")
@NoArgsConstructor
@EqualsAndHashCode(of = {"vehicleId"}, callSuper=false)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class VehicleDbo extends MappedDomainObjectBase {
    @Id
    @GeneratedValue(generator = "uuid2")
    @GenericGenerator(name = "uuid2", strategy = "uuid2")
    @Column(name = "vehicle_id", nullable = false, unique = true)
    private String vehicleId;

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "category_id", referencedColumnName = "category_id")
    @IndexedEmbedded(includeEmbeddedObjectId = true)
    private CategoryDbo category;

    @Column(name = "model")
    private String model;

    @Column(name = "brand")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    private String brand;

    @Column(name = "model_description")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    private String modelDescription;

    @Column(name = "width")
    private int width;

    @Column(name = "height")
    private int height;

    @Column(name = "body")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    private String body;

    @Column(name = "end_year")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    @FieldBridge(impl = LongBridge.class)
    private int endYear;

    @Column(name = "fuel_type")
    private String fuelType;

    @Column(name = "fuel_type_enum")
    @Enumerated(EnumType.ORDINAL)
    private FuelTypeEnum fuelTypeEnum;

    @Column(name = "equipment_type")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    private String equipmentType;

    @Column(name = "long_model_description")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    private String longModelDescription;

    @Column(name = "doors")
    private int doors;

    @Column(name = "acceleration")
    private double acceleration;

    @Column(name = "hp")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    @FieldBridge(impl = LongBridge.class)
    private int hp;

    @Column(name = "udc")
    private double udc;

    @Column(name = "eudc")
    private double eudc;

    @Column(name = "vehicle_type")
    private int vehicleType;

    @Column(name = "auto_class")
    private String autoClass;

    @Column(name = "start_year")
    @Field(index = org.hibernate.search.annotations.Index.YES, analyze = Analyze.YES, store = Store.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer"))
    @FieldBridge(impl = LongBridge.class)
    private int startYear;

    @Column(name = "seats")
    private int seats;

    @Column(name = "cylinders")
    private int cylinders;

    @Column(name = "ccm")
    private int ccm;
}

```

3- I use elastic search integration.

```auto
spring.jpa.properties.hibernate.search.default.indexmanager=elasticsearch
spring.jpa.properties.hibernate.search.default.elasticsearch.aws.signing.enabled=true
spring.jpa.properties.hibernate.search.default.elasticsearch.host=${ELASTIC_SEARCH_URL}
spring.jpa.properties.hibernate.search.default.elasticsearch.aws.access_key=${ELASTIC_SEARCH_AWS_ACCESS_KEY}
spring.jpa.properties.hibernate.search.default.elasticsearch.aws.secret_key=${ELASTIC_SEARCH_AWS_SECRET_KEY}
spring.jpa.properties.hibernate.search.default.elasticsearch.aws.region=${ELASTIC_SEARCH_AWS_REGION}
spring.jpa.properties.hibernate.search.default.elasticsearch.index_schema_management_strategy=create
spring.jpa.properties.hibernate.search.default.elasticsearch.required_index_status=yellow
spring.jpa.properties.hibernate.search.default.elasticsearch.read_timeout=600000
spring.jpa.properties.hibernate.search.default.elasticsearch.index_management_wait_timeout=600000

```

4- For example when searching with only “206+” it returns empty. if a search with “206” it returns all match because of wildcard match.

This is the database records for 206+

```auto
SELECT long_model_description FROM car.vehicle where long_model_description like '%206+%';

206+ 1.4 Comfort
206+ 1.4 Urban Move
206+ 1.4 Sportium
206+ 1.4 HDI Sportium
206+ 1.4 Sportium
206+ 1.4 HDI Urban Move
206+ 1.4 HDI Comfort
206+ 1.4 HDI Envy
206+ 1.4 HDI Sportium
206+ 1.4 Envy

```

---

<div class="post-metadata">

### Author: ![yrodiere](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/yrodiere/32/1760_2.png) [@yrodiere](https://discourse.hibernate.org/u/yrodiere)
#### Post date: [February 21, 2020, 8:27am UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/4 "2020-02-21T08:27:56Z")

</div>

Ok, the problem is here:

```auto
protected Analyzer analyzer = new WhitespaceAnalyzer();

```

And here:

```auto
        List<String> keywords = this.tokenizeString(this.analyzer, searchText);

```

And here:

```auto
            for (String keyword : keywords) {
                booleanJunction.must(qb.keyword().wildcard().onFields(fields).matching(keyword + "*").createQuery());
            }

```

You’re tokenizing the input string with a whitespace analyzer. This will just split the string on whitespaces, so for `206+` it won’t do anything. Then you’re creating one wildcard query per token, in this case `206+*`. Since parameters to wildcard queries (in your case, `206+`) are not analyzed, Elasticsearch will look for all documents that contain a token starting with `206+`.

Problem is, with the analyzer you picked for field at indexing time, I’m almost certain `206+` is transformed to just `206`, because (IIRC) `StandardFilterFactory` removes special characters such as `+`. So the index does _not_ contain `206+`, only `206`.

The solution to your problem would be to use the same analyzer for queries as for indexing. So replace this:

```auto
protected Analyzer analyzer = new WhitespaceAnalyzer();

```

With this:

```auto
// Same analyzer as "searchTextAnalyzer"
protected Analyzer analyzer = new CustomAnalyzer.Builder()
        .withTokenizer( StandardTokenizerFactory.class )
        .addTokenFilter( LowerCaseFilterFactory.class )
        .addTokenFilter( StandardFilterFactory.class )
        .build();

```

This will, however, ignore the `+` sign completely in searches, which is probably not what you want. You should also consider tuning your analyzer, maybe replace the standard filter with just `AsciiFoldingFilter` + `LowerCaseFilter`. Be sure to update both analyzers (the one configured in Hibernate Search through annotations, and the one you instantiate directly).

Another, probably better solution would be to get rid of all local analysis, and to delegate everything to Elasticsearch. For this, you will have to use a more advanced technique that does not use the wildcard query, but instead uses the `EdgeNGramFilter` in the analyzer and overrides the analyzer at query time. See [here](https://stackoverflow.com/a/46904863/6692043) for more information.

---

<div class="post-metadata">

### Author: ![kamokaizen](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/kamokaizen/32/959_2.png) [@kamokaizen](https://discourse.hibernate.org/u/kamokaizen)
#### Post date: [February 22, 2020, 2:44pm UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/5 "2020-02-22T14:44:09Z")

</div>

I changed `StandardFilterFactory` with `AsciiFoldingFilter`, and recreate both index and custom analyzer. it works now, thank you @yrodiere. Appreciate

---

<div class="post-metadata">

### Author: ![Humoyun\_Norboboev](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/humoyun_norboboev/32/1363_2.png) [@Humoyun\_Norboboev](https://discourse.hibernate.org/u/Humoyun_Norboboev)
#### Post date: [January 27, 2021, 9:47am UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/6 "2021-01-27T09:47:29Z")

</div>

Hi. I have also such kind of problem. I tried to use your solution, but it didn’t work, it seems I’m doing smth wrong. Could you help to find what is problem exactly with my code? Thanks for any help.

```auto
@Indexed
@Getter
@Setter
@Entity`Preformatted text`
@DynamicInsert
@DynamicUpdate
@Table(name = "contracts")
@Audited(targetAuditMode = RelationTargetAuditMode.NOT_AUDITED, withModifiedFlag = true)
@org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
@AnalyzerDef(name = "searchTextAnalyzer",
        tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
        filters = {@TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = ASCIIFoldingFilterFactory.class)})
public class _Contract extends _Entity {
    
    @Fields(value = {@Field(analyze = Analyze.YES, bridge = @FieldBridge(impl = StringBridge.class), index = Index.YES, analyzer = @Analyzer(definition = "searchTextAnalyzer")),
            @Field(analyze = Analyze.NO, name = "codeSort", index = org.hibernate.search.annotations.Index.NO)})
    @SortableField(forField = "codeSort")
    @Column(unique = true)
    private String code;
...

```

---

<div class="post-metadata">

### Author: ![yrodiere](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/yrodiere/32/1760_2.png) [@yrodiere](https://discourse.hibernate.org/u/yrodiere)
#### Post date: [January 27, 2021, 10:00am UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/7 "2021-01-27T10:00:40Z")

</div>

@Humoyun_Norboboev Sure we can help, but please create your own topic.

Include in particular:

- The code that creates and executes the search query.
- The code of the the relevant entities, including in particular `@Indexed` / `@Field` annotations.
- Do you use the experimental Elasticsearch integration, or just Lucene (the default)?
- An example of a search string that doesn’t work **with the text that it should match**. E.g. you’re looking for “Peugeot 206+”, but what is the name of the car in your database? It it “Peugeot 206+” too, or something slightly different such as “peugeot 206+”?

---

<div class="post-metadata">

### Author: ![yrodiere](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.hibernate.org/yrodiere/32/1760_2.png) [@yrodiere](https://discourse.hibernate.org/u/yrodiere)
#### Post date: [January 27, 2021, 10:00am UTC](https://discourse.hibernate.org/t/hibernate-search-on-special-characters/3768/8 "2021-01-27T10:00:46Z")

</div>