Its a more of a year we are full flash live in production with HS6 + elastic server.
We are proud of choosing this technical stuff with all the advantages.
recently we had performance testing where we extrapolated the data and does testing the search functionality for next year data.
we have concern over the elastic scalability and resilience in production by next year.
we have discuss the business about the decision to considering the purge of old data which they are not agree.
do we have some good documentation on elastic auto scaling and clustering with or with azure cloud?
You mean autoscaling of Elasticsearch? I’d recommend having a look on the Elasticsearch docs/forums. This should be relatively independent from the fact you’re using Hibernate Search. Hibernate Search mostly takes care of transfering data to and from Elasticsearch, but all storage and query execution is delegated to Elasticsearch.
Out of curiosity, what’s the problem, the storage size? Slow queries? Something else?
As to scaling Hibernate Search itself, and in particular indexing in Hibernate Search, you can have a look at outbox-polling coordination, which allows spreading the load over multiple nodes, and could theoretically be used to have dedicated nodes do only indexing. It will increase the stress on your database, though (more reads).
so the problem is when we have high load in the system search api taking too much time to retrieve the data and also at some point elastic failed to index large data set
so we wanted to auto scale elastic as per use in kubernates.
Scaling may help with that, indeed – especially replication if you see this slowdown mostly under high load, or sharding if you see it only with large indexes.
You might want to investigate the particular queries causing trouble, though. If the bottleneck is loading data from the database, scaling up your Elasticsearch cluster obviously won’t help. Similarly, there might be some problems with the queries themselves that mean they perform poorly and always will (thinking of regexp predicates in particular, but there are other performance traps).
If it’s about a single document which is too large or with a single field that is too large, adding more Elasticsearch nodes will not help. I’d need to see the error to say for sure what would help.
If it’s about a bulk indexing request that failed to complete in time (timed out, or rejected because the server cannot handle a request at the moment), then tuning Hibernate Search could help, but having more Elasticsearch nodes might help as well, indeed (at least, if you use node discovery).