Elasticsearch high search query/fetch time

[micsnare (theresa)]

Hi,

please forgive me if this is the wrong place to ask this…but according the elastic-hq my elasticsearch cluster has a rather high search-query and search-fetch time

Search - Query: is ~ 237.34ms
Search - Fetch:  is ~ 36.23ms (!!)

especially the last one is according to the elastichq plugin rather critical.

My elasticsearch node however is not really busy (low load or memory consumption).

Any idea where to look at or what the root cause could be if the system is actually in a healthy state?!

cheers,
theresa

[jochen]

What are the hardware specs of the machines running Elasticsearch?
What’s the configuration of the Elasticsearch nodes in the cluster?
Do you have any actual problems or are you trying to do some “predictive maintenance”?

[micsnare (theresa)]

Hi Jochen,

thanks a lot for your quick response!!
the Elasticsearch nodes have the following hardware specs:
10 vcores
32GB memory

whereas 12GB are reservered for the java heap-size.
it was previously only set to 6GB but last week the elasticsearch cluster crashed because the elasticsearch node had an HeapDumpOnOutOfMemoryError
I’m currently investigating why this happened to be able to prevent this from happening in the future,… so more or less predictive maintenance. I have currently aggregated 1.4TB of log data

The elasticsearch configuration is:

cluster.name: graylog
path.data: /opt/elasticsearch/
network.host: 0.0.0.0
transport.tcp.port: 9300
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["1.2.3.4:9300"]
index :
refresh_interval: 1s

Again, thanks for all your help so far!

cheers,
theresa