Problem elasticsearch

Hello everyone,

I collect a very large amount of logs, when I do a search (search in last 5, 10.15 minutes …) I do not receive the logs. (just for information, my server logs are powerful, i.e. it is not a resource problem on the server side)

While retrieving data for this widget, the following error(s) occurred:

In this case, is it necessary Add more resources to Elasticsearch or adjust the output settings from Graylog to Elasticsearch ?? Can somebody please explain it to me ?

Hello @Labidi,

Based on the messages it looks like either your elasticsearch instance/cluster is at the least not accessible, but also possibly not healthy or not able to keep up (resources) with the amount of data it is receiving. It appears the graylog server is journaling messages more quickly than elasticsearch can process them, hence the warning about some messages being flushed from the journal before they can be indexed.

Make sure ES is reachable by Graylog. If it is, check health and performance.

Good luck!

1 Like

Hello @ttsandrew

Thanks for your feedback, ES is reachable by Graylog and is green

I need a solution. the problem that I receive a lot of logs

Is ES served up on the same host as graylog? Or is it a separate host? If they are on separate hosts then the messages queueing on Graylog more quickly than they can be indexed by ES could be a problem with the network connection between them.

How is performance of Graylog? Do you have any pipeline rules:

Graylog and elasticsearch in the same host ! I assure you that there is not a network problem, graylog has enough ram 12 GB and 8 vcpu and has large amount of storage…

Ok. So where is the issue? Do you have any pipeline rules? Is the storage able to keep up with both journaling and indexing messages?

now it is working properly. its operation is correct I can load (at least 5 minutes …) and is ok,
NO I don’t have a pipeline rules :confused: . of course, I indexed the logs for a period of 4 months, I indexed it per month, but it’s getting heavy. so I indexed it daily, (for a period of 4 months)

If there is no evidence of compute exhaustion, there’s no network issue to worry about, and you don’t have any special processing like a pipeline rule then I would suspect storage performance. You mention that you are processing a lot of messages. If you look at iotop/atop/top (wa column) do you see many processes waiting for disk?

Related, we found this guide very valuable for monitoring performance to identify any issues:

We also followed the Elasticsearch guidelines for sizing based on our deployment to ensure that we are within or as near to as possible the recommendations for shard count / size and adjusted Graylog indices as necessary.

1 Like

I’ll check the document and get back to you , thanks @ttsandrew

1 Like

For posterity, this was resolved over in this thread.

1 Like

yes , thanks a lot @ttsandrew

1 Like