I have noticed that for the past couple days, when Graylog goes to rollover indexes, the connection to Elastic stops on my master node, but my secondary node keeps processing messages. During this time, messages are all sent to the journal, and Graylog is not able to run any index processing. Restarting the graylog-server service brings everything back to normal.
Environment:
Graylog v2.4.3 (2 nodes)
Elastic cluster v5.6.6 (3 nodes, separate from Graylog)
Please post the complete logs of all your Graylog and Elasticsearch nodes.
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html
Hmm. It rolled over properly this Saturday and Sunday. Will watch it today and see if it happens on a day with more traffic.
Looks like I may be hitting the high water mark for my Elastic storage. Going to trim back the number of logs and try again tomorrow.
Okay, so I was able to trace this back to Elastic storage. The Graylog rolls to new indices, but Elastic throws a warning about breaching 85% storage utilization. I reduced the number max of indices in the set, and it looks like the daily rollover is happier now.
Thanks.
My approach would’ve been to give the Elasticsearch cluster more disk space, but I’m happy that it works for you. 
Yes, increasing disk space would be optimal, but we are still in proof of concept mode, and are limited in available space until we formally approve the project.
My Elastic cluster has three nodes with 1TB of storage each, and we are looking at about 10 days of storage to fill this up (with 1 replica per index).
I am seeing high journal utilization during the day, but my Elastic nodes are maxing at 60% CPU and about 70% JVM (total cluster combined heap 12GB).
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
