can you tell a little bit more about your environment? Is your Graylog environment just a single machine? Are you using the graylog virtual appliance or have you separated the Graylog server from the elasticsearch data node?
It seems to be due to high I/O that the elasticsearch database cannot process the log events quickly enough so that the Graylog server has to buffer those locally in the message journal.
But, I’m by no means an expert, so please take this with a grain of salt.
How much data are you currently ingesting into your graylog environment?
You might also want to check if you’re running out of diskspace, it could be that your message journal is low on diskspace. If you haven’t changed this, than the message journal will be located in: /var/lib/graylog-server/journal
If it’s not there then you can look up the exact path in your graylog server.conf
Sorry if my question is to wide, i’m new on these things
Hi theresa, is Graylog a single machine, virtual appliance (2 core, 2ghz, 4GB of ram and 40GB of disk) nothing separated of this.
The log traffic is about 2 or 3 GB per 24hs. About the disk space i make indices about 1GB (15 indices)
i got a lot of regex extractor, somethings like this : regex_value: .(msg=“SSL tunnel shutdown”)
I don’t know how see replicas. I have just one node. The quantity of nodes, depends of what?
So, make it clean, if i want to “reset” and kill all problems (shrads and etc) all without lose extractors and alarms. What sould i do.
In tail -f /var/log/graylog/elasticsearch/current i don’t see errors
In tail -f /var/log/graylog/server/current i don’t see errors neither.
But i stiil seem slow :
$ cat /opt/graylog/conf/graylog.conf | grep replicas
!#How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
elasticsearch_replicas = 1
First, change elasticsearch_replicas = 0
then reboot. After that there will be no more new unassigned shards. It also easy to get rid of the old unassigned ones, it is a single command line command. If you don’t find the answer with google, I’ll post sunday/monday.
The regex looks like it is very good normally, but the maximum number is big. It is well possible that the real problem is in extractors, so that they sometimes hang and need to be optimized.