Hi, today my GL installation stop to process message. I cant find anything from the last days in the log search. I’ve try to update GL to last 3.1.* stable version with no luck.
Elastichsearch status is ok, i can query old index message, but new message are not processed. Click on start or stop process message does nothing:
@matteolavaggi your journal has hit the full watermark, so Graylog won’t process any additional messages. What’s your disk utilization look like? I’m assuming it’s full, or very near full.
Hi, yesterday was full but i’ve rotate some index (es is on the same machine, i know is not a big setup but this is a temporary setup), but after rotating disk is not full:
Check if your cluster is not read only.
Index are ok and not locked i’ve check, recheck and unlocked again for sure ^^
ubuntu@syslog01:~ curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_a llow_delete": null}' ubuntu@syslog01:~ curl -XPUT -H “Content-Type: application/json” http://localhost:9200/_all/_settings -d ‘{“index.blocks.read_only”: false}’
Hello @matteolavaggi, just to be sure, is that space in the word “allow” in “read_only_allow_delete” there in the command you’re pasting into CLI? If so it’s not going to do what it should. Easy to overlook with tired eyes.
No space is a copy paste from terminal issue. Command is ok and ES reply with ok status.
Any idea ? Its so strange all work great, no error message in graylog, but log entry are notprocessed
NO solution => reinstall.
All from scratch.
There is a node without any running inputs. This means that you are not receiving any messages from this node at this point in time. This is most probably an indication of an error or misconfiguration. You can click here to solve this.
Port 1514 is not listening for income log . any idea?
SOLVE, the graylog configuration does not store input section. Need to manual create again!
New installation, new server, new elastic, after 1 week same problem.
No log can be searched, index status green.
Maybe your HW is not enough (ES) to store such messages, how many messages do you store every day/second? Check your CPU, RAM and disk utilization.
You’re journal is 96% full with 67 million messages in it. the oldest is 15 minutes old… that means you are getting about 75k message per second. But you have 1GB of heap. you need more RAM… possibly more CPUs, but most likely both.
Message rate is not so high, i think this is an error on how graylog show the total message in journal and the oldest data count. Log rate is about 3-4 k every minutes
Also i found that 2/3 of total error related to log message come from ES read only option / allow delete. Why dont include a fix for this directly in graylog gui?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
