The inputs of my Graylog do not have extractors (they are raw/plaintext UDP and TCP which btw I’m not familiar with). My other input is receiving messages but I currently have no use for that input. My process and output buffers are at 100%, my journal utilization is at 78%, I have 6M unprocessed messages, and when I click “Show received messages” from my two inputs that I need, I see nothing. Do I need extractors, and if so, where do I get them?
I don’t know how to fix this issue of not receiving messages at all.
The way you are describing it it sounds like Elasticsearch (backend DB for Graylog) is having an issue. Either it is not running or you have filled up the storage and it had to stop.
I don’t know what system you have so I have to guess at commands… if it is Ubuntu then you might do the following to check if elasticsearch is running:
$ systemctl status elasticsearch
if running you would get this in return:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-05-23 06:52:09 EDT; 1 weeks 3 days ago
Docs: https://www.elastic.co
Main PID: 713 (java)
Tasks: 163 (limit: 10287)
Memory: 7.4G
CGroup: /system.slice/elasticsearch.service
├─ 713 /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.c>
└─1085 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
you could check for disk space with df -h
Here is a page with a bunch of diagnostic commands plus how to give good information when asking a question:
My system is Centos 7 from what I know, but when I ran the command, I got the same result you posted except that the vendor preset is disabled.
I see in my Graylog system overview that there are alerts saying journal utilization is too high and “Elasticsearch nodes disk usage above flood stage watermark”, but I am not sure how to fix this problem. The main problem is that I can’t receive any messages from my two inputs.
When this happens, Elasticsearch sets itself to read-only and you cant write logs to it any more. So then Graylog queues up messages in the journal and processing queues until they get full or until you fix the Elasticsearch storage issues. You can search the forums (just searching for “watermark” is a good start) for a lot of commentary of this happening and how to deal with it.
Essentially you can increase disk space or change the log retention in Graylog to keep less data around in indexes. Once you solve disk space you have to run a command against Elasticsearch to flip it from read-only to read-write. I could track down the command but it is mentioned several times in the forums and it’s good practice to search for it… Let us know what you find and how you solved it… or if you get stumped further down the line.
I ran this command: curl -XGET ‘localhost:9200/_cat/allocation?v&pretty’
and got disk.avai = 11.5 gb, does that mean I have enough disk space for Elasticsearch?
UPDATE: I used the curl -X and restart commands above, which somehow got rid of the unprocessed messages. The only thing is the utilization for the journal is slowly increasing, it was at 1.12% and now is at 1.55%. However, I still am not receiving any messages from the inputs I need.
That’s a CentOS question, not really a Graylog question… It depends on how your current disk is set up… do you have extra space to expand to? Maybe you need to add extra physical disk? I don’t have anything to go on - I would suggest you use Google and search for things like this. but without knowing more about your set up (I had suggested the df -h command earlier) there are too many variables - the internet knows all though and Google is a good start…
If you don’t need the input, “Stop” it … then if you are sure after a while that you don’t need it you can delete it.
So I’m accessing Graylog from a VM, the host page for the vm says I still have 7GB of memory, 150GB storage left. For the vm hardware, I see 160GB for my hard disk.
When I run df -h, everything under “Available” is 3.9gB or below.
Is lack of storage why I am not receiving messages?
Lack of storage (Elasticsearch going above high watermark and reverting to read only) will cause your journal and processing queues to fill up… once those are full, you won’t be able to receive messages. If things are processing (we solved your original question) but you are not receiving messages on an Input, that is a new issue. If things are processing but your queues are steadily increasing, then you likely need to allocate more resources to your Graylog environment.
Messages have stopped processing again and are piling up in the unprocessed section. Would creating a new node help, and if so, how do I create a new node?
Not enough information. Adding a node is a significant step.
Has processing messages stopped completely or are you processing some messages? Did you stop the input that you didn’t want? Did Elasticsearch hit the high water mark again? Is there anything appearing in the Graylog logs - you can post the relevant logs in text (using the </> forum tool to make it look nice) Post pictures if it helps but text is preferred if possible. Have you done any research on the issues you are having other than posting in the forums? Have you tried anything other than what is suggested in this thread?
Messages seem to have stopped processing completely. I have stopped the input I don’t want, and I am not sure whether Elasticsearch hit the high mark again. There is nothing appearing in the logs at all (has been so for the past year).
I have been doing research everywhere online for the past few days and had been trying to set up a new index/stream just now. I have tried changing the index rotation, updating Graylog, deleting the disk journal, adding http.max_initial_line_length: 32k to ES config file. Nothing has worked so far.
I find it hard to believe that nothing has shown in the logs for the past year - when you hit a high water mark Graylog should show logs about that - you can view what is going on with Graylog logs with the following command:
tail -f /var/log/graylog-server/server.log
I am happy to help but I can not guess - it helps if you post what you are seeing and what work you have done as needed.
Oh wow, for anyone coming across this you basically started from scratch.
No security any/all through firewall.
allows deleting the index to make more resources available. The disk-based shard allocator may add and remove this block automatically which is set to null.
Deleted all the indices.
Restarted GL, ES and MongoDb couple times.
a document is not deleted from a segment, just marked as deleted. During a merge process of segments, a new segment is created that does not have those deletes. This flag allows to only merge segments that have deletes. Defaults to false
Let us know what you find and how you solved it… or if you get stumped further down the line. 
