No more input logs

Hello,

All of a sudden, I no longer receive any input logs on my Graylog server.

When I go to the “Nodes” I see the “Process Buffer” and the “Output Buffer” at 100% (see the screeenshot)

My server settings:
Version:3.2.6+e255fcc, codename Ethereal Elk
JVM: PID 2794, Debian 11.0.7 on Linux 4.19.0-9-amd64
CPU : 8
RAM : 8192 Mo

What I have already done:

  • stop and restart the inputs
  • restart the server
  • Boost the server (CPU & RAM)

Do you have any help for me?

I am still a beginner on graylog.

Thanks for your help.

Regards,

Hello @mlouarn, welcome!

What is the status of the graylog-server service and supporting services?

systemctl status mongod
systemctl status elasticsearch
systemctl status graylog-server

Is the elasticsearch service running? Are there any errors in the most recent service logs? (you should see these output at the bottom of the systemctl command output)

Are there any errors in your graylog server log file?

Hello,

Thank you for your quick reply,

I did the actions you asked me to do.


You can see warnings on the screenshot.

Are these warnings serious?

Thanks for your help.

Regards,

anything in the logs? what’s your elasticsearch health status?

curl -X GET “{IP or hostname}:9200/_cluster/health?pretty” (no braces in this command)

default log locations…

/var/log/graylog-server/

/var/log/elasticsearch/

Hello,

Anything in the log since september 6, 2020.

/var/log/graylog-server/

/var/log/elasticsearch/

The system does not accept any more logs because I reached the limit of 5Gb and I need a license?

But is this a limit of 5Gb of logs per day?

The 5GB limit is just for the enterprise features… it wouldn’t cause your system to stop ingesting. can you grep for any WARN or ERROR in the /var/log/graylog-server/server.log file, and did you look in the /var/log/elasticsearch/graylog.log file for any issues?

The problem is most likely listed right in one of those 2 logs.

Also, did you get your elasticsearch cluster health?

Hi,

my last log entering the server was at 10:53 am.

below the logs “server.log” of “/var/log/graylog-server/server.log” just after 10:53 am.

below the logs “graylog.log” of “/var/log/elasticsearch/graylog.log” just after 10:53 am.

This morning I restarted the machine and I recreated a log file to have only the events of this morning :
logs of graylog :

logs of elastic :

Can someone help me please ?

You are probably out of storage, Elastic Search by default stop after you use 90% of disk space. If you can, empty some space or add new one or change watermark setting on Elastic Search, and run to unblock:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d ‘{"index.blocks.read_only_allow_delete": null}'

reference: