Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic!
1. Describe your incident:
I am trying to change where Graylog is storing the incoming logs for our Palo Alto firewalls. I setup Graylog a few months ago and changed Elasticsearch.yml data and log paths to the mounted iSCSI share. I have the correct permission set on the new share. Graylog seemed to be working and logs were coming in. On the 30th Graylog stopped and the firewalls started screaming. I found the local HDD was out of space. It was a syslog.1 file in /var/log/ I moved the syslog.1 file to the mounted iSCSI restarted the server and I was able to get back into Graylog. I can see incoming logs in the upper right hand corner of the web interface but I can’t search anything.
2. Describe your environment:
OS Information: Ubuntu 24.04.1 LTS
Package Version:
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I tried restarting the server and double checking the folder permissions as well as elasticsearch.yml file.
From my point of view the /var/log/syslog.xx is not directly related to graylog.
This file should only contain messages of the operating system(Linux) itself.
However, if graylog/opensearch/… generate error messages, they could also find their way to the file.
So there might be two different issues:
First:
the syslog file grows by errors the system put in (too many errors? you’ll have to look inside)
the rotation of the file is not (or not correct) configured. You’ll have to consult your operating system related configs
Second:
as you don’t find anything in your graylog searches, there might be a different problem. This time I would guess a misconfiguration in the elasticsearch config.
on my side the path of the database (however we use opensearch instead of elastic) is: /var/lib/opensearch
Looks like this can be configured in /etrc/opensearch/opensearch.yml:
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#