Outsourcing Log-Files to NAS

Graylog Central (peer support)
1

Hello people,

I just got a general question about the log-files which are stored in graylog/elastic search under /var/log/graylog.
Is it possible to outsource them? For example on a NAS. Or is it not designed to outsource the logs itself again since graylog is a centralized logging server? I’m just asking because we have a NAS which isn’t used by anyone and it would save us some space on the VMware-Cluster.
I’m working on the OVA by the way.
I tried the following, but without success:

  • Stopped the graylog-services via graylog-ctl script
  • made an copy with cp -a of the directory /var/log/graylog
  • deleted all files in /var/log/graylog
  • mounted my NAS-SMB-share with RW-permissions in /var/log/graylog
  • copied all existing files from the backup-folder to the mountpoint
  • started the graylog-services via graylog-ctl script

The services started without an error, but when I logged in on the webinterface I get the following error in the search tab and also the notification that the elastic search cluster is not available.

Error Message:
blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];: cannot GET https://IP-ADDRESS/api/search/universal/relative?query=*&range=300&limit=150&sort=timestamp%3Adesc (500)
Search status code:
500

Thank you in advance!

2

What exactly do you mean? The logs produced by Graylog and its dependencies on the OVA themselves? These are stored under /var/log/graylog/.
Or do you mean the logs ingested by Graylog and indexed into Elasticsearch?

Please refer to http://docs.graylog.org/en/2.2/pages/configuration/file_location.html for details about the correct file locations.

Additionally, you can take a look at http://docs.graylog.org/en/2.2/pages/configuration/graylog_ctl.html#extend-disk-space for instructions how to provide more disk space in the OVA. Instead of the larger disk, you could probably also mount a network-backed storage (but I would highly recommend not to do this).

3

I’m actually getting the same error on a fresh install from the OVA.

Could not execute search

There was an error executing your search. Please check your Graylog server logs for more information.

Error Message:
blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];: cannot GET http://10.xxx.xxx.xxx:9000/api/search/universal/relative?query=*&range=300&limit=150&sort=timestamp%3Adesc (500)
Search status code:
500

I thought it was the password less than 16 characters error at first but I changed that for the UI login. I only see 3 threads with this error and all the configs seem fine according to the replies.

4

Please don’t hijack this thread but open a new one.

FWIW, our Elasticsearch cluster is unreachable and Graylog doesn’t work without it.

5

@jochen Thank you again for your help.
I figured out the services had no permission to edit those files.
Another question about the ingested and indexed logs. Could you tell me how much space those would take? In my understanding those are the files which are using all the space in graylog, aren’t they? That’s why I came up with the idea to mount a NAS into those directories beforehand.

Or did I mess up again and didn’t read properly.

Thank you in advance and sorry for those stupid questions…

6

That depends on how many messages are being ingested, processed, and indexed, and how big these messages are.

Yes, in most Graylog setups, the indexed messages take up most of the space. I’m not sure if that’s a surprise.