Disk Utilisation on Graylog

[Raynu]

Hello Graylog Community,

I am using Graylog version 4.1 and receiving logs from 151 sources and the size currently is 12 Gb average on daily basis. The problem now we are facing is disk space as the retention period is 1 year and within 5 months we have reached to 1.4 TB of disk utilisation. The main aim was to keep the security logs for forensic purposes and auditing. The highest number of messages are coming from the firewall as we have routed the traffic to Graylog for monitoring. Could I get help with understanding the below questions:

1. I have created 5 indices based on the log sources types(Network, Linux Servers, Windows, Nginx etc.) and then routed the data to different streams. Is creating multiple streams for one index type duplicate the data and result in more disk storage?
2. More the number of fields + extractors = more storage?

Basically, I am looking for suggestions on how to optimise the disk storage? Any help will be appreciated.

Thanks

[tmacgbay]

Duplication of data happens when you split data off to two indexes (i.e. a route_to_stream() that points to a different index without removing it from the default …) extractors don’t cause storage, they are only there for changing data coming in. You can work through normalizing your fields and keeping only the ones you want - windows logs are a good example of where you can do a lot of paring down. The default message can be double the size with an explanation at the bottom. Here is a quick rule to remove that portion … everything after “This event is generated” will be removed from the message.

rule "clean-some-windows"
when
    // Can add other event numbers in here, make sure to add the right search below
    to_string($message.winlog_event_id) == "4624" ||
    to_string($message.winlog_event_id) == "4648"

then
   let orig_message  = to_string($message.message);
   // Create one of these for each separate search and delete.
   let short_message = regex_replace("This event is generated[^@]*",orig_message,"");
   set_field("message", short_message );
end

Elasticsearch beats also allows you to selectively send or drop information before it’s even sent to Graylog.

Here is a snippet of something like that:

...
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security
     processors:
       - drop_event.when.or:
           - equals.winlog.event_id: "4634"
           - equals.winlog.event_id: "4658"
...

Start with the chattiest (firewall) and work to remove stuff that you know you won’t be interested in… Some firewalls you can set just to record the end of session and not the beginning - that would cut the amount in half right there…

Let us know what you do and add in as much detail as you can for future searchers!

[gsmith]

Hello @Raynu

@Raynu

I am looking for suggestions on how to optimise the disk storage?

As @tmacgbay suggested is to limit the data being ingested/sent.

I’m not sure how those streams were configured but there is a tick box as shown below.

Depending on the type of Inputs used, the more fields that are created, the more storage needed.

Example I have 50 nodes sending 30 GB a day using GELF TCP/TLS and in another environment I have over 150 devices sending 5 GB a day using Syslog UDP. GELF is convenient when creating the fields needed but it does come at a cost.

[Raynu]

@tmacgbay @gsmith
Thank you for your value able input guys. For Windows logs, I am already converting GELF format to jason which helped me to reduce a number of fields. My motive was to get all information in message field and get rid of all extra fields as I am sending that syslog output(security related logs) to our snmp monitors. Also, have already removed messages from All Messages streams.

I noticed that for a few inputs I was storing full messages, which I disabled today.

I will check with my Juniepr vendor if its possible to just send the end of the session as firewall is my biggest log source and for security reasons most critical too.