Logs size very huge

Dear All,
I have the below envirnoment
Rocky Linux 8
graylog-enterprise-5.2.5-1.x86_64 all working fine
My first input has been Fortigate 1500D firewall and using the sean content pack all working fine.
But I have seen that the disk space is growing up very quickly. about 20 GB a day.
Attached is a snapshot of my index config
Actually I wanted atleast 90 days log but due to space growth reduced to 15 days for checking the disk consumption.
Kindly advice the best config for indices or any help to recduce the huge growth

Thanks and Regards

simon

Index configuration only determines lifetime of the data and how many shards are generated.
To reduce the amount of logged data, you can use pipeline rules to

  • drop messages that are not relevant
  • reduce the set of fields that are stored

Dear Patrickmann

Thanks for your quick reply
By the way I am using the Fortigate Syslog Pipeline Content pack ver 7 by Sean Whalen and I do believe it does remove unnecessary fields .
So appreciate your kind advice

regards

simon

That content pack does indeed remove a number of unnecessary fields.
Perhaps you can be more aggressive though, since you know what really is important for you and what can be safely discarded.

Thanks once again for the early reply
I will do the needful and carefully check the unnecessary firelds
really apprecite

Regards

simon

Hi Simon,
depending on your setup you might want also to drop unneccesary logs. An example: DNS is very noisy, and if you have a DNS Query log you will not need all the firewall logs for clients doing DNS resolution.

1 Like