Disk Usage on Graylog

#1

Hello :slight_smile:

I have today a graylog installation on tests in my structure.

And only one of our servers is generating around 9GB of daily logs.

Reading some documents about Graylog, I noticed that ElasticSearch is the element that uses the most disk.

I create a custom index mapping ( http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#custom-index-mappings ) removing the analysis of the full_message, but i have one question.

How to minimize the disk usage on Graylog ? The index mapping is a solution ?

Anyone have tips ?

(Jochen) #2

What do you want to achieve exactly?

While Elasticsearch can (and by default does) compress data on disk, it will usually still take more disk space than plaintext files because of secondary data structures.

(Daniel) #3

i managed to trim some space by removing some unneeded fields and dismissing some insignificant messages

(Daniel) #4

do we need full_message field after data are extracted to respective fields? could it just be removed with pipeline rule?

#5

Maniel, thanks for your reply.

I believe this is what i need. How do you remove this unneeded fields ? :smile:

#6

Jochen, i need to trim some messages or remove some unneed fields from my index.

I believe removing some of these fields I minimize the use of the disk.

But my doubt is how i remove these fields. Using the index mapping or we have another method to do this ?

I use the tips of this repository : https://github.com/jordansissel/experiments/tree/master/elasticsearch/disk

but all of this tips are for logstash and have a low impact on my disk usage.

(Daniel) #7

you can use a pipeline rule with remove_field function

(Jochen) #8

This completely depends on your own requirements and use cases. Sometimes it might be viable to remove some fields for efficiency, sometimes it violates compliance requirements.

(Matt) #9

Could you create a rule somewhere within Graylog to automatically delete certain un-needed logs? This would certainly help with disk space and is something I have been look at how to do. For example, delete all informational event logs from graylog after 1 month?

(Jochen) #10

That’s what index sets are for: http://docs.graylog.org/en/2.2/pages/configuration/index_model.html