I have today a graylog installation on tests in my structure.
And only one of our servers is generating around 9GB of daily logs.
Reading some documents about Graylog, I noticed that ElasticSearch is the element that uses the most disk.
I create a custom index mapping (
http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#custom-index-mappings ) removing the analysis of the full_message, but i have one question.
How to minimize the disk usage on Graylog ? The index mapping is a solution ?
Anyone have tips ?
March 23, 2017, 3:09pm
What do you want to achieve exactly?
While Elasticsearch can (and by default does) compress data on disk, it will usually still take more disk space than plaintext files because of secondary data structures.
March 23, 2017, 5:41pm
i managed to trim some space by removing some unneeded fields and dismissing some insignificant messages
March 23, 2017, 5:42pm
do we need full_message field after data are extracted to respective fields? could it just be removed with pipeline rule?
Maniel, thanks for your reply.
I believe this is what i need. How do you remove this unneeded fields ?
Jochen, i need to trim some messages or remove some unneed fields from my index.
I believe removing some of these fields I minimize the use of the disk.
But my doubt is how i remove these fields. Using the index mapping or we have another method to do this ?
I use the tips of this repository :
but all of this tips are for logstash and have a low impact on my disk usage.
March 23, 2017, 7:32pm
you can use a pipeline rule with
March 24, 2017, 9:05am
This completely depends on your own requirements and use cases. Sometimes it might be viable to remove some fields for efficiency, sometimes it violates compliance requirements.
March 24, 2017, 9:48am
Could you create a rule somewhere within Graylog to automatically delete certain un-needed logs? This would certainly help with disk space and is something I have been look at how to do. For example, delete all informational event logs from graylog after 1 month?
March 24, 2017, 10:24am
That’s what index sets are for: