Kindly guide me to implement best compression for logs in graylog.
I am using graylog 3 and elastic 6.3.
Hello, are you referring to use of the Enterprise Archive Functionality?
Could you provide more details about what you are asking exactly?
Thanks for reply
I am using graylog 3.0.2 and elastic 6.8.20.
But Graylog Enterprise Plugin is not installed.
Can i implement best compression?
If there is any way to do please guide me the steps.
What do you mean by best compression? (What are you trying to compress?)
Best compression means to implement deflate rather than LZ4. And i want to compress the logs collected by graylog which is stored by elastic because the logs sizes takes too much spaces.
I understand. An archiving feature is included in the Graylog Enterprise Plugin, which can automatically compress older messages to a variety of formats.
You can get a free 5gb Enterprise Liscence here, should you want to use this feature - Free Enterprise
Sorry to say but 5gb cannot fulfilled my requirements. Can you tell me another way to implement best compression.
If you have a budget, I’d suggest looking at the pricing of a larger Graylog Enterprise liscence, or more storage.
If you don’t, I would suggest that you could set an aggressive retention policy of deleting messages older than x days in Graylog via System → Indicies window.
Hello,
What I have found out is the amount of fields being generate. This can be a direct result of message filling up also. For example I have 175 devices sending logs to graylog while using 4.8 GB a day and retaining logs for 30 Days. Were using Syslog UDP . Another DMZ, were using GELF TCP/TLS which generates a ton of fields. Its nice but we have 48 devices and were pushing 30 GB a day. If your using a default Index template its probably set as dynamic which gives elasticsearch free rein over creating these fields for you. Also, keep in mind what @tellistone suggest, either set retention policy or upgrade you license.