Reduce size of daily logs from different devices

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Hi!

Is there a way to reduce the size of logs that we receive on daily basis. I am adding more devices to graylog and don’t want to hit the 5G line.

2. Describe your environment:

  • OS Information:
    Ubuntu 18.04

  • Package Version:
    4.1

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?

4. How can the community help?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hello,

One way is reduce the amount of fields being generated.
This also depends on the type of input you use.

I use UDP GELF for windows, syslog udp from my switches and Raw/Plaintext UDP from my Firewall.
I read somewhere that it can be done by using pipeline but I am not able to find any doc yet.

Hello there

Liscence usage relates to data saved to Elasticsearch only - not data received to Graylog. You can therefore use Graylog to enrich your data, or in this case filter your data, so less gets saved to Elastic.

With this in mind, some typical ways to shave data usage:

  1. make sure “store full_message” is not enabled on any input

  2. drop any messages you don’t actually need using a pipeline rule. For example, you might drop any message with [DEBUG] tag to prevent high verbosity hitting your ingestion limit.

  3. drop any fields you don’t need using a pipeline rule. For example, you might be writing long stack traces to a field. If you drop the field, the message will use less ES storage.

1 Like

Thanks. I have implemented few of these things now.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.