Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic!
1. Describe your incident:
Hi!
Is there a way to reduce the size of logs that we receive on daily basis. I am adding more devices to graylog and don’t want to hit the 5G line.
2. Describe your environment:
OS Information:
Ubuntu 18.04
Package Version:
4.1
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I use UDP GELF for windows, syslog udp from my switches and Raw/Plaintext UDP from my Firewall.
I read somewhere that it can be done by using pipeline but I am not able to find any doc yet.
Liscence usage relates to data saved to Elasticsearch only - not data received to Graylog. You can therefore use Graylog to enrich your data, or in this case filter your data, so less gets saved to Elastic.
With this in mind, some typical ways to shave data usage:
make sure “store full_message” is not enabled on any input
drop any messages you don’t actually need using a pipeline rule. For example, you might drop any message with [DEBUG] tag to prevent high verbosity hitting your ingestion limit.
drop any fields you don’t need using a pipeline rule. For example, you might be writing long stack traces to a field. If you drop the field, the message will use less ES storage.