How to filter or drop unwanted data before it consumes Graylog traffic license?

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

I am running Graylog 7.0.5 and noticed that a large amount of incoming data is not actually useful for my use cases.
Even though I do not search or store this data long-term, it still counts against my Graylog traffic license, which is causing unnecessary license consumption.

My goal is to prevent completely unwanted messages from being ingested by Graylog, so they do not consume traffic/license at all.

2. Describe your environment:

  • OS Information:
    Linux (CentOS / Rocky Linux / RHEL-like)

  • Package Version:
    Graylog 7.0.5
    OpenSearch backend
    MongoDB

  • Service logs, configurations, and environment variables:
    Logs are mainly collected via inputs (for example: Syslog / GELF / Beats)

  • Some sources constantly send low-value or noisy logs (health checks, debug logs, frequent info logs, etc.)

  • Pipelines are already used for field extraction and routing

3. What steps have you already taken to try and solve the problem?
I have considered:

  • Stream rules

  • Pipeline rules (dropping messages in pipelines)

4. How can the community help?
I would like to understand:

  • Is there any supported way in Graylog 7.x to filter or drop messages before they consume the traffic license?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @yi.yao,

Messages can be dropped within a pipeline using the example rule below, you will want to edit it so that it is targeted to the logs you wish to drop.

rule "Drop Messages"
when
contains(to_string($message.message), "useless_information")
then
   drop_message();
end
3

Your ingestion is counted as the logs are stored to disk, so if you use either of those methods to drop message it will not count against your license. It does not matter how much is processed, just how much is stored.

4

Remember too- depending on what you’re using to send logs you can filter at that point too- NXLog, Beats and syslog all have the ability to filter/restrict data sent, which has the added benefit of less traffic on the wire as well.