Pipeline rule not droping message from stream

Graylog Central (peer support)
1

1. Describe your incident:
I’m trying to drop a message from being stored when this IP address is in the message: 1.1.1.1 (example IP address). I have it connected to my helios stream but when I view the live stream, messages with the Ip address are still in there. Essentially my file server sends all file logs to graylog but graylog stores the log data on that file server, which sends a log back to graylog and the whole thing repeats. Before you ask, I can’t turn off logging on that volume. I need to drop all logs (as seen below) containing the IP address in the message from being put into the helios stream/index and sent to disk.

2. Describe your environment:

  • OS Information:
    Ubuntu 20.04 with Graylog, opensearch, and mongo in docker containers
  • Package Version:
    graylog/graylog:5.1
    mongo:6.0.5-jammy
    opensearchproject/opensearch:2

3. What steps have you already taken to try and solve the problem?
Here is my pipeline rule:

rule "Discard Messages with IP: 1.1.1.1"
when
    contains(to_string($message.message), "1.1.1.1")
then
    drop_message();
end

mpc
mpc1663×550 23.3 KB

More images in following replies (one media embed limit per post)

4. How can the community help?
Any thoughts, ideas, answers, directions you can point me in will be greatly appreciated. Thanks.

2

dot2
dot2838×460 16.6 KB

3

dot1
dot1822×478 16.3 KB

4

pipe
pipe1677×284 15.2 KB

5

rule
rule1684×837 59.2 KB

6

Hey @northportio

Perhaps try this settings on your stage 0.

From

image
image636×518 15.6 KB

To

image
image695×526 29.8 KB

7

When im having issues like this i like to back up a bit. Change the rule to do something very sinple like like sets a field processed=yes. This way you can see if the pipeline is doing anyrhing at all, and it shows very easily in the simulator.

At the time the pipeline is running are you sure its in that stream (easy way to check is attach it to all your streams and see if it starts working.

1 Like
8

Tried that but no difference.

9

In the simulator, it shows correctly, it sets the field value to “test” based on the IP address in the message field. However, nothing is changed when I go to the helios stream. Could this be because the pipeline pulls from the stream rather than changes/pushes to the stream? Thanks.

10

Unless you have a route_to_stream function in there it won’t move the messages. So if they are in the helios stream, the pipeline is attached to that stream, then they will be in there when done. Now just to check (because it comes up a lot) you are looking for changes only new messages received after you updated the pipeline and saved it right?

11

Wait is there a reason you have your stream rule processor after the pipeline processor? It looks like you have stream processing after pipeline, which means the messages may not be in the stream when the pipeline runs, which could mean the pipeline never runs at all.

1 Like
12

Good catch :+1: @Joel_Duffield

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.