Pipeline Drop Message Not Working

[bluescreenofwin]

Hi everyone,

Running Graylog 4.3.x. I have been trying to drop specific messages from a stream using a few different qualifiers. Example rules I’ve tried.

rule "ESXi Drop Verbose Messages Based on Facility"
when
   regex("\\bverbose\\b",to_string($message.message)).matches==true
then
   drop_message();
end

rule "ESXi Drop Verbose Messages Based on Facility"
when
   contains(to_string($message.message), “verbose”)
then
   drop_message();
end

rule "ESXi Drop Verbose Messages Based on Facility"
when
   contains(to_string($message.facility_num), “20”) && contains(to_string($message.level), “7”)
then
   drop_message();
end

I’ve verified that the Message Filter Chain executes before Pipeline Processor.

I’ve ran debug on the drop_message function and get a lot of

PIPELINE DEBUG: Passed value is NULL

Message log example from graylog

herp.derp.corpo.lab Hostd: verbose hostd[2103333] [Originator@6876 sub=Libs opID=6f9a4033-01-01-dd-a71b] VigorTransport_ClientSendRequest: opID=6f9a4033-01-01-dd-a71b seq=38258: Sending GuestStats.SetNotificationTime request.

Raw message example

    1.2.3.4.40224 > corpo.lab.syslog: SYSLOG, length: 301
        Facility local4 (20), Severity debug (7)
        Msg: 2023-01-12T19:44:43.485Z herp.derp.corpo.lab Hostd: verbose hostd[2102275] [Originator@6876 sub=AdapterServer opID=2c963ec1 use  r=vpxuser] New request: target='vim.InternalStatsCollector:ha-internal-stats-collector', method='queryLatestVmStats', session='5233d284-30c7-c2c  7-8965-d51487011d27'\0x0a
11:44:43.485745 IP (tos 0x0, ttl 62, id 61847, offset 0, flags [none], proto UDP (17), length 255)

Rule is in a pipeline with no other stages and the rule is set to match at least one rule before continuing. Regardless of which rule, it looks as though some messages are processed but it’s very few (should be hundreds if not thousands) and I can confirm that none of the messages are being dropped if I search for them in the stream.

Any help would be appreciated

[bluescreenofwin]

Forgot to add that I tested the rules in a pipeline simulation and it comes back as successfully dropping (despite the messages not actually being dropped).

[gsmith]

Hey @bluescreenofwin

Have you tried to use a different Stage (i.e., -1). Also is the message in multiple Streams?

[ihe]

Hi,
just to be sure: you have your messages routed via “Manage Rules” in the stream-overview?

to find out, which of your rules hits I’d recommend to do the debug without deleting, but adding another field

rule "ESXi Drop Verbose Messages Based on Facility"
when
   regex("\\bverbose\\b",to_string($message.message)).matches==true
then
   set_field("tobedeleted", "by Regex");
end

and so on with your contains rule 1 and 2.

One thing I’d like to mention is that rules are identified by their name. Your three rules all have the same name. Graylog will not be able to handle them the same time. Rule 2 with the simple contains looks best from my point of view.

[gsmith]

@ihe

Your three rules all have the same name.

Good catch , I looked over that.

[bluescreenofwin]

Hi @gsmith,

Yes I tried different stages. The message is only in one stream and I remove from all messages.

@ihe Only one rule exists at a time. I create the rule in Graylog pipe rule editor and when want to try a new rule I copy+paste them into a txt file, delete the contents of the rule and craft a new rule.

All 3 rules “work” in their various iterations, they find some matches but fail to drop. Is there a better way to debug that provides more information?

[gsmith]

Hey

Check this out

• Drop message before "All Messages" stream - #2 by Ponet

Correct me if Im wrong but if your routing a message/log then its now indexed which I think defeats the purpose here. Are you trying to drop a message to save room or to minimize the amount of data ingested per day?

[bluescreenofwin]

@gsmith Yup, you got it. Trying to delete message(s) from this stream. I already have all my logic configured to the stream for all the sources of interest and just wanted to delete messages from these specific sources without having to recreate the “if source this or if source this or if source this…” ad nauseam in a rule/pipeline. Doesn’t look like I’ll be able to though?

[gsmith]

hey @bluescreenofwin

I cant lab this out tonight, I have other thing to do at work tonight I was going to start Monday. What I seen most do was drop the message from " All Messages" stream before it gets caught up in the message processor chain.

@bluescreenofwin

Doesn’t look like I’ll be able to though?

Oh it can be done, not sure whats going on with your setup thou. why i was going to lab it out and find what might be causing this issue.

[tmacgbay]

@bluescreenofwin

“

In my rule editor, it doesn’t allow quotes like this… you can’t save the rule if you have them… but YOU have them in the rules that you have posted… it’s possible that the version of Graylog you have is allowing them… or that its just an artifact for how you are managing rules in and out of Graylog. Definitely if you have quotes like those, you would not get the results you want.

So … “ quotes bad … " quotes good.

When you do a debug() message, include text with it so it’s clearer where you are in your testing.

rule "ESXi Drop Verbose Messages Based on Facility"
when
   contains(to_string($message.message), “verbose”)
then
 //  debug ("++++ This is the THING!: ", to_string(the_thing)); //random example
   debug("Message is about to be dropped..."); 
   drop_message();
end

It’s strange that it is not dropped - the other thing to consider is you may have more than one pipeline the message is going through - it may be going to all messages index and the VMware Index and in the end you are only dropping the message in the VMware pipeline/Index. But it would still be findable in All Messages - that could be that the “Remove matches from ‘All Messages’ stream” isn’t checked in the VMware Stream… or whatever alternative stream you have the pipeline on…

[bluescreenofwin]

Hi everyone,

Long weekend. Trying the suggestions now that I am back in the office. Will report back in a few.

[bluescreenofwin]

No quotes like that are in the rule but I know what you mean. Sometimes when I copy text from the graylog forums (to or from) those quotes appear in the paste. Assumed it might be my text editor. Graylog always highlights them though and I delete the illegal characters (otherwise can’t save the rule).
Current rule:

rule "ESXi Drop Verbose Messages"
when
   contains(to_string($message.facility_num), "20", false) && contains(to_string($message.level), "7", false)
then
debug("Message is about to be dropped..."); 
   drop_message();
end

I checked my other pipelines for overlap. The messages aren’t in “all messages” but rather the specific stream. The stream has the rule “remove from all messages” checked. I can double check that no other pipelines are manipulating the logs.

Adding the debug string

debug(“Message is about to be dropped…”);

produces the same error “Passed value is NULL”

Just to add: I appreciate the help from everyone.

[gsmith]

Hey @bluescreenofwin

Not sure whats going on with dropping a message, and not sure if you seen this.

Unable to Drop syslog messages based on message contents

HI All, I am encountering an issue with pipeline configuration where I seem to be unable to drop Syslog messages based on the message contents. I have done some googling / searching of the forum and the other suggestions don’t seem to be helping. I am running Graylog 4.1.14-1 on Ubuntu 22.04.1 LTS An example of the message type I am trying to drop is below: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet1/0/32 (not half duplex), with -Device Name- GigabitEthernet0 (hal…

Kind of looks like you issue

[bluescreenofwin]

It does look similar. I’ll try deleting/recreating the rule and report back.