Pipelines, accept and drop messages

1. Describe your incident:

Hello friends.
Sorry for the bad formation of english.
Before posting my question here. I say that I searched, however, I did not find the answer I needed.
So I’m taking a beating in the pipelines.
What I need to do, I think is something simple:

I have a netflow entry and graylog wants to ingest everything.
But not to ingest everything, I’m trying with pipelines the following:

stage 0
in the netflow entry;
accepts everything that goes in or out of port “1521”;

stage 1
in the netflow entry;
drop everything else;

So I just want to ingest what goes to or from port “1521”. And nothing else.

— stage 0

rule "netflow accept"
when
 has_field("nf_src_port") AND to_string($message.nf_src_port) == "1521" OR
 has_field("nf_dst_port") AND to_string($message.nf_dst_port) == "1521"
then
end

— stage 1

rule "netflow discard"
when
 from_input("63120a152e8cd91dd0973b40") == true
then
 drop_message();
end

But I was not successful.

Any tip is appreciated.
Grateful.

2. Describe your environment:
Ubuntu 20:04
“Graylog 4.3.5” over Docker

3. What steps have you already taken to try to resolve the issue?
I searched other threads with similar subjects to find a solution

4. How can the community help?
She always helps. I think what I need is just to google past content to troubleshoot some issues.

2 Likes

Hello @isotecviac2022

I havent tested this out, but maybe something like this

[Stage 0]
rule "Netflow"
 when
    has_field("nf_src_port") AND contains(to_string($message.nf_src_port),"1521") OR
    has_field("nf_dst_port") AND contains(to_string($message.nf_dst_port),"1521")	        
 then
    route_to_stream(id:"5d8acba383d72e04cba96317");
 end
 
[stage 1]
rule "drop Message"
when
   has_field($message.gl2_source_input) AND contains(to_string($message.gl2_source_input),"63120a152e8cd91dd0973b40") == true 
then 
  drop_message();
end

The idea is to route the messages into a different stream, then stage 1 drop the rest.
@tmacgbay maybe able to shed some light on this.

Hello @gsmith ,
Thanks for the comeback!
I tried your tip, but to no avail. He ends up accepting everything.
Your guess gave me the idea to do it this way:

rule "netflow"
when
 //has_field($message.gl2_source_input) AND contains(to_string($message.gl2_source_input),"63120a152e8cd91dd0973b40") == true
 //has_field($message.gl2_source_input) AND contains(to_string($message.gl2_source_input),"63120a152e8cd91dd0973b40") == true
 from_input(id:"63120a152e8cd91dd0973b40") == true AND
 has_field("nf_src_port") AND NOT contains(to_string($message.nf_src_port),"1521") OR
 has_field("nf_dst_port") AND NOT contains(to_string($message.nf_dst_port),"1521")
 //NOT has_field("nf_src_port") AND to_string($message.nf_src_port) == "161" OR
 //NOT has_field("nf_dst_port") AND to_string($message.nf_dst_port) == "161" AND
then
 drop_message();
 //remove_from_stream(id:"63129fc92e8cd91dd09877b9");
 //route_to_stream(id:"63129fc92e8cd91dd09877b9");
end

However, I wasn’t successful either.

This simple pipeline can match “from input”, but not “has_field”.

rule "netflow"
when
 from_input(id:"63120a152e8cd91dd0973b40") == true AND
 has_field("$message.nf_src_port") AND NOT contains(to_string($message.nf_src_port),"1521") OR
 has_field("$message.nf_dst_port") AND NOT contains(to_string($message.nf_dst_port),"1521")
then
 drop_message();
end

Look:

In this change:

rule "netflow"
when
 from_input(id:"63120a152e8cd91dd0973b40") == true AND
 NOT has_field("$message.nf_src_port") AND contains(to_string($message.nf_src_port),"1521") OR
 NOT has_field("$message.nf_dst_port") AND contains(to_string($message.nf_dst_port),"1521")
then
 drop_message();
end

Now it seems to be discarding the match string I would like to store. In other words, the opposite of what I would like.

In these formats I also could not get it right:

rule "netflow"
when
 !has_field("$message.nf_src_port") AND contains(to_string($message.nf_src_port),"1521") OR
 !has_field("$message.nf_dst_port") AND contains(to_string($message.nf_dst_port),"1521")
then
 remove_from_stream(id:"63129fc92e8cd91dd09877b9");
end

rule "netflow"
when
 has_field("$message.nf_src_port") AND !contains(to_string($message.nf_src_port),"1521") OR
 has_field("$message.nf_dst_port") AND !contains(to_string($message.nf_dst_port),"1521")
then
 remove_from_stream(id:"63129fc92e8cd91dd09877b9");
end

rule "netflow"
when
 has_field("$message.nf_src_port") AND (to_string($message.nf_src_port) != "1521") OR
 has_field("$message.nf_dst_port") AND (to_string($message.nf_dst_port) != "1521")
then
 remove_from_stream(id:"63129fc92e8cd91dd09877b9");
end

In search, I get what I want with the following queries:

Filtering by input:
gl2_source_input:"63120a152e8cd91dd0973b40" AND (nf_src_port:"1521" OR nf_dst_port:"1521")

Filtering by streams:
streams:"63129fc92e8cd91dd09877b9" AND (nf_src_port:"1521" OR nf_dst_port:"1521")

So accomplishing this feat through the pipeline of ingesting the above search result and discarding everything else. For me it is still a mystery.

It looks like you are so close!! Your query shows the clue here. The key being that you need to group the OR statements with parenthesis, otherwise the AND/OR from other statements bleed over, just like you did in both queries.

rule "netflow"
when
 from_input(id:"63120a152e8cd91dd0973b40")       == true  AND
    (
        contains(to_string($message.nf_src_port),"1521")  OR
        contains(to_string($message.nf_dst_port),"1521")
    )

then
    drop_message();
end

We can go through some of the syntax/logic errors of the other posts as to why they didn’t work if you like but this example above should:

drop any message that comes in on input 63120a152e8cd91dd0973b40 and has 1521 in either nf_src_port -or- nf_dst_port fields

2 Likes

Dear Mr. @tmacgbay
His advice was very valuable. It was just a matter of logic as you said.
Here I managed to make it work:

rule "netflow"
when
 from_input(id:"63120a152e8cd91dd0973b40") == true AND NOT
 (
 contains(to_string($message.nf_src_port),"1521") OR
 contains(to_string($message.nf_dst_port),"1521")
 )
then
 drop_message();
end

See that the “NOT” outside, negates everything inside the parentheses. As in search.
Therefore, I am accepting everything that is origin/destination “1521” and discarding everything else that does not match.

Thanks @gsmith for tagging Mr. @tmacgbay on this thread.

4 Likes

Thanks @tmacgbay for help a fellow community member find the solution. Nicely done!

3 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.