Pipeline rules - Concatenate two Grok extractor field


(Sathishdsgithub) #1

I’m trying to combine two grok fields to a single field.

Example: Currently my grok extractors has field “src_ip” and “dst_ip”. Now I want to use the pipeline rules to combine both the field src_ip and dst_ip to a single field with name “src_dst_ip”.

How do I achieve this with pipeline rules? , Please post me the example.rule creation.Thanks

I’m trying with the below rule in the pipeline and it does not show in a search field.

rule "Combine src and dst field"
when
    has_field("$message.src_ip") && has_field("$message.dst_ip")
    
then
let src_dst_ip = concat( "$message.src_ip" , "$message.dst_ip" ) ;
end

Expectation

src_ip = 192.168.1.1

dst_ip = 10.1.1.1

src_dst_ip = 192.168.1.1 , 10.1.1.1


(Jochen) #2

You only define a variable named src_dst_ip with the literal content $message.src_ip$message.dst_ip in your rule but you don’t assign that anywhere.

You need to use the set_field() function to assign a value to a message field.


(Sathishdsgithub) #3

Hi Jochen,

Thanks for the reply. I’m little confused about the set_field() function and I don’t know how to assign a value to it.

This is my sample raw message that is reporting to gray log

“2017-06-12 05:40:50.837390 IP 192.168.96.1.5366 > 192.168.96.141.12900: Flags [S], seq 1252815841, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0”

And my Grok Extractor looks like this

grok_pattern: %{TIMESTAMP_ISO8601:timestamp} IP %{IP:src_ip}.%{INT:src_port} > %{IP:dst_ip}.%{INT:dst_port}(?[^$]+)

Output of Grok pattern for the field src_ip and dst_ip

dst_ip

192.168.96.141

src_ip

192.168.96.1

In this, if you see I have two field src_ip and dst_ip. Now I wanted to use pipeline rule to combine this two field value.

Please let me know how do I accomplish this with pipeline rules

"set_field syntax

set_field(field: string, value: any, [message: Message])"


(Jochen) #4

See here for some examples for the set_field() function:


(Philipp Ruland) #5

Hey @sathishdsgithub,

this would be your code:

rule "Combine src and dst field"
when
    has_field("$message.src_ip") && has_field("$message.dst_ip")
then
    let src_dst_ip = concat( "$message.src_ip", " , ", "$message.dst_ip" ) ;
    set_field(field:"src_dst_ip", value: src_dst_ip)";
end

Explanation:

I added " , " because this will add the comma you stated above (src_dst_ip = 192.168.1.1 , 10.1.1.1)
let src_dst_ip = concat( "$message.src_ip", " , ", "$message.dst_ip" ) ;

This will add a field to the message named src_dst_ip with the value in the variable src_dst_ip.
set_field(field:"src_dst_ip", value: src_dst_ip)";

Greetings - Phil


(Sathishdsgithub) #6

Hi Phil,

Thanks a lot for providing me the information.

I have applied the pipeline rules as shown below, but this is not showing in my search field. Do I need to make any additional configurations? I have followed the exact steps mentioned in the below link

http://docs.graylog.org/en/2.2/pages/pipelines/usage.html

rule “Combine src and dst field"
when
has_field(”$message.src_ip") && has_field("$message.dst_ip")

then
let src_dst_ip = concat( “$message.src_ip”, “$message.dst_ip” ) ;
set_field(field:“src_dst_ip”, value: src_dst_ip);
end


(Philipp Ruland) #7

Did you attach the rule to a pipeline, that is attached to a stream that the messages in question are running through? If yes, can you look inside the Graylog Interface under Indexer errors and the Graylog Logfile for errors? And post them if some occurred?

Greetings - Phil


(Jochen) #8

"$message.dst_ip" is a literal string, $message.dst_ip is a constant pointing to the content of the “dst_ip” field of the current message.


(Philipp Ruland) #9

Lol, I never noticed, that the reference was quoted, oops, my bad :smiley:


(Sathishdsgithub) #10

Hi Phil , Jochen

:slight_smile:

Final working code as shown below

rule "Combine src and dst field"
when
has_field(“src_ip”) && has_field(“dst_ip”)

then
let src_ip_comma = concat(to_string($message.src_ip), “-”);
let src_dst = concat(src_ip_comma,to_string($message.dst_ip));
set_field(field:“src_dst_ip”, value: src_dst);
end

Output

src_dst_ip

192.168.96.1-192.168.96.141


(system) #11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.