Pipelines and grok clarification

(Oidz1234) #1

Hello,

I am writing a graylog pipeline and have decided to use grok to extract fields.

The pipeline will run if a certain field exists. However a lot of messages have this field and the grok pattern only matches a portion of the messages.

My question is: If the grok pattern does not match the message then what happens ? does the message get dropped ? Or does nothing happen to that message ?

I hope someone can clarify
Thanks you !

0 Likes

(Oidz1234) #2

Maybe to piggyback on this question, could I have multiple grok patterns and if one matches then set the fields to the matching one ?

0 Likes

(Jan Doberstein) #3

what is your processing order in System > Configuration ?

your processing pipeline should run after the message filter chain

0 Likes

(Oidz1234) #4
AWS Instance Name Lookup
Pipeline Processor
Message Filter Chain
GeoIP Resolver

It is currently setup like this; if I swap the two around will it work ?

0 Likes

(Jan Doberstein) #5

make it

Message Filter Chain
Pipeline Processor
GeoIP Resolver

and disable the AWS Lookup if you do not use it.

0 Likes

(Oidz1234) #6

Thanks Jan,
So this will make it so that even if the pipeline does not match then they will still be routed to the stream ? I guess I will need to modify my pipeline attachments to relevant streams then, they are currently all attached to all messages. Guessing this is not best practice…

0 Likes

#7

If you are trying out the basics its ok. But if you want to build a growing environment, you should definitely sort your streams and pipeline connections :smiley:

1 Like

(Oidz1234) #8

Ok I have swapped around the connections, thanks for your pointer !

0 Likes

(system) closed #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

0 Likes