Hi everyone, we have :
“input” -> “Stream” -> “Pipeline Rules” who extract fields and everything working well.
We want to add somes little extractor for simple extraction / manipulation on fields created on pipelines. When whe try function (like grek email pattern) it’s work, but when we save the extractor, there is no matching.
There is a specific order to use extractors ? is it compatible with pipeline, especially when pipeline extract fields from full_message ?
Check the order of message processors in your Graylog cluster on the System / Configurations page in the web interface.
If the “header_From” message field is created in a pipeline rule, the Pipeline Processor has to run before the extractors in the Message Filter Chain.
Hi @jochen thanks for your helps i tried but if i do that, fields ares not created, i need to let chain processor before pipeline.
Here is my conf ( pipeline works greats but not extractors):
regards
Looks similar to this post :
If a use pipeline, i need to create “pipeline rules” to replace existing extractors … no ?
Only if there’s an unresolvable conflict in the order of message processors, e. g. pipeline rules create fields which are created by extractors but the extractors require fields created by (other) pipeline rules.
Otherwise just adjust the order of message processors according to your use case.
Strange… it’s not my case. i have :
on my beat agent on servers, i had a field to route my logs.
on graylog side i have1 beats inputs -> 3 streams routed by a field value (added on beats agent side) -> each stream use a rule with csv plugin extractor to parse cvs quotes and creates fields … that’s all.
I retry 5 minutes ago …when i switch pipeline before message processing like that :
i lost instantly all the created fields .
Please post the complete configuration of all pipeline rules, extractors, stream rules, and the stream connections of all pipelines.
Here it is :
My processor order:
My global input
Extractor configuration of my global input (i have only one input) :
extractor preview work’s weel on “header_from” field, it’s a field created on my pipeline by csv plugin extractor (see below).
my 3 streams :
Configuration for 1 stream ( 3 ares similar, the route is done by a field value):
3 pipelines :
Configuration for 1 pipeline( 3 ares similar, the difference is on csv field extract):
my “Logtype ACCT csv extractor” rule :
rule "Logtype_ACCT"
when
true
then
let csv_fields = "type,timeLogged,timeQueued,orig,rcpt,orcpt,dsnAction,dsnStatus,dsnDiag,dsnMta,bounceCat,srcType,srcMta,dlvType,dlvSourceIp,dlvDestinationIp,dlvEsmtpAvailable,dlvSize,vmta,jobId,envId,queue,vmtaPool,header_From";
let csv_parsed = csv(csv_fields: csv_fields, csv_text: to_string($message.message), csv_separator: ",", dummy_value:"not_used");
set_fields(csv_parsed);
end
All my fields are createds, pipeline work greats (you have header_from using on the extractor at the end of the scrennshot) :
but i lost my fields if i change processor order and i put pipeline before chain processor.
thanks @jochen for your help, regards.
Stream matching (at least the one with the stream rules) happens in the Message Filter Chain.
If the Pipeline Processor runs first, the messages haven’t been routed into their respective streams yet.
Also see the following related pull request on GitHub:
https://github.com/Graylog2/graylog2-server/pull/4186
Not sure to understand 
I need to wait for this pull request to put the chain processor before ?
regards
Ideally yes, but it’s not sure that it will ever be merged.
In fact, it would be simpler if you moved the extractor logic into pipeline rules and put the Message Filter Chain before the Pipeline Processor.
That should be fairly easy if you’re only using the Grok extractor:
http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#grok
it’s clear now ! Thanks @jochen.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
