I’ve moved on to pipelines after using extractors but finding I can only have one extractor per input.
I’m trying to use a grok pattern to extract common apache fields. I’ve looked up previous posts and have found one that helped me get started:
Jan Doberstein
Grayling
Oct '18
I use the following in my setup - as you can see you need to put the Grok Pattern into quotes (")
rule "extract nginx"
when
has_field("tags") AND contains(to_string($message.type), "nginx")
then
let message_field = to_string($message.message);
let action = grok(pattern: "%{COMBINEDAPACHELOG}", value: message_field, only_named_captures: true);
set_fields(action);
end
I have stream rules setup, so my processing order is:
pipeline processor
message filter chain
My pipeline rule looks like the following:
rule "Common Apache Log"
when
$message.source == "cbdata"
then
let message_field = to_string($message.message);
let action = grok(pattern: "%{COMMONAPACHELOG}", value: message_field, only_named_captures: true);
set_fields(action);
end
The pipeline rule is applied to the “all messages” stream due to my processing order.
It seems to simulate properly when I paste in raw messages, however all messages that should be getting this pipeline applied seem to not show up in graylog after the pipeline rule is in place.
Here is an example raw data simulation:
On the simulation results, it says “Not stored”, and stored in index: “message is not stored”
I’m not sure what I’m doing that is preventing the messages from being stored. The rules look like they would do the job, but I might be missing something else in the pipeline rules that I sort of massaged from a couple different sources including Jan.