Pipeline Grok function errors


(Oidz1234) #1

Hello,

I am trying to write my first pipeline. This being a grok to extract the apache log message into fields.

Here is my pipeline attempt:

rule "apache grok"
when
has_field("application_name")
then

let apache_grok = grok(%{COMMONAPACHELOG}, to_string($message.message), true);

set_fields(apache_grok);
end

For some reason I am getting many errors to do with the formatting of the message. I am not sure what to do in this situation as I have pretty much copied the example from the documentation.

If anyone has any pointers for me that would be great :slight_smile:

Thank you !


(Jan Doberstein) #2

if you share the errors you have, we might find out what creates them.


(Oidz1234) #3

Hi Jan,

I think the best way to show this is with a screenshot:

The error message also changes depending on where I insert whitespace. (even if it is not modifying the line) I am not sure if this is expected behaviour or not.


(Jan Doberstein) #4

I use the following in my setup - as you can see you need to put the Grok Pattern into quotes (")

rule "extract nginx"
when
   has_field("tags") AND contains(to_string($message.type), "nginx")
then
 let message_field = to_string($message.message); 
 let action = grok(pattern: "%{COMBINEDAPACHELOG}", value: message_field, only_named_captures: true);
 set_fields(action);

end

(Oidz1234) #5

Thank you so much Jan ! That worked great.

I am not sure if this is the right place to ask this or if I should start a new thread, but is it recommended to have another pipeline rule after this to route it to a stream ? Or to change the process ordering so that “message filter chain” runs before pipeline processor ?


(Oidz1234) #6

it might also be worth pointing out that the graylog docs don’t say to include quotes for the grok function.

// Let "nginxaccessfields" hold the Map returned by the grok function
// Use the "set_fields" function to use the "nginxaccessfields" object to set individual field names and values.
let nginxaccessfields = grok(%{NGINXACCESS}, to_string($message.message), true);
set_fields(nginxaccessfields);

http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#grok


(Jan Doberstein) #7

you are very welcome to contribute to the documentation: https://github.com/Graylog2/documentation

With PR to correct or issue to report that.


(Oidz1234) #8

Seems that you beat me to it !

Ah well, at least it’s fixed now :slight_smile:


(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.