Need help in pipeline rule configuration

(Anmol Sharma) #1

I am new to the pipeline feature in Graylog. I am trying to set up a pipeline rule but after going through soo many sources not able to find a suitable one.

Rule definition: If the “message:” contains “failed” then add a new field.

Message format is like: (field: value)
message: [2018-07-19 10:33:10,053] admin finish [2486:failed] AllServers admin/- “-/-”[-]

I need help with 2 things basically:

  1. writing the “when” condition for matching if “failed” string present in the message.
  2. extracting job id (2486) from the message and store it into a variable.

(Jan Doberstein) #2


I would build a GROK Pattern (with the help of ) to extract the needed values. But the structure is not very parsing friendly.

When you reveal the application that is writing such a logfile you might find a user solution already in the community.

(Anmol Sharma) #3

Thank you for the GROK reference. I have written a the following GROK pattern suitable for the log message mentioned above.

[%{TIMESTAMP_ISO8601:logdate}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}

Using above GROK pattern, I will get “jobID” and “status”.

But, how do I use it with pipeline function “grok” to create a rule. I need only a simple example for understanding.

(Jan Doberstein) #4

you might find this useful:

(Anmol Sharma) #5

I have found a similar use-case on thread Pipeline rule: Escaping brackets in grok template but unanswered.
I have tried creating a pipeline rule for the use case described above but getting the errors undermentioned. Please let me know where am I going wrong and it’s workaround.

Rule definition:
rule “process_when_message_contains_failed”
(has_field(“message”) AND contains(“failed”))
let message_field = to_string($message.message);
let grokpattern = “[%{TIMESTAMP_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}”;
let nexus = grok(pattern: $grokpattern, value: $message_field, only_named_captures: true);

Error message:

(Jan Doberstein) #6

You should be carefull when copy&paste and not reflect if that fits your usecase

rule "process_when_message_contains_failed"
  has_field("message") AND contains("failed",to_string($message.message))
  let extract = grok(pattern: "[%{TIMESTAMP_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}" , value: to_string($message.message), only_named_captures: true);

I did not check if the pattern is wroking - but this is without error in the editor.

(Anmol Sharma) #7

Thanks @jan it worked without any error.

(Anmol Sharma) #8


I did some modifications to the rule definition as shown below.

rule “process_when_message_contains_failed”
has_field(“rundeck”) AND contains(“executionslog”, to_string($message.rundeck))
set_field(“pipeline”, “pass”);
let extract = grok(pattern: “[%{GREEDYDATA:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{DATA:state}] %{DATA:rundeckProject} %{GREEDYDATA}” , value: to_string($message.message));

The rule is actually executing without error:

  1. the 1st rule action set_field(“pipeline”, “pass”); is preformed successfully.
  2. but the 3rd rule action “set_fields(extract)” is not setting the “fields & values” in the message stored.

I guess there might me some logical error either in the 2nd or 3rd rule actions. Please help me in resolving this.

Based on your last reply:
I was getting errors like below in pattern %{TIMESTAMP_ISO8601:logtime} and %{WORD:state} in pipeline processing, therefore I have modified these to as mentioned above in this message.

(Jan Doberstein) #9

I guess that the [ and ] should be escaped like \[ and \] - That escaping should be done for the fixed ones in the pattern.

As you can see the GROK is expandet to REGEX and the already present [] are added and got interpreted.

(Anmol Sharma) #10

@jan Thanks for the help. Adding double escape characters to [ and ] did the job.

Here are the final rule definitions:

Thanks once again @jan for help.

(system) #11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.