I am new to the pipeline feature in Graylog. I am trying to set up a pipeline rule but after going through soo many sources not able to find a suitable one.
Rule definition: If the “message:” contains “failed” then add a new field.
Message format is like: (field: value)
message: [2018-07-19 10:33:10,053] admin finish [2486:failed] AllServers admin/- “-/-”[-]
I need help with 2 things basically:
writing the “when” condition for matching if “failed” string present in the message.
extracting job id (2486) from the message and store it into a variable.
I would build a GROK Pattern (with the help of https://grokdebug.herokuapp.com/ ) to extract the needed values. But the structure is not very parsing friendly.
When you reveal the application that is writing such a logfile you might find a user solution already in the community.
@jan
I have found a similar use-case on thread Pipeline rule: Escaping brackets in grok template but unanswered.
I have tried creating a pipeline rule for the use case described above but getting the errors undermentioned. Please let me know where am I going wrong and it’s workaround.
Rule definition:
rule “process_when_message_contains_failed”
when
(has_field(“message”) AND contains(“failed”))
then
let message_field = to_string($message.message);
let grokpattern = “[%{TIMESTAMP_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}”;
let nexus = grok(pattern: $grokpattern, value: $message_field, only_named_captures: true);
end
You should be carefull when copy&paste and not reflect if that fits your usecase
rule "process_when_message_contains_failed"
when
has_field("message") AND contains("failed",to_string($message.message))
then
let extract = grok(pattern: "[%{TIMESTAMP_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}" , value: to_string($message.message), only_named_captures: true);
set_fields(extract);
end
I did not check if the pattern is wroking - but this is without error in the editor.
I did some modifications to the rule definition as shown below.
rule “process_when_message_contains_failed”
when
has_field(“rundeck”) AND contains(“executionslog”, to_string($message.rundeck))
then
set_field(“pipeline”, “pass”);
let extract = grok(pattern: “[%{GREEDYDATA:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{DATA:state}] %{DATA:rundeckProject} %{GREEDYDATA}” , value: to_string($message.message));
set_fields(extract);
end
The rule is actually executing without error:
the 1st rule action set_field(“pipeline”, “pass”); is preformed successfully.
but the 3rd rule action “set_fields(extract)” is not setting the “fields & values” in the message stored.
I guess there might me some logical error either in the 2nd or 3rd rule actions. Please help me in resolving this.
Based on your last reply:
I was getting errors like below in pattern %{TIMESTAMP_ISO8601:logtime} and %{WORD:state} in pipeline processing, therefore I have modified these to as mentioned above in this message.
