# Need help in pipeline rule configuration

**URL:** <https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [July 21, 2018, 3:35pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060 "2018-07-21T15:35:26Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 21, 2018, 3:35pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/1 "2018-07-21T15:35:26Z")

</div>

I am new to the pipeline feature in Graylog. I am trying to set up a pipeline rule but after going through soo many sources not able to find a suitable one.

Rule definition: If the “message:” contains “failed” then add a new field.

**Message format is like: (field: value)**  
message: [2018-07-19 10:33:10,053] admin finish [2486:failed] AllServers admin/- “-/-”[-]

**I need help with 2 things basically:**

1. writing the “when” condition for matching if “failed” string present in the message.
2. extracting job id (2486) from the message and store it into a variable.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [July 23, 2018, 7:47am UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/2 "2018-07-23T07:47:14Z")

</div>

@anmolsharma

I would build a GROK Pattern (with the help of [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) ) to extract the needed values. But the structure is not very parsing friendly.

When you reveal the application that is writing such a logfile you might find a user solution already in the community.

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 23, 2018, 11:59am UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/3 "2018-07-23T11:59:59Z")

</div>

@jan  
Thank you for the GROK reference. I have written a the following GROK pattern suitable for the log message mentioned above.

**[%{TIMESTAMP\_ISO8601:logdate}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}**

Using above GROK pattern, I will get “jobID” and “status”.

But, how do I use it with pipeline function “grok” to create a rule. I need only a simple example for understanding.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [July 23, 2018, 12:06pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/4 "2018-07-23T12:06:25Z")

</div>

you might find this useful:

[https://community.graylog.org/search?q=grok%20pattern](https://community.graylog.org/search?q=grok%20pattern)

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 23, 2018, 2:19pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/5 "2018-07-23T14:19:49Z")

</div>

@jan  
I have found a similar use-case on thread [Pipeline rule: Escaping brackets in grok template](https://community.graylog.org/t/pipeline-rule-escaping-brackets-in-grok-template/3633) but unanswered.  
I have tried creating a pipeline rule for the use case described above but getting the errors undermentioned. Please let me know where am I going wrong and it’s workaround.

_Rule definition:_  
rule “process\_when\_message\_contains\_failed”  
when  
(has\_field(“message”) AND contains(“failed”))  
then  
let message\_field = to\_string($message.message);  
let grokpattern = “[%{TIMESTAMP\_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}”;  
let nexus = grok(pattern: $grokpattern, value: $message\_field, only\_named\_captures: true);  
end

_Error message:_  
 ![Menu_284](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/1/1780aa00ba3497f0f465f0b039ad6de950fa27bb.png)

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [July 23, 2018, 2:37pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/6 "2018-07-23T14:37:06Z")

</div>

You should be carefull when copy&paste and not reflect if that fits your usecase

```auto
rule "process_when_message_contains_failed"
when
  has_field("message") AND contains("failed",to_string($message.message))
then
  let extract = grok(pattern: "[%{TIMESTAMP_ISO8601:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{WORD:state}] %{DATA:rundeckProject} %{GREEDYDATA}" , value: to_string($message.message), only_named_captures: true);
  set_fields(extract);
end

```

I did not check if the pattern is wroking - but this is without error in the editor.

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 23, 2018, 3:18pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/7 "2018-07-23T15:18:14Z")

</div>

Thanks @jan it worked without any error.

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 24, 2018, 1:26pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/8 "2018-07-24T13:26:20Z")

</div>

@jan

I did some modifications to the rule definition as shown below.

rule “process\_when\_message\_contains\_failed”  
when  
has\_field(“rundeck”) AND contains(“executionslog”, to\_string($message.rundeck))  
then  
set\_field(“pipeline”, “pass”);  
let extract = grok(pattern: “[%{GREEDYDATA:logtime}] %{USERNAME:eventUser} %{WORD:event} [%{NUMBER:jobID}:%{DATA:state}] %{DATA:rundeckProject} %{GREEDYDATA}” , value: to\_string($message.message));  
set\_fields(extract);  
end

The rule is actually executing without error:

1. the 1st rule action set\_field(“pipeline”, “pass”); is preformed successfully.
2. but the 3rd rule action “set\_fields(extract)” is not setting the “fields & values” in the message stored.

I guess there might me some logical error either in the 2nd or 3rd rule actions. Please help me in resolving this.

Based on your last reply:  
I was getting errors like below in pattern %{TIMESTAMP\_ISO8601:logtime} and %{WORD:state} in pipeline processing, therefore I have modified these to as mentioned above in this message.

 ![Selection_285](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d88935f0579944c9861f114e8196943fd7d18051.png)

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [July 24, 2018, 2:26pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/9 "2018-07-24T14:26:16Z")

</div>

I guess that the `[` and `]` should be escaped like `\[` and `\]` - That escaping should be done for the fixed ones in the pattern.

As you can see the GROK is expandet to REGEX and the already present [] are added and got interpreted.

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [July 24, 2018, 4:19pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/10 "2018-07-24T16:19:45Z")

</div>

@jan Thanks for the help. Adding double escape characters to [and] did the job.

Here are the final rule definitions:

 ![Selection_288](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/9d32c2241fb76115b5ef7f9d2d641ffd290918c4.png)

 ![Selection_287](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/1/1d9062d9f45ece2d5a9748a47a65101d3c9096bb.png)

Thanks once again @jan for help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [August 7, 2018, 4:19pm UTC](https://community.graylog.org/t/need-help-in-pipeline-rule-configuration/6060/11 "2018-08-07T16:19:52Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.