# Grok in pipeline

**URL:** <https://community.graylog.org/t/grok-in-pipeline/21860>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [November 26, 2021, 7:21am UTC](https://community.graylog.org/t/grok-in-pipeline/21860 "2021-11-26T07:21:49Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![samkh](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@samkh](https://community.graylog.org/u/samkh)\
**Post date:** [November 26, 2021, 7:21am UTC](https://community.graylog.org/t/grok-in-pipeline/21860/1 "2021-11-26T07:21:49Z")

</div>

Hi,

i have configured a pipeline that attached to a stream. in the pipeline is a stage0 that has one rule.  
the rule is:  
rule “format logs”  
when  
true  
then  
let message = to\_string($message.message);  
let fw = grok(pattern: “%{SYSLOGTIMESTAMP} %{GREEDYDATA} fw=”%{DATA:source}" pri=%{INT:priority} msg="%{DATA:identifer}: %{DATA:loglevel}: %{DATA:message}"", value: “message”, only\_named\_captures: true);  
set\_fields(fw);  
end

i did change the order in the system configuration that the message filter chain before the pipeline processor.  
but the message still be not extracted with the grok function.  
when i use the pattern in as a grok extractor, it works fine.  
does anybody has a hint for me?

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [November 26, 2021, 8:43pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/2 "2021-11-26T20:43:44Z")

</div>

A couple of tips I use to help with GROK:

Use the debug() function to figure out what is going on in your pipeline rule - maybe something like this:

```auto
debug(concat("=== This is fw: ", to_string(fw)));

```

and watch for it in your Graylog logs with:

`tail -f /var/log/graylog-server/server.log`

Also helpful, create your GROK under system -\>GROK patterns and just reference it - makes for a cleaner rule.

Lastly, practice your grok against the message at [this site](http://grokdebug.herokuapp.com/).

---

<div class="post-metadata">

**Author:** ![samkh](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@samkh](https://community.graylog.org/u/samkh)\
**Post date:** [November 29, 2021, 7:52am UTC](https://community.graylog.org/t/grok-in-pipeline/21860/3 "2021-11-29T07:52:35Z")

</div>

thank you for your answer.  
i got in the logs the following line:  
2021-11-29T08:42:59.233+01:00 INFO [Function] PIPELINE DEBUG: === This is fw: {}  
i created a grok pattern, but it is still not working.  
also practice the grok pattern against a message as a extractor in the graylog clutser and it works as a extractor.

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [November 29, 2021, 12:59pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/4 "2021-11-29T12:59:07Z")

</div>

Please repost the rule using the forum formatting tools - specifically `</>` As it looks above, you have too many double quotes in the grok statement but it’s hard to tell without being properly formatted. There should only be double quotes at the beginning and end of the `pattern:` section any others need to be escaped if you are looking for them. Also it would help if you posted a sample message to test against.

---

<div class="post-metadata">

**Author:** ![samkh](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@samkh](https://community.graylog.org/u/samkh)\
**Post date:** [November 29, 2021, 1:13pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/5 "2021-11-29T13:13:44Z")

</div>

```auto
rule "format mellanox"
when
  true
then
  let message = to_string($message.message);
  debug(concat("=== This is message: ", to_string(message)));
  let fw = grok(pattern: "%{SYSLOGTIMESTAMP} %{GREEDYDATA} fw=\"%{DATA:source}\" pri=%{INT:priority} msg=\"%{DATA:identifer}: %{DATA:loglevel}: %{DATA:message}\"", value: "message", only_named_captures: true);
  set_fields(fw);

  debug(concat("=== This is fw: ", to_string(fw)));
end

```

that was the rule.  
An example of a message:

```auto
Nov 29 13:49:47 id=firewall time="2021-11-29 13:49:47" fw="fqdn.example" pri=5 msg="snmpd[8925]: [snmpd.NOTICE]: Got SNMP request from ip x.x.x.x"

```

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [November 29, 2021, 5:33pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/6 "2021-11-29T17:33:59Z")

</div>

> [@samkh](#):
>
> ` let message = to_string($message.message);`

This can be confusing when you call your variable ‘message’ use something like ‘msg’ or ‘the\_thing’ or something like that.

`let the_thing = to_string($message.message);`

(personal pref)

> [@samkh](#):
>
> `...%{DATA:message}\""....`

This part of your GROK is where I think it is erroring out (and relevant to my first point) - your message already has a `message` field… best to name that something else.

---

<div class="post-metadata">

**Author:** ![samkh](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@samkh](https://community.graylog.org/u/samkh)\
**Post date:** [November 30, 2021, 8:57am UTC](https://community.graylog.org/t/grok-in-pipeline/21860/7 "2021-11-30T08:57:35Z")

</div>

i have done the changes but the problem still there.

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [November 30, 2021, 2:32pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/8 "2021-11-30T14:32:14Z")

</div>

Does that include moving your GROK to grok patterns and referencing it? I had forgotten - any escaped character done in pipeline has to be double escaped `\\"` … which you don’t have to do if you move the full GROK to the GROK Patterns section and reference it in your pipeline.

---

<div class="post-metadata">

**Author:** ![samkh](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@samkh](https://community.graylog.org/u/samkh)\
**Post date:** [November 30, 2021, 2:36pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/9 "2021-11-30T14:36:11Z")

</div>

i move the full Grok to Grok Pattern und i edit the value in the grok function to “to\_string(the\_thing)” after that it works fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [December 14, 2021, 2:36pm UTC](https://community.graylog.org/t/grok-in-pipeline/21860/10 "2021-12-14T14:36:52Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.