# Pipeline processes messages but does not grok them

**URL:** <https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules, debuggingpl, grok-patternspl, sidecar\
**Created:** [June 28, 2019, 10:16am UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981 "2019-06-28T10:16:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ErikHolgersson](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@ErikHolgersson](https://community.graylog.org/u/ErikHolgersson)\
**Post date:** [June 28, 2019, 10:16am UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981/1 "2019-06-28T10:16:13Z")

</div>

Hello,

I have set up Graylog to process different logs from the same machine with Filebeat and Graylog pipelines. However, the pipelines don’t seem to filter anything with the grok I set up in them and instead the full original messages go through.

My configs / rules look as follows:

Sidecar-Filebeat:  
# Needed for Graylog  
fields\_under\_root: true  
fields.collector\_node\_id: {sidecar.nodeName} fields.gl2\_source\_collector: {sidecar.nodeId}

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "PATH/TO/GLASSFISH/LOG"
  fields.log_type: "glassfish"

- type: log
  enabled: true
  paths:
    - "PATH/TO/JETTY/LOG"
  fields.log_type: "jetty"
output.logstash:
   hosts: ["localhost:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

```

And my Jetty-Pipeline (format is the same for all pipelines with different keywords and patterns obviously)

```
rule "Divide and jetty"
when
    true
then
    let message_field = to_string($message.message);
    let parsed_fields = grok(pattern: "%{TIMESTAMP_ISO8601}|%{LOGLEVEL} ?|%{GREEDYDATA:Handler}| - %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:level} ?# %{NUMBER:msg_nr}: (?<thread>[%{NUMBER}] (%{NUMBER})) %{GREEDYDATA:msg}", value:message_field, only_named_captures: true);
    set_fields(parsed_fields);
 end

```

This pipeline is applied to my all messages stream and according to Graylog, there’s through-put, but no extracted output.

---

<div class="post-metadata">

**Author:** ![ErikHolgersson](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@ErikHolgersson](https://community.graylog.org/u/ErikHolgersson)\
**Post date:** [June 28, 2019, 10:18am UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981/2 "2019-06-28T10:18:55Z")

</div>

The

```
when
  true

```

part of the pipeline-rule was for testing purposes. The original line would have been

```
when
  has_field("filebeat_fields_log_type")&&contains(to_string($message.filebeat_fields_log_type, "jetty")

```

I thought it was a matter of escaping the |'s inside the pattern, but if I try to escape them as \|, I can’t update my pipeline as Graylog seems to have a problem with the \ character inside a rule.

---

<div class="post-metadata">

**Author:** ![ErikHolgersson](https://avatars.discourse-cdn.com/v4/letter/e/2acd7d/32.png) [@ErikHolgersson](https://community.graylog.org/u/ErikHolgersson)\
**Post date:** [July 1, 2019, 3:11pm UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981/3 "2019-07-01T15:11:02Z")

</div>

Found the mistake, I had to double escape the special characters in my grok-pattern instead of copy pasting what I built on the grok debugger.

An alternative solution would have been to use the pipeline replace function and add whitespace to the log line where the |'s would have been.

Maybe this could be added in the pipeline or rule doc, since |'s are common as seperators in some log formats and special characters can happen anywhere.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [July 2, 2019, 2:21am UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981/4 "2019-07-02T02:21:09Z")

</div>

He Erik,

the docs are open source … you can edit them on github if you have an addition … or at least submit some issue for that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [July 16, 2019, 2:21am UTC](https://community.graylog.org/t/pipeline-processes-messages-but-does-not-grok-them/10981/5 "2019-07-16T02:21:09Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.