Pipeline processes messages but does not grok them

[ErikHolgersson (Erik W.)]

Hello,

I have set up Graylog to process different logs from the same machine with Filebeat and Graylog pipelines. However, the pipelines don’t seem to filter anything with the grok I set up in them and instead the full original messages go through.

My configs / rules look as follows:

Sidecar-Filebeat:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: {sidecar.nodeName} fields.gl2_source_collector: {sidecar.nodeId}

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "PATH/TO/GLASSFISH/LOG"
  fields.log_type: "glassfish"

- type: log
  enabled: true
  paths:
    - "PATH/TO/JETTY/LOG"
  fields.log_type: "jetty"
output.logstash:
   hosts: ["localhost:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

And my Jetty-Pipeline (format is the same for all pipelines with different keywords and patterns obviously)

rule "Divide and jetty"
when
    true
then
    let message_field = to_string($message.message);
    let parsed_fields = grok(pattern: "%{TIMESTAMP_ISO8601}|%{LOGLEVEL} ?|%{GREEDYDATA:Handler}| - %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:level}  ?# %{NUMBER:msg_nr}: (?<thread>[%{NUMBER}] (%{NUMBER})) %{GREEDYDATA:msg}", value:message_field, only_named_captures: true);
    set_fields(parsed_fields);
 end

This pipeline is applied to my all messages stream and according to Graylog, there’s through-put, but no extracted output.

[ErikHolgersson (Erik W.)]

The

when
  true

part of the pipeline-rule was for testing purposes. The original line would have been

when
  has_field("filebeat_fields_log_type")&&contains(to_string($message.filebeat_fields_log_type, "jetty")

I thought it was a matter of escaping the |'s inside the pattern, but if I try to escape them as \|, I can’t update my pipeline as Graylog seems to have a problem with the \ character inside a rule.

[ErikHolgersson (Erik W.)]

Found the mistake, I had to double escape the special characters in my grok-pattern instead of copy pasting what I built on the grok debugger.

An alternative solution would have been to use the pipeline replace function and add whitespace to the log line where the |'s would have been.

Maybe this could be added in the pipeline or rule doc, since |'s are common as seperators in some log formats and special characters can happen anywhere.

[jan (Jan Doberstein)]

He Erik,

the docs are open source … you can edit them on github if you have an addition … or at least submit some issue for that.