Drop one specific logline in filebeat (multiline in use)

I am using with sidecar filebeat configuration multiline patterns, which works fine, kinda.
But I have one line which gets wraps up, which I do not want to have in that message and should be dropped.

My current filebeat.yml in sidecar looks like this:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["ADDRESS:PORT"]
   ssl.verification_mode: full
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
 - windows
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - PATH.log
    - PATH.txt
  ignore_older: 3h
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}.[0-9]{2}.[0-9]{2}'
  multiline.negate: true
  multiline.match: after


Message in Graylog → 2nd Line should always be removed (dropped)

2023.09.19 10:15:51.888["PATH"]["STRING"]["NUMBER"]
[2023.09.19 10:15:50.886 (0 )  (Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0)] #---> should be dropped
["			STRING"]
["			STRING"]
["			STRING"]
["			STRING"]

I tried with exclude_lines before mutline. But that did not work, as multiline is in used.

exclude_lines: '^\[[0-9]{4}.[0-9]{2}.[0-9]{2}'

After that I tried to work with drop_fields in processors, as drop_event would drop the entire event and not just this single line?

processors:
  - drop_fields:
      when:
      - regexp:
            message: "\[[0-9]{4}.[0-9]{2}.[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} \(0 \) \(Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0\)\]"
      - regexp:
      fields: "\[[0-9]{4}.[0-9]{2}.[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} \(0 \) \(Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0\)\]"
      ignore_missing: true

Also tried different regex

\[[0-9]+\.[0-9]+\.[0-9]+\s[0-9]+:[0-9]+:[0-9]+\.[0-9]+\s\(0\s\)\s\(Type:\s0\sSys:\s0\sDp:\s0\sEl:\s0\s:\s0\.\.0\)\]

Any suggestion, how to drop the second line? What am I missing or doing wrong?

Hey @bavarian

Im just taking a guess if the field was called Type perhaps something like this.

processors:
  - drop_event:
      when:
          - equals:
                event.Type: "0"

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.