I am using with sidecar filebeat configuration multiline patterns, which works fine, kinda.
But I have one line which gets wraps up, which I do not want to have in that message and should be dropped.
My current filebeat.yml in sidecar looks like this:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["ADDRESS:PORT"]
ssl.verification_mode: full
path:
data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
- windows
filebeat.inputs:
- type: log
enabled: true
paths:
- PATH.log
- PATH.txt
ignore_older: 3h
multiline.type: pattern
multiline.pattern: '^[0-9]{4}.[0-9]{2}.[0-9]{2}'
multiline.negate: true
multiline.match: after
Message in Graylog → 2nd Line should always be removed (dropped)
2023.09.19 10:15:51.888["PATH"]["STRING"]["NUMBER"]
[2023.09.19 10:15:50.886 (0 ) (Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0)] #---> should be dropped
[" STRING"]
[" STRING"]
[" STRING"]
[" STRING"]
I tried with exclude_lines before mutline. But that did not work, as multiline is in used.
exclude_lines: '^\[[0-9]{4}.[0-9]{2}.[0-9]{2}'
After that I tried to work with drop_fields in processors, as drop_event would drop the entire event and not just this single line?
processors:
- drop_fields:
when:
- regexp:
message: "\[[0-9]{4}.[0-9]{2}.[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} \(0 \) \(Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0\)\]"
- regexp:
fields: "\[[0-9]{4}.[0-9]{2}.[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} \(0 \) \(Type: 0 Sys: 0 Dp: 0 El: 0 : 0..0\)\]"
ignore_missing: true
Also tried different regex
\[[0-9]+\.[0-9]+\.[0-9]+\s[0-9]+:[0-9]+:[0-9]+\.[0-9]+\s\(0\s\)\s\(Type:\s0\sSys:\s0\sDp:\s0\sEl:\s0\s:\s0\.\.0\)\]
Any suggestion, how to drop the second line? What am I missing or doing wrong?