Filebeat Multiline not combining lines

[sqlswerver]

1. Describe your incident:
I have been attempting to follow the Filebeat docs regarding multiline parsing. Initially, I was following the latest version before realizing that Sidecar currently deploys 7.11.1. I am now following that version of the docs, still with no luck in it parsing correctly.

2. Describe your environment:
Graylog 5.0, Opensearch 2.6, latest Sidecar available

Log begins with a date and timestamp such as: 2022-10-03 07:14:04,467
The log continues, mostly without a pattern, until the next timestamp is met on a new line.

3. What steps have you already taken to try and solve the problem?
I have attempted the following config below with no different results from leaving multiline out altogether:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["graylog01:5044"]
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
filebeat.inputs:
- type: filestream
  paths:
    - C:\Path\To\*.log
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["environment", "testing"]

[tmacgbay]

Since you are using filestream rather than log for your input type, you should use the updated multiline terminology. Rather than me retyping - go to this link, it has all the information you need!

NOTE: check the syntax too - looks like you haven’t defined multiline as a parser…

[sqlswerver]

Thanks, I initially used that config though. It previously looked like this with the same result:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["graylog01:5044"]
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
filebeat.inputs:
- type: filestream
  paths:
    - C:\Path\To\*.log
  parsers:
    - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      negate: true
      match: after
  tags: ["environment", "testing"]

I had changed to what I posted after changing the docs version from 8.6 to 7.11.

[gsmith]

Hey @sqlswerver

Does this look like your Filebeat Log Collector configuration?

Example:

[gsmith]

Hey @sqlswerver

I did a mockup in my lab with your configurations and the logs came through. Think you might be missing a step, not sure thou.

results:

[sqlswerver]

Thanks for the replies. My configuration looks identical to yours currently and it’s not that I’m not receiving logs or my tags are incorrect, it’s the fact that the logs are not coming across as multiline. See below:

Results:

In this case I would expect the 2023-03-13 15:27:57,109 message and the lines below it within the file up until another timestamp to be a single message.

[sqlswerver]

I’ve figured out the issue. In the version of Filebeat that sidecar is using, 7.11, filestream is still experimental and the multiline functionality is not supported for it. After changing back to log and setting the multiline. fields, it is now working as expected. Is there going to be an update to the Beats included in Sidecar at any point?

[gsmith]

Hello @sqlswerver

@sqlswerver

Is there going to be an update to the Beats included in Sidecar at any point?

Normally there will be an update, but not sure when. Glad you reolved it.

[tmacgbay]

It is posibile to use newer versions of beats - just note that field naming can change between versions… I think between 7.11 and the current, it some fields changed … then changed back in later versions because of outcry… at least I happened to see it that way.