FileBeat multiline How to

Graylog Central (peer support)
,
1

Hi all,
I’m using Graylog 4.1.6 and I’m using filebeat to send leg message into Graylog.
My problem is how to configure fielbeat in order to send multiline messages.

Consider a simple json text

{
  "preview": false,
  "offset": 2491152,
  "result": {
    "_raw": "row message",
    "_time": "2022-04-01T00:00:13.000+0000",
    "host": "127.0.0.0",
    "index": "source.net",
    "source": "tenant",
    "sourcetype": "gianluca.localmachine",
    "_server": "server.ip"
  }
}
{
  "preview": false,
  "offset": 2491111,
  "result": {
    "_raw": "row message",
    "_time": "2022-04-01T00:00:13.000+0000",
    "host": "127.0.0.0",
    "index": "source.net",
    "source": "tenant",
    "sourcetype": "mike.localmachine",
    "_server": "server.ip"
  }
}

and so on…

Can somebody can help me to understand how set the sidercar.yml in order to get and parse correctly the multiline log file?
Thanks a lot
Gianluca

2

You can handle that in your configuration here are the docs from Elasticsearch on filebeat->multiline

I found a random example of the multiline commands being used here… that post doesn’t solve the problem is is just an example of multiline processing in the sidecar configuration.

1 Like
3

Adding on to @tmacgbay , I also found another example. This post did look familiar.

4

Hi @gsmith @tmacgbay
thanks a lot for your help
I’m going to solve my scenario and I will share the solution here hoping that could help someone in the future

Thanks

5

Hi all
I tested

{
	"test": true
}

using the configuration below:

multiline.pattern: '^\{'
multiline.negate: true
multiline.match: after

as suggested here

but it does not works. I tested multiline.pattern: ‘^{’ too without result.
The json is splitted in more then one message
image

Any suggestions?

UPDATE
I add my collector configuration

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["myhost:5044"] 
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - s-aad 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:/splunk-logs/test/*.json
# Multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
multiline.type: pattern
multiline.pattern: '^{'
multiline.negate: false 
multiline.match: after
6

Your post and it’s edit conflict in what your multiline pattern settings are, as I read it the top one where it says this:

multiline.pattern: '^\{'
multiline.negate: true
multiline.match: after

should work the way that you want.

The other important thing to note is that yml configuration files are picky about indentation. Your multiline configurations should be indented at the same level as paths:

...
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:/splunk-logs/test/*.json
  # Multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  multiline.type: pattern
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
...

The indentation means that the multiline applies specifically to the paths: above it… this comes into play if you have a configuration file the has different settings for different paths. Some paths: may not be multiline.

1 Like
7

Thanks @tmacgbay
let me try

8

Hi,
finally I solve it

I just add the configuration from filebeat.inputs:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:/splunk-logs/aad-test/test.json

# Multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  multiline.type: pattern
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 5000
  multiline.timeout: 10

processors:
 - decode_json_fields:
     fields: ['message']
     target: ""
     process_array: true
     max_depth: 8
     overwrite_keys: true

Now I can send a json multifile to a single message text into my graylog and finally can start the parsing operation

1 Like
9

Mark it as solved for future searchers! :smiley:

1 Like
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.