How to configure filebeat from graylog to parse json?

Graylog Central (peer support)
,
1

I’m trying collector-sidecar and currently facing an issue.
I can (and probably should) configure filebeat settings from gray log site and those settings should be synchronized with all the sidecar service clients.
If I understand correctly - to enable json parsing I need to use any of JSON options available:

json.keys_under_root: true
json.add_error_key: true
json.message_key: log

But I can not find how to set json properties from http://graylog.softeng.knaslt0.priv.intermedix.com/system/collectors/configurations page

Has anyone else faced this issue?

Update: I’ve also tried the API:
http://graylog.local/api/plugins/org.graylog.plugins.collector/configurations/[collector-config-id]/inputs/[input-id]

 {
  "backend": "filebeat",
  "type": "file",
  "name": "filebeat-input",
  "properties": {
    "paths": "['C:\\temp\\log.txt']",
    "scan_frequency": "10s",
    "encoding": "plain",
    "ignore_older": "0",
    "document_type": "log",
    "exclude_lines": "[]",
    "include_lines": "[]",
    "json.keys_under_root": true,
    "tail_files": true        
  },
  "input_id": "xxx",
  "forward_to": "yyy"
}

and it gave error:

{
  "type": "ApiError",
  "message": "Invalid BSON field name json.keys_under_root"
}
2

You can use snippets for that.

http://docs.graylog.org/en/latest/pages/collector_sidecar.html#snippets

3

I’m quite new with that, from documentation it looks like it is possible to add to the end of configuration, but I need to go that under prospectors, like this:

     filebeat.prospectors:
       - input_type: log
       paths: ["test/*"]
       json.message_key: event
       json.keys_under_root: true
    output:
       logstash:
         hosts:
              graylog.local:5044

Update: For some reason I can get it indented on post.

4

You can use triple back ticks for that. See Markdown Reference for details.

5

I’ve read snippets documentation once again and there’s this statement:

It’s also conceivable to put a full configuration file into a snippet and skip all of the input and output mechanism.

So yey, it looks like what I need, so I’ve deleted filebeat input/output configuration and added configuration to snippet instead.
But it looks like even though there are no inputs/outputs for filebeat, graylog renders some empty configuration and then appends snippet in filebeat.yml:

filebeat:
  prospectors: []
path:
  data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
  logs: C:\Program Files\graylog\collector-sidecar\logs
tags:
- windows
- iis
- custom
filebeat:
  prospectors:
  - encoding: plain
    fields:
      collector_node_id: graylog-collector-sidecar
      gl2_source_collector: collector-guid
      type: log
    ignore_older: 0
    paths:
    - C:\temp\log.txt
    scan_frequency: 10s
    tail_files: true
    type: log
    json.keys_under_root:true
output:
  logstash:
    hosts:
    - graylog.local:5044
path:
  data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
  logs: C:\Program Files\graylog\collector-sidecar\logs
tags:
- windows
- iis
- custom

Am I still overlooking something?

6

I would say this is a bug, if there are no inputs/outputs configured we could skip the empty prospector array. So that a snippet can set it’s own. Feel free to open an issue in the Sidecar repository, can take a look then.

Cheers,
Marius

7

Thanks Marius, I’ve created issue in github.

8

For reference:

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.