# How to configure filebeat from graylog to parse json?

**URL:** <https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226>\
**Category:** Graylog Central (peer support)\
**Tags:** sidecar, filebeat-windows\
**Created:** [November 20, 2017, 4:00pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226 "2017-11-20T16:00:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![giedrius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/giedrius/32/1235_2.png) [@giedrius](https://community.graylog.org/u/giedrius)\
**Post date:** [November 20, 2017, 4:00pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/1 "2017-11-20T16:00:55Z")

</div>

I’m trying collector-sidecar and currently facing an issue.  
I can (and probably should) configure filebeat settings from gray log site and those settings should be synchronized with all the sidecar service clients.  
If I understand correctly - to enable json parsing I need to use any of JSON options available:

> **[Filebeat Prospectors Configuration | Filebeat Reference \[5.2\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/5.2/configuration-filebeat-options.html#config-json)**

> json.keys\_under\_root: true  
> json.add\_error\_key: true  
> json.message\_key: log

But I can not find how to set json properties from [http://graylog.softeng.knaslt0.priv.intermedix.com/system/collectors/configurations](http://graylog.softeng.knaslt0.priv.intermedix.com/system/collectors/configurations) page

Has anyone else faced this issue?

Update: I’ve also tried the API:  
[http://graylog.local/api/plugins/org.graylog.plugins.collector/configurations/[collector-config-id]/inputs/[input-id]](http://graylog.local/api/plugins/org.graylog.plugins.collector/configurations/%5Bcollector-config-id%5D/inputs/%5Binput-id%5D)

```
 {
  "backend": "filebeat",
  "type": "file",
  "name": "filebeat-input",
  "properties": {
    "paths": "['C:\\temp\\log.txt']",
    "scan_frequency": "10s",
    "encoding": "plain",
    "ignore_older": "0",
    "document_type": "log",
    "exclude_lines": "[]",
    "include_lines": "[]",
    "json.keys_under_root": true,
    "tail_files": true        
  },
  "input_id": "xxx",
  "forward_to": "yyy"
}

```

and it gave error:

```
{
  "type": "ApiError",
  "message": "Invalid BSON field name json.keys_under_root"
}

```

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [November 20, 2017, 4:08pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/2 "2017-11-20T16:08:40Z")

</div>

> [@giedrius](#):
>
> Has anyone else faced this issue?

You can use snippets for that.

[http://docs.graylog.org/en/latest/pages/collector\_sidecar.html#snippets](http://docs.graylog.org/en/latest/pages/collector_sidecar.html#snippets)

---

<div class="post-metadata">

**Author:** ![giedrius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/giedrius/32/1235_2.png) [@giedrius](https://community.graylog.org/u/giedrius)\
**Post date:** [November 20, 2017, 4:14pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/3 "2017-11-20T16:14:35Z")

</div>

I’m quite new with that, from documentation it looks like it is possible to add to the end of configuration, but I need to go that under prospectors, like this:

```auto
     filebeat.prospectors:
       - input_type: log
       paths: ["test/*"]
       json.message_key: event
       json.keys_under_root: true
    output:
       logstash:
         hosts:
              graylog.local:5044

```

Update: For some reason I can get it indented on post.

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [November 20, 2017, 4:40pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/4 "2017-11-20T16:40:12Z")

</div>

> [@giedrius](#):
>
> Update: For some reason I can get it indented on post.

You can use triple back ticks for that. See [Markdown Reference](http://commonmark.org/help/) for details.

---

<div class="post-metadata">

**Author:** ![giedrius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/giedrius/32/1235_2.png) [@giedrius](https://community.graylog.org/u/giedrius)\
**Post date:** [November 21, 2017, 7:09am UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/5 "2017-11-21T07:09:25Z")

</div>

I’ve read snippets documentation once again and there’s this statement:

> It’s also conceivable to put a full configuration file into a snippet and skip all of the input and output mechanism.

So yey, it looks like what I need, so I’ve deleted filebeat input/output configuration and added configuration to snippet instead.  
But it looks like even though there are no inputs/outputs for filebeat, graylog renders some empty configuration and then appends snippet in filebeat.yml:

```auto
filebeat:
  prospectors: []
path:
  data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
  logs: C:\Program Files\graylog\collector-sidecar\logs
tags:
- windows
- iis
- custom
filebeat:
  prospectors:
  - encoding: plain
    fields:
      collector_node_id: graylog-collector-sidecar
      gl2_source_collector: collector-guid
      type: log
    ignore_older: 0
    paths:
    - C:\temp\log.txt
    scan_frequency: 10s
    tail_files: true
    type: log
    json.keys_under_root:true
output:
  logstash:
    hosts:
    - graylog.local:5044
path:
  data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
  logs: C:\Program Files\graylog\collector-sidecar\logs
tags:
- windows
- iis
- custom

```

Am I still overlooking something?

---

<div class="post-metadata">

**Author:** ![marius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/marius/32/14_2.png) [@marius](https://community.graylog.org/u/marius)\
**Post date:** [November 21, 2017, 2:02pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/6 "2017-11-21T14:02:29Z")

</div>

I would say this is a bug, if there are no inputs/outputs configured we could skip the empty prospector array. So that a snippet can set it’s own. Feel free to open an issue in the Sidecar repository, can take a look then.

Cheers,  
Marius

---

<div class="post-metadata">

**Author:** ![giedrius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/giedrius/32/1235_2.png) [@giedrius](https://community.graylog.org/u/giedrius)\
**Post date:** [November 21, 2017, 2:39pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/7 "2017-11-21T14:39:44Z")

</div>

Thanks Marius, I’ve created issue in github.

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [November 21, 2017, 8:57pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/8 "2017-11-21T20:57:28Z")

</div>

For reference:

> <https://github.com/Graylog2/collector-sidecar/issues/208>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [December 5, 2017, 8:58pm UTC](https://community.graylog.org/t/how-to-configure-filebeat-from-graylog-to-parse-json/3226/9 "2017-12-05T20:58:06Z")

</div>

Closed on December 5, 2017, 8:58pm UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.