# Filebeat & Graylog (processors & extractors)

**URL:** <https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238>\
**Category:** Graylog Central (peer support)\
**Created:** [March 4, 2019, 12:40pm UTC](https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238 "2019-03-04T12:40:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mooseh](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@mooseh](https://community.graylog.org/u/mooseh)\
**Post date:** [March 4, 2019, 12:40pm UTC](https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238/1 "2019-03-04T12:40:10Z")

</div>

Hi Everyone

So Im running the latest graylog iso with a node running icecast and im using filebeat as a log shipper rather than using rsyslog. this is my first time using filebeat and I noticed you can create your own modules and you can can define an ingest\_pipline. but i have not been successful in getting graylog to obey the processor section in the pipeline.json.

the logs are shipping fine to graylog over a beats input and all the beats metadata and custom fields are shipping just fine, but the message is not being processed, is this something only the graylog end can do as an extractor or can beats define this as a processor so graylog can follow it?

here is my test configuration

```auto
{
    "description": "Pipeline for Icecasg2.",
    "processors": [{
      "grok": {
        "field": "message",
        "patterns":[
          "%{IPORHOST:ip_address} - %{DATA:user_name} \\[%{HTTPDATE:access_time}\\] \"%{WORD:method} %{DATA:mount}?%{URIPARAM:query} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} (?:%{NUMBER:bytes}|-)( \"%{DATA:referrer}\")?( \"%{DATA:user_agent}\") %{NUMBER:duration_seconds}?",
          "%{IPORHOST:ip_address} - %{DATA:user_name} \\[%{HTTPDATE:access_time}\\] \"-\" %{NUMBER:icecast.access.response_code} -"
          ],
        "ignore_missing": true
      }
    },{
      "remove":{
        "field": "message"
      }
    }, {
      "rename": {
        "field": "@timestamp",
        "target_field": "read_timestamp"
      }
    }, {
      "date": {
        "field": "access_time",
        "target_field": "@timestamp",
        "formats": ["dd/MMM/YYYY:H:m:s Z"]
      }
    },{
      "user_agent": {
        "field": "user_agent",
        "target_field": "user_agent",
        "ignore_failure": true
      }
    }, {
      "geoip": {
        "field": "ip_address",
        "target_field": "ip_address"
      }
    }],
    "on_failure" : [{
      "set" : {
        "field" : "error.message",
        "value" : "{{ _ingest.on_failure_message }}"
      }
    }]
}

```

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [March 4, 2019, 12:53pm UTC](https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238/2 "2019-03-04T12:53:06Z")

</div>

Graylog does not work with the configuration of `ingest_pipeline` in beats. For Graylog beats are just shippers/collectors.  
The processing is done in Graylog and need to be configured in Graylog.

---

<div class="post-metadata">

**Author:** ![mooseh](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@mooseh](https://community.graylog.org/u/mooseh)\
**Post date:** [March 4, 2019, 1:02pm UTC](https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238/3 "2019-03-04T13:02:50Z")

</div>

hi @jan

Thankyou for clearing this up for me, it was bugging me big time.  
so just ship them and configure graylog to do the work, got it! 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [March 18, 2019, 1:09pm UTC](https://community.graylog.org/t/filebeat-graylog-processors-extractors/9238/4 "2019-03-18T13:09:23Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.