Filebeat & Graylog (processors & extractors)

Graylog Central (peer support)
1

Hi Everyone

So Im running the latest graylog iso with a node running icecast and im using filebeat as a log shipper rather than using rsyslog. this is my first time using filebeat and I noticed you can create your own modules and you can can define an ingest_pipline. but i have not been successful in getting graylog to obey the processor section in the pipeline.json.

the logs are shipping fine to graylog over a beats input and all the beats metadata and custom fields are shipping just fine, but the message is not being processed, is this something only the graylog end can do as an extractor or can beats define this as a processor so graylog can follow it?

here is my test configuration

{
    "description": "Pipeline for Icecasg2.",
    "processors": [{
      "grok": {
        "field": "message",
        "patterns":[
          "%{IPORHOST:ip_address} - %{DATA:user_name} \\[%{HTTPDATE:access_time}\\] \"%{WORD:method} %{DATA:mount}?%{URIPARAM:query} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} (?:%{NUMBER:bytes}|-)( \"%{DATA:referrer}\")?( \"%{DATA:user_agent}\") %{NUMBER:duration_seconds}?",
          "%{IPORHOST:ip_address} - %{DATA:user_name} \\[%{HTTPDATE:access_time}\\] \"-\" %{NUMBER:icecast.access.response_code} -"
          ],
        "ignore_missing": true
      }
    },{
      "remove":{
        "field": "message"
      }
    }, {
      "rename": {
        "field": "@timestamp",
        "target_field": "read_timestamp"
      }
    }, {
      "date": {
        "field": "access_time",
        "target_field": "@timestamp",
        "formats": ["dd/MMM/YYYY:H:m:s Z"]
      }
    },{
      "user_agent": {
        "field": "user_agent",
        "target_field": "user_agent",
        "ignore_failure": true
      }
    }, {
      "geoip": {
        "field": "ip_address",
        "target_field": "ip_address"
      }
    }],
    "on_failure" : [{
      "set" : {
        "field" : "error.message",
        "value" : "{{ _ingest.on_failure_message }}"
      }
    }]
}
2

Graylog does not work with the configuration of ingest_pipeline in beats. For Graylog beats are just shippers/collectors.
The processing is done in Graylog and need to be configured in Graylog.

3

hi @jan

Thankyou for clearing this up for me, it was bugging me big time.
so just ship them and configure graylog to do the work, got it! :+1:

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.