Outbound Proxy settings

Graylog Add-ons
1

So I have a few problems with the AWS Plugin and configuring an outbound proxy.

I have upgraded to GrayLog 2.2.3 (from 2.1.3) with AWS plugin 1.3.2 and have enabled the proxy setting in the configuration that read the server.conf file line http_proxy_uri.

After some figuring out of the actual URLs the AWS plugin is using and the permission required I seem to be connecting to the Kineses flowlog streams I created from the AWS Plugin documentation.

Now I can see in the server.log that the plugin is encountering a parse error when trying to ingest the flowlogs from an AWS ELB:

[27]: index [graylog_5], type [message], id [8e95caa1-28e6-11e7-a325-06831b6fd7b7], message [MapperParsingException[failed to parse [protocol]]; nested: NumberFormatException[For input string: "TCP"];]
[28]: index [graylog_5], type [message], id [8e95caa2-28e6-11e7-a325-06831b6fd7b7], message [MapperParsingException[failed to parse [protocol]]; nested: NumberFormatException[For input string: "ICMP"];]

I haven’t modified the ELB logging format so I would have expected the Plugin to be able to parse that data. Has anyone got an tips that might help resolve this issue?

2

You have to make sure that the protocol message field is always a string.

You can use a custom Elasticsearch index mapping for this purpose: http://docs.graylog.org/en/2.2/pages/configuration/elasticsearch.html#custom-index-mappings

3

Hi @jochen,

Thanks for that so “current write-active index is graylog_5” and if I do this:

 {
  "template": "graylog_*",
  "mappings" : {
    "message" : {
       "properties" : {
        "protocol" : {
          "type" : "string"
        }
      }
    }
  }
}

curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'

So from the documentation that looks like it will only effect new indexes and not the current index. When I look at the mapping for ‘graylog_5’ it has not changed so how to move forward?

4

This is correct.

You can manually trigger an index rotation in the web interface on the System → Index Sets page.

5

Thanks @jochen - rolling the index did indeed pick up the new mapping plus some extra aws properties.

6

@jochen - just as an after thought do we have a way of finding out exactly which AWS permissions the plugin actually needs?

I have added some very permissive IAMs policies but would like to tighten them. I could do it the old fashioned way and reduce to nil then add them up as the log complains about them but…

7

The required IAM permissions are listed in the plugin’s README file:

8

Thanks @jochen - I hadn’t got that far down the guide.