AWS raw logs plugin: What fields should come out of the box with entity translation?


(Bronius Motekaitis) #1

Hi - Please help me understand what fields will always be expected for the AWS Logs plugin and whether other fields might be available in some cases out of the box.

Backstory:
We migrated from single iron rack host to multi-host on AWS cloud, and we switched from syslog message forwarding to AWS log forward over Kinesis using Graylog 2.4 and the “AWS Logs” plugin that ships with it. I see only the following fields in every message regardless of which log group produces the message:

  • aws_log_group
  • aws_log_stream
  • message
  • source
  • timestamp

This is with Entity Translation checked but without AWS keys configured in Graylog config. Is it normal that just these 4 fields and raw message are available out of the box? It is understandable if it is, in which case I have some pipeline processing to write…

What gives me reason to believe there might be more is the screenshot in plugin docs suggests a whole bunch more fields, but maybe this is just what comes with Flow Logs (which I am not using). Hoping for more fields, I continued with the suggested IAM user and Policy and configured Graylog with the new AWS keys. After this, my input stopped reporting messages (0 throughput). I removed AWS keys, and the Input picked right back up again.

Thanks
-Bronius


(system) #2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.