Hi - Please help me understand what fields will always be expected for the AWS Logs plugin and whether other fields might be available in some cases out of the box.
Backstory:
We migrated from single iron rack host to multi-host on AWS cloud, and we switched from syslog message forwarding to AWS log forward over Kinesis using Graylog 2.4 and the “AWS Logs” plugin that ships with it. I see only the following fields in every message regardless of which log group produces the message:
- aws_log_group
- aws_log_stream
- message
- source
- timestamp
This is with Entity Translation checked but without AWS keys configured in Graylog config. Is it normal that just these 4 fields and raw message are available out of the box? It is understandable if it is, in which case I have some pipeline processing to write…
What gives me reason to believe there might be more is the screenshot in plugin docs suggests a whole bunch more fields, but maybe this is just what comes with Flow Logs (which I am not using). Hoping for more fields, I continued with the suggested IAM user and Policy and configured Graylog with the new AWS keys. After this, my input stopped reporting messages (0 throughput). I removed AWS keys, and the Input picked right back up again.
Thanks
-Bronius