We are using AWS plugin to gather cloudtrail messages. The messages are being successfully received.
I’ve set up a very simple stream with one rule: Field source must match exactly aws-cloudtrail
Received messages successfully pass the “Test against stream” test, but are not routed to the stream.
Please provide the complete configuration of the AWS CloudTrail input, the complete stream rules, and some example messages.
Here are some cuts and pastes and edits of what you are requesting:
AWS CloudTrail input:
X Global
Should this input start on all nodes
Title
CloudTrail US_EAST_1
AWS SQS Region
US_EAST_1
The AWS region the SQS queue is in.
AWS S3 Region
US_EAST_1
The AWS region the S3 bucket containing CloudTrail logs is in.
SQS queue name
cloudtrail-logs
The SQS queue that SNS is writing CloudTrail notifications to.
Allow throttling this input.(optional)
If enabled, no new messages will be read from this input until Graylog catches up with its message load. This is typically useful for inputs reading from files or message queue systems like AMQP or Kafka. If you regularly poll an external system, e.g. via HTTP, you normally want to leave this disabled.
AWS access key(optional)
XXXXX
Access key of an AWS user with sufficient permissions. (See documentation)
AWS secret key(optional)
Secret key of an AWS user with sufficient permissions. (See documentation)
AWS assume role ARN(optional)
The role ARN with required permissions (cross account access)
Override source(optional)
The source is a hostname derived from the received packet by default. Set this if you want to override it with a custom string.
——
The only rule for the stream is:
Field source must match exactly aws-cloudtrail
——
Here is the graylog view of a message that passes “Test against stream”, but is not sent to the stream:
aws_region
us-east-1
aws_source
true
event_id
0b4d6605-4bf4-409c-860a-8e07b3068b01
event_name
DescribeInstanceStatus
event_source
ec2.amazonaws.com
event_type
AwsApiCall
full_message
[instancesSet={items=[{instanceId=i-01130318f45e7f672}]}, filterSet={}, includeAllInstances=false]
message
ec2.amazonaws.com:DescribeInstanceStatus in us-east-1 by elasticbeanstalk.amazonaws.com / null
recipient_account_id
531234332176
request_id
9b6b8a28-68a6-40aa-af8d-7b35d35b2dd9
source
aws-cloudtrail
source_address
elasticbeanstalk.amazonaws.com
timestamp
2018-06-13T13:08:25.000Z
user_account_id
531234332176
user_agent
elasticbeanstalk.amazonaws.com
user_principal_arn
arn:aws:sts::531234332176:assumed-role/aws-elasticbeanstalk-service-role/elasticbeanstalk
user_principal_id
AROAIGZ6MKRZWMLJNL6F4:elasticbeanstalk
user_session_creation_date
2018-06-13T12:36:27Z
user_session_mfa_authenticated
false
user_type
AssumedRole
I have still had no luck in routing cloud trail messages to a stream.
I’ve added several rules, which all pass “Test against stream”, but nothing makes it into the stream.
The present rules are:
A message must match at least one of the following rules
Field source must match regular expression .*cloudtrail
Field aws_region must match regular expression .*
Field aws_source must match exactly true
Test messages pass all three rules, but are not getting routed to the stream.
what is your processing order in System > Configuration ?
The Stream matching is done in the Message Filter Chain should that be after the AWS Lookup this will not work.
You should then change the order of processing or do the routing with the processing pipelines (if they are after the aws lookup)
The processiing order was indeed the problem. It was:
AWS Instance Name Lookup
GeoIP Resolver
Pipeline Processor
Message Filter Chain
I changed it to:
GeoIP Resolver
Pipeline Processor
Message Filter Chain
AWS Instance Name Lookup
And the expected routing is taking place. Can I expect any side effects from changing the order?
Thank you very much for your help.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.