Cloudtrail input not running


(tom) #1

I am currently trying to ingest input from AWS cloudtrail logs but the permanent state is “not running” with no connection notifications in syslog or graylog on the ec2 instance running the test graylog instance. I have set up SQS/S3 and followed the instructions on https://github.com/Graylog2/graylog-plugin-aws for cloudtrail. S3 and this graylog ec2 instance are able to reach eachother but is there further network/permissions configuration needed for the input to ingest properly?


(Jochen) #2

What’s in the logs of your Graylog node?


(tom) #3

Here is the graylog logs:
There was nothing relevant in syslog

2018-03-05_15:53:38.25703 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:38.25859       at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25860       at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25957       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25981       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26175       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26205       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26255       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26393       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26466       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26498       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26564       at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26594       at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26861       at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26900       at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26952       at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.27059       at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.25856 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-05_15:53:43.25900 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:43.25944       at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26030       at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26078       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26144       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26198       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26264       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26301       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26415       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26471       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26813       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26856       at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26945       at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27017       at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27049       at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27081       at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27155       at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26039 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-05_15:53:48.26078 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:48.26137       at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26226       at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26275       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26351       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26427       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26461       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26534       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26568       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26643       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26669       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27083       at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27164       at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27241       at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27276       at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27334       at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27400       at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]

(Jochen) #4

Have you seen these error messages and do you understand what they’re saying?


(tom) #5

That appeared to be an IAM issue not configured correctly. I updated the IAM role and attached it and now the input is ingesting logs from S3 but at random times you can’t view the logs in graylog and it says it is no longer receiving input nor can you view the input. Why does it randomly switch on and off? The throughput randomly goes to 0 msg/per second. Is that intended? Show messages shows nothing. Note when it does show messages it seems to be functioning correctly pulling logs from the correct S3 bucket successfully.


(tom) #6

Here are updated logs:

2018-03-06_15:55:38.90187 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-06_15:55:38.90232 com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueu
e; Request ID: 60d010ed-2732-5c73-8c55-43f59b085fe5)
2018-03-06_15:55:38.90319       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90364       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90417       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90452       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90498       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90533       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90575       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90610       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90674       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90708       at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90749       at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90826       at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90854       at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90895       at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90961       at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]

(tom) #7

Note the logs above no longer complain about credentials or AWS sdk but just SQS errors, which is strange because it does seem to work half the time.


(Jochen) #8

What’s the complete configruation of your CloudTrail input?


(tom) #9

For the input configuration all I have filled in is the node name, title (tomtest), SQS region (us east 1), S3 region (us east 1), and SQS name (tomtestsns). The SQS name looks to be mistyped and should be tomtestsqs but I’m unsure why it was working at one point regardless, I will try this change.


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.