I am currently trying to ingest input from AWS cloudtrail logs but the permanent state is “not running” with no connection notifications in syslog or graylog on the ec2 instance running the test graylog instance. I have set up SQS/S3 and followed the instructions on https://github.com/Graylog2/graylog-plugin-aws for cloudtrail. S3 and this graylog ec2 instance are able to reach eachother but is there further network/permissions configuration needed for the input to ingest properly?
What’s in the logs of your Graylog node?
Here is the graylog logs:
There was nothing relevant in syslog
2018-03-05_15:53:38.25703 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:38.25859 at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25860 at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25957 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.25981 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26175 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26205 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26255 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26393 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26466 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26498 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26564 at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26594 at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26861 at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26900 at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.26952 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:38.27059 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.25856 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-05_15:53:43.25900 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:43.25944 at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26030 at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26078 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26144 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26198 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26264 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26301 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26415 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26471 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26813 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26856 at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.26945 at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27017 at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27049 at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27081 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:43.27155 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26039 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-05_15:53:48.26078 com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
2018-03-05_15:53:48.26137 at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26226 at org.graylog.aws.auth.AWSAuthProvider.getCredentials(AWSAuthProvider.java:78) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26275 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26351 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26427 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26461 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26534 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26568 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26643 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.26669 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27083 at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27164 at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27241 at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27276 at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27334 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-05_15:53:48.27400 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
Have you seen these error messages and do you understand what they’re saying?
That appeared to be an IAM issue not configured correctly. I updated the IAM role and attached it and now the input is ingesting logs from S3 but at random times you can’t view the logs in graylog and it says it is no longer receiving input nor can you view the input. Why does it randomly switch on and off? The throughput randomly goes to 0 msg/per second. Is that intended? Show messages shows nothing. Note when it does show messages it seems to be functioning correctly pulling logs from the correct S3 bucket successfully.
Here are updated logs:
2018-03-06_15:55:38.90187 ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
2018-03-06_15:55:38.90232 com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueu
e; Request ID: 60d010ed-2732-5c73-8c55-43f59b085fe5)
2018-03-06_15:55:38.90319 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90364 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90417 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90452 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90498 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90533 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90575 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90610 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90674 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90708 at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90749 at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90826 at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90854 at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90895 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:47) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-03-06_15:55:38.90961 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.3.jar:?]
Note the logs above no longer complain about credentials or AWS sdk but just SQS errors, which is strange because it does seem to work half the time.
What’s the complete configruation of your CloudTrail input?
For the input configuration all I have filled in is the node name, title (tomtest), SQS region (us east 1), S3 region (us east 1), and SQS name (tomtestsns). The SQS name looks to be mistyped and should be tomtestsqs but I’m unsure why it was working at one point regardless, I will try this change.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.