I’m using the ami from graylog to quickly test it in our dev env. I can see the kinesis stream gathering flow logs via the aws GUI but am unable to get graylog to connect to the stream. I currently have a full admin account listed under the flow log input. below is the error I’m seeing in /var/log/graylog/server/current. It looks like a dns issue with connecting to dynamodb but I can dig dynamodb.us-east-1.amazonaws.com just fine. This instance is in a private subnet using a nat gateway. I’m not sure what to look at now.
2018-04-19_15:49:44.93497 INFO [Worker] Initialization attempt 7
2018-04-19_15:49:44.93585 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:45.57359 INFO [Worker] Initialization attempt 9
2018-04-19_15:49:45.57394 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:47.09861 INFO [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now STARTING
2018-04-19_15:49:47.10078 INFO [KinesisTransport] Starting Kinesis reader thread for input [AWS Flow Logs/5ad8a1660b326b0a350a69e6]
2018-04-19_15:49:47.10325 INFO [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now RUNNING
2018-04-19_15:49:47.10482 INFO [LeaseCoordinator] With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 l$
2018-04-19_15:49:47.10509 INFO [Worker] Initialization attempt 1
2018-04-19_15:49:47.10556 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:55.20718 ERROR [LeaseManager] Failed to get table status for graylog-aws-plugin-arn:aws:kinesis:us-east-1:936793730029:stream/awsFlowLogs
2018-04-19_15:49:55.20922 com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.20925 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:162) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21106 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:107) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21137 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:235) [graylog-plugin-aws-2.4.3.ja$
2018-04-19_15:49:55.21279 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:431) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21310 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:372) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21353 at org.graylog.aws.kinesis.KinesisConsumer.run(KinesisConsumer.java:168) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21502 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
2018-04-19_15:49:55.21552 at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
2018-04-19_15:49:55.21621 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
2018-04-19_15:49:55.21675 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
2018-04-19_15:49:55.21748 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-04-19_15:49:55.21797 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22104 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1068) ~[?:?]
2018-04-19_15:49:55.22140 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1034) ~[?:?]
2018-04-19_15:49:55.22203 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[?:?]
2018-04-19_15:49:55.22241 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[?:?]
2018-04-19_15:49:55.22308 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[?:?]
2018-04-19_15:49:55.22361 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[?:?]
2018-04-19_15:49:55.22423 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[?:?]
2018-04-19_15:49:55.22460 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[?:?]
2018-04-19_15:49:55.22522 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2186) ~[?:?]
2018-04-19_15:49:55.22561 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2162) ~[?:?]
2018-04-19_15:49:55.22594 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1048) ~[?:?]
2018-04-19_15:49:55.22630 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1024) ~[?:?]
2018-04-19_15:49:55.22705 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:154) ~[?:?]
2018-04-19_15:49:55.22744 ... 10 more
2018-04-19_15:49:55.22824 Caused by: java.net.UnknownHostException: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22863 at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_161]
2018-04-19_15:49:55.22923 at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
2018-04-19_15:49:55.22965 at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
2018-04-19_15:49:55.23031 at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[?:?]
2018-04-19_15:49:55.23075 at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[?:?]
2018-04-19_15:49:55.23151 at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[graylog.jar:?]
2018-04-19_15:49:55.23190 at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
2018-04-19_15:49:55.23271 at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23317 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
2018-04-19_15:49:55.23396 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
2018-04-19_15:49:55.23435 at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
2018-04-19_15:49:55.23520 at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23557 at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-04-19_15:49:55.23608 at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-04-19_15:49:55.23643 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23836 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
2018-04-19_15:49:55.23838 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) ~[?:?]
@pilotcorp Please format your first posts properly for better readability.
I believe this is an issue with the ami. when I try to wget from anything amazonaws.com related it fails due to name resolution. However I can wget anything else on the internet.
I spun up a amazon linux instance and was able to wget from amazon.
graylog server:
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.22.53
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.22.53|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 11:39:03-- https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... failed: No address associated with hostname.
wget: unable to resolve host address ‘aws.amazon.com’
regular server:
ec2-user@ip-10-135-240-154 ~]$ wget s3.amazonaws.com
--2018-04-19 16:38:23-- http://s3.amazonaws.com/
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.98.115
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.98.115|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 16:38:23-- https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... 54.239.26.209
Connecting to aws.amazon.com (aws.amazon.com)|54.239.26.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’
after rebuilding on a standard ec2 I’m now getting Iam permission issues regarding the dynamodb. I’m not sure what is wrong with my policy as I’m using a full admin account to test from but I will open up a ticket with aws on that for help.
com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:iam::909382730029:user/graylog is not authorized to perfo
rm: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:909382730029:table/graylog-aws-plugin-arn:aws:kinesis:us-east-1:909382730029:stream/awsFlowLogs
maybe the SDK Version for AWS is to old - we already have a PR with a newer version present: https://github.com/Graylog2/graylog-plugin-aws/pull/75
But that is just a shoot in the dark.
regards
Jan
Hello all, the iam permission issue is resolved. When I was filling in the inputs I was putting the full arn for “Kinesis Stream Name”. You should just put the name of the stream name. Linking the best document I found documenting required permissions when using the KCL as a consumer
Kinesis IAM Permissions
If you found this unclear from the (sparse) documentation for the AWS plugin, please add your findings to the README of the AWS plugin.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.