I’m using the ami from graylog to quickly test it in our dev env. I can see the kinesis stream gathering flow logs via the aws GUI but am unable to get graylog to connect to the stream. I currently have a full admin account listed under the flow log input. below is the error I’m seeing in /var/log/graylog/server/current. It looks like a dns issue with connecting to dynamodb but I can dig dynamodb.us-east-1.amazonaws.com just fine. This instance is in a private subnet using a nat gateway. I’m not sure what to look at now.
2018-04-19_15:49:44.93497 INFO [Worker] Initialization attempt 7
2018-04-19_15:49:44.93585 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:45.57359 INFO [Worker] Initialization attempt 9
2018-04-19_15:49:45.57394 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:47.09861 INFO [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now STARTING
2018-04-19_15:49:47.10078 INFO [KinesisTransport] Starting Kinesis reader thread for input [AWS Flow Logs/5ad8a1660b326b0a350a69e6]
2018-04-19_15:49:47.10325 INFO [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now RUNNING
2018-04-19_15:49:47.10482 INFO [LeaseCoordinator] With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 l$
2018-04-19_15:49:47.10509 INFO [Worker] Initialization attempt 1
2018-04-19_15:49:47.10556 INFO [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:55.20718 ERROR [LeaseManager] Failed to get table status for graylog-aws-plugin-arn:aws:kinesis:us-east-1:936793730029:stream/awsFlowLogs
2018-04-19_15:49:55.20922 com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.20925 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:162) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21106 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:107) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21137 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:235) [graylog-plugin-aws-2.4.3.ja$
2018-04-19_15:49:55.21279 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:431) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21310 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:372) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21353 at org.graylog.aws.kinesis.KinesisConsumer.run(KinesisConsumer.java:168) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21502 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
2018-04-19_15:49:55.21552 at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
2018-04-19_15:49:55.21621 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
2018-04-19_15:49:55.21675 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
2018-04-19_15:49:55.21748 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-04-19_15:49:55.21797 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22104 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1068) ~[?:?]
2018-04-19_15:49:55.22140 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1034) ~[?:?]
2018-04-19_15:49:55.22203 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[?:?]
2018-04-19_15:49:55.22241 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[?:?]
2018-04-19_15:49:55.22308 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[?:?]
2018-04-19_15:49:55.22361 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[?:?]
2018-04-19_15:49:55.22423 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[?:?]
2018-04-19_15:49:55.22460 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[?:?]
2018-04-19_15:49:55.22522 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2186) ~[?:?]
2018-04-19_15:49:55.22561 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2162) ~[?:?]
2018-04-19_15:49:55.22594 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1048) ~[?:?]
2018-04-19_15:49:55.22630 at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1024) ~[?:?]
2018-04-19_15:49:55.22705 at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:154) ~[?:?]
2018-04-19_15:49:55.22744 ... 10 more
2018-04-19_15:49:55.22824 Caused by: java.net.UnknownHostException: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22863 at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_161]
2018-04-19_15:49:55.22923 at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
2018-04-19_15:49:55.22965 at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
2018-04-19_15:49:55.23031 at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[?:?]
2018-04-19_15:49:55.23075 at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[?:?]
2018-04-19_15:49:55.23151 at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[graylog.jar:?]
2018-04-19_15:49:55.23190 at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
2018-04-19_15:49:55.23271 at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23317 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
2018-04-19_15:49:55.23396 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
2018-04-19_15:49:55.23435 at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
2018-04-19_15:49:55.23520 at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23557 at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-04-19_15:49:55.23608 at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-04-19_15:49:55.23643 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23836 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[graylog.jar:?]
2018-04-19_15:49:55.23837 at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
2018-04-19_15:49:55.23838 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) ~[?:?]
@pilotcorp Please format your first posts properly for better readability.
I believe this is an issue with the ami. when I try to wget from anything amazonaws.com related it fails due to name resolution. However I can wget anything else on the internet.
I spun up a amazon linux instance and was able to wget from amazon.
graylog server:
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.22.53
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.22.53|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 11:39:03-- https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... failed: No address associated with hostname.
wget: unable to resolve host address ‘aws.amazon.com’
regular server:
ec2-user@ip-10-135-240-154 ~]$ wget s3.amazonaws.com
--2018-04-19 16:38:23-- http://s3.amazonaws.com/
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.98.115
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.98.115|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 16:38:23-- https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... 54.239.26.209
Connecting to aws.amazon.com (aws.amazon.com)|54.239.26.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’
after rebuilding on a standard ec2 I’m now getting Iam permission issues regarding the dynamodb. I’m not sure what is wrong with my policy as I’m using a full admin account to test from but I will open up a ticket with aws on that for help.
com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:iam::909382730029:user/graylog is not authorized to perfo
rm: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:909382730029:table/graylog-aws-plugin-arn:aws:kinesis:us-east-1:909382730029:stream/awsFlowLogs
maybe the SDK Version for AWS is to old - we already have a PR with a newer version present: https://github.com/Graylog2/graylog-plugin-aws/pull/75
But that is just a shoot in the dark.
regards
Jan
Hello all, the iam permission issue is resolved. When I was filling in the inputs I was putting the full arn for “Kinesis Stream Name”. You should just put the name of the stream name. Linking the best document I found documenting required permissions when using the KCL as a consumer
Kinesis IAM Permissions
If you found this unclear from the (sparse) documentation for the AWS plugin, please add your findings to the README of the AWS plugin.