Error pulling flow logs via AWS

I’m using the ami from graylog to quickly test it in our dev env. I can see the kinesis stream gathering flow logs via the aws GUI but am unable to get graylog to connect to the stream. I currently have a full admin account listed under the flow log input. below is the error I’m seeing in /var/log/graylog/server/current. It looks like a dns issue with connecting to dynamodb but I can dig dynamodb.us-east-1.amazonaws.com just fine. This instance is in a private subnet using a nat gateway. I’m not sure what to look at now.

2018-04-19_15:49:44.93497 INFO  [Worker] Initialization attempt 7
2018-04-19_15:49:44.93585 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:45.57359 INFO  [Worker] Initialization attempt 9
2018-04-19_15:49:45.57394 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:47.09861 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now STARTING
2018-04-19_15:49:47.10078 INFO  [KinesisTransport] Starting Kinesis reader thread for input [AWS Flow Logs/5ad8a1660b326b0a350a69e6]
2018-04-19_15:49:47.10325 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now RUNNING
2018-04-19_15:49:47.10482 INFO  [LeaseCoordinator] With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 l$
2018-04-19_15:49:47.10509 INFO  [Worker] Initialization attempt 1
2018-04-19_15:49:47.10556 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:55.20718 ERROR [LeaseManager] Failed to get table status for graylog-aws-plugin-arn:aws:kinesis:us-east-1:936793730029:stream/awsFlowLogs
2018-04-19_15:49:55.20922 com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.20925       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:162) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21106       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:107) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21137       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:235) [graylog-plugin-aws-2.4.3.ja$
2018-04-19_15:49:55.21279       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:431) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21310       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:372) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21353       at org.graylog.aws.kinesis.KinesisConsumer.run(KinesisConsumer.java:168) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21502       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
2018-04-19_15:49:55.21552       at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
2018-04-19_15:49:55.21621       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
2018-04-19_15:49:55.21675       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
2018-04-19_15:49:55.21748       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-04-19_15:49:55.21797 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22104       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1068) ~[?:?]
2018-04-19_15:49:55.22140       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1034) ~[?:?]
2018-04-19_15:49:55.22203       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[?:?]
2018-04-19_15:49:55.22241       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[?:?]
2018-04-19_15:49:55.22308       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[?:?]
2018-04-19_15:49:55.22361       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[?:?]
2018-04-19_15:49:55.22423       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[?:?]
2018-04-19_15:49:55.22460       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[?:?]
2018-04-19_15:49:55.22522       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2186) ~[?:?]
2018-04-19_15:49:55.22561       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2162) ~[?:?]
2018-04-19_15:49:55.22594       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1048) ~[?:?]
2018-04-19_15:49:55.22630       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1024) ~[?:?]
2018-04-19_15:49:55.22705       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:154) ~[?:?]
2018-04-19_15:49:55.22744       ... 10 more
2018-04-19_15:49:55.22824 Caused by: java.net.UnknownHostException: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22863       at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_161]
2018-04-19_15:49:55.22923       at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
2018-04-19_15:49:55.22965       at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
2018-04-19_15:49:55.23031       at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[?:?]
2018-04-19_15:49:55.23075       at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[?:?]
2018-04-19_15:49:55.23151       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[graylog.jar:?]
2018-04-19_15:49:55.23190       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
2018-04-19_15:49:55.23271       at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23317       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
2018-04-19_15:49:55.23396       at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
2018-04-19_15:49:55.23435       at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
2018-04-19_15:49:55.23520       at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23557       at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-04-19_15:49:55.23608       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-04-19_15:49:55.23643       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23836       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
2018-04-19_15:49:55.23838       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) ~[?:?]

@pilotcorp Please format your first posts properly for better readability.

https://community.graylog.org/faq#format-markdown

I believe this is an issue with the ami. when I try to wget from anything amazonaws.com related it fails due to name resolution. However I can wget anything else on the internet.
I spun up a amazon linux instance and was able to wget from amazon.

graylog server:

Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.22.53
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.22.53|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 11:39:03--  https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... failed: No address associated with hostname.
wget: unable to resolve host address ‘aws.amazon.com’

regular server:

ec2-user@ip-10-135-240-154 ~]$ wget s3.amazonaws.com
--2018-04-19 16:38:23--  http://s3.amazonaws.com/
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.98.115
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.98.115|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 16:38:23--  https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... 54.239.26.209
Connecting to aws.amazon.com (aws.amazon.com)|54.239.26.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

after rebuilding on a standard ec2 I’m now getting Iam permission issues regarding the dynamodb. I’m not sure what is wrong with my policy as I’m using a full admin account to test from but I will open up a ticket with aws on that for help.

com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:iam::909382730029:user/graylog is not authorized to perfo
rm: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:909382730029:table/graylog-aws-plugin-arn:aws:kinesis:us-east-1:909382730029:stream/awsFlowLogs

maybe the SDK Version for AWS is to old - we already have a PR with a newer version present: https://github.com/Graylog2/graylog-plugin-aws/pull/75

But that is just a shoot in the dark.

regards
Jan

Hello all, the iam permission issue is resolved. When I was filling in the inputs I was putting the full arn for “Kinesis Stream Name”. You should just put the name of the stream name. Linking the best document I found documenting required permissions when using the KCL as a consumer
Kinesis IAM Permissions

If you found this unclear from the (sparse) documentation for the AWS plugin, please add your findings to the README of the AWS plugin.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.