Error pulling flow logs via AWS


(Jason) #1

I’m using the ami from graylog to quickly test it in our dev env. I can see the kinesis stream gathering flow logs via the aws GUI but am unable to get graylog to connect to the stream. I currently have a full admin account listed under the flow log input. below is the error I’m seeing in /var/log/graylog/server/current. It looks like a dns issue with connecting to dynamodb but I can dig dynamodb.us-east-1.amazonaws.com just fine. This instance is in a private subnet using a nat gateway. I’m not sure what to look at now.


(Jason) #2
2018-04-19_15:49:44.93497 INFO  [Worker] Initialization attempt 7
2018-04-19_15:49:44.93585 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:45.57359 INFO  [Worker] Initialization attempt 9
2018-04-19_15:49:45.57394 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:47.09861 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now STARTING
2018-04-19_15:49:47.10078 INFO  [KinesisTransport] Starting Kinesis reader thread for input [AWS Flow Logs/5ad8a1660b326b0a350a69e6]
2018-04-19_15:49:47.10325 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now RUNNING
2018-04-19_15:49:47.10482 INFO  [LeaseCoordinator] With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 l$
2018-04-19_15:49:47.10509 INFO  [Worker] Initialization attempt 1
2018-04-19_15:49:47.10556 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:55.20718 ERROR [LeaseManager] Failed to get table status for graylog-aws-plugin-arn:aws:kinesis:us-east-1:936793730029:stream/awsFlowLogs
2018-04-19_15:49:55.20922 com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.20925       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:162) ~[graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21106       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:107) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21137       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:235) [graylog-plugin-aws-2.4.3.ja$
2018-04-19_15:49:55.21279       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:431) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21310       at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:372) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21353       at org.graylog.aws.kinesis.KinesisConsumer.run(KinesisConsumer.java:168) [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21502       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
2018-04-19_15:49:55.21552       at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
2018-04-19_15:49:55.21621       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
2018-04-19_15:49:55.21675       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
2018-04-19_15:49:55.21748       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-04-19_15:49:55.21797 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22104       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1068) ~[?:?]
2018-04-19_15:49:55.22140       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1034) ~[?:?]
2018-04-19_15:49:55.22203       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[?:?]
2018-04-19_15:49:55.22241       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[?:?]
2018-04-19_15:49:55.22308       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[?:?]
2018-04-19_15:49:55.22361       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[?:?]
2018-04-19_15:49:55.22423       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[?:?]
2018-04-19_15:49:55.22460       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[?:?]
2018-04-19_15:49:55.22522       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2186) ~[?:?]
2018-04-19_15:49:55.22561       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2162) ~[?:?]
2018-04-19_15:49:55.22594       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1048) ~[?:?]
2018-04-19_15:49:55.22630       at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1024) ~[?:?]
2018-04-19_15:49:55.22705       at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:154) ~[?:?]
2018-04-19_15:49:55.22744       ... 10 more
2018-04-19_15:49:55.22824 Caused by: java.net.UnknownHostException: dynamodb.us-east-1.amazonaws.com
2018-04-19_15:49:55.22863       at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_161]
2018-04-19_15:49:55.22923       at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
2018-04-19_15:49:55.22965       at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
2018-04-19_15:49:55.23031       at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[?:?]
2018-04-19_15:49:55.23075       at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[?:?]
2018-04-19_15:49:55.23151       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[graylog.jar:?]
2018-04-19_15:49:55.23190       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
2018-04-19_15:49:55.23271       at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23317       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
2018-04-19_15:49:55.23396       at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
2018-04-19_15:49:55.23435       at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
2018-04-19_15:49:55.23520       at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23557       at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-04-19_15:49:55.23608       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-04-19_15:49:55.23643       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23836       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
2018-04-19_15:49:55.23838       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1189) ~[?:?]

(Jochen) #3

@pilotcorp Please format your first posts properly for better readability.

https://community.graylog.org/faq#format-markdown


(Jason) #4

I believe this is an issue with the ami. when I try to wget from anything amazonaws.com related it fails due to name resolution. However I can wget anything else on the internet.
I spun up a amazon linux instance and was able to wget from amazon.

graylog server:

Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.22.53
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.22.53|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 11:39:03--  https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... failed: No address associated with hostname.
wget: unable to resolve host address ‘aws.amazon.com’

regular server:

ec2-user@ip-10-135-240-154 ~]$ wget s3.amazonaws.com
--2018-04-19 16:38:23--  http://s3.amazonaws.com/
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.98.115
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.98.115|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://aws.amazon.com/s3/ [following]
--2018-04-19 16:38:23--  https://aws.amazon.com/s3/
Resolving aws.amazon.com (aws.amazon.com)... 54.239.26.209
Connecting to aws.amazon.com (aws.amazon.com)|54.239.26.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

(Jason) #5

after rebuilding on a standard ec2 I’m now getting Iam permission issues regarding the dynamodb. I’m not sure what is wrong with my policy as I’m using a full admin account to test from but I will open up a ticket with aws on that for help.

com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:iam::909382730029:user/graylog is not authorized to perfo
rm: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:909382730029:table/graylog-aws-plugin-arn:aws:kinesis:us-east-1:909382730029:stream/awsFlowLogs


(Jan Doberstein) #6

maybe the SDK Version for AWS is to old - we already have a PR with a newer version present: https://github.com/Graylog2/graylog-plugin-aws/pull/75

But that is just a shoot in the dark.

regards
Jan


(Jason) #7

Hello all, the iam permission issue is resolved. When I was filling in the inputs I was putting the full arn for “Kinesis Stream Name”. You should just put the name of the stream name. Linking the best document I found documenting required permissions when using the KCL as a consumer
Kinesis IAM Permissions


(Jochen) #8

If you found this unclear from the (sparse) documentation for the AWS plugin, please add your findings to the README of the AWS plugin.


(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.