Error pulling flow logs via AWS

(Jason) #1

I’m using the ami from graylog to quickly test it in our dev env. I can see the kinesis stream gathering flow logs via the aws GUI but am unable to get graylog to connect to the stream. I currently have a full admin account listed under the flow log input. below is the error I’m seeing in /var/log/graylog/server/current. It looks like a dns issue with connecting to dynamodb but I can dig just fine. This instance is in a private subnet using a nat gateway. I’m not sure what to look at now.

(Jason) #2
2018-04-19_15:49:44.93497 INFO  [Worker] Initialization attempt 7
2018-04-19_15:49:44.93585 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:45.57359 INFO  [Worker] Initialization attempt 9
2018-04-19_15:49:45.57394 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:47.09861 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now STARTING
2018-04-19_15:49:47.10078 INFO  [KinesisTransport] Starting Kinesis reader thread for input [AWS Flow Logs/5ad8a1660b326b0a350a69e6]
2018-04-19_15:49:47.10325 INFO  [InputStateListener] Input [AWS Flow Logs/5ad8a1660b326b0a350a69e6] is now RUNNING
2018-04-19_15:49:47.10482 INFO  [LeaseCoordinator] With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 l$
2018-04-19_15:49:47.10509 INFO  [Worker] Initialization attempt 1
2018-04-19_15:49:47.10556 INFO  [Worker] Initializing LeaseCoordinator
2018-04-19_15:49:55.20718 ERROR [LeaseManager] Failed to get table status for graylog-aws-plugin-arn:aws:kinesis:us-east-1:936793730029:stream/awsFlowLogs
2018-04-19_15:49:55.20922 com.amazonaws.SdkClientException: Unable to execute HTTP request:
2018-04-19_15:49:55.20925       at ~[graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21106       at [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21137       at [graylog-plugin-aws-2.4.3.ja$
2018-04-19_15:49:55.21279       at [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21310       at [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21353       at [graylog-plugin-aws-2.4.3.jar:?]
2018-04-19_15:49:55.21502       at java.util.concurrent.Executors$ [?:1.8.0_161]
2018-04-19_15:49:55.21552       at [?:1.8.0_161]
2018-04-19_15:49:55.21621       at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:1.8.0_161]
2018-04-19_15:49:55.21675       at java.util.concurrent.ThreadPoolExecutor$ [?:1.8.0_161]
2018-04-19_15:49:55.21748       at [?:1.8.0_161]
2018-04-19_15:49:55.21797 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request:
2018-04-19_15:49:55.22104       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException( ~[?:?]
2018-04-19_15:49:55.22140       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper( ~[?:?]
2018-04-19_15:49:55.22203       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute( ~[?:?]
2018-04-19_15:49:55.22241       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer( ~[?:?]
2018-04-19_15:49:55.22308       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute( ~[?:?]
2018-04-19_15:49:55.22361       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500( ~[?:?]
2018-04-19_15:49:55.22423       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute( ~[?:?]
2018-04-19_15:49:55.22460       at com.amazonaws.http.AmazonHttpClient.execute( ~[?:?]
2018-04-19_15:49:55.22522       at ~[?:?]
2018-04-19_15:49:55.22561       at ~[?:?]
2018-04-19_15:49:55.22594       at ~[?:?]
2018-04-19_15:49:55.22630       at ~[?:?]
2018-04-19_15:49:55.22705       at ~[?:?]
2018-04-19_15:49:55.22744       ... 10 more
2018-04-19_15:49:55.22824 Caused by:
2018-04-19_15:49:55.22863       at ~[?:1.8.0_161]
2018-04-19_15:49:55.22923       at ~[?:1.8.0_161]
2018-04-19_15:49:55.22965       at ~[?:1.8.0_161]
2018-04-19_15:49:55.23031       at com.amazonaws.SystemDefaultDnsResolver.resolve( ~[?:?]
2018-04-19_15:49:55.23075       at com.amazonaws.http.DelegatingDnsResolver.resolve( ~[?:?]
2018-04-19_15:49:55.23151       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect( ~[graylog.jar:?]
2018-04-19_15:49:55.23190       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect( ~[graylog.jar:?]
2018-04-19_15:49:55.23271       at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23317       at sun.reflect.DelegatingMethodAccessorImpl.invoke( ~[?:1.8.0_161]
2018-04-19_15:49:55.23396       at java.lang.reflect.Method.invoke( ~[?:1.8.0_161]
2018-04-19_15:49:55.23435       at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke( ~[?:?]
2018-04-19_15:49:55.23520       at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) ~[?:?]
2018-04-19_15:49:55.23557       at org.apache.http.impl.execchain.MainClientExec.establishRoute( ~[graylog.jar:?]
2018-04-19_15:49:55.23608       at org.apache.http.impl.execchain.MainClientExec.execute( ~[graylog.jar:?]
2018-04-19_15:49:55.23643       at org.apache.http.impl.execchain.ProtocolExec.execute( ~[graylog.jar:?]
2018-04-19_15:49:55.23836       at org.apache.http.impl.client.InternalHttpClient.doExecute( ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute( ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at org.apache.http.impl.client.CloseableHttpClient.execute( ~[graylog.jar:?]
2018-04-19_15:49:55.23837       at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute( ~[?:?]
2018-04-19_15:49:55.23838       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest( ~[?:?]

(Jochen) #3

@pilotcorp Please format your first posts properly for better readability.

(Jason) #4

I believe this is an issue with the ami. when I try to wget from anything related it fails due to name resolution. However I can wget anything else on the internet.
I spun up a amazon linux instance and was able to wget from amazon.

graylog server:

Resolving (
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: [following]
--2018-04-19 11:39:03--
Resolving ( failed: No address associated with hostname.
wget: unable to resolve host address ‘’

regular server:

ec2-user@ip-10-135-240-154 ~]$ wget
--2018-04-19 16:38:23--
Resolving (
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: [following]
--2018-04-19 16:38:23--
Resolving (
Connecting to (||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

(Jason) #5

after rebuilding on a standard ec2 I’m now getting Iam permission issues regarding the dynamodb. I’m not sure what is wrong with my policy as I’m using a full admin account to test from but I will open up a ticket with aws on that for help. User: arn:aws:iam::909382730029:user/graylog is not authorized to perfo
rm: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:909382730029:table/graylog-aws-plugin-arn:aws:kinesis:us-east-1:909382730029:stream/awsFlowLogs

(Jan Doberstein) #6

maybe the SDK Version for AWS is to old - we already have a PR with a newer version present:

But that is just a shoot in the dark.


(Jason) #7

Hello all, the iam permission issue is resolved. When I was filling in the inputs I was putting the full arn for “Kinesis Stream Name”. You should just put the name of the stream name. Linking the best document I found documenting required permissions when using the KCL as a consumer
Kinesis IAM Permissions

(Jochen) #8

If you found this unclear from the (sparse) documentation for the AWS plugin, please add your findings to the README of the AWS plugin.

(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.