We have a 4 node graylog cluster.
We’re currently ingesting from a few inputs, but we have 3 inputs in “Local inputs” that are in the state Not Running. When I click “Start input”, I get the message that the command was sent and it will start shortly, but it never does. No error is displayed. I’m tailing the server.log, and nothing shows up there that seems related to starting the input, but I am continuously being spammed with a AWSCloudTrail error even though that input claims to not be running. I’m more interested in the mesos logs at this point anyway.
Here’s a screen shot:
They were running for quite some time, but now noticed we were no longer getting any messages. I do see that on that particular host syslog has files mounted for the logs I’d actually be looking to ingest from graylog. It seems like something is up from syslog->graylog.
I’m pretty stumped how to diagnose this further and would appreciate any tips on what to look for. Unfortunately we’re left in a place where the original setup and admin was done in isolation.
you should really do a tail -f on your Graylog server.log and then hit the “start input” button. The error that is shown should guide you to a solution.
In addition that would help other (users of that forum) to guide you and not just guess what your problems might be.
From doing a tail, I see a stream of the following (below).
Note: this happens when I start the “Mesos Logs” local input. It’s also worth noting that the “AWS Cloudtrail” input claims to be “Not Running” in the ui. Also seeing a journal size warning but that’s it. I’ve tried both Chrome and Firefox, just in case the “Start Input” UI was being wonky.
Thanks!
2017-12-06T21:08:59.138Z ERROR [CloudTrailSubscriber] Could not read CloudTrail log file for <app-aws-cloudtrail>. Skipping.
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: java.io.StringReader@5033b2b; line: 1, column: 10008201]
at [Source: java.io.StringReader@5033b2b; line: 1, column: 9999999] (through reference chain: org.graylog.aws.inputs.cloudtrail.json.CloudTrailRecordList["Records"]->java.util.ArrayList[12158]->org.graylog.aws.inputs.cloudtrail.json.CloudTrailRecord["recipientAccountId"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:262) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:277) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3807) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2797) ~[graylog.jar:?]
at org.graylog.aws.inputs.cloudtrail.messages.TreeReader.read(TreeReader.java:20) ~[graylog-plugin-aws.jar:?]
at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:107) [graylog-plugin-aws.jar:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: java.io.StringReader@5033b2b; line: 1, column: 10008201]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:458) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:1959) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:1946) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:260) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:31) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260) ~[graylog.jar:?]
... 12 more
As you can see it’s now running after the restart.
Local Inputs not running and not starting contains the log info. It really was just that, over and over, and over. Again, outsider looking in I find it interesting that the log was getting spammed about CloudTrail when the UI claimed that import source was not running. That seems to be a good hint to me that something was a bit gummed up. The other periodic log message was a journal utilization warning.
