Local Inputs not running and not starting

We have a 4 node graylog cluster.
We’re currently ingesting from a few inputs, but we have 3 inputs in “Local inputs” that are in the state Not Running. When I click “Start input”, I get the message that the command was sent and it will start shortly, but it never does. No error is displayed. I’m tailing the server.log, and nothing shows up there that seems related to starting the input, but I am continuously being spammed with a AWSCloudTrail error even though that input claims to not be running. I’m more interested in the mesos logs at this point anyway.

Here’s a screen shot:
image

They were running for quite some time, but now noticed we were no longer getting any messages. I do see that on that particular host syslog has files mounted for the logs I’d actually be looking to ingest from graylog. It seems like something is up from syslog->graylog.

I’m pretty stumped how to diagnose this further and would appreciate any tips on what to look for. Unfortunately we’re left in a place where the original setup and admin was done in isolation.

Thanks,
John

Hej John,

you should really do a tail -f on your Graylog server.log and then hit the “start input” button. The error that is shown should guide you to a solution.

In addition that would help other (users of that forum) to guide you and not just guess what your problems might be.

regards
Jan

From doing a tail, I see a stream of the following (below).
Note: this happens when I start the “Mesos Logs” local input. It’s also worth noting that the “AWS Cloudtrail” input claims to be “Not Running” in the ui. Also seeing a journal size warning but that’s it. I’ve tried both Chrome and Firefox, just in case the “Start Input” UI was being wonky.

Thanks!

2017-12-06T21:08:59.138Z ERROR [CloudTrailSubscriber] Could not read CloudTrail log file for <app-aws-cloudtrail>. Skipping.
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
 at [Source: java.io.StringReader@5033b2b; line: 1, column: 10008201]
 at [Source: java.io.StringReader@5033b2b; line: 1, column: 9999999] (through reference chain: org.graylog.aws.inputs.cloudtrail.json.CloudTrailRecordList["Records"]->java.util.ArrayList[12158]->org.graylog.aws.inputs.cloudtrail.json.CloudTrailRecord["recipientAccountId"])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:262) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:277) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3807) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2797) ~[graylog.jar:?]
	at org.graylog.aws.inputs.cloudtrail.messages.TreeReader.read(TreeReader.java:20) ~[graylog-plugin-aws.jar:?]
	at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:107) [graylog-plugin-aws.jar:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value
 at [Source: java.io.StringReader@5033b2b; line: 1, column: 10008201]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:458) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:1959) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:1946) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:260) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:31) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260) ~[graylog.jar:?]
	... 12 more

Not sure if it’s helpful, but my syslog conf has
destination d_graylog_mesos { syslog( "127.0.0.1" port(10516) ts_format(iso) frac-digits(3)); };

I see that the syslog proc has our mesos logs properly captured and up to date.

Well, one of our operators bounced the graylog server and the local inputs just started running again :-/

Alls well that ends well. I’d be willing to still gather any info that might aide in determining root cause if there is anyone interested.

What’s the full configuration of the Syslog TCP input in Graylog?
What’s in the logs of your Graylog nodes?

Here’s the full input config:
image

As you can see it’s now running after the restart.

Local Inputs not running and not starting contains the log info. It really was just that, over and over, and over. Again, outsider looking in I find it interesting that the log was getting spammed about CloudTrail when the UI claimed that import source was not running. That seems to be a good hint to me that something was a bit gummed up. The other periodic log message was a journal utilization warning.

Is syslog-ng running on the same machine as Graylog?

Yes, syslog-ng is running on the same machine as graylog.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.