We have a 4 node graylog cluster.
We’re currently ingesting from a few inputs, but we have 3 inputs in “Local inputs” that are in the state Not Running. When I click “Start input”, I get the message that the command was sent and it will start shortly, but it never does. No error is displayed. I’m tailing the server.log, and nothing shows up there that seems related to starting the input, but I am continuously being spammed with a AWSCloudTrail error even though that input claims to not be running. I’m more interested in the mesos logs at this point anyway.
Here’s a screen shot:
They were running for quite some time, but now noticed we were no longer getting any messages. I do see that on that particular host syslog has files mounted for the logs I’d actually be looking to ingest from graylog. It seems like something is up from syslog->graylog.
I’m pretty stumped how to diagnose this further and would appreciate any tips on what to look for. Unfortunately we’re left in a place where the original setup and admin was done in isolation.