The server running graylogs/elasticsearch lost its NFS mount where the elasticsearch indicies are stored.
After a reboot of the server, almost everything is working fine, except the following:
DHCP messages are coming in on the input.
I see that the DHCP stream is receiving messages based on the stream page. The number is higher than zero.
When I enter the DHCP stream, there are no messages from after the reboot, but there are messages from before the reboot.
When I check the DHCP index, I see that the number of documents in the DHCP index are increasing.
I created a copy of the DHCP stream as DHCP stream 2. When I enter DHCP stream 2 I can see the current entries that are coming in. I can’t see the old entries, as expected.
How can I fix the original stream so that it shows old and new messages (and I assume all of the messages since the network issue since the messages are making it into the index).
sounds like a timestamp issue, you may want to see if you can adjust the query timeframe in the future to check for those messages.
Curious about how you’re managing to use NFS for your elastic data directory. Last time I attempted, elastic couldn’t get a lock on the mount, which others have experienced as well.
you need to check the elasticsearch logs for any error messages.
In addition having elasticsearch on NFS is not a good idea … but that will not help you now.
You will find something in your logs, either Graylog or Elasticsearch and that should give you the idea what the problem is. I can’t imagine and I will not build a lab setup out of that.
