Recently updated to the newest Graylog version 2.3.1, running ES 5.6 and I can not longer search past 23:23 CST. I can see the indices are growing, there aren’t any errors in elasticsearch logs (cluster is green), nothing in graylog, nothing in the journal, and I can event see the logs being processed and can manually perform a query and see that graylog is shipping the logs into the indices. For whatever reason though, the last log I am able to see is at 23:23. I’ve seen it before where it was a time zone issue, but if I go back 8 hours, and turn on live updates, i’m not even getting new messages popping in. The side bar where it states how many total messages were found in the past 8 hours is growing, but no new messages pop up into the messages feed. I’m assuming it has something to do with having to manually clear out the sessions from my MongoDB because I was getting the timeout error that doesn’t appear to have a solution to, referenced here https://github.com/Graylog2/graylog2-server/issues/2559.
Is there a point of reference/file that is telling graylog that message x is the last available message to search for or something?
It appears I was able to fix the issue. To help others I’ll identify what I did to fix this:
What happened:
For some reason, Graylog was not including the “gralog_deflector” index. (Graylog_deflector index is the default name of the current index that is being written too.)
This was verified by connecting to my mongoDB, and running:
use graylog
db.index_ranges.find()
I believe there should be an entry in there with your latest index. You can check your current index name on the web app under System>indices>“Default Index” and the index name should be the current “active write index”. Whatever the name of that index is, there should be an entry located in the db.index_ranges.find() output from your mongodb. If there is not an entry, go to the web app>indices>“Default Index set” and in the top right there is a blue button called maintenance.
Select that button and choose the recalculate index ranges button, and that should re-add that index into your db.index_range.find() output. Your alerts should start appearing again.
