I’m hitting a problem that seems very basic, so I’m hoping someone can point out something obvious I’ve missed. I see the problem appear in several places, but I’ll give the simplest form of the problem here:
I click on Search. I select a particular stream of messages.
The default view of a “Message Count” chart and “All Messages” list appears with a number of messages represented. However, the timestamp shown in the messages doesn’t align with the time shown in the Message Count chart. For reference, I’m using a user account with a -5 timezone (Chicago), and the timestamps in the messages are displayed as such. But in the chart, the messages are displayed 3 hrs earlier.
When I mouse over the chart results above, such as the last far right bar, the pop-up places those messages at 2020-06-14 23:00. But the last message in the list below is 2020-06-15 02:00 (in Chicago timezone). Does the chart use a different timezone?
To make the mystery more strange, I have to select a large time frame or nothing shows up in the chart, even if there are messages that appear in the list. Here is the same query with only a 2 day time frame.
The incoming messages for this stream are GELF messages, with no timestamp set – purposefully so that the graylog server will be the authority on the time, since the sending agents’ clocks are all often wrong.
This doesn’t seem to be happening with other messages. I have other incoming logs being parsed that are represented accurately between the chart and the messages.
I’ve looked at the server logs to see if there are errors that indicate a problem, but I don’t see anything amiss. Anyone have any ideas of what’s going on here?
Thanks for any help you can offer.