I’m hitting a problem that seems very basic, so I’m hoping someone can point out something obvious I’ve missed. I see the problem appear in several places, but I’ll give the simplest form of the problem here:
I click on Search. I select a particular stream of messages.
The default view of a “Message Count” chart and “All Messages” list appears with a number of messages represented. However, the timestamp shown in the messages doesn’t align with the time shown in the Message Count chart. For reference, I’m using a user account with a -5 timezone (Chicago), and the timestamps in the messages are displayed as such. But in the chart, the messages are displayed 3 hrs earlier.
When I mouse over the chart results above, such as the last far right bar, the pop-up places those messages at 2020-06-14 23:00. But the last message in the list below is 2020-06-15 02:00 (in Chicago timezone). Does the chart use a different timezone?
To make the mystery more strange, I have to select a large time frame or nothing shows up in the chart, even if there are messages that appear in the list. Here is the same query with only a 2 day time frame.
The incoming messages for this stream are GELF messages, with no timestamp set – purposefully so that the graylog server will be the authority on the time, since the sending agents’ clocks are all often wrong.
This doesn’t seem to be happening with other messages. I have other incoming logs being parsed that are represented accurately between the chart and the messages.
I’ve looked at the server logs to see if there are errors that indicate a problem, but I don’t see anything amiss. Anyone have any ideas of what’s going on here?
I think, that you always have to sync time using NTP on all hosts sending logs. It’s a must, so you can’t except that timestamps will be fine.
Graylog always convert timestamps internally to UTC and render in timezone you setup for user. For user admin you need to setup timezone in /etc/graylog/server/server.conf file with parameter: root_timezone
Do you login as admin user to graylog web interface? If so, check /etc/graylog/server/server.conf file a parameter: root_timezone, setup correct
If you use regular user (other than admin), check timezone settings in Profile
Check System - Overview - check Time configuration: server, web, user and compare
Check also timezone settings in Linux server, using sudo timedatectl, or change correct timezone with timedatectl set-timezone ‘Europe/Bratislava’ or so
Check also timezone setting in sending hosts, that uses GELF
Yes, thank you. I explained all of that in the first post, but to answer your questions directly here:
1 & 2. I’m not using admin. I’m using a user with a -5hr offset (Chicago.)
3 & 4. Yes, system timezone and time is set correctly.
5. As I said, the sending hosts are intentionally not sending timestamp values. (These are IoT devices that in this environment have no reliable clock.) By Graylog GELF documentation, if timestamp is not set, it will be set by the timestamp of the Graylog server. This is working as expected, and confirmed to be set correctly in the messages, as shown in the screenshot. But the chart doesn’t report them in the correct place.
My question is: what would cause Graylog to chart the timestamp 3 hours off from the messages’ timestamps? Your answers seem to indicate that this could occur if Graylog’s root timezone is set incorrectly? I’ve confirmed that, it is set correctly. And, it is interesting that other incoming logs are charting correctly, matching their message’s timestamps. Any other ideas?
My question is: what would cause Graylog to chart the timestamp 3 hours off from the messages’ timestamps? Your answers seem to indicate that this could occur if Graylog’s root timezone is set incorrectly? I’ve confirmed that, it is set correctly. And, it is interesting that other incoming logs are charting correctly, matching their message’s timestamps. Any other ideas?
It might be a bug @sirkus but I can’t be sure with that. Personal I would open a bug report over at github giving enough details that a developer is able to reproduce that:
