1. Describe your incident:
Graylog timestamps do not line up for received logs. The graylog server’s OS is set to UTC and so are the sending clients. For example, below is a log message that is received by Graylog, for which you can see the timestamp doesn’t match up and is actually in the future, meaning that I have to set the time view to “absolute” mode and fast-forward a few hours to see any recent logs.
Screenshots showing timedatectl on sending servers and graylog server, as well as syslog udp input configuration and times shown within the wbeui, as well as the log message showing mismatched times: Imgur: The magic of the Internet
2. Describe your environment:
OS Information: Debian 12, Kernel version 6.1.0-12-amd64
Package Version: 5.0.11+30bdbfa, codename Noir
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
Set root_timezone in server.conf to UTC
timedatectl set to UTC
Restarted graylog-server and the operating system
Ensured clients sending logs (via rsyslogd) are configured with UTC time using timedatectl
Restarted rsyslogd on the clients sending logs and also rebooted the systems entirely
Enabled “Allow overriding date” on syslog input with no success
4. How can the community help?
I’m hoping that the graylog community here will be able to spot the cause of this issue and point me in the right direction to resolve it, as it seems that I’ve been unsuccessful in resolving this myself.
Seams like you check most , if not all timezone settings on everything.
The user that is logged in also has the correct Timezone configured in the profile? Reason I ask is that the Admin timestamp inthe link you post is off but the other two are the same.
example:
I assume this setting is good in your GL config file?
# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
The timezone is indeed correct within the user’s profile, and also the graylog config has the root_timezone configured correctly as well. What I can’t understand is why the graylog server thinks it is in UTC-4 and not UTC like it’s supposed to be, and I have a feeling that it’s what is causing the timestamps to be out-of-sync and resulting in having to set an absolute date in order to view recent logs.
I havent had any issue with timezones but I also havent used UTC. it seams you hit all the timezone spots but what I dont under stand is the Server config shows UTC -4 and the web browser.
UTC -4 is the east coast of USA. kind of odd because Opensearch/Elasticsearch uses UTC by default and the other configurations are within the configuration file.
Something wasnt configured correctly and it hard to see from here. You are correct this is the reason why you have this issue its with timezone out of sync 100%
