Timestamp in graylog does not match logs being ingested

[djc (Dan)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
See title. The logs coming into graylog are off by 4 hours even though everything is configured for America/Detroit.

2. Describe your environment:

• OS Information: Ubuntu 22.04

• Package Version: Graylog 5.0.7 / Opensearch 2.5 / Mongodb 6.0.6

• Service logs, configurations, and environment variables:

Graylog/MongoDB server:

timedatectl status
               Local time: Tue 2023-05-16 13:42:48 EDT
           Universal time: Tue 2023-05-16 17:42:48 UTC
                 RTC time: Tue 2023-05-16 17:42:48
                Time zone: America/Detroit (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Opensearch server:

timedatectl status
               Local time: Tue 2023-05-16 13:43:33 EDT
           Universal time: Tue 2023-05-16 17:43:33 UTC
                 RTC time: Tue 2023-05-16 17:43:33
                Time zone: America/Detroit (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Fortigate:

Logs in Graylog dashbaord:

3. What steps have you already taken to try and solve the problem?
Verified all timed/date configuration. Restarted services and even restarted servers.

4. How can the community help?

Any guidance to get timestamps in graylog web dashboard to match actual log timestamp would be appreciated.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[djc (Dan)]

One thing I find interesting is that there is a pre-parsed field in the log that is:

tz = -0400

which equals the time offset between what graylog dashboard is showing and what the actual timestamp is. Is graylog somehow reading that time zone and applying it?

[patrickmann]

EDT is 4h ahead of UTC, so I think this is just a case of message times showing as UTC.
Here is a good summary of time zone issues:

Time Zones: A Logger's Worst Nightmare

[djc (Dan)]

EDT is 4 hours behind UTC. It seems the issue is with how fortigate sends its logs.

I was able to resolve the issue by modifying the timestamp field with a pipeline rule:

rule "fortigate timestamp"
when
    has_field("devname") && has_field("date") && has_field("time")
then
    let part1 = concat(to_string($message.date), " ");
    let part2 = concat(part1, to_string($message.time));
    let shift_timestamp = parse_date(value:to_string(part2), pattern:"yyyy-MM-dd HH:mm:sss", timezone:"America/Detroit");
    set_field("timestamp", shift_timestamp);
end