Timezone Help for single syslog device

[fangbuster2]

Hi all,

I have just set up a Graylog server (4.0.8) and have started ingested devices. All is fine except for one problematic device that incorrectly displays the timestamp in UTC instead of EST.

Graylog server/UI and my profile are all in eastern time and it is working and displaying correctly for all of our other logs coming in via syslog.

The problematic device comes in with a timestamp of the following:

Full raw message: <44>Jun 24 09:54:08 [redacted] Detected an unauthorized user attempting to access the Web interface from [redacted]. 0x0006

The Timestamp field shows incorrectly 2021-06-24 05:54:08.000 - if I hover over it, it displays the correct time: 2021-06-24T09:54:08.000Z which should be 9:54AM EST.

The timestamp field of the message also shows incorrectly: 2021-06-24 05:54:08.000 -4:00 - again if I hover over it, it displays the correct time: 2021-06-24T09:54:08.000Z

For messages from devices that do work but have the same raw format I have noticed these two differentiators:
Full raw message: <44> Jun 23 15:24:01 [redacted] 00419 auth: Invalid user name/password on SSH session User ‘asdf’ is trying to login from [redacted]

Here there is a space between the <44> and Jun where as in the bad message, it does not have this space - <44>Jun instead.
Additionally I found this post that referenced something similar but there was no resolution.
Syslog Timestamps

The other differentiator is that the timestamp field of a good message is the following: 2021-06-23 15:24:00.264 and if I hover over it, it displays correctly the same: 2021-06-23T19:24:00.264Z
The hover is correct UTC time and the display is our correct timezone of EST, but the thing is the .264Z is correctly set, whereas in our bad device it is always .000Z. So I’m not sure if this
is a bug, but in any event I am desperately trying to correct it so this bad device shows the correct timestamp in the WebUI so i can correlate the logs correctly.

I have created the following pipeline rule that seems to adjust the timezone in the simulator but it actually does not have any impact on the actual message in the stream or in all messages:

Created a stream to apply to all messages from the source IP of the bad device.
Created a pipeline that is connected to this stream and has Stage 0 and 1 rule that is the following:

rule “change_timezone”
when
true
then
let new_time = parse_date(to_string($message.timestamp), “yyyy-MM-dd’T’HH:mm:ss.SSS’Z’”, “en”, “Asia/Tashkent”);
set_field(“timestamp”, new_time);
end

When running the simulator against the bad device raw log: <44>Jun 24 09:54:08 [redacted] Detected an unauthorized user attempting to access the Web interface from [redacted]. 0x0006
the simulator shows that it will modify the original message timestamp from “2021-06-24T09:34:22.000Z” to “2021-06-24T04:35:22.000Z”

However when testing in the main UI, this pipeline rule does not seem to apply or change any of the timestamps. Additionally I am not sure if this will work as it is adjusting the timezone only.

I think there are a few things going on here and just looking for the best and easiest solution to modify the timestamp to account for the UTC time so the messages display correctly on an accurate
timeline so I can correlate logs without having to account for such discrepancies.

All feedback and help is most appreciated. Thank you so much!

[fangbuster2]

All: I figured out the issue, the pipeline needed to be moved after the Message Filter Chain! My pipeline rule works and adjusts the timezone/timestamp according to my rule above. Thank you and hopefully this helps someone else out as well!

[ttsandrew]

Hello @fangbuster2. Well done figuring out the issue! Thank you for sharing the solution back with the community.