Getting correct timezone for timestamp

[pasul]

Hi all,

I’m trying to pull timestamp from messages instead of using filebeat timestamp. Its mostly working ok, but I have issues with Timezones.

Timezone of my Graylog Server and User is set to Pacific/Auckland. Timestamps in my streams are currently showing as: 2020-05-18 14:31:01 +12:00

I have some logs shipped from Australia ( Melbourne ), example message:

[DEBUG] 2020-05-18 12:32:37.577 User logged in: false

I’ve setup an extractor on my Beats Input for message field. I’ve used TIMESTAMP_ISO8601 GROK Pattern.

That works nicely, for the message above I get field TIMESTAMP_ISO8601: 2020-05-18 12:32:37.577

Then I’ve setup a new pipeline rule to create a new_timestamp date field.

rule "Change AU timestamp"
when
    has_field("TIMESTAMP_ISO8601")
then
    let new_date = parse_date(to_string($message.TIMESTAMP_ISO8601), "yyyy-MM-dd HH:mm:ss.SSS");
    set_field("new_timestamp", new_date);
end

After this rule is applied, field new_timestamp is created with value: 2020-05-19 00:32:37 +12:00

It seems like it adds +12 hours to the TIMESTAMP_ISO8601 field which already contains time Australian time (+10.00).

To resolve the problem I would probably need to add timezone information to my pipeline rule.

Here are the options I’ve tried without success:

let new_date = parse_date(to_string($message.TIMESTAMP_ISO8601), "yyyy-MM-dd HH:mm:ss.SSS", "en-AU", "AEST");

let new_date = parse_date(to_string($message.TIMESTAMP_ISO8601), "yyyy-MM-dd HH:mm:ss.SSS", "Australia/Melbourne");

I’m hoping to get new_timestamp filed to be 2020-05-18 14:32:37 +12.00 or 2020-05-18 12:32:37 +10.00

Then I could use it to replace original timestamp field.

Any help or pointers would be much appreciated.

[shoothub]

I don’t know if graylog render correct timezone for other field than timestamp. Try to change it from new_timestamp to timestamp and try…

[pasul]

Thanks shoothub.
I think you can set timezones on other fields as well if you use parse_date function.
I’ve managed to resolve the issue with following format:

rule "Change AU timestamp"
when
    has_field("TIMESTAMP_ISO8601")
then
    let new_date = parse_date(
        value: to_string($message.TIMESTAMP_ISO8601), 
        pattern: "yyyy-MM-dd HH:mm:ss.SSS", 
        timezone: "Australia/Melbourne");
    set_field("timestamp", new_date);
end