Struggling With Timestamp conversion

[EyImWawkinHeah]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
We are receiving processing errors for 2 different log sources relating to timestamps.

Failure Details Example:
Value <2024-08-22T17:55:22.875418Z> caused exception: Invalid format: “2024-08-22T17:55:22.875418Z” is malformed at “T17:55:22.875418Z”.

2. Describe your environment:
2 node cluster in Graylog Cloud of 6.0.4

3. What steps have you already taken to try and solve the problem?
I have attempted to do research onto how other people have solved this issue.
I found pipeline rules extracting/parsing the timestamp to be the solution for most people but its just not working for me.

Here is the current pipeline rule I am trying out, doesn’t seem to work:

rule "testing timestamps"
when
   true
then
    let result = regex("([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})",to_string($message.message));    
    let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS","EST");
    set_field("timestamp", new_date);
end

This doesn’t appear to set the new date correctly:

The regex appears to be good.
I have also played around with the date format in new_date trying different variations. Where am I going wrong? For what its worth, I’m pretty new to pipeline rules so it might be painfully obvious what I’m doing wrong to you guys.

4. How can the community help?
Looking for a bit of guidance on next steps.

[EyImWawkinHeah]

I think I found something that works.

rule "Fix Timestamp"
when
   has_field("timestamp")
then
    let result = regex("([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})",to_string($message.timestamp));
    let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS", "America/Toronto");
    set_field("timestamp", new_date);
end

I swear I had already tried this but guess not…

Gonna let it soak over night.

[gsmith]

hey @EyImWawkinHeah

If that doesn’t work, try setting the field “timestamp” to “timestamp1”.

[EyImWawkinHeah]

That’s essentially how I found out my stuff wasn’t working, thanks for the response!