Change the timestamp format from yyyy-MM-dd HH:mm:ss.SSS Z to yyyy/MM/dd HH:mm:ss.SSS Z with pipeline rule

[hungv35 (hung)]

i want to change the timestamp format from yyyy-MM-dd HH:mm:ss.SSS Z to yyyy/MM/dd HH:mm:ss.SSS Z with pipeline rule but when i create a rule below it’s seem not work.
Graylog version 4.1
Guest OS: Ubuntu 20.
Time configuration

here my pipeline rule:

here my example message:

Thank you.

[patrickmann]

When modifying the timestamp I like to assign to a temporary field first while testing. Then you don’t get caught out by messages not showing up in the search because e.g. they end up being in a different timezone.

Also check the processing failure stream for error messages.

[hungv35 (hung)]

thanks for reply, i got this error message

[gsmith]

Hey @hungv35

I have this in my tool kit not sure it it will work for ya.

rule "replace timestamp"
when
    has_field("timestamp")
then    
    let new_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS","CST"); ///Centeral time Zone
    set_field("timestamp1", new_date);
end

[hungv35 (hung)]

hmmm, i tried as you said, but it still get the same error message. is there any possibility to change the timestamp format as i want

[gsmith]

hey,
timestamp field is default by elasticsearch index template.
Try creating a new field called something like “new_timestamp” see if that works.

[hungv35 (hung)]

yes, i tried, but message error is still, and the new field not show.

[gsmith]

can you show the updated version of your pipeline?
EDIT Im going to test it out in my lab.

[hungv35 (hung)]

yeah sure, but i think the new format timestamp is incorrect, my idea is when i have a message with timestamp like 2023-01-13 11:10:48.000 +07:00 and the pipeline will change it to (dd/MM/yyyy hh:mm:ss.S timezone. example: 13/01/2023 11:10:48 +07:00)

[gsmith]

Hey,
After testing a few of these pipeline I cant seam to get your format.

Correct me is im wrong but you have this

yyyy-MM-dd HH:mm:ss.SSS Z

and you want this

yyyy/MM/dd HH:mm:ss.SSS Z

I tries a few of my pipes , and it still didnt work.
Example of one I know that works but ended up with same error.

rule "add ingestion timestamp"
when
 true
then
 let ingestion_time = now();
 let timestamp_time = parse_date(to_string($message.timestamp), "yyyy/MM/dd'T'HH:mm:ss.SSSZ");
 set_field("timestamp_text", to_string($message.timestamp,"default_text"));
 set_field("timestamp2b", $message.timestamp);
 set_field("timestamp2", timestamp_time);
 set_field("ingestion_time",ingestion_time);
 set_field("ingestion_time_millis", ingestion_time.millis); set_field("timestamp_millis",to_date($message.timestamp).millis);
end

I think Elasticsearch doesnt like these "/" perhaps @tmacgbay would know better.

[hungv35 (hung)]

yeah, thank you for your support.
i just curl -X GET ‘http://localhost:9200/graylog_deflector/_mapping?pretty’
and got this one, is there any way to change format timestamp on elasticsearch

[gsmith]

Hey,

I really dont know. I would take a guess probably , first I would see if elasticearch would be able to to that. Perhaps create a new index template for a new index set. This way is something goes horribly wrong your not going to mess up all your indices.