Change the timestamp format from yyyy-MM-dd HH:mm:ss.SSS Z to yyyy/MM/dd HH:mm:ss.SSS Z with pipeline rule

i want to change the timestamp format from yyyy-MM-dd HH:mm:ss.SSS Z to yyyy/MM/dd HH:mm:ss.SSS Z with pipeline rule but when i create a rule below it’s seem not work.
Graylog version 4.1
Guest OS: Ubuntu 20.
Time configuration
here my pipeline rule:

here my example message:

Thank you.

When modifying the timestamp I like to assign to a temporary field first while testing. Then you don’t get caught out by messages not showing up in the search because e.g. they end up being in a different timezone.

Also check the processing failure stream for error messages.

thanks for reply, i got this error message

Hey @hungv35

I have this in my tool kit not sure it it will work for ya.

rule "replace timestamp"
    let new_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS","CST"); ///Centeral time Zone
    set_field("timestamp1", new_date);

hmmm, i tried as you said, but it still get the same error message. is there any possibility to change the timestamp format as i want

timestamp field is default by elasticsearch index template.
Try creating a new field called something like “new_timestamp” see if that works.

yes, i tried, but message error is still, and the new field not show.

can you show the updated version of your pipeline?
EDIT Im going to test it out in my lab.

yeah sure, but i think the new format timestamp is incorrect, my idea is when i have a message with timestamp like 2023-01-13 11:10:48.000 +07:00 and the pipeline will change it to (dd/MM/yyyy hh:mm:ss.S timezone. example: 13/01/2023 11:10:48 +07:00)

1 Like

After testing a few of these pipeline I cant seam to get your format.

Correct me is im wrong but you have this

yyyy-MM-dd HH:mm:ss.SSS Z

and you want this

yyyy/MM/dd HH:mm:ss.SSS Z

I tries a few of my pipes , and it still didnt work.
Example of one I know that works but ended up with same error.

rule "add ingestion timestamp"
 let ingestion_time = now();
 let timestamp_time = parse_date(to_string($message.timestamp), "yyyy/MM/dd'T'HH:mm:ss.SSSZ");
 set_field("timestamp_text", to_string($message.timestamp,"default_text"));
 set_field("timestamp2b", $message.timestamp);
 set_field("timestamp2", timestamp_time);
 set_field("ingestion_time_millis", ingestion_time.millis); set_field("timestamp_millis",to_date($message.timestamp).millis);

I think Elasticsearch doesnt like these "/" perhaps @tmacgbay would know better.

1 Like

yeah, thank you for your support.
i just curl -X GET ‘http://localhost:9200/graylog_deflector/_mapping?pretty
and got this one, is there any way to change format timestamp on elasticsearch


I really dont know. I would take a guess probably , first I would see if elasticearch would be able to to that. Perhaps create a new index template for a new index set. This way is something goes horribly wrong your not going to mess up all your indices.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.