I am trying to change the timestamp of a log with the following pipeline rule:
rule "change_timezone"
when
true
then
let original_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
let new_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "en_US", "America/Bogota");
set_field("original_date", original_date);
set_field("timestamp", new_date);
end
When I try this rule in the Simulator it works fine.
However this rule does not work on logs in the connected Stream.
The pipeline rule says there are many errors (one log = one error).
So I don’t know what’s wrong and I don’t know how to get errors in order to investigate.
I manage to debug the rule: I set log level to debug (subsystem graylog) and I configure my stream to output to LoggingOutput. So I saw the error in the field gl2_processing_error:
For rule ‘change_timezone’: In call to function ‘parse_date’ at 5:22 an exception was thrown: Invalid format: “2018-09-19T11:00:00.000+02:00” is malformed at “+02:00”
So the field prints “2018-09-19T11:00:00.000Z” but when you use the function to_string() it prints “2018-09-19T11:00:00.000+02:00”. I think it’s because I set root_timezone to GMT+2 in server.conf.
So it seems Simulator doesn’t care about timezone because it worked fine. I think it’s a bug because we should have the same result in Simulator and with real logs.
I tried the date pattern “yyyy-MM-dd’T’HH:mm:ss.SSSZZ” but I did not manage to get the result I want.
Here is the solution:
rule "change_timezone"
when
has_field("timestamp")
then
let result = regex("^(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3})", to_string($message.timestamp));
let original_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS");
let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS", "en_US", "Asia/Qatar");
set_field("original_date", original_date);
set_field("timestamp", new_date);
end
