Pipeline rule not working parse_date

Graylog Central (peer support)
,
1

Hello,

I am trying to change the timestamp of a log with the following pipeline rule:

rule "change_timezone"
when
  true
then
  let original_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
  let new_date = parse_date(to_string($message.timestamp), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "en_US", "America/Bogota");
  set_field("original_date", original_date);
  set_field("timestamp", new_date);
end

When I try this rule in the Simulator it works fine.
However this rule does not work on logs in the connected Stream.
The pipeline rule says there are many errors (one log = one error).

So I don’t know what’s wrong and I don’t know how to get errors in order to investigate.

2

how does the error look like? Without knowing your data and without the error nobody would be able to help you.

3

I manage to debug the rule: I set log level to debug (subsystem graylog) and I configure my stream to output to LoggingOutput. So I saw the error in the field gl2_processing_error:
For rule ‘change_timezone’: In call to function ‘parse_date’ at 5:22 an exception was thrown: Invalid format: “2018-09-19T11:00:00.000+02:00” is malformed at “+02:00”

So the field prints “2018-09-19T11:00:00.000Z” but when you use the function to_string() it prints “2018-09-19T11:00:00.000+02:00”. I think it’s because I set root_timezone to GMT+2 in server.conf.

So it seems Simulator doesn’t care about timezone because it worked fine. I think it’s a bug because we should have the same result in Simulator and with real logs.

I tried the date pattern “yyyy-MM-dd’T’HH:mm:ss.SSSZZ” but I did not manage to get the result I want.

Here is the solution:

rule "change_timezone"
when
  has_field("timestamp")
then
  let result = regex("^(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3})", to_string($message.timestamp));
  let original_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS");
  let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSS", "en_US", "Asia/Qatar");
  set_field("original_date", original_date);
  set_field("timestamp", new_date);
end
1 Like
4

if you mind that you found a bug, please open a bug issue

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.