Timestamp Pipeline/rewrite from Filebeat

[abraxas]

I’m using filebeat to retrieve logs written to a file every few minutes. The logs come in JSON format and are handled properly. There’s a field created called “CreationTime” representing the time in PST. I can convert that string to a datetime with a pipeline rule like the following:

set_field("NewDateTime",(parse_date(to_string(new_date),"YYYY-MM-dd'T'HH:mm:ss")));

This works as expected, however, when I use a rule like:
set_field("timestamp",(parse_date(to_string(new_date),"YYYY-MM-dd'T'HH:mm:ss")));

It gets converted to UTC (as expected) and then renders in my browser as PST/PDT. The result of this is that even though the event happened at, say, 8AM PST, it shows up as happening at 1AM PST.

I’ve tried adding timezones in to the parse_date function but it doesn’t seem to make any difference. In the image below “DDDDD” represents the CreationTime field set to a date, timestamp represents the attempt to overwrite the time stamp.

For me in the timezone I’m located in, these should match but they do not.

I’ve tried variations on this to no avail…help would be TREMENDOUSLY appreciated as I have spent hours looking for a solution to this. I’ve tried “UTC”,“PST”,"+08:00","+16:00", “+0800” and none of them seem to help…

  let new_date = to_string($message.CreationTime);
  set_field("DDDDD",(parse_date(to_string(new_date),"YYYY-MM-dd'T'HH:mm:ss","en","UTC")));
  set_field("timestamp",(parse_date(to_string(new_date),"YYYY-MM-dd'T'HH:mm:ss","en","UTC")));

[shoothub]

Try to use this snippet:

let new_time = parse_date(value: to_string($message.CreationTime), pattern:“yyyy-MM-dd’T’HH:mm:ss”, timezone:“Europe/Bratislava”);
set_field(“timestamp”, new_time);

Set timezone parameter to your real timezone.

If will not work, try to use function flex_parse_date which try to guest correct format for you:
https://docs.graylog.org/en/3.1/pages/pipelines/functions.html#flex-parse-date

[abraxas]

Thanks @shoothub, I gave it a shot and it worked out!

Appreciate it a TON, I’d been slaving over that for a long time.