Epoch time fields in pipeline rule

[mmurdock (Mark M.)]

I’m parsing some Cisco UCM call detail records into Graylog using a pipeline rule:

rule "parse ucm cdr log"
when
  has_field("ucm_type") && to_string($message.ucm_type) == "cdr"
then
  let message_field = to_string($message.message);
  let parsed_fields =  grok("..<omitted for brevity>..", message_field);
  set_fields(parsed_fields);
  let epoch = parse_date("1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:mm:ss.SSSZ", "UTC");
  let ts_seconds = seconds(to_long($message.ucm_dateTimeOrigination));
  set_field("ucm_dateTimeOrigination", epoch + ts_seconds);
  let ts_seconds = seconds(to_long($message.ucm_dateTimeConnect));
  set_field("ucm_dateTimeConnect", epoch + ts_seconds);
  let ts_seconds = seconds(to_long($message.ucm_dateTimeDisconnect));
  set_field("ucm_dateTimeDisconnect", epoch + ts_seconds);
end

3 fields are in epoch time format, so I’m using the method above of adding those seconds to a 1970 date and then setting the fields with the date values. If I view the fields in the Graylog Message pane, they appear to be UTC dates (which is fine). However, if I run Quick Values against any of those fields, they show up in the original seconds/epoch format.

Am I doing something wrong in my pipeline rule?

Thanks,
Mark

[derPhlipsi (Philipp Ruland)]

Heyo @mmurdock,

I don’t think that you’re doing anything wrong.
Let Graylog do a statistical chart against the timestamp field of the message itself. It’ll convert the timestamps to epoch seconds. I think this has to do with some internal conversion or fallbacks of the displaying libraries.

But I have some pointers for you:

How to transform a timestamp in ms to datetime format?

Graylog 3.0.0 will come with a pipeline function for that: parse_unix_milliseconds(). In Graylog 2.x, there’s unfortunately no simple way to achieve this, since parse_date() doesn’t support creating a date from a UNIX epoch timestamp. You can try the workaround described in Epoch time to readable datetime stamp.

Epoch time to readable datetime stamp

Yes, but it’s a bit of a hack. The idea is to convert the number of seconds (here in the field “unix_timestamp”) to a time period with seconds() and then add it to the UNIX epoch: rule "UNIX timestamp" when has_field("unix_timestamp") then // UNIX epoch let epoch = parse_date("1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"); let ts_seconds = seconds(to_long($message.unix_timestamp)); set_field("timestamp", epoch + ts_seconds); end Of course having a function convert…

But I guess you already saw this topic since you are using the seconds() function

Greetings,
Philipp

[mmurdock (Mark M.)]

My hope was that the date/time fields would behave like the default timestamp field does in Quick Values:

But as you can see with this ucm_dateTimeStamp (which was set using code similar to what I posted above), it only appears as an integer:

Yet when I view that field in an event, it appears as a timestamp:

I’m not sure why there is that inconsistency?

Thanks,
Mark

[derPhlipsi (Philipp Ruland)]

Well, it’s the other way round for the normal timestamp field

Why that is, I don’t know. You could open up a Graylog Github Issue to get the devs to have a look at it

Greetings,
Philipp

[jan (Jan Doberstein)]

you did not wrote what Version of Graylog you are using …

But 2.4.2 had fixed exactly that: https://github.com/Graylog2/graylog2-server/issues/4509

[mmurdock (Mark M.)]

Jan,

Yes sorry, what an obvious thing for me to leave out. I am running version v2.4.6+ceaa7e4.

Thanks,
Mark

[jan (Jan Doberstein)]

would you please open a new issue, refer to #4509 and this thread including all information that a developer could reproduce the issue.

thank you