Epoch time fields in pipeline rule


(Mark M.) #1

I’m parsing some Cisco UCM call detail records into Graylog using a pipeline rule:

rule "parse ucm cdr log"
when
  has_field("ucm_type") && to_string($message.ucm_type) == "cdr"
then
  let message_field = to_string($message.message);
  let parsed_fields =  grok("..<omitted for brevity>..", message_field);
  set_fields(parsed_fields);
  let epoch = parse_date("1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:mm:ss.SSSZ", "UTC");
  let ts_seconds = seconds(to_long($message.ucm_dateTimeOrigination));
  set_field("ucm_dateTimeOrigination", epoch + ts_seconds);
  let ts_seconds = seconds(to_long($message.ucm_dateTimeConnect));
  set_field("ucm_dateTimeConnect", epoch + ts_seconds);
  let ts_seconds = seconds(to_long($message.ucm_dateTimeDisconnect));
  set_field("ucm_dateTimeDisconnect", epoch + ts_seconds);
end

3 fields are in epoch time format, so I’m using the method above of adding those seconds to a 1970 date and then setting the fields with the date values. If I view the fields in the Graylog Message pane, they appear to be UTC dates (which is fine). However, if I run Quick Values against any of those fields, they show up in the original seconds/epoch format.

Am I doing something wrong in my pipeline rule?

Thanks,
Mark


(Philipp Ruland) #2

Heyo @mmurdock,

I don’t think that you’re doing anything wrong.
Let Graylog do a statistical chart against the timestamp field of the message itself. It’ll convert the timestamps to epoch seconds. I think this has to do with some internal conversion or fallbacks of the displaying libraries.

But I have some pointers for you:


But I guess you already saw this topic since you are using the seconds() function :slight_smile:

Greetings,
Philipp


(Mark M.) #3

My hope was that the date/time fields would behave like the default timestamp field does in Quick Values:

But as you can see with this ucm_dateTimeStamp (which was set using code similar to what I posted above), it only appears as an integer:

Yet when I view that field in an event, it appears as a timestamp:

I’m not sure why there is that inconsistency?

Thanks,
Mark


(Philipp Ruland) #4

Well, it’s the other way round for the normal timestamp field :smiley:

Why that is, I don’t know. You could open up a Graylog Github Issue to get the devs to have a look at it :slight_smile:

Greetings,
Philipp


(Jan Doberstein) #5

you did not wrote what Version of Graylog you are using …

But 2.4.2 had fixed exactly that: https://github.com/Graylog2/graylog2-server/issues/4509


(Mark M.) #6

Jan,

Yes sorry, what an obvious thing for me to leave out. I am running version v2.4.6+ceaa7e4.

Thanks,
Mark


(Jan Doberstein) #7

would you please open a new issue, refer to #4509 and this thread including all information that a developer could reproduce the issue.

thank you


(system) closed #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.