Epoch time to readable datetime stamp

Graylog Central (peer support)
1

Hi,

I’m trying to convert an epoch time (just seconds since epoch, not milliseconds) to a readable timestamp. I’ve searched and I should be able to do this using something like this;

%{NUMBER:,metrictimestamp;date;yyyy-MM-dd HH:mm}%

But when I “try” this in graylog I get this status message in return;

metrictimestamp
1488545635
metrictimestamp_grokfailure
java.text.ParseException: Unparseable date: “1488545635”

Any idea what I’m doing wrong?

2

Hey @NeefRoel,

your problem is a known bug:

Neither the SimpleDateFormat Converter nor the Flexible Date Converter available for Extractors are able to use epoch-timestamps. I guess you will have to wait until this bug is fixed or you use logstash to parse the epoch-timestamp.
See here and here for more information :slight_smile:

Greetings - Phil

3

Folks,

I am not at the state to fix the graylog bugs yet. :smiley:

But as work around, I wrote custom graylog plugin function for Graylog Processing Piepline to parse the Epoch time to date then store it in desired field.

You can start here https://www.graylog.org/blog/71-writing-your-own-graylog-processing-pipeline-functions.

Hope it help,

ye

4

Hey @yett,

would you please share your plugin with the Graylog Community? This would be highly appreciated I guess. The marketplace is a good location for that.

Of course you don’t have to do that if you don’t want to or your plugin contains sensitive data. :slight_smile:

Greetings - Phil

5

@derPhlipsi,

Sounds like a plan. I will explore more to ensure there is not duplicate and share the plugin.

Cheers,

Ye

2 Likes
6

I’d like that… I’m not a software developer and it looks like writing the plug-in looks more like writing a bit of a java source code…

7

I have a log sending epoch timestamps and initially thought that flex_parse_date() got me really close:

2017-11-16_16:36:33.60367 INFO  [Function] PIPELINE DEBUG: original: 1510850193
2017-11-16_16:36:34.53309 INFO  [Function] PIPELINE DEBUG: rewrit: 2017-11-16T15:10:00.000Z

But by playing with this online demo interface linked from Graylog Pipeline Functions docs:
http://natty.joestelmach.com/try.jsp
I see by the breakdown that it’s misinterpreting the hh:mm:ss and supplying today as the part it thinks is missing from the date string. :-/ …

Any other ideas for parsing epoch timestamps?

8

Yes, but it’s a bit of a hack. :wink:

The idea is to convert the number of seconds (here in the field “unix_timestamp”) to a time period with seconds() and then add it to the UNIX epoch:

rule "UNIX timestamp"
when
  has_field("unix_timestamp")
then
  // UNIX epoch
  let epoch = parse_date("1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:mm:ss.SSSZ");
  let ts_seconds = seconds(to_long($message.unix_timestamp));
  set_field("timestamp", epoch + ts_seconds);
end

Of course having a function converting directly from UNIX time to a proper timestamp would be preferable, but that doesn’t exist out of the box in Graylog.

This being said, feel free to create a feature request at Issues · Graylog2/graylog-plugin-pipeline-processor · GitHub

3 Likes
9

This is excellent. Makes good sense and is fun to implement! :wink:

I actually ended up augmenting my application output to additionally include an ISO date format, but I will remember your workaround when I run into this again where I don’t have such flexibility – thanks!