Neither the SimpleDateFormat Converter nor the Flexible Date Converter available for Extractors are able to use epoch-timestamps. I guess you will have to wait until this bug is fixed or you use logstash to parse the epoch-timestamp.
See here and here for more information
I have a log sending epoch timestamps and initially thought that flex_parse_date() got me really close:
2017-11-16_16:36:33.60367 INFO [Function] PIPELINE DEBUG: original: 1510850193
2017-11-16_16:36:34.53309 INFO [Function] PIPELINE DEBUG: rewrit: 2017-11-16T15:10:00.000Z
But by playing with this online demo interface linked from Graylog Pipeline Functions docs: http://natty.joestelmach.com/try.jsp
I see by the breakdown that it’s misinterpreting the hh:mm:ss and supplying today as the part it thinks is missing from the date string. :-/ …
The idea is to convert the number of seconds (here in the field “unix_timestamp”) to a time period with seconds() and then add it to the UNIX epoch:
rule "UNIX timestamp"
// UNIX epoch
let epoch = parse_date("1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:mm:ss.SSSZ");
let ts_seconds = seconds(to_long($message.unix_timestamp));
set_field("timestamp", epoch + ts_seconds);
Of course having a function converting directly from UNIX time to a proper timestamp would be preferable, but that doesn’t exist out of the box in Graylog.
This is excellent. Makes good sense and is fun to implement!
I actually ended up augmenting my application output to additionally include an ISO date format, but I will remember your workaround when I run into this again where I don’t have such flexibility – thanks!