Problem with date extractor converter

[WhiteBear]

Hi all,
I’m trying to parse a date string with format yyyy/MM/dd HH:mm:ss
I think the format is correct because the extractor “try” button report this for the messages

But when I set the converter like yyyy-MM-dd HH:mm:ss

I got this error in log: failed to parse date field [2022-06-09T15:39:24.000Z] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]

If i set the converter like yyyy-MM-dd HH:mm:ss.SSS i got the error ERROR [Extractor] Could not apply converter [DATE] of extractor <StartTime (015fc1e1-e777-11ec-832a-0050569363a3)>:
java.lang.IllegalArgumentException: Invalid format: “2022/06/10 10:48:39” is malformed at “/06/10 10:48:39”

if I explode the message content I found the time field with the error “Invalid Date”, but the field in search is showed up as DateTime.

Any hint?

Thanks

[gsmith]

Hello

Can you show an accurate example of this message, leaving out private info?

@WhiteBear

I got this error in log: failed to parse date field [2022-06-09T15:39:24.000Z] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]

I have seen this issue around here. If my memory is correct it has something to do with mill seconds at the end of Date/Time. It possible to use a GROK pattern or Pipeline to normalize the date time on the Input.

yyyy-MM-dd HH:mm:ss.SSS Z

Found that here.

ERROR [Extractor] Could not apply converter [DATE] of extractor

Greetings. I am seeing errors in my log file even though my extractor is properly converting epoch dates. I’m wondering why the errors are triggering (and how to stop them). Environment: Graylog 3.1.4+1149fe1 Elastic Search 5.6 Error message: 2021-03-23T16:42:09.404-05:00 ERROR [Extractor] Could not apply converter [DATE] of extractor [9f96c3c4-6aee-11ea-ab2c-0efa831ef056]. java.lang.IllegalArgumentException: Invalid format: "1616510609950" is malformed at "9950" at org.joda.time.format.…

And Here

Issues with date type converter format string

Hello guys! Sorry i was very busy the last days and had no time to test out stuff. But i finally got it working again using a mix of extractor and pipeline. Heres how i did it: Create an extractor to copy the timestamp from the message into a second timestamp field. [grafik] Create a pipeline on the stream that gets the messages you want the timestamps to get changed. [grafik] Edit: You can also use has_field(“msg_timestamp”) instead of true if you want to. Change the message proces…

And here

How to transform a timestamp in ms to datetime format?

How to transform a timestamp in ms to datetime format? for example, i want to tranform a timestamp like 1514736000000 to a datetime format 2018-01-01 00:00:00.000. also can we directly compare the datetime with > < == or != ?

[WhiteBear]

Hi,
thanks, I’ve already read those links but with no luck.

I’m really new to graylog so, maybe, there’s something I can’t understand.
This is an example message

field1,field2,field3,field4,field5,field6,field7,field8,field9,field10,field11,field12,field13,field14,field15,field16,field17,field18,field19,field20,field21,field22,field23,field24,field25,field26,field27,field28,field29,field30,field31,field32,field33,field34,field35,2022/06/13 16:16:48,field37,field38,field39,field40,field41,field42,field43,field44,field45,field46,field47,field48,field49,field50,field51,field52,field53,field54,field55,field56,field57,field58,field59,field60,field61,field62,field63,field64,field65,field66,field67,field68,field69,field70,field71,field72,field73,field74,field75

In position #36 we find the date in format yyyy/MM/dd HH:mm:ss and the extractor using split by comma is working

If I open a message in “preview” of all messages the date is reported as invalid as already said

But if I open the message clicking on it (the linked url is like https://myserver:9000/messages/graylog_5/762ebd00-eb23-11ec-8f8f-0050569363a3) I get the right value

Even if I create a new extractor of type “copy input”

I got the same problem, the field in elasticsearch is identified by date, why?

Edit:
I’ve tried, as suggested, to copy the original field to another field and do some pipeline conversion like

rule "parse date"
when
has_field("StartTime")
then
let new_date = parse_date(to_string($message.StartTime2), "yyyy-MM-dd'T'HH:mm:ss.SSSSZ");
/*let new_date = parse_date(to_string($message.StartTime2), "yyyy/MM/dd HH:mm:ss");*/
set_field("StartTime3", new_date);
end

but the log has again the errors

Error: In call to function ‘parse_date’ at 5:15 an exception was thrown: Invalid format: “2022/06/13 17:33:09” is malformed at “/06/13 17:33:09”)
or
failed to parse date field [2022-06-13T17:36:46.000Z] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]

Edit2:

If I create an extractor copying a string field to another field, for example StarTime4 and than I set the value with the pipeline code

rule "parse date"
when
has_field("StartTime")
then
let new_field = to_string($message.StartTime);
set_field("StartTime4", new_field);
end

the value is visible, but I cannot search as it was a datetime because it’s a string…

[gsmith]

Hello,

Looking at pipelines on how to convert Date/time and since were deal with epoch time have you seen these links? You can just use a pipeline to convert the date/time and remove the extractor I believe or unless you have to use an extractor.

1.Fortigate changed eventtime from seconds to nanoseconds - #2 by shoothub

Or perhaps this?

2.Graylog Knowledge Base - Pipeline Rule Samples

Is it possible to give a more accurate example of the raw message received and insure personal info is either replaced or removed?

Here is an example of mine.

rule "Epoch Convert"
when
  has_field("eventtime")
then
  let ts_millis = to_long($message.eventtime) / 1000;
  let new_date = parse_unix_milliseconds(ts_millis);
  set_field("epoch_timestamp", new_date);
  //set_field("timestamp", new_date);
end

[WhiteBear]

The right hint was remove the extractor and only use the pipeline.
Using this code

rule "parse date"
when
has_field("StartTime")
then
let new_field = to_string($message.StartTime);
let new_field2 = parse_date(
                value: new_field,
                pattern: "yyyy/MM/dd HH:mm:ss",
                timezone: "Europe/Rome");
set_field("StartTimeEC", new_field2);
end

it works!

Just one little thing, when I click on “StartTimeEC” value like this

and I select “add to query”, the value is in UTC time

so I need to do all the query using time -2 hours compared to Europe/Rome timezone. That’s not a big deal, but any hint on this?

Thanks