Hi,
I have asked this before but the thread this was asked on is now closed. apologies for the long post, I’m trying to understand this behaviour.
Basically I have a sidecar input sending RADUS accounting messages, the logs from it are recorded to Graylog and I have in my sidecar config a processor that treats the message as CSV - decode_csv_fields
I use “extract_arrays” to provide the association with the csv columns and a ‘column name’
All this works perfectly - accept for the parsing of the date.
The message contains a date string in the format of ‘yyyy/MM/dd hh:MM:ss’
This date sting has no Timezone associated to it, but it is BST (British Summer Time)
My Graylog is configured as the same timezone as the source message (configured to the same NTP source). As is my client desktop.
Ive tried using the sidecars to get the timestamp I want. However, this appears to be automatically parsed - for example,
message
Interim-Update,user1@realm,2021/10/11 09:16:36,0,BNG-2,30B87900029966615E4F3B,X.X.X.X,Framed-User,PPP,,,X.X.X.X,255.255.255.255,,,,3795710611,20,3327212325,,30783831,,61022672,PPPoEoQinQ,193687867,lag-11:2231.315,Radius,369577,2axx:xxxx:xx:xxxx::/56
the date extracted from the above is
2021/10/11 09:06:36
however, the value that is appearing in the field is
Event_Timestamp
2021-10-11 10:16:36.000 +01:00
I then tried using a pipeline, here I created the pipeline with a single rule.
rule "Event_Timestamp"
when
has_field("Event_Timestamp")
then
let new_date = parse_date(to_string($message.Event_Timestamp), "yyyy/MM/dd HH:mm:ss", "BST", "Europe/Isle_od_Man") - hours(1);
set_field("Event-Timestamp", new_date);
end
Which does actually work and provide the date I want - However, this is not ideal as the BST is going back to GMT at the end of the month - and this will affect my dates once again.
I then took a look at an extractor -
{
"extractors": [
{
"title": "Event_1",
"extractor_type": "split_and_index",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "Event_1",
"extractor_config": {
"index": 3,
"split_by": ","
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "Event_Timestamp",
"extractor_type": "copy_input",
"converters": [
{
"type": "date",
"config": {
"date_format": "yyyy/MM/dd HH:mm:ss",
"time_zone": "Europe/Isle_of_Man",
"locale": "und"
}
}
],
"order": 0,
"cursor_strategy": "copy",
"source_field": "Event_1",
"target_field": "EventTimestamp",
"extractor_config": {},
"condition_type": "none",
"condition_value": ""
}
],
"version": "4.1.0"
}
As you will see there are two extractors.
I tried to use a single extractor where I perform split on the ‘,’ and use index ‘3’
I get the date in the extractor preview.
I save this, but the extracted date is once again automatically parsed and ends up being 1hour ahead of what I want.
If I try using a single extractor - but this time with a date converter, I get this in my error logs
[type=illegal_argument_exception, reason=failed to parse date field [2021-10-11T10:16:36.000Z] with format [yyyy/MM/dd HH:mm:ss........
I tried to change the format to ‘yyyy-MM-dd’T’HH:mm:ss.SSSZ’ but then the error said
java.lang.IllegalArgumentExpection: Invalid format: "2021/10/11 07:55:24" at "/10/11 07:55:24"
But this time the error appears to be a java error…
By using the two extractors as above is the only wat I have found to get the date I want.
I recall seeing a post about automatic date parsing but I can no longer find it. Is this such a thing?
I would be keen to see if im not alone with this peculiar parsing issue and I would be grateful if anyone can provide some feedback on whether what I am doing is “the best way” or is there something else I can try?
