rule “replace timestamp”
when
true
then
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”);
set_field(“timestamp”, new_date);
end
The field http_time is extracted via the JSON extractor and looks like this:
2019-03-08T11:10:38
So it should match the given format exactly. Yet when I use the Pipeline rule like that, the connected streams just drop every new message while I don’t get any error messages in the Graylog log.
I already checked that the Extractor is run before the Pipeline, so the field http_time should be populated. Also when I alter the parsing format in the pipeline rule so that it doesn’t match I get the correct processing error messages in the log and the messages don’t get dropped.
I’m running Graylog version 2.4.5.
I’m not sure if I just forgot something somewhere along the line, but I’m really out of ideas right now.
Unfortunately there are also no entries in the Elasticsearch Log.
I also just now tried to write the datetime into another field called “test”. Still the same result.
I also built in a Pipeline step with debug($message.http_time). That gives me entries in the Graylog log:
INFO [Function] PIPELINE DEBUG: Passed value is NULL.
Which would indicate that the field is empty, if I’m not mistaken. That shouldn’t be the case because the JSON extractor runs before the Pipeline Rule. Also when I intentionally mismatch the patterns it still gives me the right field value in the error message. So that’s another thing that’s not quite clear to me.
To get a bit more info I altered my pipeline rule a bit for now:
rule “replace timestamp”
when
true
then
debug($message.http_time);
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”);
debug(new_date);
set_field(“timestamp”, new_date);
end
That gives me following output in the log:
2019-03-08T13:36:28.577+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:26
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:26.000Z
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:27
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:27.000Z
I’m not sure why but now debug($message.http_time) actually has a value unlike before. That value gets parsed to the new format and seemingly set for the timestamp field. Only the messages don’t show up in the web interface.
Could it be that some erroneous time zone parsing sends the messages to the wrong time of day so my user can’t see it in the specified search time?
The messages are now coming in with the timestamp parsed almost exactly, just always 1 hour in the future. So it really is a time zone issue.
I’ve tried this now:
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”,“EU”,“Germany/Berlin”);
And in the logs you can see that the date is parsed correctly just like in my last post. Only the timestamp in the end is one hour apart. The server.conf is also set to CET. I think what is happening is that Graylog assumes the parsed timestamp from the message is in UTC, so it automatically adds another hour onto it, because the server is set in CET.
Unfortunately my version posted above doesn’t work apparently. Any other ideas would be greatly appreciated!
And in the logs you can see that the date is parsed correctly just like in my last post. Only the timestamp in the end is one hour apart. The server.conf is also set to CET. I think what is happening is that Graylog assumes the parsed timestamp from the message is in UTC, so it automatically adds another hour onto it, because the server is set in CET.
if no timezone is given, Graylog will always assume that this is UTC. The Timezone you can set in the Graylog server.conf is for the hardcoded user and not for the server. So that all is intentional. Please see the parse_date function how to use this. Or search this community as this is very common setting that needs to be fixed.
If I understood it right, the documentation says that if there is no time zone given in the actual timestamp, you can give one to the new parsed variable by appending it like this:
I tried that in multiple variations and with multiple different time zone codes but it just doesn’t change anything. This was my most recent variation:
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, “CET”);
rule “replace timestamp”
when
true
then
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, "CET");
set_field(“timestamp”, new_date);
end
I think I’ve got it working now. The variation you posted was the same I had running already, just with different quotation marks. That gave me some syntax errors and while checking that out I noticed that there’s always the Locale expected first after the parsing pattern and then the time zone afterwards.
So my rule now looks like this:
rule “replace timestamp”
when
true
then
debug($message.http_time);
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, “DE”, “CET”);
set_field(“timestamp”, new_date);
debug($message.timestamp);
end
And it seems to work. The debug messages in the log show the parsed timestamp with the correct +0100 time zone.
There are still some mismatched timestamps appearing in the web interface but I guess those are the ones that were already processed an hour ago and only just now slip in with the others.
Thank you for the help! Let’s hope that was all for now to get it work.
