Replacing default Graylog timestamp via Pipeline Processor

Hey there,

I have been trying to overwrite the default Graylog timestamps for our Apache 2.2 log messages with the actual timestamp of the respective message.

I used the solution from this post as a start: Searching imported logs by log timestamp, not time Graylog received the log

My own rule now looks like follows:

rule “replace timestamp”
when
true
then
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”);
set_field(“timestamp”, new_date);
end

The field http_time is extracted via the JSON extractor and looks like this:

2019-03-08T11:10:38

So it should match the given format exactly. Yet when I use the Pipeline rule like that, the connected streams just drop every new message while I don’t get any error messages in the Graylog log.

I already checked that the Extractor is run before the Pipeline, so the field http_time should be populated. Also when I alter the parsing format in the pipeline rule so that it doesn’t match I get the correct processing error messages in the log and the messages don’t get dropped.

I’m running Graylog version 2.4.5.
I’m not sure if I just forgot something somewhere along the line, but I’m really out of ideas right now.

Thanks already in advance!

Did you checked if you have errors in the Elasticsearch File?

When you write the date in another field, how does that look?

Unfortunately there are also no entries in the Elasticsearch Log.

I also just now tried to write the datetime into another field called “test”. Still the same result.

I also built in a Pipeline step with debug($message.http_time). That gives me entries in the Graylog log:

INFO [Function] PIPELINE DEBUG: Passed value is NULL.

Which would indicate that the field is empty, if I’m not mistaken. That shouldn’t be the case because the JSON extractor runs before the Pipeline Rule. Also when I intentionally mismatch the patterns it still gives me the right field value in the error message. So that’s another thing that’s not quite clear to me.

what is your processing order in System > Configuration?

To get a bit more info I altered my pipeline rule a bit for now:

rule “replace timestamp”
when
true
then
debug($message.http_time);
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”);
debug(new_date);
set_field(“timestamp”, new_date);
end

That gives me following output in the log:

2019-03-08T13:36:28.577+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:26
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:26.000Z
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:27
2019-03-08T13:36:28.578+01:00 INFO [Function] PIPELINE DEBUG: 2019-03-08T13:36:27.000Z

I’m not sure why but now debug($message.http_time) actually has a value unlike before. That value gets parsed to the new format and seemingly set for the timestamp field. Only the messages don’t show up in the web interface.

Could it be that some erroneous time zone parsing sends the messages to the wrong time of day so my user can’t see it in the specified search time?

The messages are now coming in with the timestamp parsed almost exactly, just always 1 hour in the future. So it really is a time zone issue.

I’ve tried this now:

let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”,“EU”,“Germany/Berlin”);

And in the logs you can see that the date is parsed correctly just like in my last post. Only the timestamp in the end is one hour apart. The server.conf is also set to CET. I think what is happening is that Graylog assumes the parsed timestamp from the message is in UTC, so it automatically adds another hour onto it, because the server is set in CET.

Unfortunately my version posted above doesn’t work apparently. Any other ideas would be greatly appreciated!

Thanks!

And in the logs you can see that the date is parsed correctly just like in my last post. Only the timestamp in the end is one hour apart. The server.conf is also set to CET. I think what is happening is that Graylog assumes the parsed timestamp from the message is in UTC, so it automatically adds another hour onto it, because the server is set in CET.

if no timezone is given, Graylog will always assume that this is UTC. The Timezone you can set in the Graylog server.conf is for the hardcoded user and not for the server. So that all is intentional. Please see the parse_date function how to use this. Or search this community as this is very common setting that needs to be fixed.

If I understood it right, the documentation says that if there is no time zone given in the actual timestamp, you can give one to the new parsed variable by appending it like this:

parse_date(value: string, pattern: string, [timezone: string])

I tried that in multiple variations and with multiple different time zone codes but it just doesn’t change anything. This was my most recent variation:

let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, “CET”);

Which gives me following output:

So it seems, the parsed message is still seen as UTC and then there is an hour added for my admin user that is set to CET.

Did I mess up the syntax or is there something else I have to consider?

Thank you very much for helping me here! Timezone stuff is often a bit confusing to me.

The following should do the trick

rule “replace timestamp”
when
    true
then
    let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, "CET");
    set_field(“timestamp”, new_date);
end

You might want to add some debug information until it works like you expect. Something like you find in Can't overwrite timestamp through pipeline-rule

I think I’ve got it working now. The variation you posted was the same I had running already, just with different quotation marks. That gave me some syntax errors and while checking that out I noticed that there’s always the Locale expected first after the parsing pattern and then the time zone afterwards.

So my rule now looks like this:

rule “replace timestamp”
when
true
then
debug($message.http_time);
let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, “DE”, “CET”);
set_field(“timestamp”, new_date);
debug($message.timestamp);
end

And it seems to work. The debug messages in the log show the parsed timestamp with the correct +0100 time zone.
There are still some mismatched timestamps appearing in the web interface but I guess those are the ones that were already processed an hour ago and only just now slip in with the others.

Thank you for the help! Let’s hope that was all for now to get it work.

sorry just copy&pasted from some slides I had and not checked the " are correct or not. but glad that you found the issue.

No problem. That was what pointted me to the error in the end :wink:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.