I have all timezones for all servers unified as “Europe/London”.
System/Overview/Time configuration shows correct time:
User admin :2020-08-25 17:22:34 +01:00
Your web browser:2020-08-25 17:22:34 +01:00
Graylog server:2020-08-25 17:22:34 +01:00
Additionally, the messages I am sending into Graylog via Filebeat are also ‘Europe/London’ timezone.
The problem is that Graylog is only showing default UTC timezone for message entries and any time queries, which is an hour behind.
So if the time now is 17:22 “Europe/London”, then Graylog will display and stream messages from 1 hour ago at 16:22. Graylog acts as if these messages are the latest messages from now (current time 17:22), but is actually showing messages from 1 hour ago.
I hope this makes sense, would very much appreciate some feedback on how to resolve this.
Here is a screenshot of what I mean. Timestamp field of Graylog and timestamp of my message are in-sync, but they are from 1 hour ago, even though search time is set to the “last 5 minutes”.
To be clear, I have no extractors running that is manipulating the data from the input. All buffers are 0 so there is no delay in processing data.
Logs are sent into Graylog without any timezone specified, e.g. “2020-08-27T14:28:53,229”, but the timestamp is logged as the time on the server itself which is “Europe/London”. From what I have read here if no timezone is specified in the message timestamp then Graylog assumes the time is UTC.
So the problem is that in the “All Messages” stream (or any stream) the timestamp in Graylog client shows current time(timezone), but the message shown will be from 1 hour ago.
For example if I stop the input so that no new messages are processed, Graylog will continue to display incoming messages that are from 1 hour ago as if they are new messages. Messages will continue to stream in to the client (“All Messages” stream) for 1 hour into the future until and then will stop when it catches up to the time.
Does this sound like a bug with Graylog, or am I missing something?
Would very much appreciate feedback to help resolve this as currently Graylog is unusable in this state.
Europe/London is different from UTC, due to Summertime. You need to convert the timestamp to UTC, if the incoming messages are in Europe/London timezone.
I have set up a pipeline attached to a stream which has two stages.
Stage 0 - Extract fields from the message using grok pattern. This extracts timestamp from the message.
Stage 1 - Replace default Graylog timestamp with timestamp field extracted from message using this function:
rule "replace_timestamp"
when
true
then
let new_timestamp = parse_date(
value: to_string($message.atstimestamp),
pattern: "yyyy-MM-dd'T'HH:mm:ss,SSS",
timezone: "Europe/London");
set_field("timestamp", new_timestamp);
end
This function works because I can see the default Graylog timestamp now matches the timestamp of the message, however this doesn’t fix the issue because the messages that are processed and displayed by Graylog is still from 1 hour ago.
Additionally, I’m not sure why the “@timestamp” field is 2 hours behind or where that field comes from. I assume it’s some Graylog default, but why is it different to the other default timestamp?
I did try to set the timezone in this pipeline function to “UTC” timezone, however this did not fix the issue. I’m not sure I understand why it would fix the issue either?
It takes an hour for a message to appear in a search?
The timestamp fields show incorrect times?
If the timestamp field is interpreted in the wrong way, the message will appear in the wrong time in the search results. But in that case, it does not take an hour for it to appear there, but it is added to the past.
Thanks all for your suggestions, this issue is now resolved as Graylog is now showing correct timestamp which matches the timestamp of the messages. I believe the “replace_timestamp” pipeline function is what fixed the issue in the end, however there was some delay in seeing the results so I could not see instantly that it was working.
