1. Describe your incident:
I’m trying to figure out how to have logs from my windows server come in formatted properly. The logs are in Common Log Format and GrayLog is collecting every line , including the source, date/time, etc as the message and the date/time mismatch.
3. What steps have you already taken to try and solve the problem?
I’ve searched around the graylog database but have not found a clear definitive guide.
4. How can the community help?
I’m looking for either some direction towards a guide that can help me set up my filebeat so that the time/date is being properly captured as well as excluded from the message field.
Yes, I’m using a beats input with the filebeat config I referenced.
Graylog is getting the log file but it’s not processing the timestamp correctly, I’d like to configure the filebeat so that the collector is using the timestamp within the log file message.
I.E. when I set the collector up today I had a 24k entries dating back to January all coming in with the timestamp of today @ noon
Timestamps of newly created logs are also offset by a few seconds.
Oh I see, So FileBeat went re-read the whole log file again.
When you stated it incorrect, off by 5 minutes, 3 hours, etc…?
Normally when timestamp is incorrect ,let’s say 1-2 hours. Check your Graylog server date/time, the remote device send messages/logs, the user logged in has the correct time zone.
It is preferred that Graylog has some type of time sync (i.e. NTP).
Filebeat has a registry file to track the events it has already sent to outputs.
Are you sure Filebeat has access to it? Or does it get deleted between container runs?
Could you please share the debug logs of filebeat?
EDIT: There was anothe community member had docker recently, They had to set there Docker container to the Server Date/time
I’ll need to take a look at that as it is a docker container, but yeah there are two major issues
any events processed are timestamped at the time that filebeat read the file, not the timestamp on the log entry itself.
I.e. when I apply a new filebeat to a system, it’s grabbing every single log entry (even those very old and time-stamping the entry on graylog as the time the collection ran)
This is usually off by a second or two from when the line was written and timestamp in the captured message.
The log file Itself is defaulting to UTC and off by a few hours for that. I can’t recall if I mounted the timezone/clock data through to the container so I’ll need to check that.