Syslog Timestamps

[niknjc (Nick)]

Hello everyone. New to the Graylog community. I am running into issues with time settings. I see many of these issues posed frequently but can’t seem anything specifically related to my issue. I’ll do my best to explain what is going on.

I have a single syslog input. This input is accepting syslogs from our switches. Most of these switches are of the same vendor. I’m seeing that one specific version of the firmware installed on these switches removes an extra space at the beginning of the message and its enough to trigger Graylog to ignore/remove the timestamp on syslog inputs. This causes Garylog to change the timestamp to the local server so I need to do a relative search for the logs. Not ideal for building dashboards and reports. I can verify the timestamp is included in the message if I just switch the input to raw text.
Here is an example of a working message
raw input message:
<46> Nov 25 03:18:32 message
Looks like this:

You’ll see the timestamp is still stored in the message and it converts the “Timestamp” to local time of the server.

Now, if I remove the space between <46>Nov… the time is stripped from the message and Graylog changes the time to the local server rather than preserving the timestamp. Here is an example:
raw input message:
<46>Nov 30 03:18:32 message
Looks like this: (unable to attach a second picture, but the “message” field does not include timestamp)
message
message
You’ll see here the timestamp is removed from the message. Graylog does not convert the Timestamp back to local time.

Another thing to note. The raw input messages is not what I want to use for my inputs. I like the DNS resolution that comes with the syslog input. So my thoughts are to create some pipeline processing rule that adds a space to the raw message after the <46> if it’s not there. However I think the syslog input is already processing the messages before it would hit a pipeline rule. Thoughts?

[commandline-be (JL)]

taking a guess here, what you may need to do is perform extraction on the said field and adapt a grok pattern to ignore the match on the space

[niknjc (Nick)]

Interesting. I suspected that I would need to setup some sort of extraction for this. The only issue here is that the syslog input does not store the message like “<46> Nov 25 03:18:32 message” rather “Nov 25 03:18:32 message”. My thought is that the syslog input is already doing some sort of extraction before it’s stored. So basically the extra “<46>” (with and without the space) is not stored in the message.

[niknjc (Nick)]

So a small update to this. I enabled “Store full message” on the input. Now I can see the “full_message” field for the logs. Now I need to figure out how to get the space added after the <46> in the “full_message” and save that as the new “message”. Doing this I’m hopeful Graylog will store the correct timestamps. Problem for me is I’ll need to figure that out. Back to Google I go. I’ll post an update as I move along. If anyone stumbles across this and has any grok patterns or rules already built, feel free to help out.