I am forwarding syslog messages from ta TP-link SX3008F switch. I was sure they where arriving at the GrayLog input … however they did not show up in the streams After spending some hours, it turned out that GrayLog did fill the timestamp field with a wrong value
In fact not only the time field contained wrong values. E.g. source is also wrong. …
Here an message example and some incorrect results:
<134>2024-01-17 19:31:18 SX3008F 65730 Port 3 SFP Module bias current recover from the high warning threshold.
Timestamp: 2024-01-17 01:00:00.000
source: 19:31:18
2. Describe your environment:
OS Information:
Ubuntu LTS (as VM)
Package Version:
very latest GrayLog and OpenSearch versions
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I spend some hours finding the cause of the problem. After some verdict in relation timestamp, I found the messages / the cause by searching from yesterday to tomorrow.
4. How can the community help?
I especially need to fix/override the timestamp. Best solution is of course in the time could be extracted from the message, but just using the (local) message arrival time would all ready do the job
I would also like to know if the incoming message is wrong or graylog is wrong. So that it is clear who should fix the issue
I more or less worked around the problem using two pipeline rules:
rule-1 <<
rule “Correct_SX3008F_TimeStamp”
when
contains(
value: to_string($message.“full_message”),
search: “SX3008F”,
ignore_case: false
)
then
set_field(“timestamp”, now(“Europe/Amsterdam”));
end
rule-2
rule “process_sx3008”
when
has_field(
field : “full_message”
)
AND
contains(
value: to_string($message.“full_message”),
search: " SX3008F",
ignore_case: false
)
then
let gl2_fragment_grok_results = grok(
pattern: “^%{NOTSPACE}>%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:source}%{GREEDYDATA:application_msg}”,
value: to_string($message.“full_message”),
only_named_captures: true
);
set_fields(
fields: gl2_fragment_grok_results
);
end
Note that the first rule is not really correct I have no solution yet for the required summer / winter time correction, which as result that the time is e.g. 12:00 where in reality it is 13:00
And of course (less severe) I use the message arriving time and not the time in the message text.
That this kind of workarounds are necessary … hum, should not be …
I think there is a general recommendation to have your Graylog instance OS and devices that you are logging all set up as UTC, then you can offset in reporting. Not sure if that is helpful to you or not, but timezone differences can be a real pain.
As described I worked around the problem. I used the same method for an other switch type.
I agree that all systems at UTC has advantages, but I like to see all messages in local time on all my equipment. Up to now that is not really a problem. GrayLog inputs allow you to set the source time zone. For a company having equipment in different country’s the situation is of course different.
I have no solution yet for the required summer / winter time correction, which as result that the time is e.g. 12:00 where in reality it is 13:00