Delayed Syslog From Cisco Switch

[JonB9393 (JonB)]

Hello, I’ve done about 16 hours of research on this and tried everything under the sun. I have some Cisco 2960 switches that are set to send their syslogs to graylog. The syslog messages come in just fine, however they come in an hour later. Or they come in on time, but show up in my searches exactly one hour later. Basically Graylog is adding +1 hour to the entry. But why?

My graylog server overview has the same exact time for user admin, web browser, and graylog server

Here is some of my cisco sh run config:

service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone
!
no aaa new-model
clock timezone EST -5
clock summer-time EST recurring
!
logging trap debugging
logging facility syslog
logging host 10.10.0.140 transport udp port 1515



    Trap logging: level debugging, 1482 message lines logged
        Logging to 10.10.0.140  (udp port 1515,  audit disabled,
              authentication disabled, encryption disabled, link up),
              35 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled

Log Buffer (4096 bytes):

OS: Centos Stream 9

Name : graylog-server
Version : 5.1.4
Release : 1
Architecture : x86_64
Size : 411 M
Source : graylog-server-5.1.4-1.src.rpm
Repository : @System
From repo : graylog
Summary : Graylog server
URL : https://www.graylog.org/
License : SSPL
Description : Graylog server

Available Packages
Name : graylog-server
Version : 5.1.12
Release : 1
Architecture : x86_64
Size : 296 M
Source : graylog-server-5.1.12-1.src.rpm
Repository : graylog
Summary : Graylog server
URL : https://www.graylog.org/
License : SSPL
Description : Graylog server

I’m def stumped on this one. It would be one thing to be off a few hours, but 1 hour is odd. My Timezone is American/Detroit so -5 and my centos server is this:

[root@icr-graylog5 /]# timedatectl
               Local time: Tue 2024-03-12 16:07:39 EDT
           Universal time: Tue 2024-03-12 20:07:39 UTC
                 RTC time: Tue 2024-03-12 20:07:39
                Time zone: America/Detroit (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
[root@icr-graylog5 /]#

Thanks in advance

Example

[Arie (Arie van den Heuvel)]

Graylog is able to convert the syslog-time to localtime, so your timezone should be set correct in your profile. What we do is put the time in the event to the timestamp in graylog to be sure.

Then graylog presents the syslog-time of the message inide timestamp and in the search the time is presented in your localtime.

In a pipeline it is something like this:

let pf = grok(
pattern: "%{TIMESTAMP_ISO8601:timestamp},
value: to_string($message.message),
only_named_captures: true

good luck

[Joel_Duffield]

How long had this been happening, could it be a time change thing, ie a really old firmware with the old dates of change etc.

Also its generally less headache to have your devices set to UTC, you dont have to, but it can make your life easier with weird things.

[JonB9393 (JonB)]

Thanks for all the responses. I finally figured it out. I had to remove that line on my switches that started with “service timestamps log datetime…”

Once I did that, no more timestamps were in my buffer logs by doing a “sh log” and they injested in graylog right away and on time which caused my alerts to fire.

Crazy, it took me days to figure this out, but I’m glad its over!! For record, I’m running 2960’s and a few 3750’s in my farm.