No Messages in Syslog UDP/5141 Input

[phosgene]

Hello,

I am very new to Graylog, and I’m having trouble with the Syslog UDP input I just configured on my server. The Syslog packets arrive at the server, but they do not get processed by the Syslog UDP input. Do I need to configure anything more than just the input to start seeing traffic on it? Sorry if this has been asked a million times.

This is the config for the Syslog UDP input:

allow_override_date:
 true
bind_address:
 10.10.110.43
expand_structured_data:
 false
force_rdns:
 false
number_worker_threads:
 2
override_source:
 <empty>
port:
 5141
recv_buffer_size:
 262144
store_full_message:
 false

This is me sending two syslog messages from my router:

router#sho run | sec logging host
logging host 10.10.110.43 transport udp port 5141
router#send log 3 hello world
router#send log 3 hello world
router#sho log | inc SYS-3-
Dec 22 17:53:02.812: %SYS-3-USERLOG_ERR: Message from tty133(user id: user): hello world
Dec 22 17:53:03.864: %SYS-3-USERLOG_ERR: Message from tty133(user id: user): hello world

These are the same two syslog messages arriving on the Debian 10 server that is the Graylog host:

user@graylog$ sudo tcpdump -vv -n -i enp1s0 port 5141
tcpdump: listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:53:03.812232 IP (tos 0x0, ttl 255, id 10, offset 0, flags [none], proto UDP (17), length 126)
    10.10.110.1.57197 > 10.10.110.43.5141: [udp sum ok] UDP, length 98
17:53:04.864210 IP (tos 0x0, ttl 255, id 11, offset 0, flags [none], proto UDP (17), length 126)
    10.10.110.1.57197 > 10.10.110.43.5141: [udp sum ok] UDP, length 98
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

This is the firewall config on said server:

user@graylog$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
9000                       ALLOW IN    Anywhere
5140                       ALLOW IN    Anywhere
5141                       ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
9000 (v6)                  ALLOW IN    Anywhere (v6)
5140 (v6)                  ALLOW IN    Anywhere (v6)
5141 (v6)                  ALLOW IN    Anywhere (v6)

It seems like I’ve made a config error on the Graylog server, but I’m not sure where to begin with fixing it. Any help would be greatly appreciated.

[phosgene]

Hello,

So I found out what the actual problem is, but I still don’t know how to fix it…

It seems like Graylog is 24 hours out of sync with when the logs are actually received, despite the system times being exactly the same between the router sending the syslogs and the Graylog server itself.

Does anyone know how to fix this? Below are some example screenshots.

[phosgene]

I also just configured the year to be present in the timestamps for the syslog messages being sent from my router, but still no change…

[phosgene]

Ah… So searching absolute time into the future, I can see the “hello world” message I just sent in the previous post

[aaronsachs]

All of this smacks of a timezone mismatch somewhere. As a rule, we recommend setting everything to UTC to avoid timezone chicanery and of course, running NTP. Have you checked both of those off your list to rule out timezones being the issue?

[phosgene]

I think the graylog server assumes the timestamp from the router is UTC +0, and then adds UTC +11 to that timestamp (as per the root_timezone in server.conf) pushing it out 11 hours into the future.

This is what I see when I hover my mouse over “Timestamp” in the message:

I don’t think this router (a Cisco 1941) can send UTC formatted time, so is there a way to correct for this in Graylog? I don’t want to use UTC +0 on the router, because I still need the logs to be readable on that device in memory.

[cawfehman]

try changing the input type to a raw/plaintext UDP input instead of syslog udp.

Also, you say your receiving on UDP 5141, but the input is named syslog tcp5140.

[phosgene]

Thanks cawfehman and aaronsachs…

So the issue is definitely with the syslog messages from the router containing no UTC timezone information, and Graylog assuming that it is UTC +0.

If I remove “localtime” from the “service timestamps” configuration on the router (so that it sets the timestamps as UTC +0) then Graylog appends the +11 hours in server.conf and shows the correct time, but now the time is wrong on the logs stored in the router’s memory:

I suppose I can just add +11 or +10 hours in my brain when reading the logs on the router, depending on daylight saying time, but it’d be nice if I could get them to match up.

[cawfehman]

the 1941 is a long running ISR for Cisco (release in 2009)… are you running an older version of IOS on it? perhaps upgrading it will help.

[phosgene]

I’m not running the oldest IOS image (version 15.6(3)M3 released 28-Jul-2017), but this is a lab device and it does not have a service contract with Cisco, so I can’t download the latest M9 release for it.

[shoothub]

Cisco doesn’t conform syslog standard, so better is to use Raw input.

Check this great article:

Working with Cisco ASA / Nexus on Graylog

[dickinsonzach]

This is an interesting item. I believe Cisco will only log to the standard 514 port. So if you setup a Raw input, it will be consuming that standard port. Well, that wouldn’t play well with ALL the other devices that use 514, but do Syslog standards.

I guess you would need a 2nd IP to run the Raw Input on 514.

Thank you, Zach.

[aaronsachs]

Ehhhh, except that 514 is a privileged port and would require running Graylog as root which we advise against . Sticking with a raw input and then writing a rule to parse as the case may be would be a less risky approach IMO.

[dickinsonzach]

Well, you wouldn’t run Graylog on 514. You would run Graylog on 5140 and then do an OS port forward of 514 to 5140. But the original port has to be 514 for Cisco, so whatever Input you forward it to needs to be a Raw input.

In our environment we had a device only support 514, but needed a Raw input. We had another device that only supported 514 and supported Syslog. We had to add a second IP to the server. Both with 514 open.

Not a big deal. Just a bit of extra fiddling. All this may be moot if you have a Graylog cluster or proxy or something.