Hello, everyone!
I have a bit of a weird problem.
I have a graylog server (running Graylog 2.4.3) that works perfectly with a syslog TCP input. We have a centralized rsyslog server that all of our instances send logs to, and then the central logs server sends to graylog. This has worked for the better part of a year.
So, I want to switch this to UDP, but when I do, I am unable to get any messages into graylog. This input is able to start, but it does not receive any log messages.
I’ve also tried using a raw UDP input. Here are all the inputs configurations:
The rsyslog server seems to be configured correctly - the line
action(type="omfwd" Target="graylog1" Port="5140" Protocol="tcp")
was changed to
action(type="omfwd" Target="graylog1" Port="5141" Protocol="udp")
and I can see with tcpdump that log messages are getting to my graylog server successfully (this is all in staging):
[00:13:05][0][root@graylog1.<domain>:/var/log/graylog-server]$ tcpdump udp port 5141 -vvv -X
[...]
(from another box:)
[23:52:06][0][root@dashboard1.<domain>:~]# echo '<14>central-logs GRAYLOG-TEST-FOR-FORUM-POST' | nc -v -u -w 0 graylog1 5141
[...]
00:10:43.239253 IP (tos 0x0, ttl 64, id 53484, offset 0, flags [DF], proto UDP (17), length 73)
dashboard1.<domain>.46775 > graylog1.<domain>.5141: [udp sum ok] UDP, length 45
0x0000: 4500 0049 d0ec 4000 4011 1fbc 0a00 1a21 E..I..@.@......!
0x0010: 0a00 1bdb b6b7 1415 0035 e81d 3c31 343e .........5..<14>
0x0020: 6365 6e74 7261 6c2d 6c6f 6773 2047 5241 central-logs.GRA
0x0030: 594c 4f47 2d54 4553 542d 464f 522d 464f YLOG-TEST-FOR-FO
0x0040: 5255 4d2d 504f 5354 0a RUM-POST.
(among many other legitimate logs coming in)
Netstat shows graylog listening on 5140 (for TCP) and 5141 (for UDP):
[00:14:48][0][root@graylog1.<domain>:/var/log/graylog-server]$ netstat -nutlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 661/rpcbind
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 1203/java
tcp 0 0 0.0.0.0:5140 0.0.0.0:* LISTEN 1010/java
tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN 1203/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1060/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1336/master
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 1262/nrpe
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 1010/java
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1012/mongod
tcp6 0 0 :::111 :::* LISTEN 661/rpcbind
tcp6 0 0 :::22 :::* LISTEN 1060/sshd
tcp6 0 0 :::5666 :::* LISTEN 1262/nrpe
udp 0 0 0.0.0.0:20916 0.0.0.0:* 647/dhclient
udp 0 0 0.0.0.0:5140 0.0.0.0:* 1010/java
udp 0 0 0.0.0.0:5141 0.0.0.0:* 1010/java
udp 0 0 0.0.0.0:68 0.0.0.0:* 647/dhclient
udp 0 0 0.0.0.0:111 0.0.0.0:* 661/rpcbind
udp 0 0 0.0.0.0:123 0.0.0.0:* 1453/chronyd
udp 0 0 0.0.0.0:323 0.0.0.0:* 1453/chronyd
udp 0 0 0.0.0.0:514 0.0.0.0:* 423/rsyslogd
udp 0 0 0.0.0.0:819 0.0.0.0:* 661/rpcbind
udp6 0 0 :::111 :::* 661/rpcbind
udp6 0 0 :::123 :::* 1453/chronyd
udp6 0 0 :::323 :::* 1453/chronyd
udp6 0 0 :::514 :::* 423/rsyslogd
udp6 0 0 :::819 :::* 661/rpcbind
udp6 0 0 :::17626 :::* 647/dhclient
Iptables allows these ports:
[00:14:53][0][root@graylog1.<domain>:/var/log/graylog-server]$ iptables-save
# Generated by iptables-save v1.4.21 on Thu Dec 20 00:18:07 2018
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [565643:164476349]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p icmp -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5140 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5141 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Thu Dec 20 00:18:07 2018
And nothing weird is in the graylog log files.
I’ve searched the forums - often this is a case of not being able to bind to a port below 1024 (and this is well above). Sometimes it’s a problem with TCP and TLS, but this is UDP.
I’m kind of at a loss. The messages are clearly making it to the box, and graylog is listening on that port, but … nothing’s happening. Any ideas where I should look?