UDP inputs not working


#1

Hello, everyone!

I have a bit of a weird problem.

I have a graylog server (running Graylog 2.4.3) that works perfectly with a syslog TCP input. We have a centralized rsyslog server that all of our instances send logs to, and then the central logs server sends to graylog. This has worked for the better part of a year.

So, I want to switch this to UDP, but when I do, I am unable to get any messages into graylog. This input is able to start, but it does not receive any log messages.

I’ve also tried using a raw UDP input. Here are all the inputs configurations:

The rsyslog server seems to be configured correctly - the line

action(type="omfwd" Target="graylog1" Port="5140" Protocol="tcp")

was changed to

action(type="omfwd" Target="graylog1" Port="5141" Protocol="udp")

and I can see with tcpdump that log messages are getting to my graylog server successfully (this is all in staging):

[00:13:05][0][root@graylog1.<domain>:/var/log/graylog-server]$ tcpdump udp port 5141 -vvv -X

[...]

(from another box:)
[23:52:06][0][root@dashboard1.<domain>:~]# echo '<14>central-logs GRAYLOG-TEST-FOR-FORUM-POST' | nc -v -u -w 0 graylog1 5141

[...]

00:10:43.239253 IP (tos 0x0, ttl 64, id 53484, offset 0, flags [DF], proto UDP (17), length 73)
    dashboard1.<domain>.46775 > graylog1.<domain>.5141: [udp sum ok] UDP, length 45
        0x0000:  4500 0049 d0ec 4000 4011 1fbc 0a00 1a21  E..I..@.@......!
        0x0010:  0a00 1bdb b6b7 1415 0035 e81d 3c31 343e  .........5..<14>
        0x0020:  6365 6e74 7261 6c2d 6c6f 6773 2047 5241  central-logs.GRA
        0x0030:  594c 4f47 2d54 4553 542d 464f 522d 464f  YLOG-TEST-FOR-FO
        0x0040:  5255 4d2d 504f 5354 0a                   RUM-POST.

(among many other legitimate logs coming in)

Netstat shows graylog listening on 5140 (for TCP) and 5141 (for UDP):

[00:14:48][0][root@graylog1.<domain>:/var/log/graylog-server]$ netstat -nutlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      661/rpcbind
tcp        0      0 127.0.0.1:9200          0.0.0.0:*               LISTEN      1203/java
tcp        0      0 0.0.0.0:5140            0.0.0.0:*               LISTEN      1010/java
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN      1203/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1060/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1336/master
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN      1262/nrpe
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      1010/java
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      1012/mongod
tcp6       0      0 :::111                  :::*                    LISTEN      661/rpcbind
tcp6       0      0 :::22                   :::*                    LISTEN      1060/sshd
tcp6       0      0 :::5666                 :::*                    LISTEN      1262/nrpe
udp        0      0 0.0.0.0:20916           0.0.0.0:*                           647/dhclient
udp        0      0 0.0.0.0:5140            0.0.0.0:*                           1010/java
udp        0      0 0.0.0.0:5141            0.0.0.0:*                           1010/java
udp        0      0 0.0.0.0:68              0.0.0.0:*                           647/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           661/rpcbind
udp        0      0 0.0.0.0:123             0.0.0.0:*                           1453/chronyd
udp        0      0 0.0.0.0:323             0.0.0.0:*                           1453/chronyd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           423/rsyslogd
udp        0      0 0.0.0.0:819             0.0.0.0:*                           661/rpcbind
udp6       0      0 :::111                  :::*                                661/rpcbind
udp6       0      0 :::123                  :::*                                1453/chronyd
udp6       0      0 :::323                  :::*                                1453/chronyd
udp6       0      0 :::514                  :::*                                423/rsyslogd
udp6       0      0 :::819                  :::*                                661/rpcbind
udp6       0      0 :::17626                :::*                                647/dhclient

Iptables allows these ports:

[00:14:53][0][root@graylog1.<domain>:/var/log/graylog-server]$ iptables-save
# Generated by iptables-save v1.4.21 on Thu Dec 20 00:18:07 2018
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [565643:164476349]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p icmp -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5140 -j ACCEPT
-A INPUT -s <IP RANGE> -i eth0 -p tcp -m tcp --dport 5141 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Thu Dec 20 00:18:07 2018

And nothing weird is in the graylog log files.

I’ve searched the forums - often this is a case of not being able to bind to a port below 1024 (and this is well above). Sometimes it’s a problem with TCP and TLS, but this is UDP.

I’m kind of at a loss. The messages are clearly making it to the box, and graylog is listening on that port, but … nothing’s happening. Any ideas where I should look?


(Jan Doberstein) #2

Debug now:

echo ‘<14>sourcehost message text’ | nc -v -u -w 0 localhost 514

First send a message from the Graylog Server to the input - is that going in? Ten some other Server on that Network, is that working? If not - find the reason.

From what I can see in your iptables, you only allow TCP and now UDP, that might already be the reason…


#3

Should be udp, not tcp


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.