Graylog ignoring certain UDP messages

I have two inputs configured (replacing old unmanaged ELK stack) and I am getting messages on port 514. I have setup two inputs on 5514 (one for TCP and one for UDP) just using plain text for now as to not reject or misread messages until I have their format correct.

If I use tcpdump on UDP and watch the interface there’s a ton of messages going to 514 but only a certain few are actually getting recorded.

If I do a dump I can see requests like:

15:10:34.754495 IP (tos 0x0, ttl 64, id 59249, offset 0, flags [DF], proto UDP (17), length 196)
    10.X.X.35.48046 > 10.X.X.4.514: [udp sum ok] SYSLOG, length: 168
	Facility daemon (3), Severity info (6)
	Msg: Apr 20 15:10:34 server-r2u35 squeezed[1635]: [1431705.82] Failed to free 9216 KiB of memory: the following domains have failed to meet their targets: [ 11, 18, 24 ]
	0x0000:  3c33 303e 4170 7220 3230 2031 353a 3130
	0x0010:  3a33 3420 7365 7276 6572 2d72 3275 3335
	0x0020:  2073 7175 6565 7a65 645b 3136 3335 5d3a
	0x0030:  205b 3134 3331 3730 352e 3832 5d20 4661
	0x0040:  696c 6564 2074 6f20 6672 6565 2039 3231
	0x0050:  3620 4b69 4220 6f66 206d 656d 6f72 793a
	0x0060:  2074 6865 2066 6f6c 6c6f 7769 6e67 2064
	0x0070:  6f6d 6169 6e73 2068 6176 6520 6661 696c
	0x0080:  6564 2074 6f20 6d65 6574 2074 6865 6972
	0x0090:  2074 6172 6765 7473 3a20 5b20 3131 2c20
	0x00a0:  3138 2c20 3234 205d

This is a syslog message leaving a xen server host and coming to the syslog server. I can see it hitting port 514 on the syslog server.

Here is the command I am running:
tcpdump -nnvvS src 10.99.2.35 and dst port 514 -i enp6s0

Looking at these dumps it would make sense that the UDP raw input I created should be receiving these but it’s not

I’m assuming you’re forwarding 514 to 5514 on your firewall? Can you have the xen server send directly to 5514?

is the tcp input working?

Both UDP and TCP is working just fine for a few servers, but the xen hosts are pushing out a huge amounts of messages per IP. I was wondering if perhaps there’s some sort of flood protection it could be hitting or rate limit?

Random tips:
You can also check graylog’s log. eg. if the message missformatted it can drop out. Also You can have some indexing problem (system - overview menu).
Check the TCP way. no guarantee for UDP forwarding. And also under the java GC graylog drops udp (don’t worry, its under 1%…)

Clearly this problem is port forwarding related… I am doing a port forward in iptables but it’s not working properly. I have logged into the hosts in question and changed their port to use 5514 and poof it works just fine now.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.