Graylog ignoring certain UDP messages

Graylog Central (peer support)
1

I have two inputs configured (replacing old unmanaged ELK stack) and I am getting messages on port 514. I have setup two inputs on 5514 (one for TCP and one for UDP) just using plain text for now as to not reject or misread messages until I have their format correct.

If I use tcpdump on UDP and watch the interface there’s a ton of messages going to 514 but only a certain few are actually getting recorded.

If I do a dump I can see requests like:

15:10:34.754495 IP (tos 0x0, ttl 64, id 59249, offset 0, flags [DF], proto UDP (17), length 196)
    10.X.X.35.48046 > 10.X.X.4.514: [udp sum ok] SYSLOG, length: 168
	Facility daemon (3), Severity info (6)
	Msg: Apr 20 15:10:34 server-r2u35 squeezed[1635]: [1431705.82] Failed to free 9216 KiB of memory: the following domains have failed to meet their targets: [ 11, 18, 24 ]
	0x0000:  3c33 303e 4170 7220 3230 2031 353a 3130
	0x0010:  3a33 3420 7365 7276 6572 2d72 3275 3335
	0x0020:  2073 7175 6565 7a65 645b 3136 3335 5d3a
	0x0030:  205b 3134 3331 3730 352e 3832 5d20 4661
	0x0040:  696c 6564 2074 6f20 6672 6565 2039 3231
	0x0050:  3620 4b69 4220 6f66 206d 656d 6f72 793a
	0x0060:  2074 6865 2066 6f6c 6c6f 7769 6e67 2064
	0x0070:  6f6d 6169 6e73 2068 6176 6520 6661 696c
	0x0080:  6564 2074 6f20 6d65 6574 2074 6865 6972
	0x0090:  2074 6172 6765 7473 3a20 5b20 3131 2c20
	0x00a0:  3138 2c20 3234 205d

This is a syslog message leaving a xen server host and coming to the syslog server. I can see it hitting port 514 on the syslog server.

Here is the command I am running:
tcpdump -nnvvS src 10.99.2.35 and dst port 514 -i enp6s0

Looking at these dumps it would make sense that the UDP raw input I created should be receiving these but it’s not

2

I’m assuming you’re forwarding 514 to 5514 on your firewall? Can you have the xen server send directly to 5514?

is the tcp input working?

3

Both UDP and TCP is working just fine for a few servers, but the xen hosts are pushing out a huge amounts of messages per IP. I was wondering if perhaps there’s some sort of flood protection it could be hitting or rate limit?

4

Random tips:
You can also check graylog’s log. eg. if the message missformatted it can drop out. Also You can have some indexing problem (system - overview menu).
Check the TCP way. no guarantee for UDP forwarding. And also under the java GC graylog drops udp (don’t worry, its under 1%…)

5

Clearly this problem is port forwarding related… I am doing a port forward in iptables but it’s not working properly. I have logged into the hosts in question and changed their port to use 5514 and poof it works just fine now.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.