One of my devices data will not display in Graylog

Graylog Central (peer support)
1

So I have a strange problem and I was hoping someone may have some additional troubleshooting steps I can try.

I am sending about 15 devices to a brand new graylog installation.

However for device number 16, which is one of my wireless controllers, the data never appears in the graylog GUI. I have followed the traffic throughout my network using wireshark, and onto the graylog server itself using tshark. I then created a firewall rule that logged all traffic from this host and saw it passing through the rule.

However the traffic never appears in the messages journal, or the GUI. All of my other hosts that use the udp/ 5514 connector work fine. But not this one.

As a disclaimer I do have NAT on the firewall-cmd forwarding udp/514 to udp/5514.

Even my other wireless controller works perfectly fine with the exact same configuration as the one that doesn’t work. I literally copied and pasted the config with no luck.

I am at a loss as to what to try next. I have removed the input and re-added it several times, I have deleted all rules from firewall—cmd and recreated everything. I have rebooted the server. Nothing I have done can get this controller working.

This controllers data does appear perfectly fine on my super old graylog server that has not been updated in probably 4 years… but I need to get it on the new server because that old hardware is getting retired.

Curious if anyone has ever seen this before or has something I can try?

Thanks

2

What is the time that is set on the “not working” device?

Maybe the time difference let you think nothing comes in but the messages are a few hours in the past or future? What can be found in the Graylog server.log?

3

Please link any other discussion topics about the same issue by yourself next time.

For reference:

4

The time is exactly the same on the device and the graylog server. In fact they are using the same NTP server.

if i look at the graylog-server/server.log it just shows me restarting the listeners and things like that.

Nothing that would indicate a problem. Is there a way to put this in debug mode or something and get more data in the log?

5

Just as a quick update, i updated Graylog to the new version that was released the other day with no changes.

I am going to try and setup a dedicated input for this controller and see if that makes any difference.

Thanks

6

So i created three new inputs, and moved all of my UDP 514 devices to different inputs on different ports with different nat rules forwarding and logging the traffic. Two of the inputs work perfectly fine. The third one the traffic never shows up in Graylog. The last time I see the traffic is inside the firewall log. I am completely stumped.

Here are the firewall logs from the device that is not working…

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=197 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=177

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=225 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=205

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=149 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=129

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=255 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=235

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=217 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=197

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=255 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=235

Jul 19 09:46:43 graylog2 kernel: wlc-firewalld-logIN=eth0 OUT= MAC=00:15:5d:14:8c:17:80:ac:ac:ac:3a:40:08:00 SRC=10.18.0.2 DST=10.39.99.209 LEN=217 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=UDP SPT=514 DPT=514 LEN=197

7

He Brian,

I personal would start to investigate from the source of the messages until I reach Graylog check all systems where the messages cross and you will find the reason why the messages are not reaching Graylog.

8

So thats what I did… i wiresharked the port for the wireless controller… and saw the syslog packets.
Then i wiresharked the port going to the graylog server… and saw the syslog packets.
then i ran tshark ON the graylog server… and saw the packets.
then i created a rich firewall rule on the CentOS 7 graylog server and logged the packets from the controller that would not appear to a log file… and saw the packets being accepted.

The packets are getting to the graylog server and passing through the firewall and being accepted.

But they never appear anywhere inside graylog.

I am at a bit of a loss…

9

when you switch that input to a RAW/Plaintext input, does that change?

Did you search all messages or in a frame years in the past and the future?

10

So, i tried changing the input, and searched everywhere, the messages just never appeared.

I shut down the VM and created a new one, kept everything the same… and it works fine on the new VM.

I don’t understand why… but unfortunately was out of time to get this project moving.

Thanks everyone for their help.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.