Messages getting to server but not to Graylog

[fl-mike]

I have messages making it to the server, but are not being picked up by Graylog. My issue appears to be identical to an issue reported here: Server receive logs but don't go in Graylog but there was no reported resolution on that topic. My test results are practically identical to the tests in that topic.

My graylog server, running on Ubuntu 16.04, has been collecting on a single interface successfully since installation last year. I have a need to collect from a separate isolated network and configured my 2nd NIC, eth1 on 10.1.1.2. It is connected to a firewall at 10.1.1.1 which is allowing UDP 514 traffic to the Graylog server. I can see the messages making it to the server via a tcpdump. The devices sending the logs had previously sent to a ‘test’ Graylog server I had set up last year, so I’m certain the logs are formatted properly.

I am running authbind and native port 514 traffic is being picked up on eth0 just fine, but not on my eth1 connection. I tried an iptables redirect to port 1514 with the same results.

It appears that Graylog is listening:

$ sudo netstat -nutlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1258/sshd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1130/mongod
tcp6 0 0 :::22 :::* LISTEN 1258/sshd
tcp6 0 0 172.16.4.22:9000 :::* LISTEN 1169/java
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1150/java
tcp6 0 0 ::1:9200 :::* LISTEN 1150/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1150/java
tcp6 0 0 ::1:9300 :::* LISTEN 1150/java
udp6 0 0 172.16.4.22:514 :::* 1169/java
udp6 0 0 172.16.4.22:514 :::* 1169/java
udp6 0 0 172.16.4.22:514 :::* 1169/java
udp6 0 0 10.1.1.2:514 :::* 1169/java
udp6 0 0 172.16.4.22:514 :::* 1169/java
udp6 0 0 10.1.1.2:1514 :::* 1169/java
udp6 0 0 172.16.4.22:2055 :::* 1169/java

When I send a test message from the server via “echo “Test message” | nc -u 10.1.1.2 514” the message shows up in Graylog, and the input connector reflects 26.0B passed.

I upgraded to Graylog 3 yesterday in the hopes that it might make a difference, but everything is the same with regards to this input.

I’m really stuck on this.

My eth1 input connector in Graylog is already set to RAW text.

UFW is not running and my iptables are as follows:

$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

[fl-mike]

Anybody? Please? It would benefit me greatly to be able to get messages from our second network.

[jan (Jan Doberstein)]

sorry to say @fl-mike but it looks like nobody is able to help you debugging this. You need to check yourself if something in the OS is the issue or if you have simple not created the input in Graylog.

Special because direct message via netcat works it might be something in your networking/firewall or similar. At least from what you have written I did not see any other issue.

[fl-mike]

I appreciate the reply, even if you don’t have a known solution.

You mentioned “not created the input in Graylog”, but since netcat is putting messages all the way into graylog, then I couldn’t have missed that, right, or is there something else I am overlooking? I’ve been staring at this problem for so many days now that I feel like I’ve probably overlooked something silly.

I’m thinking something is going on at the OS or Java level but my Linux knowledge is weak. I did remove and reinstall Java earlier today but no change there. I’ll keep at it and will definitely post the solution if I ever find one.

[jan (Jan Doberstein)]

from a logical perspective it is more likely that this is located in your OS or environment and not JAVA. As you would not be the only one having this issues.