No log input from all sources

Graylog Central (peer support)
1

Hi Guys,

Installed Graylog 3.0 using this guide, configured syslog inputs over both TCP and UDP, configured other machines to send logs to the node, but still not seeing anything coming in. Spend a few hours pouring over the configs trying to nail the issue, but not getting any closer.
Could it have something to do with the fact that elastic is only listening on IPv6?

$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.98.75:9000      0.0.0.0:*               LISTEN      5003/java
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      5277/mongod
tcp        0      0 127.0.0.1:5514          0.0.0.0:*               LISTEN      5003/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4985/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      5273/master
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      4989/java
tcp6       0      0 ::1:9200                :::*                    LISTEN      4989/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      4989/java
tcp6       0      0 ::1:9300                :::*                    LISTEN      4989/java
tcp6       0      0 :::22                   :::*                    LISTEN      4985/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      5273/master
udp        0      0 127.0.0.1:323           0.0.0.0:*                           4483/chronyd
udp        0      0 127.0.0.1:5514          0.0.0.0:*                           5003/java
udp        0      0 127.0.0.1:5514          0.0.0.0:*                           5003/java
udp6       0      0 ::1:323                 :::*                                4483/chronyd

GUI top right: “In 0 / Out 0 msg/s”.

Client:

# tcpdump -vvv -nn -i ens160 | grep 5514
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
172.20.31.112.40268 > 192.168.98.75.5514: Flags [S], cksum 0xeea6 (incorrect -> 0x686f), seq 175522271, win 29200, options [mss 1460,sackOK,TS val 207319937 ecr 0,nop,wscale 7], length 0
192.168.98.75.5514 > 172.20.31.112.40268: Flags [S.], cksum 0x0a39 (correct), seq 215325562, ack 175522272, win 28960, options [mss 1349,sackOK,TS val 60076965 ecr 207319937,nop,wscale 2], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [.], cksum 0xee9e (incorrect -> 0xa8c5), seq 1, ack 1, win 229, options [nop,nop,TS val 207319944 ecr 60076965], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [P.], cksum 0xef0d (incorrect -> 0xcbe3), seq 1:112, ack 1, win 229, options [nop,nop,TS val 207319944 ecr 60076965], length 111
192.168.98.75.5514 > 172.20.31.112.40268: Flags [.], cksum 0x8ceb (correct), seq 1, ack 112, win 7240, options [nop,nop,TS val 60076973 ecr 207319944], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [P.], cksum 0xf002 (incorrect -> 0xf9c1), seq 112:468, ack 1, win 229, options [nop,nop,TS val 207319948 ecr 60076973], length 356
192.168.98.75.5514 > 172.20.31.112.40268: Flags [.], cksum 0x8a74 (correct), seq 1, ack 468, win 7508, options [nop,nop,TS val 60076976 ecr 207319948], length 0

Graylog server:

# tcpdump -vvv -nn -i eno16777984 | grep 5514
tcpdump: listening on eno16777984, link-type EN10MB (Ethernet), capture size 262144 bytes
172.20.31.112.40268 > 192.168.98.75.5514: Flags [S], cksum 0x68de (correct), seq 175522271, win 29200, options [mss 1349,sackOK,TS val 207319937 ecr 0,nop,wscale 7], length 0
192.168.98.75.5514 > 172.20.31.112.40268: Flags [S.], cksum 0xeea6 (incorrect -> 0x09ca), seq 215325562, ack 175522272, win 28960, options [mss 1460,sackOK,TS val 60076965 ecr 207319937,nop,wscale 2], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [.], cksum 0xa8c5 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 207319944 ecr 60076965], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [P.], cksum 0xcbe3 (correct), seq 1:112, ack 1, win 229, options [nop,nop,TS val 207319944 ecr 60076965], length 111
192.168.98.75.5514 > 172.20.31.112.40268: Flags [.], cksum 0xee9e (incorrect -> 0x8ceb), seq 1, ack 112, win 7240, options [nop,nop,TS val 60076973 ecr 207319944], length 0
172.20.31.112.40268 > 192.168.98.75.5514: Flags [P.], cksum 0xf9c1 (correct), seq 112:468, ack 1, win 229, options [nop,nop,TS val 207319948 ecr 60076973], length 356
192.168.98.75.5514 > 172.20.31.112.40268: Flags [.], cksum 0xee9e (incorrect -> 0x8a74), seq 1, ack 468, win 7508, options [nop,nop,TS val 60076976 ecr 207319948], length 0

Update:
After fiddling around a bit more I managed to get UDP input working. Still no TCP though. Same as https://community.graylog.org/t/tcp-syslog-does-not-work/10402

3

you really need to check if networking is working and if no firewall or anything on the way along from the sender to the recipient is vlocking anything.

4

That’s why I posted tcpdump from both machines. The logs are being sent and arrive at destination, but Graylog server is not taking them.

5

does the Graylog server write any error message in the server.log? Did you see error message count on the input? If you switch from syslog input to raw input on that port did you see messages?

6

Set up new TCP raw input, no errors on it or the server.log file after “firewall-cmd --add-port=5555/tcp” and “firewall-cmd --reload”:

2019-05-14T12:53:30.124+01:00 INFO  [InputStateListener] Input [Syslog TCP/5cd992091fbafd48b38df0a0] is now STOPPING
2019-05-14T12:53:30.200+01:00 INFO  [InputStateListener] Input [Syslog TCP/5cd992091fbafd48b38df0a0] is now STOPPED
2019-05-14T12:53:30.201+01:00 INFO  [InputStateListener] Input [Syslog TCP/5cd992091fbafd48b38df0a0] is now TERMINATED
2019-05-14T12:53:51.447+01:00 INFO  [InputStateListener] Input [Raw/Plaintext TCP/5cdaac4e1fbafd138b6c94b3] is now STARTING
2019-05-14T12:53:51.540+01:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input RawTCPInput{title=RawTCP, type=org.graylog2.inputs.raw.tcp.RawTCPInput, nodeId=699d0be8-fc68-4e62-837f-8afa1aa981d8} (channel [id: 0x44e3adc9, L:/0.0.0.0:5555]) should be 1048576 but is 425984.
2019-05-14T12:53:51.542+01:00 INFO  [InputStateListener] Input [Raw/Plaintext TCP/5cdaac4e1fbafd138b6c94b3] is now RUNNING
7

can you send a message via telnet/netcat to that port with success? From localhost, from remote host?

This is basic linux debugging @alext that you need to run now.

8

Yup, I can. It’s not the network - of this I’m certain.

# telnet 192.168.98.75 5555
Trying 192.168.98.75...
Connected to 192.168.98.75.
Escape character is '^]'.

It has something to do with graylog config, I just can’t figure out what.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.