1. Describe your incident:
I have just installed Graylog Open and I am not seeing any messages show up on the server from external devices. However, if I send a message from the server itself (i.e., logger --udp --port 514 -n idr1.mdc.com "whatever") it shows up. If I do the same from e.g., IDR2 to IDR1 it does not show up. This holds true for any external device not originating from the local server.
Doing a tcpdump shows the server is receiving logs, but it is not passing them to Graylog.
[me@MDCIDR1 ~]$ sudo tcpdump -i any udp port 514
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
20:58:50.264598 eth0 In IP mdcwifi.wlc.mdc.com.32844 > idr1.mdc.com.syslog: SYSLOG local0.warning, length: 178
20:58:50.264599 eth0 In IP mdcwifi.wlc.mdc.com.32844 > idr1.mdc.com.syslog: SYSLOG local0.notice, length: 145
20:58:54.552717 eth0 In IP mdcwifi.wlc.mdc.com.32844 > idr1.mdc.com.syslog: SYSLOG local0.warning, length: 178
Interestingly it seems like the server is only listening on IPv6? Or am I interpreting this wrong/it doesn’t mean anything?
[me@MDCIDR1 ~]$ sudo netstat -tulnp | grep :514
udp6 0 0 10.20.10.4:514 :::* 1095/java
udp6 0 0 10.20.10.4:514 :::* 1095/java
2. Describe your environment:
-
OS Information: RHEL 9.5
-
Package Version: Graylog 6.1.5-2
-
Service logs, configurations, and environment variables:
-
IDR1 - Raw UDP
bind_address: 10.20.10.4
charset_name: UTF-8
number_worker_threads: 2
override_source:
port: 5555
recv_buffer_size: 262144 -
IDR1 - Syslog - UDP
allow_override_date: true
bind_address: 10.20.10.4
charset_name: UTF-8
expand_structured_data: false
force_rdns: false
number_worker_threads: 2
override_source:
port: 514
recv_buffer_size: 262144
store_full_message: true
timezone: America/Chicago -
Sending from multiple devices… PA-850, WLC 3504, 2930M…
3. What steps have you already taken to try and solve the problem?
- Confirmed traffic is allowed by external firewall (and I assume must be allowed by the server firewall since it appears in tcpdump)
- Confirmed traffic shows up on the server in a packet capture
- Confirmed traffic shows up fine when pointed to SolarWinds Kiwi Syslog Server
- Attempted doing raw inputs on 5555
4. How can the community help?
- What are some next steps I can take to troubleshoot this further? Linux, Graylog, the database thing, they are not something I am familiar with in the slightest.
- I have the node, and the server, configured on the same VM for staging. Would separating the node and server specifically help with this issue?