1. Describe your incident:
Messages from some servers using the same rsyslog configuration files are not being shown on GL.
2. Describe your environment:
-
OS Information: Ubuntu 20.04 LTS
-
Package Version: GL 4.3.7 + OpenSearch 1.3.3
-
Service logs, configurations, and environment variables:
Input in use for the VMs:
It works and receives messages.
General behavior:
$ cat /etc/rsyslog.d/graylog.conf
*.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format
And for nginx in particular:
$ cat /etc/rsyslog.d/nginx.conf
# Ansible Managed
#
input(type="imfile"
File="/var/log/nginx/*.log"
Tag="nginx"
Severity="info"
Facility="local1")
local1.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format
The imfile module is enabled in rsyslog:
$ grep imfile /etc/rsyslog.conf
module(load="imfile" PollingInterval="10")
…and the service was restarted by Ansible after applying the changes.
3. What steps have you already taken to try and solve the problem?
Before posting this message, I checked in the forums and found this related issue:
The time-stamp don’t seem to be an issue in my case:
After checking that the time was not the problem, I left this running on both GL nodes:
tcpdump -vnX src 10.10.10.51 -w 20220929.pcap
On one of the affected GL clients I created a test message:
root@gl-client:~# logger -t MY_TAG "This is another a test message for you"
I could see how the tcpdump on the gl-node-2 was more active, so I checked:
root@gl-node-2:~# tcpdump -qns 0 -A -r 20220929.pcap | grep MY_TAG
reading from file 20220929.pcap, link-type EN10MB (Ethernet)
1..[....<13>1 2022-09-29T09:31:46.423600+00:00 gl-client MY_TAG - - - This is another a test message for you
root@gl-node-2:~#
Messages are, indeed, being delivered from the clients to the GL cluster.
4. How can the community help?
Other VMs with the same rsyslog config are shown on the GL GUI when someone searches for them, but not this set of VMs. Am I overlooking the obvious??
TIA for your help!