TCP Messages not showing on GL despite connection is OK

[m_mlk]

1. Describe your incident:

Messages from some servers using the same rsyslog configuration files are not being shown on GL.

2. Describe your environment:

• OS Information: Ubuntu 20.04 LTS

• Package Version: GL 4.3.7 + OpenSearch 1.3.3

• Service logs, configurations, and environment variables:

Input in use for the VMs:

It works and receives messages.

General behavior:

$ cat /etc/rsyslog.d/graylog.conf 
*.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format

And for nginx in particular:

$ cat /etc/rsyslog.d/nginx.conf 
# Ansible Managed
#
input(type="imfile"
      File="/var/log/nginx/*.log"
      Tag="nginx"
      Severity="info"
      Facility="local1")

local1.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format

The imfile module is enabled in rsyslog:

$ grep imfile /etc/rsyslog.conf 
module(load="imfile" PollingInterval="10")

…and the service was restarted by Ansible after applying the changes.

3. What steps have you already taken to try and solve the problem?

Before posting this message, I checked in the forums and found this related issue:

Not receiving TCP messages

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic! 1. Describe your incident: Input Syslog TCP on port 1514 not receiving messages on graylog, but I can see traffic when I run “sudo tcpdump -i any -v port 1514” 2. Describe your environment: OS Information: Ubuntu 20.04 Package Version:graylog v. 4.2.9 Service logs, configuration…

The time-stamp don’t seem to be an issue in my case:

After checking that the time was not the problem, I left this running on both GL nodes:

tcpdump -vnX src 10.10.10.51 -w 20220929.pcap

On one of the affected GL clients I created a test message:

root@gl-client:~# logger -t MY_TAG "This is another a test message for you"

I could see how the tcpdump on the gl-node-2 was more active, so I checked:

root@gl-node-2:~# tcpdump -qns 0 -A -r 20220929.pcap  | grep MY_TAG
reading from file 20220929.pcap, link-type EN10MB (Ethernet)
1..[....<13>1 2022-09-29T09:31:46.423600+00:00 gl-client MY_TAG - - -  This is another a test message for you
root@gl-node-2:~#

Messages are, indeed, being delivered from the clients to the GL cluster.

4. How can the community help?

Other VMs with the same rsyslog config are shown on the GL GUI when someone searches for them, but not this set of VMs. Am I overlooking the obvious??

TIA for your help!

[gsmith]

Hey @m_mlk

Correct me if I’m wrong. the connections Between client and GL are fine , You identified that messages are showing up to Graylog Server, either through tcpdump or other means, but there are no message on the GUI.
I’m assume you check Elasticsearch/Opensearch is running correct?

EDIT: Another thought, have you tried a different input for testing, perhaps Raw Plaintext input instead of GELF?

[m_mlk]

Hi @gsmith

GL is working fine. Let’s say we had 500 servers sending stuff to the GL cluster in the past using GELF/tcp. All those show up flawlessly. I added a new set of VMs (ca. 20 items) and those are not being shown at all.

I have run searches for the source, host_name, applicaton_name, etc etc. Nothing comes back…
As I said, the configuration is the same on all VMs… O_o

Thanks!

[m_mlk]

Replying to myself…
I went over the rsyslog configuration of the working nodes and compared it to the one on the ones where GL was failing to show information from…
It seems like there was some hidden character (md5sum FTW!!!); I fixed the template again and all VMs are now shown on GL as expected.

Thanks all @gsmith for your assistance anyway!

Cheers