TCP Messages not showing on GL despite connection is OK

1. Describe your incident:

Messages from some servers using the same rsyslog configuration files are not being shown on GL.

2. Describe your environment:

  • OS Information: Ubuntu 20.04 LTS

  • Package Version: GL 4.3.7 + OpenSearch 1.3.3

  • Service logs, configurations, and environment variables:

Input in use for the VMs:


It works and receives messages.

General behavior:

$ cat /etc/rsyslog.d/graylog.conf 
*.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format

And for nginx in particular:

$ cat /etc/rsyslog.d/nginx.conf 
# Ansible Managed

local1.* @@graylog-lb:1514;RSYSLOG_SyslogProtocol23Format

The imfile module is enabled in rsyslog:

$ grep imfile /etc/rsyslog.conf 
module(load="imfile" PollingInterval="10")

…and the service was restarted by Ansible after applying the changes.

3. What steps have you already taken to try and solve the problem?

Before posting this message, I checked in the forums and found this related issue:

The time-stamp don’t seem to be an issue in my case:


After checking that the time was not the problem, I left this running on both GL nodes:

tcpdump -vnX src -w 20220929.pcap

On one of the affected GL clients I created a test message:

root@gl-client:~# logger -t MY_TAG "This is another a test message for you"

I could see how the tcpdump on the gl-node-2 was more active, so I checked:

root@gl-node-2:~# tcpdump -qns 0 -A -r 20220929.pcap  | grep MY_TAG
reading from file 20220929.pcap, link-type EN10MB (Ethernet)
1..[....<13>1 2022-09-29T09:31:46.423600+00:00 gl-client MY_TAG - - -  This is another a test message for you

Messages are, indeed, being delivered from the clients to the GL cluster.

4. How can the community help?

Other VMs with the same rsyslog config are shown on the GL GUI when someone searches for them, but not this set of VMs. Am I overlooking the obvious??

TIA for your help!

Hey @m_mlk

Correct me if I’m wrong. the connections Between client and GL are fine , You identified that messages are showing up to Graylog Server, either through tcpdump or other means, but there are no message on the GUI.
I’m assume you check Elasticsearch/Opensearch is running correct?

EDIT: Another thought, have you tried a different input for testing, perhaps Raw Plaintext input instead of GELF?

Hi @gsmith

GL is working fine. Let’s say we had 500 servers sending stuff to the GL cluster in the past using GELF/tcp. All those show up flawlessly. I added a new set of VMs (ca. 20 items) and those are not being shown at all.

I have run searches for the source, host_name, applicaton_name, etc etc. Nothing comes back…
As I said, the configuration is the same on all VMs… O_o


Replying to myself…
I went over the rsyslog configuration of the working nodes and compared it to the one on the ones where GL was failing to show information from…
It seems like there was some hidden character (md5sum FTW!!!); I fixed the template again and all VMs are now shown on GL as expected.

Thanks all @gsmith for your assistance anyway!


1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.