Problem ingesting rsyslog and NXLog behind an NGINX load balancer

Hello everyone, and thanks for that template. That’s a very good idea !

I am still struggling with this cluster / load balancer since my previous post.

Things moved and I fixed some stuff - Thanks @gsmith !

I was trying to get sidecars for Linux and Windows to work behind an NGINX loadbalancer, but I couldn’t make it, so I gave up (problem explain in the previous topic).

I then decided to use rsyslog for Linux. It was faster at first, not as convenient but Linux native. So I could totally work with that.

And for Windows, I tried NXLog. Which is pretty good too and work well with Windows event manager.

But, I can’t send those logs to my Graylog cluster through my NGINX Load Balancer. I’ts been days and days…I’m propably doing things wrong : feel free to tell me !

Description of your problem

  • I can’t get rsyslog and NXlogs to send log files via the NGINX load balancer.
  • The connection between rsyslog clients and the NGINX load balancer initializes correctly when starting the rsyslog service, but it seems that error messages come when I end the SSH connection.
  • If I bypass the load balancer to send the logs directly to one of the Graylog nodes, I get the logs without any problem, without losing the connection.
  • I’m still getting logs from NXLog, but it can’t seem to make a decent connection.
  • I have no problem with other inputs using the same NGINX configurations (RAW UDP, PAN OS).

Description of steps you’ve taken to attempt to solve the issue

This setup is already a “backup plan”: I can’t send the logs over TLS in the loadbalancer. This works fine on my pre-production server, where Graylog / MongoDB and Elasticsearch are on the same machine. But first things first, I’m first trying to make this work.

  • Tried to use a more traditional format for rsyslog.
  • Reloaded rsyslog.service
  • Reloaded NGINX
  • Stop / Start Input
  • tcpdump on the relevant ports
  • Sending logs directly to one of the node.

Environmental information

192.168.1.20 = Graylog UI Load Balancer- Apache
192.168.1.21 = Graylog Inputs Load Balancer - NGINX
192.168.1.22 = Graylog Node 1
192.168.1.23 = Graylog Node 2
192.168.1.24 = Graylog Node 3
192.168.1.25 = Elasticsearch 1
192.168.1.26 = Elasticsearch 2
192.168.1.27 = Elasticsearch 3

Configuration Files

  • /etc/rsyslog.d/my.conf
*.* action      (

                type     = "omfwd"
                target   = "192.168.1.21"
                port     = "10514"
                protocol = "tcp"
                template = "RSYSLOG_SyslogProtocol23Format"
                action.resumeRetryCount = "-1"
                queue.fileName          = "rsyslog_metsys_tcp"
                queue.type              = "linkedList"
                queue.size              = "10000"
                queue.saveonshutdown    = "on"

                )
  • /etc/nginx/nginx.conf
# rsyslog TCP
        upstream graylog_rsyslog_tcp
        {
                server 192.168.1.22:10514 max_fails=3 fail_timeout=30s;
                server 192.168.1.23:10514 max_fails=3 fail_timeout=30s;
                server 192.168.1.24:10514 max_fails=3 fail_timeout=30s;
        }

        server
        {
                listen 10514;
                proxy_pass graylog_rsyslog_tcp;
                proxy_timeout 10s;
                error_log /var/log/nginx/graylog_rsyslog_tcp.log;
        }

# NXLog TCP
        upstream nxlog_tcp
        {
                server 192.168.1.22:10517 max_fails=3 fail_timeout=30s;
                server 192.168.1.23:10517 max_fails=3 fail_timeout=30s;
                server 192.168.1.24:10517 max_fails=3 fail_timeout=30s;
        }

        server
        {
                listen 10517;
                proxy_pass nxlog_tcp;
                proxy_timeout 10s;
                error_log /var/log/nginx/graylog_nxlog_tcp.log;
  • nxlog.conf
define ROOT     C:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _gelf>
    Module      xm_gelf
</Extension>

<Input in>
    Module      im_msvistalog
	<QueryXML>	

		[ SOME EVENT IDs FROM WINDOWS SERVER ]

	</QueryXML>
</Input>

<Output graylog_gelf>
    Module      om_tcp
    Host        192.168.1.21
    Port        10517
	OutputType	GELF_TCP
</Output>

<Route 1>
    Path in => graylog_gelf
</Route>

Errors

  • GELF Input for NXLog on 10517
    No active connection, but lots of connections in total. For 1 server.
Throughput / Metrics
1 minute average rate: 0 msg/s
Network IO: 0B 0B (total: 1.7MiB 0B )
Active connections: 0 (669 total)
Empty messages discarded: 0
  • Syslog TCP Input on 10514
    No active connection, but lots of connections in total. For 8 servers.
Throughput / Metrics
1 minute average rate: 0 msg/s
Network IO: 0B 0B (total: 55.2MiB 0B )
Active connections: 0 (2942 total)
Empty messages discarded: 0
  • /var/log/syslog
rsyslogd: omfwd: TCPSendBuf error -2027, destruct TCP Connection to 10.100.11.21:10514 [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsyslog.com/e/2007 ]
rsyslogd: action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
rsyslogd: omfwd: TCPSendBuf error -2027, destruct TCP Connection to 10.100.11.21:10514 [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsyslog.com/e/2007 ]
rsyslogd: action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]

Operating system information

  • Debian 10

Package versions

  • Graylog = 4.1.2+20cd592, codename Noir
  • MongoDB = version v5.0.1
  • Elasticsearch = “number” : “7.10.2”, “build_flavor” : "oss"
  • Service logs, configuration, and environment variables
  • See the docs site for all file locations

Thanks to all of those who took some of their time to read me !
It’s the end of the day, and I’m pretty tired. I hope I hope my explanations are clear !

See you !

Hello Again,

I’ve been looking over this post but haven’t found a solution yet. I’m focusing on just NXlog first.
I don’t see anything really wrong with you Nxlog configuration file. You did mention that if you use other protocols like UDP, etc… as an INPUT it works fine, you just having troubles with GELF/TCP input, am I correct?

I want to ask some questions.

  • What does you NXlog internal log file show, since you using TCP maybe it will show more about any connection issue.

  • Have you tried to enable debugging on NXlog client?

  • Are you just masking the URL? I assume it was supposed to be something like this.
proxy_pass http://192.168.1.21:9000
  • Is your Graylog server using HTTPS?

  • How did you configure your INPUT?

  • Is your firewall enabled?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.