Collecting log data from HAProxy

I have set up an input in Graylog that is supposed to collect log data from a HAProxy machine via rsyslog.

Have successfully set up similar log inputs in Graylog for gathering logs from other services such as Apache and MariaDB, but specifically for HAProxy, I see the proxy machine going haywire when I enable the rsyslog config for sending logs to Graylog over UDP.

By ‘haywire’ what I mean is that the proxy server starts rapidly writing lines such as the following, to /var/log/haproxy.log file:

Proxy01 haproxy Nov 18 16:30:59 Proxy01 haproxy Nov 18 16:30:59 ...

Within a short while, the root partition had filled up due to this particular log file having grown to several gigabytes in size.

On the HAProxy, I have these two config files in /etc/rsyslog.d/ where the first one:

/etc/rsyslog.d/10-graylog_haproxy.conf (custom rsyslog config created by me, for forwarding HAProxy logs to Graylog):

*.*@192.168.5140;RSYSLOG_SyslogProtocol23Format

module(load="imfile" PollingInterval="10")

# File 1
input(type="imfile"
      File="/var/log/haproxy.log"
      Tag="haproxy"
      Severity="info")

if $programname == 'haproxy' then @192.168.:5143

/etc/rsyslog.d/49-haproxy.conf (native rsyslog config created by haproxy installer):

# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
  /var/log/haproxy.log
  stop
}

I must admit I have not tried much in terms of fixing the problem, since I don’t really know the cause and am a newbie when it comes to using rsyslog as well as Graylog (although I believe the problem may be with the former).
I have however looked at the graylog server log, but didn’t find anything that I could relate to this issue.
Also had a look at rsyslog service journal on the HAProxy box, and there I just see a whole bunch of entries like this one:

Nov 18 14:22:11 Proxy01 rsyslogd[8446]: message too long (8110) with configured size 8096, begin of message is: Nov 18 14:22:10 DK

I am on the lastest version of Graylog and Elasticsearch (OSS) and both the Graylog server and the HAProxy machines are running Debian 10.

Any help will be greatly appreciated :slight_smile:

Hello,
I feel your pain with Rsyslog, I actually used it a long time ago and had to drop it in place of something easier to configure w/o issues. Have you considered using Graylog-Sidecar or another log shipper like nxlog or FileBeat?

Not sure all what’s going on but here is my rsyslog.conf file with an example.

[root@nextcloud-web1 html]# cat  /etc/rsyslog.conf | egrep -v "^\s*(#|$)"
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
kern.debug /var/log/firewall.log
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
# *.* @@8.8.8.8:51411
*.* @8.8.8.8:51411 <--- UDP connection
[root@nextcloud-web1 html]#

Example of the difference /w the @ symbol and the @@ symbol

*.* @@8.8.8.8:51411 ---> TCP connection
*.* @8.8.8.8:51411  ---> UDP connection

I create a syslog UDP input using port 51411.

If the message size is to long (Think default is 1024 bytes) maybe something like this below.

$MaxMessageSize (some_number)k

must be set before network declaration (at the top of rsyslog.conf file) :

$ModLoad imtcp
$InputTCPServerRun 5143

Well the thing is I think setting up rsyslog has been very simple (as soon as I figured out the right starting point).

I’ve read about Sidecar and Filebeat but these approaches seem more complicated to me - and not native to Linux like rsyslog is…

As such, I would really like to get log collection working via rsyslog, and am pretty close I think, since apart from HAProxy logs only 1 or 2 additional application logs are missing from my Graylog installation at this point.

It just has me a bit worried that the rsyslog config I set up for HAProxy managed to practically kill the proxy server by filling the haproxy.log with huge amounts of garbage data in a matter of minutes/hours.
I suppose it makes one think if relying on rsyslog for collecting all logs is simply not sufficiently robust.

I have said the same thing to my boss, I wanted to use something native with Linux and not a third party application. As we progress in log management I realized Rsyslog was unable to perform some request we need and if Rsyslog was able to do what we wanted it was a pain in the @$$ to configure.
I did find some documentation about audit logs with Rsyslog. Maybe something there might help.

https://docs.graylog.org/v1/docs/syslog

https://www.thegeekdiary.com/how-to-send-audit-logs-to-remote-rsyslog-server-in-centos-rhel-67/

Sorry I cant give you a direct answer maybe someone here has done this before.

Apparently what I did with having two configs for haproxy logging to rsyslog was causing some sort of loop. So I got rid of the /etc/rsyslog/rsyslog.d/10-graylog_haproxy.conf and instead added the line “.@192.168.30.6:5143” into the default / already existing 49-haproxy.conf in same directory:

Create an additional socket in haproxy’s chroot in order to allow logging via

/dev/log to chroot’ed HAProxy processes

$AddUnixListenSocket /var/lib/haproxy/dev/log

Send HAProxy messages to a dedicated logfile

:programname, startswith, “haproxy” {
/var/log/haproxy.log
.@192.168.30.6:5143
stop
}

In doing that, I now have HAProxy logs coming into Graylog and no crazyness happening in the haproxy.log file. Case closed :slight_smile:

Nice, Thanks for posting your resolution to this issue.