Collecting log data from HAProxy

[foss4ever]

I have set up an input in Graylog that is supposed to collect log data from a HAProxy machine via rsyslog.

Have successfully set up similar log inputs in Graylog for gathering logs from other services such as Apache and MariaDB, but specifically for HAProxy, I see the proxy machine going haywire when I enable the rsyslog config for sending logs to Graylog over UDP.

By ‘haywire’ what I mean is that the proxy server starts rapidly writing lines such as the following, to /var/log/haproxy.log file:

Proxy01 haproxy Nov 18 16:30:59 Proxy01 haproxy Nov 18 16:30:59 ...

Within a short while, the root partition had filled up due to this particular log file having grown to several gigabytes in size.

On the HAProxy, I have these two config files in /etc/rsyslog.d/ where the first one:

/etc/rsyslog.d/10-graylog_haproxy.conf (custom rsyslog config created by me, for forwarding HAProxy logs to Graylog):

*.*@192.168.5140;RSYSLOG_SyslogProtocol23Format

module(load="imfile" PollingInterval="10")

# File 1
input(type="imfile"
      File="/var/log/haproxy.log"
      Tag="haproxy"
      Severity="info")

if $programname == 'haproxy' then @192.168.:5143

/etc/rsyslog.d/49-haproxy.conf (native rsyslog config created by haproxy installer):

# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
  /var/log/haproxy.log
  stop
}

I must admit I have not tried much in terms of fixing the problem, since I don’t really know the cause and am a newbie when it comes to using rsyslog as well as Graylog (although I believe the problem may be with the former).
I have however looked at the graylog server log, but didn’t find anything that I could relate to this issue.
Also had a look at rsyslog service journal on the HAProxy box, and there I just see a whole bunch of entries like this one:

Nov 18 14:22:11 Proxy01 rsyslogd[8446]: message too long (8110) with configured size 8096, begin of message is: Nov 18 14:22:10 DK

I am on the lastest version of Graylog and Elasticsearch (OSS) and both the Graylog server and the HAProxy machines are running Debian 10.

Any help will be greatly appreciated

[gsmith]

Hello,
I feel your pain with Rsyslog, I actually used it a long time ago and had to drop it in place of something easier to configure w/o issues. Have you considered using Graylog-Sidecar or another log shipper like nxlog or FileBeat?

Not sure all what’s going on but here is my rsyslog.conf file with an example.

[root@nextcloud-web1 html]# cat  /etc/rsyslog.conf | egrep -v "^\s*(#|$)"
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
kern.debug /var/log/firewall.log
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
# *.* @@8.8.8.8:51411
*.* @8.8.8.8:51411 <--- UDP connection
[root@nextcloud-web1 html]#

Example of the difference /w the @ symbol and the @@ symbol

*.* @@8.8.8.8:51411 ---> TCP connection
*.* @8.8.8.8:51411  ---> UDP connection

I create a syslog UDP input using port 51411.

If the message size is to long (Think default is 1024 bytes) maybe something like this below.

$MaxMessageSize (some_number)k

must be set before network declaration (at the top of rsyslog.conf file) :

$ModLoad imtcp
$InputTCPServerRun 5143

[foss4ever]

Well the thing is I think setting up rsyslog has been very simple (as soon as I figured out the right starting point).

I’ve read about Sidecar and Filebeat but these approaches seem more complicated to me - and not native to Linux like rsyslog is…

As such, I would really like to get log collection working via rsyslog, and am pretty close I think, since apart from HAProxy logs only 1 or 2 additional application logs are missing from my Graylog installation at this point.

It just has me a bit worried that the rsyslog config I set up for HAProxy managed to practically kill the proxy server by filling the haproxy.log with huge amounts of garbage data in a matter of minutes/hours.
I suppose it makes one think if relying on rsyslog for collecting all logs is simply not sufficiently robust.

[gsmith]

@foss4ever

I’ve read about Sidecar and Filebeat but these approaches seem more complicated to me - and not native to Linux like rsyslog is…

I have said the same thing to my boss, I wanted to use something native with Linux and not a third party application. As we progress in log management I realized Rsyslog was unable to perform some request we need and if Rsyslog was able to do what we wanted it was a pain in the @$$ to configure.
I did find some documentation about audit logs with Rsyslog. Maybe something there might help.

Graylog support for auditd in syslogs

Hi All, Are there any resources to support auditd logging within graylog with Centos hosts to parse out the fields correctly. I have tried enabling auditd logging via syslog to a udp syslog input as shown below but the fields vary and are not parsed corecttly. [linux-audit] Would anothe roption be to use something like NXLog imfile module to read the file and then send it to a GELF input. Would this work as the json should contain the fields. Alternatively, would it be better to use a GROK…

https://docs.graylog.org/v1/docs/syslog

Sending data from Solaris using rsyslog

Hello, I want to send user audit data (logins, su and sudo usage) from our Solaris 11.4 systems to Graylog. So I installed and configured rsyslog 8.2002.0. This is working in that way, that all information is logged to log files as configured. But not all information written to logs is send to Graylog. For example there are a log of auth.info entires in /var/log/auth.log, but they are not send to Graylog. I get messages with authpriv.notice and kern.warnings. I read docs, tried a lot things y…

How to send Audit Logs to Remote Rsyslog Server in CentOS/RHEL 6,7 – The Geek...

Sorry I cant give you a direct answer maybe someone here has done this before.

[foss4ever]

Apparently what I did with having two configs for haproxy logging to rsyslog was causing some sort of loop. So I got rid of the /etc/rsyslog/rsyslog.d/10-graylog_haproxy.conf and instead added the line “.@192.168.30.6:5143” into the default / already existing 49-haproxy.conf in same directory:

Create an additional socket in haproxy’s chroot in order to allow logging via

/dev/log to chroot’ed HAProxy processes

$AddUnixListenSocket /var/lib/haproxy/dev/log

Send HAProxy messages to a dedicated logfile

:programname, startswith, “haproxy” {
/var/log/haproxy.log
.@192.168.30.6:5143
stop
}

In doing that, I now have HAProxy logs coming into Graylog and no crazyness happening in the haproxy.log file. Case closed

[gsmith]

Nice, Thanks for posting your resolution to this issue.