Ideas for Syslog collection without losing events

Currently we are using rsyslog to forward events coming into a server through syslog UDP 514. We use something similar to the following format into a syslog input on the graylog server:

*.* @10.10.10.10:514

On occasion we need to perform maintenance on both the graylog server and the server that is forwarding syslog. What are some ideas that we can put in place that will minimize the event loss when either device is ungoing maintenance?

I was thinking about putting a redis or filebeat service on the syslog forwarder, however I wasn’t sure if that could accept syslog or needed to tail a file. Ideally we wouldn’t have rsyslog write everything to a file to save space on the server, however this could be done with decent logrotation.

Hey @karlt,

by nature, UDP are fire and forget packets. This makes them lightweight and fast, but they will just be dropped if they don’t arrive and the sender won’t be notified of this. I would recommend using TCP-Syslog between your forwarder and Graylog. Rsyslog can buffer messages if a TCP-Syslog connection cannot be established. This would make the connection between your forwarder and Graylog more reliable.

But there is nothing you could really do if you would have to take down the rsyslog server. The only thing I can think of is using a second server for the duration of the downtime. We for example use a dedicated load-balancer in front of two Graylog servers. If one is down for maintenance or some error, the other one will take over / the full load.

Greetings - Phil

If you’re not afraid to run the additional infrastructure, you can ship syslog to Graylog via Apache Kafka, AMQP (e. g. RabbitMQ), or ZeroMQ:

2 Likes

I wrote up a how-to awhile ago that details setting up HA with pacemaker on CentOS 7 for rsyslog.

http://alias454.com/setting-up-a-multi-tiered-log-infrastructure-part-9/
http://alias454.com/setting-up-a-multi-tiered-log-infrastructure-part-10/

Basically, you setup an HA pair running rsyslog that feeds into an rsyslog backend using tcp-syslog,

Take a look and let me know if you have any specific questions.
Regards,
Brandon

1 Like