To add further Detail i’m running a 3 host Graylog Cluster on OpenSUSE 15.2 running GrayLog 4.1 and ElasticSearch 7.10.
The GrayLog Cluster is sitting behind a Loadbalancer running Haproxy 2.4.0 with the following config configured to loadbalance syslog for VMWare ESXi (Firewalld is configured to forward port 514 from the source hosts to port 5514 on the loadbalancer):
log-forward graylog-vmware-lb
bind *:5514 transparent
dgram-bind *:5514 transparent
log ipv4-host1:5514 format rfc3164 sample 1:4 local0 info
log ipv4-host2:5514 format rfc3164 sample 2:4 local0 info
log ipv4-host3:5514 format rfc3164 sample 3:4 local0 info
A VCenter Appliance is using the same input with no issue.
I have similar outputs for udp syslog from Linux Servers and general network infrastructure which work fine, it’s only VMWare ESXi Hosts which are showing the Source as “localhost”
Any guidance would be much appreciated as i’m still new to centralised logging.
I tried RFC 5424 first but didn’t even get the localhost entry, the source for all ESXi logs was coming in as “-”. I tried RFC 3164 after seeing this article from VMWare ESXi Log Message Formats | VMware and wondered if that might have been why i wasn’t getting a source address, and whilst it did improve it’s still not working as expected.
I have no idea why there are two timestamps coming in on the log messages from ESXi, i followed the instructions on the VMWare Knowledge Base to set up syslog as in this article: Configure Syslog on ESXi Hosts. I have not made any custom changes that should affect the format.
Apart from the VMWare specific syslog configuration, the inputs and loadbalancing are configured exactly the same as used for my other UDP Syslog traffic which doesn’t have this issue.
I finally found the issue to be caused by my HAProxy Configuration changing the log format.
After removing the Syslog Format from the log config, it now looks like this and my logs are now appearing in Graylog as expected:
log-forward graylog-vmware-lb bind *:5514 transparent dgram-bind *:5514 transparent log ipv4-host1:5514 sample 1:3 local0 info log ipv4-host2:5514 sample 2:3 local0 info log ipv4-host3:5514 sample 3:3 local0 info
The issue with the double timezone was caused by ESXi 7 only supporting UTC Timezone and the HAProxy Loadbalancer was reformatting the log messages as EDT which caused the issue to occur.