ESXi 7 Syslog Format incorrect

Hi,

I’ve configured a UDP Syslog input on port 5514 and it is receiving messages, however messages from ESXi Hosts are coming in, in the following format:

_VMware_EventID
2113595
facility
user-level
facility_num
1
full_message
<12>Jul 30 06:54:39 localhost 2021-07-30T10:54:39.137Z FQDN VSANMGMTSVC: info vsand[2113595] [opID=Thread-2 VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
level
4
message
[opID=Thread-2 VsanSystemImpl::ConfigInfoPrintLog] configInfo: skipped
source
localhost
timestamp
2021-07-30 11:54:39.000 +01:00

Is there a Pipeline rule or similar i can use to force it to use the FQDN contained within the message instead of where it’s input localhost?

To add further Detail i’m running a 3 host Graylog Cluster on OpenSUSE 15.2 running GrayLog 4.1 and ElasticSearch 7.10.

The GrayLog Cluster is sitting behind a Loadbalancer running Haproxy 2.4.0 with the following config configured to loadbalance syslog for VMWare ESXi (Firewalld is configured to forward port 514 from the source hosts to port 5514 on the loadbalancer):

log-forward graylog-vmware-lb
bind *:5514 transparent
dgram-bind *:5514 transparent
log ipv4-host1:5514 format rfc3164 sample 1:4 local0 info
log ipv4-host2:5514 format rfc3164 sample 2:4 local0 info
log ipv4-host3:5514 format rfc3164 sample 3:4 local0 info

A VCenter Appliance is using the same input with no issue.

I have similar outputs for udp syslog from Linux Servers and general network infrastructure which work fine, it’s only VMWare ESXi Hosts which are showing the Source as “localhost”

Any guidance would be much appreciated as i’m still new to centralised logging.

Maybe ESXi 7.0 uses RFC 5424 format and you try to convert it to RFC 3164. Because I don’t see right reason, why there are two timestamps in message?

I tried RFC 5424 first but didn’t even get the localhost entry, the source for all ESXi logs was coming in as “-”. I tried RFC 3164 after seeing this article from VMWare ESXi Log Message Formats | VMware and wondered if that might have been why i wasn’t getting a source address, and whilst it did improve it’s still not working as expected.

I have no idea why there are two timestamps coming in on the log messages from ESXi, i followed the instructions on the VMWare Knowledge Base to set up syslog as in this article: Configure Syslog on ESXi Hosts. I have not made any custom changes that should affect the format.

Apart from the VMWare specific syslog configuration, the inputs and loadbalancing are configured exactly the same as used for my other UDP Syslog traffic which doesn’t have this issue.

I finally found the issue to be caused by my HAProxy Configuration changing the log format.

After removing the Syslog Format from the log config, it now looks like this and my logs are now appearing in Graylog as expected:

log-forward graylog-vmware-lb
bind *:5514 transparent
dgram-bind *:5514 transparent
log ipv4-host1:5514 sample 1:3 local0 info
log ipv4-host2:5514 sample 2:3 local0 info
log ipv4-host3:5514 sample 3:3 local0 info

The issue with the double timezone was caused by ESXi 7 only supporting UTC Timezone and the HAProxy Loadbalancer was reformatting the log messages as EDT which caused the issue to occur.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.