So pulling data from a VMware VCSA directly into Graylog. The input is “Syslog TCP”, similar settings on the VCSA appliance. Logs are coming in fine however, numerous ones are being fragmented into multiple broken events.
I’m pretty certain this is being caused by the “new entry on new line” code for Graylog, which I find odd since the Syslog TCP protocol (SyslogProtocol23Format) begins every new line with a <syslog ID>. I’ve validated this via Wireshark, where the single event was on a single line in Wireshark but when copied out of it I can visibly see the line break. This shows up as an 0a in hex (linefeed character).
<14>1 2019-04-26T13:18:12.119920-05:00 XXXX vpxd 4670 - - Event [111883040] [1-1] [2019-04-26T18:18:12.119587Z] [vim.event.PermissionUpdatedEvent] [info] [VSPHERE.LOCAL\XXXX] [] [111883040] [Permission changed for 'XXXX' on 'Datacenters'.
Role changed from 'Read-only' to role 'Administrator'. Propagate changed from 'Enabled' to 'Enabled'.]
The documents say that properly formatted rsyslog entries should work flawlessly on Graylog. Am I missing something here to prevent these entries from being cut up? Any easy answers out there that doesn’t involve a man in the middle? Can a pipe strip linefeeds before it reaches the stream?
So before this topic auto-closes. I wasn’t able to solve this and ultimately we began evaluating vRealize Log Insight in place of Graylog due to this problem. I haven’t fully ruled out Graylog yet, but malformed event entries is a huge show stopper and I’ve slowed my troubleshooting attempts at resolving this.
In the syslog RFC it says a log ends with a line break that’s why your log is split.
You need to strip linefeed before the log reaches Graylog.
For example you could use SEC (https://simple-evcorr.github.io/).
This document does not impose any mandatory restrictions on the MSG or PARAM-VALUE content. As such, they MAY contain control characters, including the NUL character.
You are right it’s not in the RFC but it’s a standard.
The RFC does not precise how to handle line feed and does not precise how to end a MSG.
Rsyslog’s doc says: Industry-strandard plain text tcp syslog uses the LF to delimit syslog frames. However, some users brought up the case that it may be useful to define a different delimiter and totally disable LF as a delimiter (the use case named were multi-line messages). This mode is non-standard and will probably come with a lot of problems. However, as there is need for it and it is relatively easy to support, we do so. Be sure to turn this setting to “on” only if you exactly know what you are doing. You may run into all sorts of troubles, so be prepared to wrangle with that!
You need to configure Rsyslog to send a one-line log to Graylog.
What’s your Rsyslog input ? file ? network ? Can you share your configuration ?
Not worth troubleshooting my config at this point frantz. I just have a VMware VCSA (vCenter Server Appliance) pointing it’s built in appliance-based syslog (PhotonOS) towards the syslog server.
When I switched from Graylog to another suite, the messages that were previously broken by new-line control codes are retained. However, I responded to your message because I found myself reading the RFC on an odd bit of time.
Any of the information I placed here is because I actually enjoyed the product of Graylog. But the new-line focus instead of a focus on properly formatted Syslog headers (which all begin with ) ruined the product for me and added an unnecessary layer of troubleshooting. I’m just not going down the path of setting up a middle-man server to reformat everything. And I think that feedback may or may not be useful to the developers.
If you are curious, the config was very simple. Network based Rsyslog input:
