1. Describe your incident:
Windows GELF Input not working - zlib compressed stream
I have an existing Graylog running version 4.2.4 and stood up a new box to take its place running 5.1.4. I used all the same input settings as the old graylog but I am not seeing any windows logs coming through using UDP GELF using the new 5.1.4 host.
2. Describe your environment:
OS Information:
Ubuntu 20.04
16GB, 4 core, 100GB disk
Package Version:
Graylog 5.14
Service logs, configurations, and environment variables:
GELF Input Configuration
bind_address: 0.0.0.0
port: 5414
rec_buffer_size: 262144
decompress_size_limit: 8388608
charset_name:
3. What steps have you already taken to try and solve the problem?
I have captured logs on port 5414 and verified they were zlib compressed stream with the magic header value of “78 9c” which means default compression. I also took the stream in cyberchef and validated that using zlib to decompress, it resulted in the correct json log format.
The documentation states “Graylog nodes detect the compression type in the GELF magic byte header automatically.”, so I would assume it should just work since the magic byte header is correct.
I have tried re-creating it and tried putting in different charsets (although from the cyberchef I know it is UTF-8).
This same exact stream works on my 4.2.4 host but not on the 5.1.4 instance.
Looked through the graylog server log and there aren’t any errors that I could find.
4. How can the community help?
Has anyone successfully used zlib compressed GELF input streams in 5.1.4?
Is there another setting I happened to overlook?
Is there a way to tell Graylog it is a compressed stream?
Can you clarify what the issue is you are having or if you are getting any errors? I’m having trouble following along with what your issue is specifically.
What is not working? (or rather what is happening vs what you expect to happen)
how are you sending data to grayog? (nxlog? which version? what is your config look like?)
Sorry for not being clear and thanks for the reply.
I currently have a Windows gelf input collection working and collecting fine running graylog version 4.2.4. I powered off the old graylog host and stood up a new box using graylog 5.1.4 and configured the windows gelf input collector on it and it isn’t showing any logs or any collection errors, it’s just a blank stream. I have other syslog streams also going to the host and they are working correctly and showing data. I have captured the network traffic on the correct port for the windows gelf logs going to the box, so I know they should be there for graylog to process.
The sending host is using an older version of nxlog (2.9.1716) and the configuration is sending logs correctly to the old version of graylog so I assume it should be the same format the new version of graylog is expecting. Unless there was a change in the GELF spec that would cause an issue.
The nxlog conf is using the xm_gelf extension and an output type of GELF_UDP and the om_udp module.
That is very interesting. I’ve had a lot of weird quicks and issues with nxlog that i eventually abandoned it for beats. One of my favorite features of beats is it has a built in back log so it loses connection to graylog it can store those messages and send them later.
It does look like there have been a lot of improvements and fixes with nxlog though, especially with version 3. My recommendation is to try updating to a newer version of nxlog to see if that helps (i believe the 2.x version of nxlog had a bug where it could not send GELF TCP which has been fixed in 3.x). The other thing i would try is using GELF TCP (dependent upon upgrading to nxlog 3.x though)
if upgrading nxlog does not work, my next step would be to look at the nxlog config and see if anything looks interesting there.
