UDP Messages Not Getting Logged

I’m trying to send GELF messages to the UDP input on my Graylog server, using a custom C++ library that I wrote. Some of the messages are being silently ignored. Is there a log somewhere that I can look at, which will tell me if the packets are being received, and if so, why they are being dropped? I tried tailing the log output of the docker container, but nothing relating to my messages appears there.

Alternately, is there a standardized tool available to validate a GELF message, to determine what about it, if anything, might be broken?

He @elkvis

UDP messages might be dropped - that can happens because of UDP.

If some messages make it into Graylog and some not, you might want to debug the system itself. Check for RX Errors on the interface, or similar debugging.

The thing that seems to be more-or-less guaranteed to elicit a failure is including a stack trace in my GELF message. If I have a _stackTrace field set to something meaningless like “Stack Trace,” the message is logged reliably every time. If that field contains an actual stack trace, it fails almost every single time. I’ve tried URL-encoding the stack trace, base-64 encoding it, lots of other things, and If there is an actual stack trace present, it fails. This is why I was hoping for a message validator of some sort, to see what’s actually broken. The message length seems to be irrelevant. If I log a very long message, it also works, as long as there’s no stack trace.

I guess the stack trace is a multi line event? that might be the reason.

The stack trace is multi-line, but newlines are being properly encoded with \r and \n as necessary. That doesn’t make sense. The official GELF documentation gives an actual example of a valid message with \n in one of the fields.

Well, newlines appear to be the problem after all. The absurd part of it is that my full_message field contains \n characters, and gets logged without issue. If I have a newline in an additional field, with a ‘_’ prefix, such as my _stackTrace field, even when properly encoded, the whole message gets rejected, with no diagnostic information logged anywhere. There is no mention of this behavior anywhere in the GELF format documentation :rage:

//known issue, GL drops UDP under java’s GC process
If you can avoid to use UDP.

If it was a GC problem, I would expect it to be much more random. I can consistently make it happen if I include a newline character in the value of an additional field.