IO Netty Handler ssl.NotSslRecordException Not an SSL/TLS Record


Thought I share something with you all.

A strange issue occurred today after running updates a remote client and updating Graylog Server to 4.2.2.
The remote client has Nxlog installed, and its configurations are GELF TCP/TLS. I use it for demonstrations and testing log shippers. It’s also used for Cloud storage (NextCloud). Nothing real important but I do have files and testing certificates on it. Basically, it my junk drawer.

My pre-flight instruction mainly consists of creating a check point for these Virtual Machines before the upgrade/updates occur. I executed updates on the remote client then checked logs right after. Didn’t find anything suspicious. Checked Graylog Web UI and found the messages were being received from my remote client. So, I thought I was all good.

Next, applied updates to Graylog server and updated my kernel. This would mean I had to reboot Graylog, and again I did my pre-flight checks.
Once Graylog was reboot I always start Tail’ing my graylog log file. Old habits, but good ones.

tail -f /var/log/graylog-server/server.log

In the beginning of the log file no problems were noticed. Once all the inputs were started, I noticed the following error. Not just one but a lot. To give you an idea my log file was almost 56 MB in a 20+ minutes…

2021-12-03T20:14:03.707-06:00 ERROR [AbstractTcpTransport] Error in Input [GELF TCP/5e265ada83d72ec570ab5fe2] (channel [id: 0x232a06ea, L:/ ! R:/]) 
(cause io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3c343e4465632020332032303a31343a3033206e657874636c6f75642d77656231206b65726e656c3a2044524f5020494e5055543a20494e3d65746830204f55543d2

First thing I did was research the following error (cause io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record)

Google led me to my certificates for GELF TCP/TLS. So, I replaced them on the remote client with ones that worked in my environment. Unfortunately, after restarting nxlog the issue was still there.

I was digging into my nxlog files and the only logs that were shown were “The connection was successful”. By this time my Graylog log file was getting bigger, so I stopped nxlog on the remote client.

To my dismay, I was still receiving error/s in my logs from the remote client.

:thinking: This is peculiar I stopped my log shipper, and my remote client is still sent logs.

So now I knew my Graylog server was not at fault, so there must be some dark magic lurking around.
I did a sweep on my remote client looking for FileBeat, Graylog Sidecar, etc… Nothing was found. I even shut down my remote server and the log message/ errors in Graylog Log file stopped.

:thinking: :thinking: Then I remembered there is rsyslog (you dirty bastard). So I did a status check

systemctl status rsyslog

Well, Well, Well… It’s on and running.
So, the uptime on this machine was a year + and the original configuration were still there pointing to the GELF TCP/TLS port.

The moral of the story is, the error
(cause io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record )
is now defined in my documentation as Graylog received messages in a input from a client in the wrong format.
I just lost 5 hours of my life. Hope this helps someone else.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.