Hi guys, just wanted to share something I’ve run into today.
My Graylog cluster consists of four boxen: one serves as the GUI frontend, the other three are spread across the network, to receive data from logging agents. I’ve kept an eye on System > Nodes and System > Inputs for a few days now and noticed that one out of the three receivers consistently had “Active Connections: 0 (0 total)”. That struck me as odd, because:
- The cluster has been running for more than a week.
- The collector configurations are set to “load balancing: true”, with all three receiver hosts listed as BEATS receivers.
- No errors were being shown for the receivers, or the nodes.
There was no indication that there was something really wrong… That is, until I turned to /var/log/graylog-server/server.log on the host without incoming connections. This logfile was consistently spamming Java errors:
WARN [AbstractNioSelector] Failed to initialize an accepted socket.
java.io.IOException: extra data given to DerValue constructor
at sun.security.util.DerValue.init(DerValue.java:384)
at sun.security.util.DerValue.<init>(DerValue.java:294)
at java.crypto.EncryptePrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:84)
at org.graylog2.plugin.inputs.transports.util.KeyUtil.createKeySpec (...)
at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadPrivateKey(...)
Suggesting there was an issue with the TLS private key for that particular BEATS receiver.
Going over the Input configuration again, I noticed the following which was the cause of the problems:
TLS Cert File: /etc/graylog/server/node1.crt
TLS Private Key File: /etc/graylog/server/node1.crt
While setting up the input, I had inadvertently copy/pasted the certificate filename into the box for the private key! Changing it to the following fixed it.
TLS Private Key File: /etc/graylog/server/node1.pem
I would have expected an error to show up in the GUI, warning me that the input wasn’t running. But I’m glad I found the cause after all. I hope this post may be helpful to others who run into the same situation 
.
