One out of three receiver hosts showed 0 connections

Hi guys, just wanted to share something I’ve run into today.

My Graylog cluster consists of four boxen: one serves as the GUI frontend, the other three are spread across the network, to receive data from logging agents. I’ve kept an eye on System > Nodes and System > Inputs for a few days now and noticed that one out of the three receivers consistently had “Active Connections: 0 (0 total)”. That struck me as odd, because:

  • The cluster has been running for more than a week.
  • The collector configurations are set to “load balancing: true”, with all three receiver hosts listed as BEATS receivers.
  • No errors were being shown for the receivers, or the nodes.

There was no indication that there was something really wrong… That is, until I turned to /var/log/graylog-server/server.log on the host without incoming connections. This logfile was consistently spamming Java errors:

WARN [AbstractNioSelector] Failed to initialize an accepted socket. extra data given to DerValue constructor
  at java.crypto.EncryptePrivateKeyInfo.<init>(
  at org.graylog2.plugin.inputs.transports.util.KeyUtil.createKeySpec (...)
  at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadPrivateKey(...)

Suggesting there was an issue with the TLS private key for that particular BEATS receiver.

Going over the Input configuration again, I noticed the following which was the cause of the problems:

TLS Cert File: /etc/graylog/server/node1.crt
TLS Private Key File: /etc/graylog/server/node1.crt

While setting up the input, I had inadvertently copy/pasted the certificate filename into the box for the private key! Changing it to the following fixed it.

TLS Private Key File: /etc/graylog/server/node1.pem

I would have expected an error to show up in the GUI, warning me that the input wasn’t running. But I’m glad I found the cause after all. I hope this post may be helpful to others who run into the same situation :blush:.

you might want to open a feature request in Github that such a setting should be checked.

ACK, will get on that as soon as possible.

Done :slight_smile:

Thanks for your help!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.