I am trying to get TLS encryption of Sidecar/Beats communication working in Graylog but despite trying many suggestions from the web and Graylog community forum I simply cannot get any messages coming in on the Beats input when I attempt to enable TLS on it.
OS & package Information:
Sidecar collector config used:
Troubleshooting steps taken:
Have tried entering the path for my certificates, both into the input and sidecar collector config, but cannot get any messages coming in on the input.
As for putting “ssl” directives into the sidecar collector config, I have questions about that - specifically what to enter for these three lines:
output.logstash.ssl.certificate_authorities output.logstash.ssl.certificate output.logstash.ssl.key
First, I simply don’t know what should actually be entered for ssl.certificate_authorities. Do I even need to declare that when I’m not needing actual authentication of the Beats input?
Secondly - for the ssl.certificate and ssl.key do I just copy the same cert and key file that I’m using for TLS on the Graylog server itself (/etc/graylog/server/cert.pem, /etc/graylog/server/pkcs8-encrypted.pem) to the client that is to send logs to Grayog and then expect that to work?
I did try doing that:
…while also omitting the ssl.certificate_authorities directive, but that only results in the client being unable to parse the pkcs8-encrypted.pem key (assumingly because it has not been told the passphrase for the key file) as can be seen when tailing the sidecar.log file on the client:
time="2022-01-13T10:52:39+01:00" level=error msg="[filebeat] Collector output: Exiting: error initializing publisher: 1 error: tls: failed to parse private key\n"
So I guess the client would need to be told about the passphrase for that key file via the sidecar collector config, but I haven’t been able to find any info on how to do that
I also tried providing my pkcs8-plain.pem cert instead of the encrypted one, but while that gives no errors in the sidecar.log client-side, it does produce these errors in the Graylog server log:
2022-01-13T10:57:48.016+01:00 ERROR [AbstractTcpTransport] Error in Input [Beats/61dedecb00219622cef00a65] (channel [id: 0xdf977c3f, L:/192.168.30.6:5140 ! R:/10.16.0.250:47778]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE)
Which sort of tell me that this is not the right direction…?
Here is my Beats input config (where I did also try and enter cert paths and password as mentioned above):
How can the community help?
What has me a bit frustrated is reading this guidance from official Graylog docs on securing Sidecar/Beats communication:
The Communication between Sidecar and Graylog will be secured if your API uses SSL .
To secure the communication between the Collector and Graylog you just need to mark
in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input.Now in the Sidecar Beats Output Configuration you just mark
Enable TLS Support and
Insecure TLS connection . After this is saved, the communication between Beats and Graylog will use TLS.
So I can follow the above steps until this part:
Now in the Sidecar Beats Output Configuration you just mark
Enable TLS Support and
Insecure TLS connection
My issue here is that I simply cannot find anywhere in the Graylog GUI where I could enable TLS support and Insecure TLS connection on the Sidecar collector config! Is this an enterprise-only feature, or what am I missing here??
At this point it’s tempting to just abandon this TLS security stuff, since I’ve spent far too much time on it already, but I feel like I’m quite close since I now have the web interface as well as the graylog-sidecar API using TLS encryption and thus only encrypting the actual log data being sent into Graylog from the client by graylog-sidecar is missing.
I have some idea that my problem might lie with the certificates and my lack of understanding of those…
Any help would be greatly appreciated.