Securing Sidecar/beats communication

[foss4ever]

SUMMARY

I am trying to get TLS encryption of Sidecar/Beats communication working in Graylog but despite trying many suggestions from the web and Graylog community forum I simply cannot get any messages coming in on the Beats input when I attempt to enable TLS on it.

OS & package Information:

Debian 10
Graylog 4.2.4+b643d2b
graylog-sidecar 1.1.0-1
filebeat 7.16.2

Sidecar collector config used:

Troubleshooting steps taken:

Have tried entering the path for my certificates, both into the input and sidecar collector config, but cannot get any messages coming in on the input.

As for putting “ssl” directives into the sidecar collector config, I have questions about that - specifically what to enter for these three lines:

output.logstash.ssl.certificate_authorities
output.logstash.ssl.certificate
output.logstash.ssl.key

First, I simply don’t know what should actually be entered for ssl.certificate_authorities. Do I even need to declare that when I’m not needing actual authentication of the Beats input?

Secondly - for the ssl.certificate and ssl.key do I just copy the same cert and key file that I’m using for TLS on the Graylog server itself (/etc/graylog/server/cert.pem, /etc/graylog/server/pkcs8-encrypted.pem) to the client that is to send logs to Grayog and then expect that to work?

I did try doing that:

…while also omitting the ssl.certificate_authorities directive, but that only results in the client being unable to parse the pkcs8-encrypted.pem key (assumingly because it has not been told the passphrase for the key file) as can be seen when tailing the sidecar.log file on the client:

time="2022-01-13T10:52:39+01:00" level=error msg="[filebeat] Collector output: Exiting: error initializing publisher: 1 error: tls: failed to parse private key\n"

So I guess the client would need to be told about the passphrase for that key file via the sidecar collector config, but I haven’t been able to find any info on how to do that

I also tried providing my pkcs8-plain.pem cert instead of the encrypted one, but while that gives no errors in the sidecar.log client-side, it does produce these errors in the Graylog server log:

2022-01-13T10:57:48.016+01:00 ERROR [AbstractTcpTransport] Error in Input [Beats/61dedecb00219622cef00a65] (channel [id: 0xdf977c3f, L:/192.168.30.6:5140 ! R:/10.16.0.250:47778]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE)

Which sort of tell me that this is not the right direction…?

Here is my Beats input config (where I did also try and enter cert paths and password as mentioned above):

How can the community help?

What has me a bit frustrated is reading this guidance from official Graylog docs on securing Sidecar/Beats communication:

The Communication between Sidecar and Graylog will be secured if your API uses SSL .
To secure the communication between the Collector and Graylog you just need to mark Enable TLS
in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input.Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection . After this is saved, the communication between Beats and Graylog will use TLS.

So I can follow the above steps until this part:
Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection

My issue here is that I simply cannot find anywhere in the Graylog GUI where I could enable TLS support and Insecure TLS connection on the Sidecar collector config! Is this an enterprise-only feature, or what am I missing here??

At this point it’s tempting to just abandon this TLS security stuff, since I’ve spent far too much time on it already, but I feel like I’m quite close since I now have the web interface as well as the graylog-sidecar API using TLS encryption and thus only encrypting the actual log data being sent into Graylog from the client by graylog-sidecar is missing.
I have some idea that my problem might lie with the certificates and my lack of understanding of those…

Any help would be greatly appreciated.

[gsmith]

Hello @foss4ever

I feel your pain on this, It took a while for myself to understand how to get TCP/TLS to work with self-signed certificates.
I need to ask you a few question first so I know how your actually setting up you certificates and configuration.

1. Before you decided to use TLS/TCP, did the Beats INPUT work and did you see messages arriving in the INPUT.
2. If you did then I know your Sidecar is configured correctly, If not we have more problems/issues.
3. I assume the Sidecar is for Linux Device and is this a remote device or local install?
4. Think your using the wrong certs because I’m judge this by the certificate name that’s in your post. What instruction did you use to create your certificate?
5. Could you briefly show us the steps taken when you create those certs?

The reason I ask those question about certificates is because of this error.

@foss4ever

OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE)

Next here is an example in the FileBeat config file

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log
output.logstash:
   hosts: ["8.8.8.8:5044"]
   ssl.certificate_authorities: ["/etc/filebeat/certs/cert.pem"]
   ssl.certificate:             "/etc/filebeat/certs/graylog-certificate.pem"
   ssl.key:                     "/etc/filebeat/certs/graylog-key.pem"
   ssl.key_passphrase:          "PASSWORD"
   ssl.supported_protocols:     "TLSv1.2"
   
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

You can also look here, perhaps it will help.

GitHub -...

And some extra info

GitHub - graylog-labs/shadowCA: Create a CA and Certificates to use in your...

Hope that helps

[foss4ever]

HI @gsmith

Thanks for helping out again

I’ll go through your questions one by one, below.

1. Before you decided to use TLS/TCP, did the Beats INPUT work and did you see messages arriving in the INPUT.
   Answer: Yes and yes
2. I assume the Sidecar is for Linux Device and is this a remote device or local install?
   Answer: the sidecar is for a Debian 10 VM in a different VLAN from the Graylog server
3. What instruction did you use to create your certificate? Could you briefly show us the steps taken when you create those certs?
   I’ll refer you to my previous post: Unable to start inputs after implementing HTTPS on Graylog web interface - #5 by foss4ever

[foss4ever]

A small update:

If I change my sidecar collector config to this:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/syslog
  type: log
output.logstash:
   hosts: ["graylog.r2p.com:5140"]
   ssl.verification_mode: none
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Then I can get messages coming in on the Beats input.

The change I made was adding the directive ssl.verification_mode and setting it to “none”.

However, changing that to full (which I think might be the default anyway), causes the input to stop working again and I’m back to getting the “SSL bad certificate” error in graylog server log:

2022-01-14T12:35:16.209+01:00 ERROR [AbstractTcpTransport] Error in Input [Beats/61dedecb00219622cef00a65] (channel [id: 0xaadd0519, L:/192.168.30.6:5140 ! R:/10.16.0.250:37474]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE)

Not sure of the importance of this, as I can only assume that setting ssl verification mode to “none” is a really bad move, security-wise and thus not something to be used in a production environment.

[gsmith]

@foss4ever

Looks like your using the wrong certificates.

openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

So now the three you will need are

• cert.pem
• graylog-certificate.pem
• graylog-key.pem

I seen you did this which is good -alias host.domain -file cert.pem

And I quote.

The resulting graylog-certificate.pem and graylog-key.pem can be used in the Graylog configuration file.

Then in gl.config

• http_tls_cert_file = /path/to/graylog-certificate.pem
• http_tls_key_file = /path/to/graylog-key.pem
• http_tls_key_password = secret

Then you Filebeat config because your using a token for Sidecar.

output.logstash:
   hosts: ["8.8.8.8:5044"]
   ssl.certificate_authorities: ["/etc/filebeat/certs/cert.pem"] <--- And maybe this here
   ssl.certificate:             "/etc/filebeat/certs/graylog-certificate.pem"  <-- here
   ssl.key:                     "/etc/filebeat/certs/graylog-key.pem"  <---here

[foss4ever]

I gave up on using the Graylog-suggested solution and instead created the certs that I needed using EasyRSA (on a different machine).

In this way, I believe I got things working now, albeit without authentication of the client which was not really the goal anyway.

So my procedure ended up being this:

Create certs on dedicated CA server and transfer CA .crt file as well as SRV .crt and .p8 files to Graylog server.

Put CA crt file in /usr/local/share/ca-certificates//

Do sudo update-ca-certificates to make Graylog server trust this CA.

Convert SRV cert and key file to PEM format and set graylog user as owner of these files.

Enable TLS encryption in Graylog server config and restart Graylog service.

Now, for getting TLS encryption working on the client / sidecar, I did the following:

Put CA crt file in /usr/local/share/ca-certificates//
Do sudo update-ca-certificates to make Graylog server trust this CA.

Install graylog-sidecar and filebeat.

Set up graylog-sidecar config with:
server_url
server_api_token

Restarting graylog-sidecar service.

Finally, I went onto the Gralog server web interface and set up my Beats input:

bind_address:
0.0.0.0
no_beats_prefix:
false
number_worker_threads:
4
override_source:

port:
5140
recv_buffer_size:
1048576
tcp_keepalive:
false
tls_cert_file:
/etc/graylog/server/pki/issued/GRAYLOGSRV_public.pem
tls_client_auth:
disabled
tls_client_auth_cert_file:

tls_enable:
true
tls_key_file:
/etc/graylog/server/pki/private/GRAYLOGSRV_private_encrypted.pem
tls_key_password:

---

I then created and applied a custom sidecar collector configuration for the client /sidecar:
(The only thing I added was the line “ssl.verification_mode: full”)

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/syslog
  type: log
output.logstash:
   hosts: ["graylog.<mydomain>.com:5140"]
   ssl.verification_mode: full
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

…And that was it - I now have TLS encryption working on the sidecar client log input (at least as far as I can tell)

I also created a clients certificate and key from my CA server to be used for adding authentication to the sidecar/filebeat coming into Graylog, but this part I could not get working - I kept on getting some SSL error “bad certificate” in Graylog server logs and the input stop working when trying to enable TLS client auth on the Beats input. I would be glad to hear any ideas as to what I might’ve been doing wrong, here…

[foss4ever]

So I just now managed to also get client authentication working as well

This is what I did:

First, the amended Beats input:

bind_address:
 0.0.0.0
no_beats_prefix:
 false
number_worker_threads:
 4
override_source:
 <empty>
port:
 5140
recv_buffer_size:
 1048576
tcp_keepalive:
 false
tls_cert_file:
 /etc/graylog/server/pki/issued/GRAYLOGSRV_public.pem
tls_client_auth:
 required
tls_client_auth_cert_file:
 /etc/graylog/server/pki/ca.crt
tls_enable:
 true
tls_key_file:
 /etc/graylog/server/pki/private/GRAYLOGSRV_private_encrypted.pem
tls_key_password:
********

Basically I just changed tls_client_auth to “required” which was a no-brainer, and then the other change I made took me a bit longer to figure out that you have to actually point to the trusted CA .crt file of your self-made Certificate Authority.

And then on the Sidecar collector config, I made changes like so:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/syslog
  type: log
output.logstash:
   hosts: ["graylog.<mydomain>.com:5140"]
   ssl.certificate_authorities: /usr/local/share/ca-certificates/<mycustomca>/ca.crt
   ssl.certificate: /etc/graylog/sidecar/pki/issued/<mycustomclientcert>.crt
   ssl.key: /etc/graylog/sidecar/pki/private/<mycustomclientkey>.key
   ssl.key_passphrase: <passphrase_for_key>
   ssl.verification_mode: full
   ssl.client_authentication: required
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

So what was added to Sidecar collector config was the following lines:

ssl.certificate_authorities: /usr/local/share/ca-certificates//ca.crt
Above path may differ, depending on Linux distro.

ssl.certificate: /etc/graylog/sidecar/pki/issued/.crt
Here, you point to the public key portion of your client certificate.

ssl.key: /etc/graylog/sidecar/pki/private/.key
Here, you point to the private key portion of your client certificate.

ssl.key_passphrase: <passphrase_for_key>
Provide the encryption passphrase as set for the private key.

ssl.client_authentication: required
Since we are enabling certificate-based client authentication, we might as well enforce it, too.

[gsmith]

@foss4ever
So you fix your issue ? if so Nice good job. Also if you could mark your resolve that wold be great for future search’s

[foss4ever]

Yes, I did.

However, I would really encourage Graylog themselves to improve the documentation on setting up TLS encryption for web interface / api / and for various types of log inputs.

If they had only provided a clear guide on how to do so, I might have done in a couple of hours what ended up taking me a week to figure out.
Maybe that’s just me being slow, but I’m confident that a lot of sysadmins have wasted a lot of time on this thing (which can be confirmed by searching the Graylog forum for terms such as: tls, encryption, certificates, ssl, …).

Nonetheless, thanks very much for assisting in both this and my previous posting, @gsmith !

[gsmith]

@foss4ever
I understand, and actually they are in the process of fixing the documentation. It might take a few. I did the same as you did except I had to figure out what goes where and used the GL documentation as a guide along with help from the community. It took me like two weeks and multiple posts to get it right, so I feel your pain. Now I do it like 5 time every 6 months.
No problem glad I could offer assistance