Looking for help with some errors we're receiving on our Windows TLS - [AbstractTcpTransport]

[Tom_at_lorman (Tom S)]

Hello Everyone,
We are currently running this setup:
Graylog 5.0.11+30bdbfa on <our.url.com>
Ubuntu 22.04.3 LTS
Our Graylog server is running in the cloud, we actively have linux clients connected and working via TLS
I’m been getting my Windows clients going. I currently have them working over TLS (with data being shown/uploaded) but my Linux admin is seeing constant errors in the logs.
1st error:
2023-09-15T21:01:29.738Z ERROR [AbstractTcpTransport] Error in Input [Beats/<guid_xxxxxxxxxx>] (channel [id: 0xd85478c7, L:/:5044 ! R:/:62215]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 71) ->> (this version number outputs in a few differnt numbers)
2nd error:
2023-09-15T21:01:29.610Z WARN [AbstractTcpTransport] Client auth configured, but no authorized certificates / certificate authorities configured for input [Beats/<guid_xxxxxxxxxxxx>]

Windows is using sidecar+beats. I’ll post the configs I know to share:
******* INPUT *******
Beats Beats 1 RUNNING
bind_address: 0.0.0.0
charset_name: UTF-8
no_beats_prefix: false
number_worker_threads: 8
override_source:
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: /etc/graylog/server/fullchain.pem
tls_client_auth: optional
tls_client_auth_cert_file:
tls_enable: true
tls_key_file: /etc/graylog/server/privkey.pem
tls_key_password:********

Throughput / Metrics
1 minute average rate: 18 msg/s
Network IO: 0B 0B (total: 2.3GiB 642.3MiB )
Active connections: 2 (37,262 total) *** > this total number is always rising? The Linux clients don’t increase***
Empty messages discarded: 0
Show details

******* Winbeat setup via sidecar *******

Needed for Graylog

fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
hosts: [“our.url.com:5044”]
ssl.verification_mode: full

path:
data: ${sidecar.spoolDir!“C:\Program Files\Graylog\sidecar\cache\winlogbeat”}\data
logs: ${sidecar.spoolDir!“C:\Program Files\Graylog\sidecar”}\logs

tags:

• windows

winlogbeat:
event_logs:

• name: Application
• name: System
• name: Security

[gsmith]

Hey @Tom_at_lorman

How did you configure your Beat’s log shipper? Oh NVM your using GL sidecar.

Under [logstash] I think you may need to set those certificates.

For example:

# List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
  ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
  ssl.key: "/etc/pki/client/cert.key"

EDIT: I also found this for you

Securing Sidecar/beats communication

SUMMARY I am trying to get TLS encryption of Sidecar/Beats communication working in Graylog but despite trying many suggestions from the web and Graylog community forum I simply cannot get any messages coming in on the Beats input when I attempt to enable TLS on it. OS & package Information: Debian 10 Graylog 4.2.4+b643d2b graylog-sidecar 1.1.0-1 filebeat 7.16.2 Sidecar collector config used: [image] Troubleshooting steps taken: Have tried entering the path for my certificates, both i…

[Tom_at_lorman (Tom S)]

Hello @gsmith

Myself and our Linux admin invested a few hours today into this to try to get it to work.

I think our biggest confusion is there isn’t any specific Windows documentation that we’re seeing. Everything you see seems to be for Linux clients. All the paths for those ssl.xxx point to Linux paths. We pointed our Windows client to the paths on the greylog server that contain those certs. We tried graylog default certs, we tried creating a new cert as a wildcard for our windows client domain and we created one specific cert with our clients name in it. All these attempts resulted in failure.

I believe I was mislead beforehand thinking it was working with having that single ssl.verification set to full. If I change my input from optional to required, it fails, if I changed it back to option, logs start coming through. I’m unsure how to validate if it’s actually submitting as TLS or not.

So do the windows clients need certs located locally? If that was the case I haven’t seen any example sidecare configurations like this (with paths that point to a windows path). Linux is working due to it using Syslog TCP SSL and for windows we have to use GELF TCP or BEATS (right?) - so not sure if there are other options here.

Let me paste some error outputs in case it helps diagnose the problem.

Some errors from the winbeat log on windows:

{“log.level”:“warn”,“@timestamp”:“2023-10-04T14:54:42.101-0500”,“log.logger”:“cfgwarn”,“log.origin”:{“file.name”:“tlscommon/config.go”,“file.line”:102},“message”:“DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0”,“service.name”:“winlogbeat”,“ecs.version”:“1.6.0”}

{“log.level”:“error”,“@timestamp”:“2023-10-04T14:55:45.789-0500”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: remote error: tls: certificate required”,“service.name”:“winlogbeat”,“ecs.version”:“1.6.0”}

Some errors from the graylog server:

2023-10-04T19:27:13.033Z WARN [AbstractTcpTransport] Client auth configured, but no authorized certificates / certificate authorities configured for input [Beats/-guid-]
2023-10-04T19:27:13.202Z ERROR [AbstractTcpTransport] Error in Input [Beats/-guid-] (channel [id: 0x7ee33e2a, L:/cloud-ip:5044 ! R:/local-ip:51588]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE)
2023-10-04T19:27:13.517Z WARN [AbstractTcpTransport] Client auth configured, but no authorized certificates / certificate authorities configured for input [Beats/-guid-]
2023-10-04T19:27:13.689Z ERROR [AbstractTcpTransport] Error in Input [Beats/-guid-] (channel [id: 0xe798e0d8, L:/cloud-ip:5044 ! R:/local-ip:50308]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE)
2023-10-04T19:27:14.806Z WARN [AbstractTcpTransport] Client auth configured, but no authorized certificates / certificate authorities configured for input [Beats/-guid-]
2023-10-04T19:27:15.017Z WARN [AbstractTcpTransport] Client auth configured, but no authorized certificates / certificate authorities configured for input [Beats/-guid-]
2023-10-04T19:27:15.027Z ERROR [AbstractTcpTransport] Error in Input [Beats/-guid-] (channel [id: 0x9d767e23, L:/cloud-ip:5044 ! R:/local-ip:51589]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE)
2023-10-04T19:27:15.225Z ERROR [AbstractTcpTransport] Error in Input [Beats/-guid-] (channel [id: 0x83d01efe, L:/cloud-ip:5044 ! R:/local-ip:50309]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE)

[gsmith]

hey @Tom_at_lorman

Dont be fooled by the full path of the certificates,

This can be adjusted for example:

output.logstash:
  # The Logstash hosts
  hosts: ["192.168.1.100:5044"]

  #sername: "admin"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["C:\winlogbeat\ca.pem"]

  # Certificate for SSL client authentication
  ssl.certificate: "C:\winlogbeat\cert.pem"

  # Client Certificate Key
  ssl.key: "C:\winlogbeat\cert.key"

Ensure Winlogbeat has access to certificates.

@Tom_at_lorman

So do the windows clients need certs located locally?

Where every you decide to purtyour certificate/s that Windows node/instance can access them.

@Tom_at_lorman

f that was the case I haven’t seen any example sidecare configurations like this (with paths that point to a windows path).

Winlogbeat → Graylog’s Input 5044 (SSL certificates can be used)

The Communication between Sidecar and Graylog will be secured if your API uses SSL .To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input.

This is found here.

https://go2docs.graylog.org/5-0/getting_in_log_data/graylog_sidecar.html

There are a lot of good references in the form.

Windows Sidecar TLS setup

So, i managed to make it work without TLS - Graylog Sidecar Windows beats. I receive the messages and all is good. Now, i try to make it work with TLS and i have some questions, as, i am getting this error on the windows side when i am running the sidecar.exe [image] so, the certificate i am using on GL server i installed it on the windows machine under this path [image] the setup i found for the GL part is for a linux machine though: # Needed for Graylog fields_under_root: true fields.c…