Graylog beats input TLS enable Error

Ganeshbabu · November 7, 2017, 5:10am

Hi All,

I am successfully enabled SSL in graylog and application is up and running fine without any issues now I am trying to secure the communication between the Collector and Graylog by enabling the TLS in beats input

As per the documentation, Graylog itself created a new self signed certificate for the input and in the sidecar beats output configuration I marked Enable TLS support & Insecure TLS connection.

After enabling all these changes I am getting the below error in logs,

2017-11-07T04:51:23.854Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STOPPING
2017-11-07T04:51:23.856Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STOPPED
2017-11-07T04:51:23.857Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now TERMINATED
2017-11-07T04:51:23.857Z WARN  [AbstractTcpTransport] TLS key file or certificate file does not exist, creating a self-signed certificate for input [Beats/59b794f268521b07e6b29b5f].
2017-11-07T04:51:23.858Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STARTING
2017-11-07T04:51:23.948Z INFO  [AbstractTcpTransport] Enabled TLS for input [Beats/59b794f268521b07e6b29b5f]. key-file="/tmp/keyutil_0.0.0.0:null_7936124129426110818.key" cert-file="/tmp/keyutil_0.0.0.0:null_6568254468981381412.crt"
2017-11-07T04:51:23.951Z WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Beats, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2017-11-07T04:51:23.952Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now RUNNING
2017-11-07T04:51:25.863Z ERROR [AbstractRotationStrategy] Cannot perform rotation at this moment.
2017-11-07T04:51:25.869Z ERROR [AbstractRotationStrategy] Cannot perform rotation at this moment.
2017-11-07T04:51:32.738Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T04:51:32.740Z ERROR [NettyTransport] Error in Input [Beats/873de4f268521b07e6b29b5f] (channel [id: 0xbda8b30d, /xx.xxx.xxx.xx:1072 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000001324385e6c92516bdb3e14c5f3ff7f927218f4450e57b615277a6a977663a385b5dec3d82841b56f3d53db329292524a3ee7bece90933018f383c53d1c1dff7cef4d3fcf66b3ff66b3ff7fbde1a2e7606a130cf41bc2ebc8d0786907def1103adb40e0914d38889d6da6622fd0b3f7a689e6321817daa139db8e6731e7d1783e3bbfe7faa3097c3e8700c7ac4d5b43cb42668540c73beea0f16978b2ae37a1b503042adb8fdbc06e33987e0abe5d5f7acf0102c1f8676894ec76ec20f0d47257fb48dc74e9c6dbadab7853d9aee32a58078d65a694acb33c59ad8c4af26a4989a9734ad294d327b55a168549113f79bcb2196ccd13227c5f2535f7167b81ce36279acb71ecdaeac41a4ce3a17f44f3e47d10705c59576f866dffc811215d2d1694a52b081c0187c37fdd96e5ddcdbbf2eee6fadb97fbebb2c4a9c56fd8b1f3b1191a6aaee61202c73be59f5efcb43efcadee052e42dbb30fa61fa191922c1229132abe52ae95d4999c13d177083cf3eb8b75f544bfee8cf76d8587d38c4e7b301a677a098de31821f0be1d8c7b85c60799a7149fcbe94d323f9c2a23caaf88f2784aa2584f9a225239915a1f354994af89f2a82f891411a9f4586707cf94581c721757440b49546444454eb45044c58180e28cfeb1b3fbdf010000ffffbe36d8b0
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:857) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Please kindly advice If I am doing anything wrong in the setup.

Should I use the graylog-certificate.pem & graylog-key.pem files generated in using HTTPS ???

Thanks,
Ganeshbabu R

jochen · November 7, 2017, 8:40am

Please post the complete configuration of the Beats input in Graylog, the complete configuration of the Graylog Collector Sidecar, and the configuration file for Filebeat which has generated by the Collector Sidecar.

Ganeshbabu · November 7, 2017, 9:15am

@jochen

Please find the below informations,

Beats input in Gralog

bind_address: 0.0.0.0
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: optional
tls_client_auth_cert_file: <empty>
tls_enable: true
tls_key_file: <empty>
tls_key_password: ********

Configuration of Graylog collector sidecar

server_url: https://graylogserver.southeastasia.cloudapp.azure.com/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - apache
    - graylogserver
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

Filebeat file

filebeat:
  prospectors:
  - document_type: log
    encoding: plain
    exclude_lines:
    - Ticket
    fields:
      data: example.org
      gl2_source_collector: f4749ffd-1f9b-4ef1-b065-a8fc32388fa1
    ignore_older: 0
    input_type: log
    paths:
    - /etc/graylog/graylogserver*.csv
    scan_frequency: 10s
    tail_files: false
output:
  logstash:
    hosts:
    - graylogserver.southeastasia.cloudapp.azure.com:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- linux
- apache
- graylogserver

jochen · November 7, 2017, 9:32am

The “logstash” output in Filebeat is missing all SSL/TLS settings.

Ganeshbabu · November 7, 2017, 11:40am

Hi @jochen

As you mentioned that logstash output in filebeat don’t have SSL settings. I reconfigured the filebeats output in graylog,

and I given the cert & key file paths and saved it. The SSL changes were reflected in the filebeat.yml file,

output:
  logstash:
    hosts:
    - graylogserver.southeastasia.cloudapp.azure.com:5044
    loadbalance: false
    ssl:
      certificate: /etc/ssl/nginx_crt.crt
      key: /etc/ssl/nginx_key.key
      verification_mode: none

To use SSL I was trying to configure the beats input with TLS enable and below is the changes made in graylog but I am getting the below error in logs after saving it,

2017-11-07T11:34:49.392Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T11:34:49.396Z ERROR [NettyTransport] Error in Input [Beats/59b794f268521b07e6b29b5f] (channel [id: 0xd64d29c2, /xx.xxx.xx.xx:1148 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3257000000013243000000b6785e6c8eb14a04311086e3abfcf5dc92648b83a94eae13acb4926be62e036e91e4b88cb2b2e4a97d015951dcc276f8e6fbfef8e09cbb73ce7d2e3864354962025e601f57054367bd9c550c84b38a6173e984834d599b49be82117dd8ef42d8f9fd731adde3573a9394b4960bcd66645b282d02cd5b7b5f5f478bc6f4ded743b1510749eec5893827da79f3dcbdf23e39707e15d6f6daa058c7118070fc2b77cc3f4fe150000ffffe19a4dd6
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:857) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2017-11-07T11:34:51.899Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T11:34:51.909Z ERROR [NettyTransport] Error in Input [Beats/59b794f26807e6b29b5f] (channel [id: 0x4d5f787f, /xx.xx.xx.xxx:51084 => /xx.x.x.x:5044])
**javax.net.ssl.SSLHandshakeException: General SSLEngine problem**
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_144]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1219) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1906) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1393) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1256) ~[graylog.jar:?]
        ... 19 more
Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available
        at sun.security.ssl.DummyX509TrustManager.checkClientTrusted(SSLContextImpl.java:1191) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1893) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1393) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1256) ~[graylog.jar:?]
        ... 19 more

Please correct me If I am doing anything wrong in the setup

Thanks,
Ganeshbabu R

Ganeshbabu · November 8, 2017, 12:06pm

@jochen

Kindly advice if I am doing anything wrong in the above SSL setup of beats input.

I couldn’t able to fix the problem

Thanks,
Ganeshbabu R

zionio · November 8, 2017, 12:32pm

Hi, i’m not sure, but it seems that certificate for beats input is not correct, as your logs said:

“java.security.cert.CertificateException: No X509TrustManager implementation available”

Have you removed the passphrase from certificate key?
Is your crt in pem encoded format ?

hope this helps.

Ganeshbabu · November 8, 2017, 1:25pm

Hi @zionio

I generated the cert & key files using the below command,

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/nginx_key.key -out /etc/ssl/nginx_crt.crt

Have you removed the passphrase from certificate key?

I haven’t given any passphrase while generated key

Is your crt in pem encoded format ?

No its not in pem format

I just followed this documentation

Please kindly advice

Thanks,
Ganeshbabu R

Ganeshbabu · November 13, 2017, 2:24pm

Hi @jochen

2017-11-07T11:34:49.396Z ERROR [NettyTransport] Error in Input [Beats/59b794f268521b07e6b29b5f] (channel [id: 0xd64d29c2, /xx.xxx.xx.xx:1148 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

Due to wrong configuration in the beats inputs I was getting the above error and I given the correct path of the cert & key files.
the filebeat are started harvesting the files and below is the details from the /var/log/collector-sidecar/filebeat

2017-11-13T13:57:11Z INFO Starting Registrar
2017-11-13T13:57:11Z INFO Start sending events to output
2017-11-13T13:57:11Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data74.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data71.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data72.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data73.csv
2017-11-13T13:57:41Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=4 filebeat.harvester.running=4 filebeat.harvester.started=4 libbeat.publisher.published_events=1130
2017-11-13T13:57:41Z ERR Connecting error publishing events (retrying): dial tcp 52.187.191.6:5044: i/o timeout

not sure whether this issue is anything related to firewall or port open…

below is the response of netstat -tuplen

It would be very helpful if you could share your thoughts.

Thanks

Ganeshbabu · November 13, 2017, 4:56pm

Below is the link which helps to setup SSL in beats input and logstash output.

system · November 27, 2017, 4:56pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Winlogbeats andTLS/SSL Confusion Graylog Central (peer support) sidecar , winlogbeat	3	922	June 4, 2021
TLS with Winlogbeats sidecar? Graylog Central (peer support) sidecar , winlogbeat	1	517	August 18, 2020
Is the Sidecar documentation up to date? Graylog Central (peer support) sidecar	4	720	November 18, 2019
Setup Beats to ship logs using TLS encryption and authentication Graylog Central (peer support) sidecar	5	2306	March 5, 2020
Securing Sidecar/beats communication Graylog Central (peer support) sidecar	10	2257	January 31, 2022

Graylog beats input TLS enable Error

Related Topics