GELF TCP Input Suddenly Broken

Graylog Central (peer support)
1

A couple of weeks ago my Graylog GELF TCP Input suddenly stopped working. I have tried several suggestions I have found on postings of similar issues with no luck, as well as a few other things. All other inputs on the server are working fine, and the HTTPS is valid and cert confirmed OK. This seemed to happen out of the blue, with no manual updates recently performed and not close to certificate expiration.

Things that have been tried:

  • Creating and implementing a brand new certificate & private key from scratch
  • Rolling back the last few Graylog patches
  • Downgrading Java OpenJDK
  • Adding enabled_tls_protocols = TLSv1.2 to the server config file

Error:

2021-07-07T08:53:22.762-04:00 WARN  [ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x4631075c, L:/SERVER:12202 - R:/CLIENT:50608]
org.bouncycastle.pkcs.PKCSException: Encountered unexpected object type: org.bouncycastle.cert.X509CertificateHolder
        at org.graylog2.plugin.inputs.transports.util.KeyUtil.privateKeyFromFile(KeyUtil.java:272) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.createSslEngine(AbstractTcpTransport.java:346) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:323) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:319) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport$1.initChannel(NettyTransport.java:105) ~[graylog.jar:?]
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [graylog.jar:?]
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474) [graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [graylog.jar:?]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [graylog.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]

Input config (note that the cert & private key are not encrypted therefore not needing a tls key password):

bind_address: 0.0.0.0
decompress_size_limit:8388608
max_message_size:2097152
number_worker_threads:4
override_source:<empty>
port:12202
recv_buffer_size:1048576
tcp_keepalive:false
tls_cert_file:/etc/graylog/server/graylog-certificate.pem
tls_client_auth:disabled
tls_client_auth_cert_file:<empty>
tls_enable:true
tls_key_file:/etc/graylog/server/graylog-key.pem
tls_key_password:
use_null_delimiter:true

OS: Debian 9
Graylog Version: 4.1.0
Java: OpenJDK 1.8.0_292

2

Hello,

It seams that you tried a lot a different things to resolve your issue. When you stated

Are you refering to the INPUT is in a stopped state? or it stopped receiving message? If so have you tried using a different port then 12202?

You using the same certificates for your INPUTs as for your HTTPS?

1 Like
3

I have tried a new input with no luck. Yes, it is the same as the HTTPS.

4

Long story short,
Graylog did not like that there was a cert with a 4year expiration. When I remade the cert following my instructions, missed the step converting the private key from pkcs5 to pkcs8…

1 Like
5

:laughing: Yeah that would do it. Glad you found the issue.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.