I tried the same for a Input. But still got a similar error.
2020-04-24T16:31:18.999+02:00 WARN [ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x86929380, L:/xxx.xxx.xxx.xxx:5044 - R:/xxx.xxx.xxx.xxx:56551]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/graylog/server/pem_key.pk8
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:350) ~[graylog.jar:?]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:107) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.createSslEngine(AbstractTcpTransport.java:329) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:305) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:301) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.NettyTransport$1.initChannel(NettyTransport.java:105) ~[graylog.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [graylog.jar:?]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [graylog.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:956) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474) [graylog.jar:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [graylog.jar:?]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:387) [graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [graylog.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: ObjectIdentifier() – data isn’t an object ID (tag = 48)
at sun.security.util.ObjectIdentifier.(ObjectIdentifier.java:257) ~[?:1.8.0_242]
at sun.security.util.DerInputStream.getOID(DerInputStream.java:314) ~[?:1.8.0_242]
at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_242]
at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_242]
at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_242]
at sun.security.x509.AlgorithmId.(AlgorithmId.java:114) ~[?:1.8.0_242]
at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_242]
at javax.crypto.EncryptedPrivateKeyInfo.(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_242]
at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1072) ~[graylog.jar:?]
at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1133) ~[graylog.jar:?]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1113) ~[graylog.jar:?]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:348) ~[graylog.jar:?]
… 26 more
I setup a Apache as a reverse proxy for SSL, this works without any Problem.