I have been struggling for the past few days on setting up tls between filebeat and graylog. I created a ca, created a wildcard csr and signed it using the ca. Took the key and cert and moved them to the /etc/graylog/server_trusted_clients and /etc/ssl/certs/
# Array of hosts to connect to.
Filebeat is running, graylog-server/side-car are running, and the input in the webUI is running.
The only error I can find so far is:
2020-06-08T11:21:46.043-04:00 WARN [ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x48addbef, L:/10.90.2.61:5044 - R:/10.90.2.61:43190]
java.security.cert.CertificateParsingException: signed overrun, bytes = 918
at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1839) ~[?:1.8.0_252]
at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195) ~[?:1.8.0_252]
at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:471) ~[?:1.8.0_252]
at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:356) ~[?:1.8.0_252]
at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:462) ~[?:1.8.0_252]
at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadCertificates(KeyUtil.java:82) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.createSslEngine(AbstractTcpTransport.java:316) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:305) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:301) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.NettyTransport$1.initChannel(NettyTransport.java:105) ~[graylog.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [graylog.jar:?]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [graylog.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:956) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [graylog.jar:?]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417) [graylog.jar:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474) [graylog.jar:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [graylog.jar:?]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [graylog.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
I know this is related to certificates just not sure what to do from here. I have turned off security and the input works just fine I can collect the logs and everything is smooth sailing.
I created a self signed CA and a csr and signed that using the CA. I created a wildcard -> *.domain.com so that I could use it on all of my servers once I get it figured out. from there I copied the machine-server.pem and machine-key.pem to /etc/ssl/certs and to /etc/graylog/server/trusted_clients. Had some issues with permission because the previous errors could not read so I chmod them to 644. Lastly Imported the CA into my jvm strusted store and restarted all services.
I tried to use the github shadowCA and ended up getting different errors.
I have looked though a few forum posts here and people had similar issues but they either did not say what/if/how it was fixed or ended up being something else. I have also read the documentation several times and it makes a little more sense each time.
The forum you linked is a little different than my problem. I am using an nginx reverse proxy with our vendor provided external wildcard cert. I am running mongo, elasticsearch, and graylog on the same server too.
I would not be opposed to starting over with the CA and crts since it will be a separate self signed setup and it isn’t working now. I am new to pki infrastructure and I am the only one at my job who does this so I end up on forms/google a lot.
I understand the concept of what should happen I need a root CA that is in my trusted jvm store that signs my csr for my servers and I pass those certs to them.
I’m limited on knowledge with this type of issue, but I have been in this situation with Graylog before and had the error you have also received. With some trial and error testing I was able to get it to work.
The first problem I had was permission on the where the certs were kept, can Graylog access them?
Second, are the certs incerted in the trusted store and is graylog configured to use the trusted store?
Modify the JVM Setting to include -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks in the GRAYLOG_JAVA_OPTS.
I put localhost because this filebeat is on the graylog server until I figure it out then I will begin setting up inputs for other servers. I thought the input had to run on 5044, in the filebeat should the port 9200 be 5044? I think I just confused myself. so the filebeat says send it to elastichsearch at X address using X port?
My intention was only to encrypt the channel between the filebeat and graylog. Since I am running elastichsearch, mongodb, and graylog on the same server with an nginx reverse proxy I figured the rest was self contained and the web was encryped leaving the path from filebeat to graylog.
@shoothub I cant believe I looked over that 100 times I would look at that and read elastichsearch for some reason. That’s why it was working when I took out the certificates under the elasticsearch section. Since it is on the same box it was able to send it over to elasticsearch any bypass graylog.
@shoothub I have made some progress. Redone the whole CA/certs using the shadowCA, removed old from jvm and imported new. Created key using the shadowCA and script for the log server. Put the cert in /etc/graylog/server/trusted_clients and /etc/ssl/certs/
As you can see in the background I have no active connections.
Check server logs 2020-06-09T14:09:37.646-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPING
2020-06-09T14:09:37.648-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now TERMINATED
2020-06-09T14:09:37.649-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STARTING
2020-06-09T14:09:37.651-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now RUNNING
2020-06-09T14:09:37.652-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPED
2020-06-09T14:10:25.645-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPING
2020-06-09T14:10:25.645-04:00 INFO [AbstractTcpTransport] Enabled TLS for input [Beats/5ebee593fd24b306de6d9ac7]. key-file="/etc/ssl/certs/log1.domain.com.key" cert-file="/etc/ssl/certs/log1.domain.com.crt"
2020-06-09T14:10:25.645-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPED
2020-06-09T14:10:25.647-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STARTING
2020-06-09T14:10:25.648-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now TERMINATED
2020-06-09T14:10:25.658-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now RUNNING
I had and older error related to memory
(channel [id: 0x5ade91b1, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984
but I fixed that, filebeat, sidecar, and server are all running so I am not sure where to look now. I feel like I made some progress.