Certificate/TLS Issue

[CloudBeard (Andy Mills)]

Hello Everyone,

I have been struggling for the past few days on setting up tls between filebeat and graylog. I created a ca, created a wildcard csr and signed it using the ca. Took the key and cert and moved them to the /etc/graylog/server_trusted_clients and /etc/ssl/certs/

On the input I have

On the filebeat.yaml I have:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  ssl.certificate_authorities: ["/etc/ssl/certs/ca.pem"]
  ssl.certificate: "/etc/ssl/certs/machine-cert.pem"
  ssl.key: "etc/ssl/certs/machine-key.pem"

Filebeat is running, graylog-server/side-car are running, and the input in the webUI is running.

The only error I can find so far is:

2020-06-08T11:21:46.043-04:00 WARN  [ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x48addbef, L:/10.90.2.61:5044 - R:/10.90.2.61:43190]
java.security.cert.CertificateParsingException: signed overrun, bytes = 918
        at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1839) ~[?:1.8.0_252]
        at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195) ~[?:1.8.0_252]
        at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:471) ~[?:1.8.0_252]
        at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:356) ~[?:1.8.0_252]
        at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:462) ~[?:1.8.0_252]
        at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadCertificates(KeyUtil.java:82) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.createSslEngine(AbstractTcpTransport.java:316) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:305) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$3.call(AbstractTcpTransport.java:301) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.NettyTransport$1.initChannel(NettyTransport.java:105) ~[graylog.jar:?]
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [graylog.jar:?]
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:956) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417) [graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474) [graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [graylog.jar:?]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [graylog.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

I know this is related to certificates just not sure what to do from here. I have turned off security and the input works just fine I can collect the logs and everything is smooth sailing.

[gsmith]

@CloudBeard
Yes its probalbly your certificates and can I ask how you set up your Certs ?
Need more info please.
Where have you looked to solve this?
Could your issue be something simular like this?

Config for NXLOG and INPUT on SSL/TLS

The private key is invalid or the password is wrong.

[CloudBeard (Andy Mills)]

I created a self signed CA and a csr and signed that using the CA. I created a wildcard -> *.domain.com so that I could use it on all of my servers once I get it figured out. from there I copied the machine-server.pem and machine-key.pem to /etc/ssl/certs and to /etc/graylog/server/trusted_clients. Had some issues with permission because the previous errors could not read so I chmod them to 644. Lastly Imported the CA into my jvm strusted store and restarted all services.

I tried to use the github shadowCA and ended up getting different errors.

I have looked though a few forum posts here and people had similar issues but they either did not say what/if/how it was fixed or ended up being something else. I have also read the documentation several times and it makes a little more sense each time.

The forum you linked is a little different than my problem. I am using an nginx reverse proxy with our vendor provided external wildcard cert. I am running mongo, elasticsearch, and graylog on the same server too.

I would not be opposed to starting over with the CA and crts since it will be a separate self signed setup and it isn’t working now. I am new to pki infrastructure and I am the only one at my job who does this so I end up on forms/google a lot.

I understand the concept of what should happen I need a root CA that is in my trusted jvm store that signs my csr for my servers and I pass those certs to them.

[gsmith]

@CloudBeard
I’m limited on knowledge with this type of issue, but I have been in this situation with Graylog before and had the error you have also received. With some trial and error testing I was able to get it to work.
The first problem I had was permission on the where the certs were kept, can Graylog access them?
Second, are the certs incerted in the trusted store and is graylog configured to use the trusted store?
Examples;

Modify the JVM Setting to include -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks in the GRAYLOG_JAVA_OPTS.

keytool -import -trustcacerts -file graylog-certificate.pem -alias dns.name.of.server -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64/jre/lib/security/cacerts

https://docs.graylog.org/en/3.2/pages/secure/sec_graylog_beats.html#ssl-tls-prework

I have not used a wild card certs yet, but have you tried not to use a wildcard cert for troubleshooting purposes?

When creating my certs for lab testing I used these links below. I did have to reconfigure the step/s to work in my environment.

GitHub - john-babio/graylog

https://docs.graylog.org/en/3.2/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

Graylog3 with https (easy tutorial)

My ambient test: -Oracle Linux 7.6 (VMware 15); -Network: NAT; -Graylog3, MongoDB and Elastic: running together; Documentation base for RPM distro. First of all, read Docs » Configuring Graylog » Using HTTPS 0- mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/ 1- openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem 2- openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem 3- openssl pkcs8 -in pk…

Hope this helps, sorry I dont have a direct answer for your issue but maybe these links might direct you in the right place.

[gsmith]

@CloudBeard
FYI our Linux Machine Input, and the links used from above. Our Graylog invironment is internal use only.For each DMZ were have a graylog server.

[CloudBeard (Andy Mills)]

Awesome thanks, I will start working through this info tomorrow and see where I end up. I know it must be something simple I am skipping or doing wrong. I will keep you posted on my progress!

[gsmith]

@CloudBeard

I know it must be something simple I am skipping or doing wrong.

Same here it was simple solution, but mind bending.

[shoothub]

On your first post, i see some wierd things:

1. You post screenshot with input listening on port 5044, but your filebeat, you have different port and localhost, is it OK?
2. You have enabled TLS client authentication. I don’t know if it was your desired choose, or you wanted only encrypted TLS channel between filebeat and graylog…

[CloudBeard (Andy Mills)]

I put localhost because this filebeat is on the graylog server until I figure it out then I will begin setting up inputs for other servers. I thought the input had to run on 5044, in the filebeat should the port 9200 be 5044? I think I just confused myself. so the filebeat says send it to elastichsearch at X address using X port?

My intention was only to encrypt the channel between the filebeat and graylog. Since I am running elastichsearch, mongodb, and graylog on the same server with an nginx reverse proxy I figured the rest was self contained and the web was encryped leaving the path from filebeat to graylog.

[shoothub]

1. You can’t send directly from filebeat to elasticsearch, you need to send it to graylog port listening on Input for Beats, in your case 5044 (like logstash input).
2. Your filebeat config file is wrong, correct one use output.logstash:
   https://docs.graylog.org/en/3.3/pages/secure/sec_graylog_beats.html#beats-shipper

output.logstash:
    hosts: ["graylog.example.org:5044"]
    ssl.certificate_authorities: ["/etc/ca.pem"]
    ssl.certificate: "/etc/client.crt"
    ssl.key: "/etc/client.key"

3. Best way is to setup filebeat without TLS encryption, and if everything work as expected, than configure encryption.

[CloudBeard (Andy Mills)]

@shoothub I cant believe I looked over that 100 times I would look at that and read elastichsearch for some reason. That’s why it was working when I took out the certificates under the elasticsearch section. Since it is on the same box it was able to send it over to elasticsearch any bypass graylog.

It is working with:

output.logstash:
hosts: [“graylog.example.org:5044”]

Next step is to fix my certificates. I am going to wipe them and build a new CA, if I am only using them for the filebeat auth do I still need to import it in my jvm keystore?

[shoothub]

If you will create new CA, of course you need to add it to jvm keystore again (and remove old CA if not used)

https://docs.graylog.org/en/3.3/pages/secure/sec_graylog_beats.html

[CloudBeard (Andy Mills)]

@shoothub I have made some progress. Redone the whole CA/certs using the shadowCA, removed old from jvm and imported new. Created key using the shadowCA and script for the log server. Put the cert in /etc/graylog/server/trusted_clients and /etc/ssl/certs/

Logstash output:
output.logstash:

The Logstash hosts

hosts: [“log1.domain.com:5044”]
ssl.certificate_authorities: ["/etc/ssl/certs/shadowCA.pem"]
ssl.certificate: “/etc/ssl/certs/log1.domain.com.crt”
ssl.key: “etc/ssl/certs/log1.domain.com.key”

As you can see in the background I have no active connections.

Check server logs 2020-06-09T14:09:37.646-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPING
2020-06-09T14:09:37.648-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now TERMINATED
2020-06-09T14:09:37.649-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STARTING
2020-06-09T14:09:37.651-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now RUNNING
2020-06-09T14:09:37.652-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPED
2020-06-09T14:10:25.645-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPING
2020-06-09T14:10:25.645-04:00 INFO [AbstractTcpTransport] Enabled TLS for input [Beats/5ebee593fd24b306de6d9ac7]. key-file="/etc/ssl/certs/log1.domain.com.key" cert-file="/etc/ssl/certs/log1.domain.com.crt"
2020-06-09T14:10:25.645-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STOPPED
2020-06-09T14:10:25.647-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now STARTING
2020-06-09T14:10:25.648-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now TERMINATED
2020-06-09T14:10:25.658-04:00 INFO [InputStateListener] Input [Beats/5ebee593fd24b306de6d9ac7] is now RUNNING

I had and older error related to memory

(channel [id: 0x5ade91b1, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984

but I fixed that, filebeat, sidecar, and server are all running so I am not sure where to look now. I feel like I made some progress.

[shoothub]

If I were you, I would first disable TLS client authentication in Input, and check if it works.

[CloudBeard (Andy Mills)]

If I change required to disabled I get the following error:

2020-06-10T14:12:17.034-04:00 WARN  [ChannelInitializer] Failed to initialize a channel. Closing: [id: 0xddcae76d, L:/10.90.2.61:5044 - R:/10.90.2.61:35000]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/ssl/certs/log1.domain.com.key

If I take out the key/cert fields from the filebeat and from the graylog input using port 5044 everything works fine.

I know this error is related to the key file, I generated this key using the shadowCA so I think it may be the format just not sure how to check or what it should be outside of .key