Client auth enabled not working with Letsencrypt certificate

Hi,

I have 2 servers, one for graylog and another for my application. I have letsencrypt certificates for both of them. I am using GELF TCP with TLS for the connection. But I am getting below error in graylog.

2019-06-14T02:19:21.380Z ERROR [AbstractTcpTransport] Error in Input [GELF TCP/5cfe59f41db840381ec37337] (channel [id: 0xde6714f7, L:/<graylog internal IP>:12201 ! R:/<Application server public IP>:53740]) (cause io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000418:SSL routines:OPENSSL_internal:TLSV1_ALERT_UNKNOWN_CA)

For graylog, nginx proxy is used and SSL certificates are deployed via nginx.
the certificate of client is placed under /etc/graylog/server/trusted_clients/

I am using same certificate in application to connect to graylog.

Can someone please help me?

my graylog server.conf settings


is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = qZx4gYX1Ofc05MMPw7DGJjoS5ShRkanZXCeaglTotqJiO8jf0mPzodsYm103cvavf7hjB1nyL7WJfHK4ycrHo1i3DhyGeh0s
root_password_sha2 = c70f30de786b86042a8081dfebce10274ea847b85aabcd29c595523551712e21
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_external_uri = http://127.0.0.1:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
elasticsearch_http_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = localhost:9200

My GELF TCP Global input settings

bind_address: 0.0.0.0
decompress_size_limit: 8388608
max_message_size: 2097152
number_worker_threads: 2
override_source: <empty>
port: 12201
recv_buffer_size: 1048576
tcp_keepalive: true
tls_cert_file: /etc/graylog/cert.pem
tls_client_auth: required
tls_client_auth_cert_file: /etc/graylog/server/trusted_clients/app_server.pem
tls_enable: true
tls_key_file: /etc/graylog/graylog-key-le.pem
tls_key_password: ********
use_null_delimiter: false

did you use any kind of not default jvm keystore? did you checked if your jvm can verify the certificate?

I used below command to add the cert in JVM store

sudo keytool -importcert -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/lib/security/cacerts -storepass <PASSWORD> -alias graylog_letsencrypt -file /etc/letsencrypt/live/graylog.example.com/fullchain.pem

What can I do to verify certificate with JVM?

Hi @jan,

For update, I resolved it by sending fullchain.pem as CACert and cert.pem as certfile.

Now I dont see any error in server.log. But I dont see any message in input stream as well.

Ports are open on both machine else I wont have received error message earlier.

Apart from server.log , where else can I check?

So, if I make client auth as disabled, messages are coming fine.

But if I make it optional or required, messages are failing silently.

I am using graypy for this.

you should check if your sending (Graypy?) is able to authenticate …

@jan

authenticate with what? graylog?

can you please tell me how to check that?
Because there is no error in server.log.

Also if I remove TLS , communication is flowing fine. So connectivity is not the issue.

you configured your Graylog that a sender needs to authenticate to send messages to Graylog - so you need to configure your sender to authenticate when it establish a connection using a certificate … as graypy is just a lib, did you have authentication with certificate implemented in your application?

@jan How to do that?

In graypy, I am sending cert file with full chain of certs.

Will that be enough?

Sorry I can’t solve your operational problem - because I do not know the product you are using.

You need to use certificates for Communication and for Authorization - again. Read the docs of Graypy how to archive that.

Actually I am doing that but it is failing silently. Apart from server.log, is there any other file I can refer for errors ?

nope - Graylog is writing just the server.log.

From what you described until now it looks that you use only the certificate for the connection (speak TLS) but not to authenticate.

If you do the second one - you might want to contact the developers of Graypy to check if you have done everything correct.

Shouldn’t the input running on Graylog that Graypy is communicating with be providing the certificate(s)?

we are configuring client auth with the correct cert file of client in graylog.

I think graypy will initiate communication with cert and fullchain file and graylog verify that provide cert is valid. If yes then it will allow the data to flow.

Am I right?

I think graypy will initiate communication with cert and fullchain file and graylog verify that provide cert is valid. If yes then it will allow the data to flow.

Wrong.

Graylog input is configured to accept only TLS connections. So your Sender needs to be able to make use of TLS Connection. Graylog and the sender need to be able to verify the certificate so accept the connection.

You enabled authentication that means: The sender needs to use a certificate to authenticate itself with a client certificate that Graylog can verify before the messages from that sender are accepted.

@jan I have provided tls_client_auth_cert_file in settings of GELF TCP. I cannot put that in server.conf because there are few connections which are using UDP.

And I am using the same certificate to connect from sender to graylog server. If it was cert issue, it comes in server.log.

But there is no error.

please read on what client authentication with certificates is.

It appears to me that you mix TLS for the connection and authentification with certificates.

@jan

In ideal setup of graylog, how to setup a secure connection between sender and graylog server?

I thought it is by enabling TLS in input stream with client Auth cert as required.
And send the same certificate from sender while initiating the connection.

@pratibhajagnere

the described way would be ideal - yes. Because no-one can tamper the messages on transport and Graylog is able to identify the sender.