"Uncaught exception in periodical" when connecting to ES over SSL


(Hugh Riley) #1

I’m trying to set up my Graylog (version 2.3.1 on CentOS 7) instance to connect to an ElasticSearch (version 5.6.3 on CentOS 7) node in the same network over SSL. However, when I start up the Graylog server, I start seeing the below error message in the log. Disabling SSL on the ES server and changing https to http in the elasticsearch_hosts option on the GL server allows it to operate as normal.

I enabled debug mode in GL and saw that the server was making the request “https://elastic:password@host-01.domain.com:9200/_cluster/health/_all?local=true&timeout=60s” and I can request the same URL with curl and get back the ES status. The SSL cert that the ES server uses is signed by an internal CA, but that CA’s fingerprint is in the cacerts file and I’m starting up GL with " -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts" (the path to said cacerts file). There’s nothing of interest, that I can tell, on the ES server side.

Any assistance would be greatly appreciated.

Error from log:

2017-10-10T17:04:14.131-05:00 ERROR [IndexRotationThread] Uncaught exception in periodical
java.lang.ClassCastException: [B cannot be cast to java.lang.String
        at org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309) ~[graylog.jar:?]
        at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112) ~[graylog.jar:?]
        at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99) ~[graylog.jar:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:463) ~[graylog.jar:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397) ~[graylog.jar:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) ~[graylog.jar:?]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) ~[graylog.jar:?]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
        at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:147) ~[graylog.jar:?]
        at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.Cluster.isConnected(Cluster.java:169) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:72) ~[graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

(Jochen) #2

It looks like the subjectAltName attribute of the certificate contains an invalid value ([B indicating a byte array instead of a string).

This looks like a bug in the Apache Commons HttpClient being used by Graylog to communicate with Elasticsearch:

EDIT: Unfortunately there’s currently no release of Apache HttpClient 4.5.x with the fix mentioned in these JIRA issues.
https://github.com/apache/httpcomponents-client/commit/c58288c9af339343a4433888a73f70669c2556ff


(Jochen) #3

I’ve created an issue for this on GitHub, so we won’t forget to fix it:
https://github.com/Graylog2/graylog2-server/issues/4232


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.