# Certificate problem, and maybe other issues **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [March 23, 2022, 6:27pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142 "2022-03-23T18:27:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [March 23, 2022, 6:27pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/1 "2022-03-23T18:27:52Z")
Hello. This is a continuation of this thread since it got closed by the system: [https://community.graylog.org/t/multiple-problems-with-graylog/22591](https://community.graylog.org/t/multiple-problems-with-graylog/22591) Based on the feedback from that thread I started looking into certificate issues and the most pressing issue was just getting the graylog web admin page to remain operational. In the test environment which an exact snapshot of production, I was able to update the PKI cert and have it signed by the CA. In the test, this had the following effects: 1.) All inputs now show as running. (previously inputs were received, but the inputs were not showing as running) 2.) The errors in the graylog logs went away. 3.) A domain controller whose logs previously were not getting to the input began getting to the input. 4.) The graylog web admin page began staying operational for more than a few minutes, indefinitely so far operational So, as a result of that apparent success, we did the same thing in the production server. At first it worked, but then the graylog web admin page became inaccessible and the graylog logs began showing a new error. The graylog logs before and after that event are as follows: ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [March 23, 2022, 10:58pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/2 "2022-03-23T22:58:10Z")
Hello @fffhurst and welcome back. Glad your Dev environment worked out, sorry to here about your production environment. > [@fffhurst](#): > > At first it worked, but then the graylog web admin page became inaccessible and the graylog logs began showing a new error. The graylog logs before and after that event are as follows: Did you use new certs? or use the ones from dev? Check permissions, does graylog have access to the certs/keystore? To help you further we would need more then… > [@fffhurst](#): > > inaccessible and the graylog logs began showing a new error. Thanks ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [March 24, 2022, 7:01pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/3 "2022-03-24T19:01:26Z")
```auto Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-03-22T15:31:27.403-04:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out 2022-03-22T15:31:30.513-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-03-22T15:31:35.513-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-03-22T15:31:37.417-04:00 ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[graylog.jar:?] at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[graylog.jar:?] at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[graylog.jar:?] at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[graylog.jar:?] at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[graylog.jar:?] at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?] at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?] at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?] at com.mongodb.DBCursor.one(DBCursor.java:683) ~[graylog.jar:?] at com.mongodb.DBCollection.findOne(DBCollection.java:829) ~[graylog.jar:?] at com.mongodb.DBCollection.findOne(DBCollection.java:792) ~[graylog.jar:?] at com.mongodb.DBCollection.findOne(DBCollection.java:739) ~[graylog.jar:?] at org.graylog2.database.PersistedServiceImpl.findOne(PersistedServiceImpl.java:128) ~[graylog.jar:?] at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:73) ~[graylog.jar:?] at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:84) ~[graylog.jar:?] at org.graylog2.periodical.NodePingThread.doRun(NodePingThread.java:62) ~[graylog.jar:?] at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] 2022-03-22T15:31:37.419-04:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out ``` ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [March 24, 2022, 7:02pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/4 "2022-03-24T19:02:24Z")
```auto 2022-03-22T15:31:37.444-04:00 WARN [ClusterEventPeriodical] Error while reading cluster events from MongoDB, retrying. com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[graylog.jar:?] at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[graylog.jar:?] at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[graylog.jar:?] at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[graylog.jar:?] at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[graylog.jar:?] at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?] at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?] at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?] at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?] at org.graylog2.events.ClusterEventPeriodical.doRun(ClusterEventPeriodical.java:150) [graylog.jar:?] at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] 2022-03-22T15:31:37.445-04:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out 2022-03-22T15:32:42.130-04:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 3.0.2 [org.graylog.aws.AWSPlugin] 2022-03-22T15:32:42.166-04:00 INFO [CmdLineTool] Loaded plugin: Collector 3.0.2 [org.graylog.plugins.collector.CollectorPlugin] 2022-03-22T15:32:42.167-04:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin] 2022-03-22T15:32:43.035-04:00 INFO [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.truststore=/SSCLant/certs/cacerts.jks -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm 2022-03-22T15:32:43.574-04:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final 2022-03-22T15:32:48.454-04:00 INFO [InputBufferImpl] Message journal is enabled. 2022-03-22T15:32:48.502-04:00 INFO [NodeId] Node ID: 9e804424-96fe-42c8-a79a-051fc7fb0963 2022-03-22T15:32:48.875-04:00 INFO [LogManager] Loading logs. 2022-03-22T15:32:48.980-04:00 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000005549547208.index, deleting and rebuilding index... 2022-03-22T15:32:50.339-04:00 INFO [LogManager] Logs loading complete. 2022-03-22T15:32:50.342-04:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal 2022-03-22T15:32:50.355-04:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers. 2022-03-22T15:32:50.375-04:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000} 2022-03-22T15:32:50.421-04:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out 2022-03-22T15:32:50.592-04:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017 2022-03-22T15:32:50.597-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 9]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2928904} 2022-03-22T15:32:50.637-04:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017 2022-03-22T15:32:51.144-04:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200] 2022-03-22T15:32:51.144-04:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager 2022-03-22T15:32:51.216-04:00 INFO [JestClientFactory] Using custom ObjectMapper instance 2022-03-22T15:32:51.216-04:00 INFO [JestClientFactory] Node Discovery disabled... 2022-03-22T15:32:51.216-04:00 INFO [JestClientFactory] Idle connection reaping disabled... 2022-03-22T15:32:51.426-04:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy . 2022-03-22T15:32:54.293-04:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-03-22T15:32:54.303-04:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy . 2022-03-22T15:32:54.356-04:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017 2022-03-22T15:32:55.755-04:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-03-22T15:32:57.508-04:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-03-22T15:32:59.093-04:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-03-22T15:33:00.624-04:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-03-22T15:33:01.226-04:00 INFO [ServerBootstrap] Graylog server 3.0.2+1686930 starting up 2022-03-22T15:33:01.226-04:00 INFO [ServerBootstrap] JRE: Red Hat, Inc. 1.8.0_282 on Linux 3.10.0-1160.15.2.el7.x86_64 2022-03-22T15:33:01.226-04:00 INFO [ServerBootstrap] Deployment: rpm 2022-03-22T15:33:01.226-04:00 INFO [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos) 2022-03-22T15:33:01.227-04:00 INFO [ServerBootstrap] Arch: amd64 2022-03-22T15:33:01.340-04:00 INFO [PeriodicalsService] Starting 27 periodicals ... 2022-03-22T15:33:01.340-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. 2022-03-22T15:33:01.360-04:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever. 2022-03-22T15:33:01.366-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s]. 2022-03-22T15:33:01.366-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2022-03-22T15:33:01.367-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s]. 2022-03-22T15:33:01.368-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2022-03-22T15:33:01.368-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. 2022-03-22T15:33:01.369-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. 2022-03-22T15:33:01.369-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2022-03-22T15:33:01.369-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2022-03-22T15:33:01.375-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. 2022-03-22T15:33:01.376-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2022-03-22T15:33:01.376-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2022-03-22T15:33:01.379-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. 2022-03-22T15:33:01.380-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2022-03-22T15:33:01.381-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever. 2022-03-22T15:33:01.382-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. 2022-03-22T15:33:01.426-04:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017 2022-03-22T15:33:01.429-04:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed. 2022-03-22T15:33:01.433-04:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017 2022-03-22T15:33:01.471-04:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017 2022-03-22T15:33:01.476-04:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:11}] to localhost:27017 2022-03-22T15:33:01.479-04:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017 2022-03-22T15:33:01.479-04:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node. 2022-03-22T15:33:01.479-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever. 2022-03-22T15:33:01.480-04:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017 2022-03-22T15:33:01.484-04:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017 2022-03-22T15:33:01.484-04:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017 2022-03-22T15:33:01.502-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever. 2022-03-22T15:33:01.519-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever. 2022-03-22T15:33:01.522-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever. 2022-03-22T15:33:01.534-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s]. 2022-03-22T15:33:01.556-04:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s]. 2022-03-22T15:33:01.561-04:00 INFO [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable. 2022-03-22T15:33:01.562-04:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. 2022-03-22T15:33:01.565-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200) 2022-03-22T15:33:01.565-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2022-03-22T15:33:01.566-04:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s]. 2022-03-22T15:33:01.577-04:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s]. 2022-03-22T15:33:01.577-04:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2022-03-22T15:33:01.694-04:00 INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable. 2022-03-22T15:33:01.902-04:00 INFO [JerseyService] Enabling CORS for HTTP endpoint 2022-03-22T15:33:16.394-04:00 INFO [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy 2022-03-22T15:33:25.333-04:00 INFO [NetworkListener] Started listener bound to [graylogFQDN:9000] 2022-03-22T15:33:25.335-04:00 INFO [HttpServer] [HttpServer] Started. 2022-03-22T15:33:25.335-04:00 INFO [JerseyService] Started REST API at 2022-03-22T15:33:25.336-04:00 INFO [ServiceManagerListener] Services are healthy 2022-03-22T15:33:25.336-04:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2022-03-22T15:33:25.337-04:00 INFO [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=41, JournalReader [RUNNING]=41, KafkaJournal [RUNNING]=41, GracefulShutdownService [RUNNING]=42, OutputSetupService [RUNNING]=43, EtagService [RUNNING]=43, InputSetupService [RUNNING]=49, ConfigurationEtagService [RUNNING]=83, LookupTableService [RUNNING]=253, StreamCacheService [RUNNING]=291, PeriodicalsService [RUNNING]=298, JerseyService [RUNNING]=24023} 2022-03-22T15:33:25.404-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now STARTING 2022-03-22T15:33:25.406-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING 2022-03-22T15:33:25.408-04:00 INFO [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now STARTING 2022-03-22T15:33:25.409-04:00 INFO [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now STARTING 2022-03-22T15:33:25.411-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now STARTING 2022-03-22T15:33:25.419-04:00 INFO [ServerBootstrap] Graylog server up and running. 2022-03-22T15:33:25.518-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5fb5b064, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-03-22T15:33:25.522-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xc3d1b600, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-03-22T15:33:25.522-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=COC Infrastructure, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xde2e80ee, L:/0.0.0.0:1514]) should be 1048576 but is 425984. 2022-03-22T15:33:25.522-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7dc5acf0, L:/0.0.0.0:12201]) should be 1048576 but is 425984. 2022-03-22T15:33:25.526-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING 2022-03-22T15:33:25.527-04:00 INFO [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING 2022-03-22T15:33:25.528-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xd2deacce, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-03-22T15:33:25.528-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xb660881d, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-03-22T15:33:25.528-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x3b949989, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-03-22T15:33:25.529-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7725d543, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-03-22T15:33:25.531-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x398e8adb, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-03-22T15:33:25.533-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7f5bf1b1, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-03-22T15:33:25.537-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x4a1a8403, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-03-22T15:33:25.538-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x504fa505, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-03-22T15:33:25.539-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xa21facfc, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-03-22T15:33:25.540-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING 2022-03-22T15:33:25.541-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING 2022-03-22T15:33:25.548-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x946b5e65, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-03-22T15:33:25.556-04:00 INFO [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING ``` ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [March 24, 2022, 7:04pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/5 "2022-03-24T19:04:10Z")
> Did you use new certs? or use the ones from dev? > Check permissions, does graylog have access to the certs/keystore I used new certs ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [March 24, 2022, 10:40pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/6 "2022-03-24T22:40:11Z")
Hello, I picked out the import Logs that were shown This would be the connection to Elasticsearch and Graylog and they can not connect. Could be a configuration issue. > [@fffhurst](#): > > Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) Check Elasticsearch health `curl -XGET http://127.0.0.1:9200/_cluster/health?pretty=true` Seams like Graylog maybe unable to connect to MongoDb, This could be a configuration issue. > [@fffhurst](#): > > ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] Not sure if rotating index were execute to resolve this, if so this maybe another configuration issue and/or permission issue. > [@fffhurst](#): > > couldn’t update field types for index set \ > org.graylog2.indexer.ElasticsearchException: Couldn’t collect indices for alias graylog\_deflector These all look like a configuration problems, either something was missed or incorrect. I’m not seeing anything that would pertain to certification issue yet. ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 4, 2022, 7:06pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/7 "2022-04-04T19:06:54Z")
> [@gsmith](#): > > Check Elasticsearch health > > `curl -XGET http://127.0.0.1:9200/_cluster/health?pretty=true` ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/9841d403881127be45e6d8f9c336de4d73473951.png) ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 4, 2022, 7:13pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/8 "2022-04-04T19:13:33Z")
So if I follow you correctly, when we updated the certificate, that was successful in terms of removing the certificate erros, and now we just need to deal with the configuration issues? ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 4, 2022, 10:31pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/9 "2022-04-04T22:31:02Z")
Hello, Hello, > [@fffhurst](#): > > and now we just need to deal with the configuration issues? If you were able to correct the certificate issue. Double check if your Graylog node is functioning correctly, if so you may not need to do anything. But if you still have the following error as shown below, The Yes, you may need to find out why that happening. My apologies default port for MongoDb is 27017 **Error prior.** > [@fffhurst](#): > > ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] > [@fffhurst](#): > > ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] You obtain a Connection refused. Are you sure mongod is running? Try to connect with mongoclient: **Testing Mongo Connection** [quote=“fffhurst, post:11, topic:23142”] ```auto ``` mongo 127.0.0.1:27000/test onnecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb 2022-04-08T13:35:43.247-0400 E QUERY [js] Error: couldn’t connect to server 127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27000 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:343:13 ```auto ``` If you have problem also with mongoclient, check your logs. This error could be due to multiple reason. You can debug it by trying the following ways but make sure you have a back up or checkpoint of the server before you execute anything: - Check if mongod is running - check if the mongo process is actually running or not using “ps aux | grep mongo” or “sudo service mongod status”. If the process isn’t running start it. - Check on which port the process is running. - run command “sudo netstat -tlupn” and check its running on which port. Generally the port is 27017 but it could be some other port as well and you might be connecting to wrong port from application/client. - Try doing a “telnet” to mongo from the application server or the client if you are able to connect to mongo shell. - “sudo telnet port” == “telnet 192.168.1.2 27017” - Check if mongo process is bind to which IP. Is it bind to 0.0.0.0 (anywhere) or to 127.0.0.1 (localhost). - “sudo netstat -tlupn | grep 27017” - There could be a possibility if you have started a mongo server from the data of a replicaset and your standalone replicaset hasn’t been initialised, in this case as well you will get this error. In that case you can delete local db and initiate the replicaset to make it standalone primary. ```auto >use local > db.dropDatabase(); { "dropped" : "local", "ok" : 1 } > rs.initiate() >myrepl:PRMIARY ``` Hope that helps ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 4, 2022, 11:53pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/10 "2022-04-04T23:53:20Z")
> [@fffhurst](#): > > Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) With Elasticsearch double check your **elasticsearch.yaml** file and Graylog **server.conf** file. Make sure they match. My setup might be different from yours. If you are using the default (localhost) maybe try using 127.0.0.1 instead and configure the two files accordingly. Example: `elasticsearch.yml` ```auto cluster.name: graylog path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 192.168.1.100 (for testing try 0.0.0.0) http.port: 9200 action.auto_create_index: false discovery.type: single-node ``` `server.conf` ```auto elasticsearch_hosts = http://192.168.1.100:9200 elasticsearch_index_prefix = graylog ``` ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 8, 2022, 6:25pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/11 "2022-04-08T18:25:23Z")
```auto sudo mongo 127.0.0.1:27000/test MongoDB shell version v4.0.9 connecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb 2022-04-08T13:35:43.247-0400 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27000 :: caused by :: Connection refused : connect@src/mongo/shell/mongo.js:343:13 @(connect):2:6 exception: connect failed sudo ps aux | grep mongo mongod 1869 1.9 0.3 1170220 64016 ? Sl Apr07 27:22 /usr/bin/mongod -f /etc/mongod.conf 20092 0.0 0.0 112816 972 pts/1 S+ 13:41 0:00 grep --color=auto mongo sudo service mongod status Redirecting to /bin/systemctl status mongod.service ● mongod.service - MongoDB Database Server Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-04-07 14:36:54 EDT; 23h ago Docs: https://docs.mongodb.org/manual Process: 1797 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS) Process: 1793 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS) Process: 1791 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS) Process: 1779 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS) Main PID: 1869 (mongod) CGroup: /system.slice/mongod.service └─1869 /usr/bin/mongod -f /etc/mongod.conf Apr 07 14:36:48 systemd[1]: Starting MongoDB Database Server... Apr 07 14:36:49 mongod[1797]: about to fork child process, waiting until server is ready for connections. Apr 07 14:36:49 mongod[1797]: forked process: 1869 Apr 07 14:36:54 systemd[1]: Started MongoDB Database Server. ``` mongod is listening on the correct tcp port per your response above. Now the graylog web admin page is staying up, but there is another problem: Whenever more than one person is viewing the graylog web admin page and does a search, the search function no longer works and returns the following error: ```auto Error Message: Unable to perform search query Details: Search status code: 500 Search response: cannot GET https://:9000/api/search/universal/relative?query=gl2_source_input%&range=0&limit=150&sort=timestamp%3Adesc (500) ``` Also looks like the Windows Event log input has not collected any logs since Feb when the graylog web admin page became inoperable. Now the graylog web admin page remains operational, even though the search function returns errors. ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 8, 2022, 10:38pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/12 "2022-04-08T22:38:49Z")
Hello, My apologies default port for MongoDb is 27017 **Error prior.** > [@fffhurst](#): > > ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] **Testing Mongo Connection** > [@fffhurst](#): > > ```auto > onnecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb > 2022-04-08T13:35:43.247-0400 E QUERY [js] Error: couldn't connect to server > 127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to > 127.0.0.1:27000 :: caused by :: Connection refused : > connect@src/mongo/shell/mongo.js:343:13 > > ``` So I assume that test didn’t work because of the wrong port used. If Graylog can not connect to Mongo using **localhost /w Port 27017** you need to find out why and what is preventing this to happen. Here are some ideas to look for. - Permissions - Certificates used - Connections > [@fffhurst](#): > > `cannot GET https://:9000/api/search/universal/relative?query=gl2_source_input%&range=0&limit=150&sort=timestamp%3Adesc (500)` 500 Internal Server Error server error response code indicates that the server encountered an unexpected condition that prevented it from fulfilling the request. I would look into your logs on Graylog server to find out more. Showing your configuration files would help if you can, Other then that I’m using a educated guess what is wrong with the Graylog server. ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 11, 2022, 5:58pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/13 "2022-04-11T17:58:23Z")
Here are the logs from the same day: ```auto 2022-04-08T14:07:25.761-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.794-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.807-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.823-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.833-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.844-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:25.853-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query (ElasticsearchException) 2022-04-08T14:07:29.083-04:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #20). 2022-04-08T14:07:29.892-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-04-08T14:07:33.687-04:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #20). 2022-04-08T14:07:34.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-04-08T14:07:39.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-04-08T14:07:44.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200 at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?] at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?] ... 11 more 2022-04-08T14:07:45.625-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200) 2022-04-08T14:07:45.626-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2022-04-08T14:07:49.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?] at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?] at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?] at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] ``` ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 11, 2022, 5:59pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/14 "2022-04-11T17:59:24Z")
I posted the config file in my original post: > [@Multiple problems with Graylog](https://community.graylog.org/t/multiple-problems-with-graylog/22591): > > 1. Describe your incident: Graylog server was up and running and receiving logs, but not from certain hosts. All inputs show not running, but all inputs are receiving messages. All sidecar logs show x.509 certificate not valid error. Recently was able to restart sidecar on one of the vm’s sending logs, but a few days after this, the graylog web admin site began to intermittently become unavailable. If you reboot the server, it becomes available for a time, a few days, then it times out if y… ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 11, 2022, 6:23pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/15 "2022-04-11T18:23:18Z")
```auto sudo mongo 127.0.0.1:27017/test MongoDB shell version v4.0.9 connecting to: mongodb://127.0.0.1:27017/test?gssapiServiceName=mongodb Implicit session: session { "id" : UUID("") } MongoDB server version: 4.0.9 > ``` ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 11, 2022, 9:22pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/16 "2022-04-11T21:22:51Z")
Hello, Those logs show that Graylog cannot connect to Elasticsearch. So I Grabbed the main error/s from the logs shown above. ```auto http://127.0.0.1:9200, retrying Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused) Couldn't update field types for index set ``` If Graylog Cannot connect to your Localhost(127.0.0.1), I would look into that. because something it preventing your connection. > [@fffhurst](#): > > ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}] This could be the cause of Graylog unable to connect to MongoDb because it also can not connect using Localhost ( 127.0.0.1). ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 12, 2022, 7:40pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/17 "2022-04-12T19:40:38Z")
All of my inputs are running and receiving logs except for one input Windows Event Logs which is a GELF TCP input. Can you please tell me how to troubleshoot this one input? The error of Graylog not connecting to the localhost is no longer there and most inputs are working. Can you let me know what you need to help me look at that? ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 12, 2022, 10:13pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/18 "2022-04-12T22:13:42Z")
Hello, > [@fffhurst](#): > > All of my inputs are running and receiving logs except for one input Windows Event Logs which is a GELF TCP input. This should show in your logs what that happening. This could be multiple reason why its not working. ---
**Author:** ![fffhurst](https://avatars.discourse-cdn.com/v4/letter/f/53a042/32.png) [@fffhurst](https://community.graylog.org/u/fffhurst)\ **Post date:** [April 13, 2022, 6:52pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/19 "2022-04-13T18:52:04Z")
Ok here are the logs: ```auto 2022-04-08T14:36:55.300-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2022-04-08T14:36:55.300-04:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed. 2022-04-08T14:36:55.301-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2022-04-08T14:36:55.302-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. 2022-04-08T14:36:55.303-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2022-04-08T14:36:55.303-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2022-04-08T14:36:55.306-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. 2022-04-08T14:36:55.311-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2022-04-08T14:36:55.311-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever. 2022-04-08T14:36:55.312-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. 2022-04-08T14:36:55.336-04:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017 2022-04-08T14:36:55.341-04:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017 2022-04-08T14:36:55.361-04:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017 2022-04-08T14:36:55.374-04:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. 2022-04-08T14:36:55.374-04:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017 2022-04-08T14:36:55.375-04:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017 2022-04-08T14:36:55.376-04:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node. 2022-04-08T14:36:55.376-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever. 2022-04-08T14:36:55.380-04:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017 2022-04-08T14:36:55.382-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever. 2022-04-08T14:36:55.387-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200) 2022-04-08T14:36:55.391-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2022-04-08T14:36:55.400-04:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017 2022-04-08T14:36:55.410-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever. 2022-04-08T14:36:55.411-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever. 2022-04-08T14:36:55.422-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s]. 2022-04-08T14:36:55.430-04:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s]. 2022-04-08T14:36:55.433-04:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s]. 2022-04-08T14:36:55.433-04:00 INFO [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable. 2022-04-08T14:36:55.434-04:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s]. 2022-04-08T14:36:55.437-04:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2022-04-08T14:36:55.519-04:00 INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable. 2022-04-08T14:36:55.764-04:00 INFO [JerseyService] Enabling CORS for HTTP endpoint 2022-04-08T14:37:05.873-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:05.979-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.021-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.069-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.144-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.184-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.211-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.266-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.288-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.328-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.378-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.441-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:06.500-04:00 ERROR [AlertScanner] Skipping alert check : Unable to perform count query {"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException) 2022-04-08T14:37:10.350-04:00 INFO [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy 2022-04-08T14:37:25.059-04:00 INFO [NetworkListener] Started listener bound to [graylogFQDN:9000] 2022-04-08T14:37:25.061-04:00 INFO [HttpServer] [HttpServer] Started. 2022-04-08T14:37:25.062-04:00 INFO [JerseyService] Started REST API at 2022-04-08T14:37:25.064-04:00 INFO [ServiceManagerListener] Services are healthy 2022-04-08T14:37:25.066-04:00 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, GracefulShutdownService [RUNNING]=37, BufferSynchronizerService [RUNNING]=41, ConfigurationEtagService [RUNNING]=41, JournalReader [RUNNING]=42, OutputSetupService [RUNNING]=60, EtagService [RUNNING]=60, KafkaJournal [RUNNING]=73, LookupTableService [RUNNING]=221, StreamCacheService [RUNNING]=225, PeriodicalsService [RUNNING]=240, JerseyService [RUNNING]=29822} 2022-04-08T14:37:25.068-04:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2022-04-08T14:37:25.131-04:00 INFO [ServerBootstrap] Graylog server up and running. 2022-04-08T14:37:25.212-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now STARTING 2022-04-08T14:37:25.247-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now STARTING 2022-04-08T14:37:25.265-04:00 INFO [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now STARTING 2022-04-08T14:37:25.271-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING 2022-04-08T14:37:25.272-04:00 INFO [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now STARTING 2022-04-08T14:37:25.419-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x191cd255, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-04-08T14:37:25.420-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x156c1807, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-04-08T14:37:25.433-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xef1484ac, L:/0.0.0.0:12201]) should be 1048576 but is 425984. 2022-04-08T14:37:25.443-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xade74359, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-04-08T14:37:25.453-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING 2022-04-08T14:37:25.462-04:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=COC Infrastructure, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xb41f9572, L:/0.0.0.0:1514]) should be 1048576 but is 425984. 2022-04-08T14:37:25.463-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xcfc38ad1, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-04-08T14:37:25.466-04:00 INFO [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING 2022-04-08T14:37:25.497-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xedc6a6e0, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-04-08T14:37:25.506-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7cb5600d, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-04-08T14:37:25.559-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x925a0f84, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-04-08T14:37:25.628-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5d2b0905, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-04-08T14:37:25.660-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x020fa2f5, L:/0.0.0.0:1514]) should be 262144 but is 425984. 2022-04-08T14:37:25.670-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x84ce32b2, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-04-08T14:37:25.746-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x9c601073, L:/0.0.0.0:12201]) should be 262144 but is 425984. 2022-04-08T14:37:25.769-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING 2022-04-08T14:37:25.780-04:00 INFO [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING 2022-04-08T14:37:25.703-04:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x0d3478cd, L:/0.0.0.0:5140]) should be 262144 but is 425984. 2022-04-08T14:37:25.815-04:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING 2022-04-08T14:37:55.482-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again. 2022-04-08T14:37:55.537-04:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again. 2022-04-08T14:38:26.489-04:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1). 2022-04-08T14:38:28.026-04:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1). 2022-04-08T14:38:55.483-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again. 2022-04-08T14:38:55.537-04:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again. 2022-04-08T14:38:57.918-04:00 INFO [Messages] Bulk indexing finally successful (attempt #2). 2022-04-08T14:38:59.061-04:00 INFO [Messages] Bulk indexing finally successful (attempt #2). 2022-04-10T17:57:35.310-04:00 INFO [AbstractRotationStrategy] Deflector index (index set ) should be rotated, Pointing deflector to new index now! 2022-04-10T17:57:35.318-04:00 INFO [MongoIndexSet] Cycling from to . 2022-04-10T17:57:35.318-04:00 INFO [MongoIndexSet] Creating target index . 2022-04-10T17:57:35.378-04:00 INFO [Indices] Successfully created index template graylog-internal 2022-04-10T17:57:35.612-04:00 INFO [MongoIndexSet] Waiting for allocation of index . 2022-04-10T17:57:35.738-04:00 INFO [MongoIndexSet] Index has been successfully allocated. 2022-04-10T17:57:35.738-04:00 INFO [MongoIndexSet] Pointing index alias to new index . 2022-04-10T17:57:35.827-04:00 INFO [SystemJobManager] Submitted SystemJob <3b14c2a0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] 2022-04-10T17:57:35.827-04:00 INFO [MongoIndexSet] Successfully pointed index alias to index . 2022-04-10T17:58:05.850-04:00 INFO [SetIndexReadOnlyJob] Flushing old index . 2022-04-10T17:58:06.176-04:00 INFO [SetIndexReadOnlyJob] Setting old index to read-only. 2022-04-10T17:58:06.237-04:00 INFO [SystemJobManager] Submitted SystemJob <4d362dc0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] 2022-04-10T17:58:06.290-04:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_260. 2022-04-10T17:58:06.291-04:00 INFO [OptimizeIndexJob] Optimizing index . 2022-04-10T17:58:08.462-04:00 INFO [MongoIndexRangeService] Calculated range of [graylog_260] in [2169ms]. 2022-04-10T17:58:08.464-04:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog_260. 2022-04-10T17:58:08.474-04:00 INFO [SystemJobManager] SystemJob <3b14c2a0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 2647ms. 2022-04-10T18:00:53.601-04:00 INFO [SystemJobManager] SystemJob <4d362dc0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 167363ms. 2022-04-10T18:01:55.349-04:00 INFO [AbstractIndexCountBasedRetentionStrategy] Number of indices (21) higher than limit (20). Running retention for 1 indices. 2022-04-10T18:01:55.417-04:00 INFO [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index 2022-04-10T18:01:55.617-04:00 INFO [DeletionRetentionStrategy] Finished index retention strategy [delete] for index in 198ms. ``` ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [April 13, 2022, 9:52pm UTC](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142/20 "2022-04-13T21:52:42Z")
Hello > [@fffhurst](#): > > except for one input Windows Event Logs which is a GELF TCP input. Can you please tell me how to troubleshoot this one input? I found this in your logs and it looks like the input is running. ```auto 2022-04-08T14:37:25.271-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING 2022-04-08T14:37:25.466-04:00 INFO [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING ``` I also seen these logs. ```auto 2022-04-08T14:37:55.482-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again. 2022-04-08T14:38:55.483-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again. ``` By chance are you have connection issues? Only time I receive those messages is when Graylog is started before Elasticsearch. [Next page](https://community.graylog.org/t/certificate-problem-and-maybe-other-issues/23142.md?page=2)