Certificate problem, and maybe other issues

[fffhurst (Franklin Hurst)]

Hello. This is a continuation of this thread since it got closed by the system:

https://community.graylog.org/t/multiple-problems-with-graylog/22591

Based on the feedback from that thread I started looking into certificate issues and the most pressing issue was just getting the graylog web admin page to remain operational. In the test environment which an exact snapshot of production, I was able to update the PKI cert and have it signed by the CA. In the test, this had the following effects:
1.) All inputs now show as running. (previously inputs were received, but the inputs were not showing as running)
2.) The errors in the graylog logs went away.
3.) A domain controller whose logs previously were not getting to the input began getting to the input.
4.) The graylog web admin page began staying operational for more than a few minutes, indefinitely so far operational

So, as a result of that apparent success, we did the same thing in the production server. At first it worked, but then the graylog web admin page became inaccessible and the graylog logs began showing a new error. The graylog logs before and after that event are as follows:

[gsmith]

Hello @fffhurst and welcome back.

Glad your Dev environment worked out, sorry to here about your production environment.

@fffhurst

At first it worked, but then the graylog web admin page became inaccessible and the graylog logs began showing a new error. The graylog logs before and after that event are as follows:

Did you use new certs? or use the ones from dev?
Check permissions, does graylog have access to the certs/keystore?

To help you further we would need more then…

@fffhurst

inaccessible and the graylog logs began showing a new error.

Thanks

[fffhurst (Franklin Hurst)]

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-03-22T15:31:27.403-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-03-22T15:31:30.513-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-03-22T15:31:35.513-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-03-22T15:31:37.417-04:00 ERROR [NodePingThread] Uncaught exception in periodical
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
	at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[graylog.jar:?]
	at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[graylog.jar:?]
	at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?]
	at com.mongodb.DBCursor.one(DBCursor.java:683) ~[graylog.jar:?]
	at com.mongodb.DBCollection.findOne(DBCollection.java:829) ~[graylog.jar:?]
	at com.mongodb.DBCollection.findOne(DBCollection.java:792) ~[graylog.jar:?]
	at com.mongodb.DBCollection.findOne(DBCollection.java:739) ~[graylog.jar:?]
	at org.graylog2.database.PersistedServiceImpl.findOne(PersistedServiceImpl.java:128) ~[graylog.jar:?]
	at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:73) ~[graylog.jar:?]
	at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:84) ~[graylog.jar:?]
	at org.graylog2.periodical.NodePingThread.doRun(NodePingThread.java:62) ~[graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
2022-03-22T15:31:37.419-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out

[fffhurst (Franklin Hurst)]

2022-03-22T15:31:37.444-04:00 WARN  [ClusterEventPeriodical] Error while reading cluster events from MongoDB, retrying.
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
	at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[graylog.jar:?]
	at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[graylog.jar:?]
	at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[graylog.jar:?]
	at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?]
	at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
	at org.graylog2.events.ClusterEventPeriodical.doRun(ClusterEventPeriodical.java:150) [graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
2022-03-22T15:31:37.445-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-03-22T15:32:42.130-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 3.0.2 [org.graylog.aws.AWSPlugin]
2022-03-22T15:32:42.166-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 3.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2022-03-22T15:32:42.167-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-03-22T15:32:43.035-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.truststore=/SSCLant/certs/cacerts.jks -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2022-03-22T15:32:43.574-04:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2022-03-22T15:32:48.454-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-03-22T15:32:48.502-04:00 INFO  [NodeId] Node ID: 9e804424-96fe-42c8-a79a-051fc7fb0963
2022-03-22T15:32:48.875-04:00 INFO  [LogManager] Loading logs.
2022-03-22T15:32:48.980-04:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000005549547208.index, deleting and rebuilding index...
2022-03-22T15:32:50.339-04:00 INFO  [LogManager] Logs loading complete.
2022-03-22T15:32:50.342-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-03-22T15:32:50.355-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-03-22T15:32:50.375-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-03-22T15:32:50.421-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-03-22T15:32:50.592-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2022-03-22T15:32:50.597-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 9]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2928904}
2022-03-22T15:32:50.637-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2022-03-22T15:32:51.144-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2022-03-22T15:32:51.144-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2022-03-22T15:32:51.216-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2022-03-22T15:32:51.216-04:00 INFO  [JestClientFactory] Node Discovery disabled...
2022-03-22T15:32:51.216-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2022-03-22T15:32:51.426-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-22T15:32:54.293-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-22T15:32:54.303-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-22T15:32:54.356-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2022-03-22T15:32:55.755-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-22T15:32:57.508-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-22T15:32:59.093-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-22T15:33:00.624-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-22T15:33:01.226-04:00 INFO  [ServerBootstrap] Graylog server 3.0.2+1686930 starting up
2022-03-22T15:33:01.226-04:00 INFO  [ServerBootstrap] JRE: Red Hat, Inc. 1.8.0_282 on Linux 3.10.0-1160.15.2.el7.x86_64
2022-03-22T15:33:01.226-04:00 INFO  [ServerBootstrap] Deployment: rpm
2022-03-22T15:33:01.226-04:00 INFO  [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos)
2022-03-22T15:33:01.227-04:00 INFO  [ServerBootstrap] Arch: amd64
2022-03-22T15:33:01.340-04:00 INFO  [PeriodicalsService] Starting 27 periodicals ...
2022-03-22T15:33:01.340-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-03-22T15:33:01.360-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-03-22T15:33:01.366-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2022-03-22T15:33:01.366-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-03-22T15:33:01.367-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-03-22T15:33:01.368-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-03-22T15:33:01.368-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-03-22T15:33:01.369-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-03-22T15:33:01.369-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-03-22T15:33:01.369-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-03-22T15:33:01.375-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-03-22T15:33:01.376-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-03-22T15:33:01.376-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-03-22T15:33:01.379-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-03-22T15:33:01.380-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-03-22T15:33:01.381-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-03-22T15:33:01.382-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-03-22T15:33:01.426-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017
2022-03-22T15:33:01.429-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-03-22T15:33:01.433-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017
2022-03-22T15:33:01.471-04:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017
2022-03-22T15:33:01.476-04:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:11}] to localhost:27017
2022-03-22T15:33:01.479-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017
2022-03-22T15:33:01.479-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-03-22T15:33:01.479-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2022-03-22T15:33:01.480-04:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2022-03-22T15:33:01.484-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2022-03-22T15:33:01.484-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017
2022-03-22T15:33:01.502-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-03-22T15:33:01.519-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2022-03-22T15:33:01.522-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2022-03-22T15:33:01.534-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-03-22T15:33:01.556-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-03-22T15:33:01.561-04:00 INFO  [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable.
2022-03-22T15:33:01.562-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2022-03-22T15:33:01.565-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2022-03-22T15:33:01.565-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-03-22T15:33:01.566-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-03-22T15:33:01.577-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-03-22T15:33:01.577-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-03-22T15:33:01.694-04:00 INFO  [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2022-03-22T15:33:01.902-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2022-03-22T15:33:16.394-04:00 INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2022-03-22T15:33:25.333-04:00 INFO  [NetworkListener] Started listener bound to [graylogFQDN:9000]
2022-03-22T15:33:25.335-04:00 INFO  [HttpServer] [HttpServer] Started.
2022-03-22T15:33:25.335-04:00 INFO  [JerseyService] Started REST API at <graylogFQDN:9000>
2022-03-22T15:33:25.336-04:00 INFO  [ServiceManagerListener] Services are healthy
2022-03-22T15:33:25.336-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-03-22T15:33:25.337-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=41, JournalReader [RUNNING]=41, KafkaJournal [RUNNING]=41, GracefulShutdownService [RUNNING]=42, OutputSetupService [RUNNING]=43, EtagService [RUNNING]=43, InputSetupService [RUNNING]=49, ConfigurationEtagService [RUNNING]=83, LookupTableService [RUNNING]=253, StreamCacheService [RUNNING]=291, PeriodicalsService [RUNNING]=298, JerseyService [RUNNING]=24023}
2022-03-22T15:33:25.404-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now STARTING
2022-03-22T15:33:25.406-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING
2022-03-22T15:33:25.408-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now STARTING
2022-03-22T15:33:25.409-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now STARTING
2022-03-22T15:33:25.411-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now STARTING
2022-03-22T15:33:25.419-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-03-22T15:33:25.518-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5fb5b064, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-03-22T15:33:25.522-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xc3d1b600, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-03-22T15:33:25.522-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=COC Infrastructure, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xde2e80ee, L:/0.0.0.0:1514]) should be 1048576 but is 425984.
2022-03-22T15:33:25.522-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7dc5acf0, L:/0.0.0.0:12201]) should be 1048576 but is 425984.
2022-03-22T15:33:25.526-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING
2022-03-22T15:33:25.527-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING
2022-03-22T15:33:25.528-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xd2deacce, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-03-22T15:33:25.528-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xb660881d, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-03-22T15:33:25.528-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x3b949989, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-03-22T15:33:25.529-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7725d543, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-03-22T15:33:25.531-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x398e8adb, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-03-22T15:33:25.533-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7f5bf1b1, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-03-22T15:33:25.537-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x4a1a8403, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-03-22T15:33:25.538-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x504fa505, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-03-22T15:33:25.539-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xa21facfc, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-03-22T15:33:25.540-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING
2022-03-22T15:33:25.541-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING
2022-03-22T15:33:25.548-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x946b5e65, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-03-22T15:33:25.556-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING

[fffhurst (Franklin Hurst)]

Did you use new certs? or use the ones from dev?
Check permissions, does graylog have access to the certs/keystore

I used new certs

[gsmith]

Hello,
I picked out the import Logs that were shown

This would be the connection to Elasticsearch and Graylog and they can not connect. Could be a configuration issue.

@fffhurst

Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)

Check Elasticsearch health

curl -XGET http://127.0.0.1:9200/_cluster/health?pretty=true

Seams like Graylog maybe unable to connect to MongoDb, This could be a configuration issue.

@fffhurst

ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Not sure if rotating index were execute to resolve this, if so this maybe another configuration issue and/or permission issue.

@fffhurst

couldn’t update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn’t collect indices for alias graylog_deflector

These all look like a configuration problems, either something was missed or incorrect. I’m not seeing anything that would pertain to certification issue yet.

[fffhurst (Franklin Hurst)]

@gsmith

Check Elasticsearch health

curl -XGET http://127.0.0.1:9200/_cluster/health?pretty=true

[fffhurst (Franklin Hurst)]

So if I follow you correctly, when we updated the certificate, that was successful in terms of removing the certificate erros, and now we just need to deal with the configuration issues?

[gsmith]

Hello,

Hello,

@fffhurst

and now we just need to deal with the configuration issues?

If you were able to correct the certificate issue. Double check if your Graylog node is functioning correctly, if so you may not need to do anything.

But if you still have the following error as shown below, The Yes, you may need to find out why that happening.

My apologies default port for MongoDb is 27017

Error prior.

@fffhurst

ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

@fffhurst

ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

You obtain a Connection refused. Are you sure mongod is running?

Try to connect with mongoclient:

Testing Mongo Connection

[quote=“fffhurst, post:11, topic:23142”]

mongo 127.0.0.1:27000/test
onnecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb
2022-04-08T13:35:43.247-0400 E QUERY [js] Error: couldn’t connect to server
127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to
127.0.0.1:27000 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:343:13

If you have problem also with mongoclient, check your logs.

This error could be due to multiple reason. You can debug it by trying the following ways but make sure you have a back up or checkpoint of the server before you execute anything:

• Check if mongod is running
• check if the mongo process is actually running or not using “ps aux | grep mongo” or “sudo service mongod status”. If the process isn’t running start it.
• Check on which port the process is running.
• run command “sudo netstat -tlupn” and check its running on which port. Generally the port is 27017 but it could be some other port as well and you might be connecting to wrong port from application/client.
• Try doing a “telnet” to mongo from the application server or the client if you are able to connect to mongo shell.
• “sudo telnet port” == “telnet 192.168.1.2 27017”
• Check if mongo process is bind to which IP. Is it bind to 0.0.0.0 (anywhere) or to 127.0.0.1 (localhost).
• “sudo netstat -tlupn | grep 27017”
• There could be a possibility if you have started a mongo server from the data of a replicaset and your standalone replicaset hasn’t been initialised, in this case as well you will get this error.
  In that case you can delete local db and initiate the replicaset to make it standalone primary.

>use local
> db.dropDatabase();
{ "dropped" : "local", "ok" : 1 }
> rs.initiate()
>myrepl:PRMIARY

Hope that helps

[gsmith]

@fffhurst

Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)

With Elasticsearch double check your elasticsearch.yaml file and Graylog server.conf file. Make sure they match.

My setup might be different from yours. If you are using the default (localhost) maybe try using 127.0.0.1 instead and configure the two files accordingly.

Example:

elasticsearch.yml

cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.1.100 (for testing try 0.0.0.0)
http.port: 9200
action.auto_create_index: false
discovery.type: single-node

server.conf

elasticsearch_hosts = http://192.168.1.100:9200
elasticsearch_index_prefix = graylog

[fffhurst (Franklin Hurst)]

sudo mongo 127.0.0.1:27000/test

MongoDB shell version v4.0.9
connecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb
2022-04-08T13:35:43.247-0400 E QUERY    [js] Error: couldn't connect to server 
127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to 
127.0.0.1:27000 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:343:13
@(connect):2:6
exception: connect failed

sudo ps aux | grep mongo

mongod    1869  1.9  0.3 1170220 64016 ?       Sl   Apr07  27:22 /usr/bin/mongod -f /etc/mongod.conf
<my_username>  20092  0.0  0.0 112816   972 pts/1    S+   13:41   0:00 grep --color=auto mongo

sudo service mongod status

Redirecting to /bin/systemctl status mongod.service
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-04-07 14:36:54 EDT; 23h ago
     Docs: https://docs.mongodb.org/manual
  Process: 1797 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS)
  Process: 1793 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 1791 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 1779 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
 Main PID: 1869 (mongod)
   CGroup: /system.slice/mongod.service
           └─1869 /usr/bin/mongod -f /etc/mongod.conf

Apr 07 14:36:48 <FQDN> systemd[1]: Starting MongoDB Database Server...
Apr 07 14:36:49 <FQDN> mongod[1797]: about to fork child process, waiting until server is ready for connections.
Apr 07 14:36:49 <FQDN> mongod[1797]: forked process: 1869
Apr 07 14:36:54 <FQDN> systemd[1]: Started MongoDB Database Server.

mongod is listening on the correct tcp port per your response above.

Now the graylog web admin page is staying up, but there is another problem:

Whenever more than one person is viewing the graylog web admin page and does a search, the search function no longer works and returns the following error:

Error Message:
Unable to perform search query
Details:
Search status code:
500
Search response:
cannot GET https://<FQDN>:9000/api/search/universal/relative?query=gl2_source_input%<number>&range=0&limit=150&sort=timestamp%3Adesc (500)

Also looks like the Windows Event log input has not collected any logs since Feb when the graylog web admin page became inoperable. Now the graylog web admin page remains operational, even though the search function returns errors.

[gsmith]

Hello,

My apologies default port for MongoDb is 27017

Error prior.

@fffhurst

ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Testing Mongo Connection

@fffhurst

onnecting to: mongodb://127.0.0.1:27000/test?gssapiServiceName=mongodb
2022-04-08T13:35:43.247-0400 E QUERY    [js] Error: couldn't connect to server 
127.0.0.1:27000, connection attempt failed: SocketException: Error connecting to 
127.0.0.1:27000 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:343:13

So I assume that test didn’t work because of the wrong port used.

If Graylog can not connect to Mongo using localhost /w Port 27017 you need to find out why and what is preventing this to happen. Here are some ideas to look for.

• Permissions
• Certificates used
• Connections

@fffhurst

cannot GET https://<FQDN>:9000/api/search/universal/relative?query=gl2_source_input%<number>&range=0&limit=150&sort=timestamp%3Adesc (500)

500 Internal Server Error server error response code indicates that the server encountered an unexpected condition that prevented it from fulfilling the request.
I would look into your logs on Graylog server to find out more.

Showing your configuration files would help if you can, Other then that I’m using a educated guess what is wrong with the Graylog server.

[fffhurst (Franklin Hurst)]

Here are the logs from the same day:

2022-04-08T14:07:25.761-04:00 ERROR [AlertScanner] Skipping alert check <DNS02 - Status/f43baf2c-a799-4d03-8347-cacf6506ed08>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.794-04:00 ERROR [AlertScanner] Skipping alert check <DNS03 - Status/608b707e-9c69-4669-b99d-690621bd2fb6>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.807-04:00 ERROR [AlertScanner] Skipping alert check <DNS04 - Status/7dec668a-9ef8-4a2c-b2d9-b5965ee9f0a8>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.823-04:00 ERROR [AlertScanner] Skipping alert check <AD01 - Status/fb5a3db2-359d-4a98-a1a3-1b55537953fe>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.833-04:00 ERROR [AlertScanner] Skipping alert check <AD02 - Status/541f241c-20f3-4c39-86f3-b9bf926f7be8>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.844-04:00 ERROR [AlertScanner] Skipping alert check <AD03 - Status/5f211d0f-07b7-4a98-b432-ef53e90a83bc>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:25.853-04:00 ERROR [AlertScanner] Skipping alert check <AD04 - Status/613f7904-6787-49fe-bbe1-5b8b307292dd>: Unable to perform count query (ElasticsearchException)
2022-04-08T14:07:29.083-04:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #20).
2022-04-08T14:07:29.892-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-04-08T14:07:33.687-04:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #20).
2022-04-08T14:07:34.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-04-08T14:07:39.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-04-08T14:07:44.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[graylog.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	... 11 more
2022-04-08T14:07:45.625-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2022-04-08T14:07:45.626-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-04-08T14:07:49.893-04:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>
org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
	at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]

[fffhurst (Franklin Hurst)]

I posted the config file in my original post:

Multiple problems with Graylog

1. Describe your incident: Graylog server was up and running and receiving logs, but not from certain hosts. All inputs show not running, but all inputs are receiving messages. All sidecar logs show x.509 certificate not valid error. Recently was able to restart sidecar on one of the vm’s sending logs, but a few days after this, the graylog web admin site began to intermittently become unavailable. If you reboot the server, it becomes available for a time, a few days, then it times out if y…

[fffhurst (Franklin Hurst)]

sudo mongo 127.0.0.1:27017/test

MongoDB shell version v4.0.9
connecting to: mongodb://127.0.0.1:27017/test?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("<UUID here>") }
MongoDB server version: 4.0.9
>

[gsmith]

Hello,
Those logs show that Graylog cannot connect to Elasticsearch. So I Grabbed the main error/s from the logs shown above.

http://127.0.0.1:9200, retrying
Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connection refused)
Couldn't update field types for index set <Default index set/5cf136a929fbc64ff418cc09>

If Graylog Cannot connect to your Localhost(127.0.0.1), I would look into that. because something it preventing your connection.

@fffhurst

ERROR [NodePingThread] Uncaught exception in periodical com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

This could be the cause of Graylog unable to connect to MongoDb because it also can not connect using Localhost ( 127.0.0.1).

[fffhurst (Franklin Hurst)]

All of my inputs are running and receiving logs except for one input Windows Event Logs which is a GELF TCP input. Can you please tell me how to troubleshoot this one input? The error of Graylog not connecting to the localhost is no longer there and most inputs are working. Can you let me know what you need to help me look at that?

[gsmith]

Hello,

@fffhurst

All of my inputs are running and receiving logs except for one input Windows Event Logs which is a GELF TCP input.

This should show in your logs what that happening. This could be multiple reason why its not working.

[fffhurst (Franklin Hurst)]

Ok here are the logs:

2022-04-08T14:36:55.300-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-04-08T14:36:55.300-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-04-08T14:36:55.301-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-04-08T14:36:55.302-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-04-08T14:36:55.303-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-04-08T14:36:55.303-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-04-08T14:36:55.306-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-04-08T14:36:55.311-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-04-08T14:36:55.311-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-04-08T14:36:55.312-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-04-08T14:36:55.336-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017
2022-04-08T14:36:55.341-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2022-04-08T14:36:55.361-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017
2022-04-08T14:36:55.374-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2022-04-08T14:36:55.374-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017
2022-04-08T14:36:55.375-04:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017
2022-04-08T14:36:55.376-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-04-08T14:36:55.376-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2022-04-08T14:36:55.380-04:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2022-04-08T14:36:55.382-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-04-08T14:36:55.387-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2022-04-08T14:36:55.391-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-04-08T14:36:55.400-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017
2022-04-08T14:36:55.410-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2022-04-08T14:36:55.411-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2022-04-08T14:36:55.422-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-04-08T14:36:55.430-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-04-08T14:36:55.433-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-04-08T14:36:55.433-04:00 INFO  [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable.
2022-04-08T14:36:55.434-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-04-08T14:36:55.437-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-04-08T14:36:55.519-04:00 INFO  [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2022-04-08T14:36:55.764-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2022-04-08T14:37:05.873-04:00 ERROR [AlertScanner] Skipping alert check <Tor Detection/d854d24c-42b6-40f8-8263-c72da0f2a1b8>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:05.979-04:00 ERROR [AlertScanner] Skipping alert check <Domain Controller Status/c53a101d-b92c-4225-939a-412ce7768ea0>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.021-04:00 ERROR [AlertScanner] Skipping alert check <Firewall status/cccb6a57-afd9-4c64-a8d7-2afc6d907d91>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.069-04:00 ERROR [AlertScanner] Skipping alert check <DHCP - Status/5175aa2c-fa4e-401a-89e6-f4947add68cf>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.144-04:00 ERROR [AlertScanner] Skipping alert check <DNS01 - Status/32c2a59d-e782-405c-b48c-2b082636442e>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.184-04:00 ERROR [AlertScanner] Skipping alert check <HBSS01 - Status/424014d8-2a3f-478e-9b1e-5640eaba2cd0>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.211-04:00 ERROR [AlertScanner] Skipping alert check <DNS02 - Status/f43baf2c-a799-4d03-8347-cacf6506ed08>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.266-04:00 ERROR [AlertScanner] Skipping alert check <DNS03 - Status/608b707e-9c69-4669-b99d-690621bd2fb6>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.288-04:00 ERROR [AlertScanner] Skipping alert check <DNS04 - Status/7dec668a-9ef8-4a2c-b2d9-b5965ee9f0a8>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.328-04:00 ERROR [AlertScanner] Skipping alert check <AD01 - Status/fb5a3db2-359d-4a98-a1a3-1b55537953fe>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.378-04:00 ERROR [AlertScanner] Skipping alert check <AD02 - Status/541f241c-20f3-4c39-86f3-b9bf926f7be8>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.441-04:00 ERROR [AlertScanner] Skipping alert check <AD03 - Status/5f211d0f-07b7-4a98-b432-ef53e90a83bc>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:06.500-04:00 ERROR [AlertScanner] Skipping alert check <AD04 - Status/613f7904-6787-49fe-bbe1-5b8b307292dd>: Unable to perform count query

{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]} (ElasticsearchException)
2022-04-08T14:37:10.350-04:00 INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2022-04-08T14:37:25.059-04:00 INFO  [NetworkListener] Started listener bound to [graylogFQDN:9000]
2022-04-08T14:37:25.061-04:00 INFO  [HttpServer] [HttpServer] Started.
2022-04-08T14:37:25.062-04:00 INFO  [JerseyService] Started REST API at <graylogFQDN:9000>
2022-04-08T14:37:25.064-04:00 INFO  [ServiceManagerListener] Services are healthy
2022-04-08T14:37:25.066-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, GracefulShutdownService [RUNNING]=37, BufferSynchronizerService [RUNNING]=41, ConfigurationEtagService [RUNNING]=41, JournalReader [RUNNING]=42, OutputSetupService [RUNNING]=60, EtagService [RUNNING]=60, KafkaJournal [RUNNING]=73, LookupTableService [RUNNING]=221, StreamCacheService [RUNNING]=225, PeriodicalsService [RUNNING]=240, JerseyService [RUNNING]=29822}
2022-04-08T14:37:25.068-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-04-08T14:37:25.131-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-04-08T14:37:25.212-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now STARTING
2022-04-08T14:37:25.247-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now STARTING
2022-04-08T14:37:25.265-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now STARTING
2022-04-08T14:37:25.271-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING
2022-04-08T14:37:25.272-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now STARTING
2022-04-08T14:37:25.419-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x191cd255, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-04-08T14:37:25.420-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x156c1807, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-04-08T14:37:25.433-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xef1484ac, L:/0.0.0.0:12201]) should be 1048576 but is 425984.
2022-04-08T14:37:25.443-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xade74359, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-04-08T14:37:25.453-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING
2022-04-08T14:37:25.462-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=COC Infrastructure, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xb41f9572, L:/0.0.0.0:1514]) should be 1048576 but is 425984.
2022-04-08T14:37:25.463-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xcfc38ad1, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-04-08T14:37:25.466-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING
2022-04-08T14:37:25.497-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xedc6a6e0, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-04-08T14:37:25.506-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x7cb5600d, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-04-08T14:37:25.559-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x925a0f84, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-04-08T14:37:25.628-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5d2b0905, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-04-08T14:37:25.660-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x020fa2f5, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-04-08T14:37:25.670-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x84ce32b2, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-04-08T14:37:25.746-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x9c601073, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-04-08T14:37:25.769-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING
2022-04-08T14:37:25.780-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING
2022-04-08T14:37:25.703-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x0d3478cd, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-04-08T14:37:25.815-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING
2022-04-08T14:37:55.482-04:00 WARN  [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2022-04-08T14:37:55.537-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2022-04-08T14:38:26.489-04:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2022-04-08T14:38:28.026-04:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2022-04-08T14:38:55.483-04:00 WARN  [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2022-04-08T14:38:55.537-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2022-04-08T14:38:57.918-04:00 INFO  [Messages] Bulk indexing finally successful (attempt #2).
2022-04-08T14:38:59.061-04:00 INFO  [Messages] Bulk indexing finally successful (attempt #2).
2022-04-10T17:57:35.310-04:00 INFO  [AbstractRotationStrategy] Deflector index <Default index set> (index set <graylog_260>) should be rotated, Pointing deflector to new index now!
2022-04-10T17:57:35.318-04:00 INFO  [MongoIndexSet] Cycling from <graylog_260> to <graylog_261>.
2022-04-10T17:57:35.318-04:00 INFO  [MongoIndexSet] Creating target index <graylog_261>.
2022-04-10T17:57:35.378-04:00 INFO  [Indices] Successfully created index template graylog-internal
2022-04-10T17:57:35.612-04:00 INFO  [MongoIndexSet] Waiting for allocation of index <graylog_261>.
2022-04-10T17:57:35.738-04:00 INFO  [MongoIndexSet] Index <graylog_261> has been successfully allocated.
2022-04-10T17:57:35.738-04:00 INFO  [MongoIndexSet] Pointing index alias <graylog_deflector> to new index <graylog_261>.
2022-04-10T17:57:35.827-04:00 INFO  [SystemJobManager] Submitted SystemJob <3b14c2a0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2022-04-10T17:57:35.827-04:00 INFO  [MongoIndexSet] Successfully pointed index alias <graylog_deflector> to index <graylog_261>.
2022-04-10T17:58:05.850-04:00 INFO  [SetIndexReadOnlyJob] Flushing old index <graylog_260>.
2022-04-10T17:58:06.176-04:00 INFO  [SetIndexReadOnlyJob] Setting old index <graylog_260> to read-only.
2022-04-10T17:58:06.237-04:00 INFO  [SystemJobManager] Submitted SystemJob <4d362dc0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2022-04-10T17:58:06.290-04:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_260.
2022-04-10T17:58:06.291-04:00 INFO  [OptimizeIndexJob] Optimizing index <graylog_260>.
2022-04-10T17:58:08.462-04:00 INFO  [MongoIndexRangeService] Calculated range of [graylog_260] in [2169ms].
2022-04-10T17:58:08.464-04:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_260.
2022-04-10T17:58:08.474-04:00 INFO  [SystemJobManager] SystemJob <3b14c2a0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 2647ms.
2022-04-10T18:00:53.601-04:00 INFO  [SystemJobManager] SystemJob <4d362dc0-b919-11ec-b2b8-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 167363ms.
2022-04-10T18:01:55.349-04:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (21) higher than limit (20). Running retention for 1 indices.
2022-04-10T18:01:55.417-04:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_241>
2022-04-10T18:01:55.617-04:00 INFO  [DeletionRetentionStrategy] Finished index retention strategy [delete] for index <graylog_241> in 198ms.

[gsmith]

Hello

@fffhurst

except for one input Windows Event Logs which is a GELF TCP input. Can you please tell me how to troubleshoot this one input?

I found this in your logs and it looks like the input is running.

2022-04-08T14:37:25.271-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING
2022-04-08T14:37:25.466-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING

I also seen these logs.

2022-04-08T14:37:55.482-04:00 WARN  [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2022-04-08T14:38:55.483-04:00 WARN  [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.

By chance are you have connection issues? Only time I receive those messages is when Graylog is started before Elasticsearch.